Using Apache Iceberg Destination with AWS Glue as Catalogue

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is trying to use Apache Iceberg Destination in Airbyte to replicate data from MySQL to Apache Iceberg tables using CDC method. They want to use AWS Glue as the Iceberg table catalogue, but the available options are Hive, Hadoop, REST, and JDBC. The user is seeking guidance on how to proceed with creating the destination connector.


Question

Hi everyone!

I am trying to use AirByte (self-hosted) to replicate data from MySQL (RDS) to Apache Iceberg tables using CDC method. I am trying to use Apache Iceberg Destination (https://docs.airbyte.com/integrations/destinations/iceberg). I want to use AWS Glue as Iceberg table catalogue, but there are only these types of iceberg catalogue: Hive, Hadoop, REST and JDBC. How do I proceed with creating destination connector.

Thanks for any help!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["apache-iceberg-destination", "aws-glue", "mysql", "cdc", "catalogue", "connector"]
2

Watch for this https://github.com/airbytehq/airbyte/pull/45707|PR to be merged.

3

Thanks <@U05L8MN8H9S>

4

Hi <@U05L8MN8H9S> I am trying to use AirByte self-hosted using EC2 for a POC, the machine running the AIrByte has all required permission to access Glue Catalogue and S3 via IAM Role.

But in Iceberg destination, S3 Key ID and S3 Access Key
are the mandatory parameters. Shouldn’t that be an optional parameter on AirByte OSS?