Cannot use EC2 IAM Role to create S3 source

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: AWS EC2
  • Memory / Disk: 30Gb
  • Deployment: Docker Compose
  • Airbyte Version: What version are you using now? 0.39.23-alpha
  • Source name/version: s3
  • Destination name/version: redshift
  • Step: The issue is happening when creating a new source
  • Description:

Hello there,

I am discovering airbyte and managed to connect two sources (google analytics and stripe) in a heartbeat, I am thrilled :slight_smile: I am having by first problem with my third source, which is S3. Here is what I did :

  • deployed Airbyte on an AWS EC2 instance using docker compose (following On AWS (EC2) | Airbyte Documentation)
  • attached an IAM Role to the EC2 instance with a Policy allowing it to list and read from all S3 buckets on my account
  • checked that the EC2 instance had access to all buckets using aws s3 ls, which worked.
  • clicked on “New source” in the Airbyte UI, selected “S3” as “Source Type”, and filled the whole form.
  • I did not fill AWS Access Key ID nor AWS Secret Access Key, as my understanding is that the good practice is to use IAM Roles attached to the EC2 instance to manage permissions instead of access keys.

As a consequence, I get the following error in the UI when I click on “Set up source”:

ClientError('An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied')

So my question is : Is there a way for airbyte to fetch the AWS keys via the attached IAM role on the EC2 instance it is running on ?

Thanks !

Today Airbyte doesn’t allow you to create a source/destination in AWS using roles.
There is a Github Issue to implement this feature: https://github.com/airbytehq/airbyte/issues/5942