Cannot use EC2 IAM Role to create S3 source

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: AWS EC2
  • Memory / Disk: 30Gb
  • Deployment: Docker Compose
  • Airbyte Version: What version are you using now? 0.39.23-alpha
  • Source name/version: s3
  • Destination name/version: redshift
  • Step: The issue is happening when creating a new source
  • Description:

Hello there,

I am discovering airbyte and managed to connect two sources (google analytics and stripe) in a heartbeat, I am thrilled :slight_smile: I am having by first problem with my third source, which is S3. Here is what I did :

  • deployed Airbyte on an AWS EC2 instance using docker compose (following On AWS (EC2) | Airbyte Documentation)
  • attached an IAM Role to the EC2 instance with a Policy allowing it to list and read from all S3 buckets on my account
  • checked that the EC2 instance had access to all buckets using aws s3 ls, which worked.
  • clicked on “New source” in the Airbyte UI, selected “S3” as “Source Type”, and filled the whole form.
  • I did not fill AWS Access Key ID nor AWS Secret Access Key, as my understanding is that the good practice is to use IAM Roles attached to the EC2 instance to manage permissions instead of access keys.

As a consequence, I get the following error in the UI when I click on “Set up source”:

ClientError('An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied')

So my question is : Is there a way for airbyte to fetch the AWS keys via the attached IAM role on the EC2 instance it is running on ?

Thanks !

2

Today Airbyte doesn’t allow you to create a source/destination in AWS using roles.
There is a Github Issue to implement this feature: https://github.com/airbytehq/airbyte/issues/5942

3

Alexandre, there is an implementation for this open https://github.com/airbytehq/airbyte/pull/14231 you can copy the branch and deploy a dev version locally to test it?

4

Oh wow ! Thanks Marcos for your answers and sorry for not answering sooner, I see that the PR has been merged, that was fast :slight_smile: I will test it and get back to you here :+1:

5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.