Setting up Airbyte on EC2 with RDS Postgres, S3, and Secrets Manager

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

User is struggling to set up Airbyte on EC2 with RDS Postgres, S3 for logging, and Secrets Manager for secrets. They have created a values.yaml file and installed Airbyte using abctl, but unsure about integrating RDS Postgres. Referenced Airbyte documentation.


Question

Hi all,

I’m fairly new to Airbyte - but pretty experienced in setting up AWS.

I have:
:white_check_mark: Completed the Airbyte Udemy course
:white_check_mark: Successfully setup an EC2 Instance with Airbyte using abctl local install
:white_check_mark: Successfully played around with a few sources and destinations.

What I’m struggling with now, and where I find the current documentation a little lacking is trying to setup an EC2 instance with:

  • The Airbyte config in RDS Postgres (instead on local postgres on the EC2)
  • Using S3 buckets for logging
  • Using AWS Secrets Manager for secrets

This is what I have done:

  • Started a new EC2 instance with an IAM role
  • Added inline policies like the ones posted in the thread on this message (with changed bucket name)
  • Created a values.yaml file like this:
  storage:
    type: "S3"
    storageSecretName: "airbyte-config-secrets"
    bucket:
      log: "MY_BUCKET_NAME"
      state: "MY_BUCKET_NAME"
      workloadOutput: "MY_BUCKET_NAME"
    s3:
      region: "eu-central-1"
      authenticationType: "instanceProfile"
  secretsManager:
    type: "awsSecretManager"
    awsSecretManager:
      region: "eu-central-1"
      authenticationType: "instanceProfile"```
And then I installed using `abctl local install --values ./values.yaml`

Does this look right? I'm not sure how to go about using the Postgres RDS instance. I have had a look at this: <https://docs.airbyte.com/deploying-airbyte/integrations/database>

I feel like I'm close, but I'm stuck currently in getting the final bits up and running.

Thanks a lot for any help :pray:

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1724082007357389) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["airbyte", "ec2", "rds-postgres", "s3", "secrets-manager", "values.yaml", "abctl", "documentation"]
</sub>
2

Thanks <@U07C8CCC68Y>. Are you able to store the database credentials in AWS Secret Manager? :pray:

3

Airbyte does not support pulling those secrets from AWS Secret Manager directly, only a k8s secret. I would suggest you look into something like: https://external-secrets.io/latest/ (we have a number of customers who are using that successfully) to achieve what you are looking to do.

4

there is additionally a --secret flag. You will want to create a a secret that contains the keys for each of those services secrets. So create a file called secrets.yaml (or whatever you would like) and it should look something like:

kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  # Database Secrets
  database-password: ## e.g. password

  # AWS S3 Secrets
  s3-access-key-id: ## e.g. AKIAIOSFODNN7EXAMPLE
  s3-secret-access-key: ## e.g. wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

  # AWS Secret Manager
  aws-secret-manager-access-key-id: ## e.g. AKIAIOSFODNN7EXAMPLE
  aws-secret-manager-secret-access-key: ## e.g. wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY```
5

if you are using instance profiles, then you should not need the credentials sections for S3 and Secret Manager