Setting up Airbyte on EC2 with RDS Postgres, S3, and Secrets Manager

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

User is struggling to set up Airbyte on EC2 with RDS Postgres, S3 for logging, and Secrets Manager for secrets. They have created a values.yaml file and installed Airbyte using abctl, but unsure about integrating RDS Postgres. Referenced Airbyte documentation.


Question

Hi all,

I’m fairly new to Airbyte - but pretty experienced in setting up AWS.

I have:
:white_check_mark: Completed the Airbyte Udemy course
:white_check_mark: Successfully setup an EC2 Instance with Airbyte using abctl local install
:white_check_mark: Successfully played around with a few sources and destinations.

What I’m struggling with now, and where I find the current documentation a little lacking is trying to setup an EC2 instance with:

  • The Airbyte config in RDS Postgres (instead on local postgres on the EC2)
  • Using S3 buckets for logging
  • Using AWS Secrets Manager for secrets

This is what I have done:

  • Started a new EC2 instance with an IAM role
  • Added inline policies like the ones posted in the thread on this message (with changed bucket name)
  • Created a values.yaml file like this:
  storage:
    type: "S3"
    storageSecretName: "airbyte-config-secrets"
    bucket:
      log: "MY_BUCKET_NAME"
      state: "MY_BUCKET_NAME"
      workloadOutput: "MY_BUCKET_NAME"
    s3:
      region: "eu-central-1"
      authenticationType: "instanceProfile"
  secretsManager:
    type: "awsSecretManager"
    awsSecretManager:
      region: "eu-central-1"
      authenticationType: "instanceProfile"```
And then I installed using `abctl local install --values ./values.yaml`

Does this look right? I'm not sure how to go about using the Postgres RDS instance. I have had a look at this: <https://docs.airbyte.com/deploying-airbyte/integrations/database>

I feel like I'm close, but I'm stuck currently in getting the final bits up and running.

Thanks a lot for any help :pray:

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1724082007357389) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["airbyte", "ec2", "rds-postgres", "s3", "secrets-manager", "values.yaml", "abctl", "documentation"]
</sub>
2

Thanks <@U07C8CCC68Y>. Are you able to store the database credentials in AWS Secret Manager? :pray:

3

Airbyte does not support pulling those secrets from AWS Secret Manager directly, only a k8s secret. I would suggest you look into something like: https://external-secrets.io/latest/ (we have a number of customers who are using that successfully) to achieve what you are looking to do.

4

there is additionally a --secret flag. You will want to create a a secret that contains the keys for each of those services secrets. So create a file called secrets.yaml (or whatever you would like) and it should look something like:

kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  # Database Secrets
  database-password: ## e.g. password

  # AWS S3 Secrets
  s3-access-key-id: ## e.g. AKIAIOSFODNN7EXAMPLE
  s3-secret-access-key: ## e.g. wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

  # AWS Secret Manager
  aws-secret-manager-access-key-id: ## e.g. AKIAIOSFODNN7EXAMPLE
  aws-secret-manager-secret-access-key: ## e.g. wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY```
5

if you are using instance profiles, then you should not need the credentials sections for S3 and Secret Manager

6 
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:CreateSecret",
                "secretsmanager:ListSecrets",
                "secretsmanager:DescribeSecret",
                "secretsmanager:TagResource",
                "secretsmanager:UpdateSecret"
            ],
            "Resource": [
                "*"
            ],
            "Condition": {
                "ForAllValues:StringEquals": {
                    "secretsmanager:ResourceTag/AirbyteManaged": "true"
                }
            }
        }
    ]
}


-----

{
  "Version": "2012-10-17",
  "Statement":
    [
      { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*" },
      {
        "Effect": "Allow",
        "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
        "Resource": "arn:aws:s3:::YOUR-S3-BUCKET-NAME"
      },
      {
        "Effect": "Allow",
        "Action":
          [
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
          ],
        "Resource": "arn:aws:s3:::YOUR-S3-BUCKET-NAME/*"
      }
    ]
}```
7

It should be right <@U07H3FMH3RD> after applying this values it didn’t work?

8

I think I’m a little confused in how to get the DB credentials for the internal Airbyte config DB to be stored in the AWS Secrets Manager and get everything to use that.

I got some hints in the <#C01AHCD885S|> channel as well - and a colleague with Kubernetes experience will also have a look at it tomorrow.

Let’s see if we can get it up and running without the need for internal PostgreSQL, Minio etc.

9

Otherwise I’ll get back :pray:

10

then you abctl install would look like abctl local install --values ./values.yaml --secret ./secrets.yaml