Troubleshooting AWS Secret Manager integration with Airbyte PostgreSQL connector

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User is facing issues setting up AWS Secret Manager integration with Airbyte PostgreSQL connector. Getting HTTP 500 error when saving PostgreSQL credentials. Looking for help on debugging the issue.


Question

Hi all,

I’m trying to setup a simple EC2 instance of Airbyte with PostgreSQL in RDS, AWS Secret Manager and logs to S3.

I’m struggling quite a bit, and getting close to giving up :disappointed:

I managed to get the PostgreSQL portion up and running now with a lot of trial error. I have then taken down everything again and try to set it up again now with AWS Secret manager using this:

  enabled: false

global:
  database:
    secretName: "airbyte-config-secrets"
    type: "external"
    host: "<POSTRGRES_AWS_HOST>"
    port: "5432"
    database: "airbyte_config"
    user: "airbyte_config"
    passwordSecretKey: "database-password"
  secretsManager:
    type: "awsSecretManager"
    awsSecretManager:
      region: "eu-central-1"
      authenticationType: "instanceProfile"```
It installs fine - I see the tables being created in PostgreSQL and I'm able to login to the UI.. I then tried setting up a connection from "Fake data" -> Postgres and in the step where I have to save the PostgreSQL credentials I now get: `An unexpected error occurred. Please report this if the issue persists. (HTTP 500)`

This worked on my previous attempt without the `secretsManager` portion in values.yaml. So 99% sure it's related to now trying to use this.

I have successfully setup an IAM role with the permission to use AWS Secrets Manager - and from the EC2 I can run `aws secretsmanager list-secrets` to list things - so I think this part should work.

I'm blank in terms of how to debug this? I cannot find any place to see more information in logs or something like that

Thanks a lot for the help :pray:

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1724154270555989) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["aws-secret-manager", "postgresql-connector", "http-500-error", "debugging"]
</sub>
2

It worked when moving to credentials instead of instanceProfile.