Error creating connection with Helm deployment

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

When attempting to create a connection in Airbyte deployed via Helm, user encounters an error stating ‘That secret was not found in the store’. User is using AWS SecretsManager and has confirmed the existence of the secret. Seeking guidance on resolving this issue.


Question

:wave: Hi all,

Just deployed community OSS via Helm (0.422.2). I’m able to add sources and destinations fine.

But when I attempt to create a connection I get an error:

That secret was not found in the store! Coordinate: airbyte_workspace_d06f5d58-1770-45d8-89a8-69a3d5f3a885_secret_13ca208c-b4ba-4117-9c65-5d5a508d4ce1_v1
I’m using AWS SecretsManager and have confirmed that that secret exists and that I can read it using the AWS credentials I provided to Airbyte.

I’ve searched for other instances of this issue but they don’t seem related to what I’m experiencing.

Any guidance would be appreciated! :pray:


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["error", "creating-connection", "helm-deployment", "aws-secretsmanager", "airbyte"]
2

_v1 difference at the end between screenshots is not relevant?

3

<@U05JENRCF7C> I thought the same. It looks wrong to me.

I did see this comment in the code: https://github.com/airbytehq/airbyte-platform/blob/1d3c7d459ac5d8121dead1bf823f957c3ba3f18a/airbyte-config/config-secrets/src/main/kotlin/secrets/SecretCoordinate.kt#L20-L27|https://github.com/airbytehq/airbyte-platform/blob/1d3c7d459ac5d8121dead1bf823f957[…]nfig/config-secrets/src/main/kotlin/secrets/SecretCoordinate.kt

4

Airbyte does write the secret and and delete it correctly. Only seems to be an issue reading it in workload-launcher.

5

I saw from <@U01HYQYV1BQ>’s https://airbytehq.slack.com/archives/C01A4CAP81L/p1722546930429579|announcement that workload-launcher is relatively new. I just tried redeploying with 0.383.1 (the version immediately before workload-launcher was introduced) and everything worked as expected.

Given I didn’t change anything else, I’m going to assume this is a bug with the new Workload architecture…

6

Likely related: https://github.com/airbytehq/airbyte/issues/43014

7

Hi, thanks for bringing this up. We’ve noticed this internally as well and will be fixing this over the next few days. I’ll update this thread when the fix is out.

8

Thanks <@U01HYQYV1BQ>, appreciate the update

9

<@U07G3ETG162> yes! This is fixed as of 0.63.17.

Interesting. That isn’t on my radar. Can I see how you are including the tags?

10

For sure. I’m following the pattern for AWS here:

https://docs.airbyte.com/deploying-airbyte/integrations/secrets|https://docs.airbyte.com/deploying-airbyte/integrations/secrets

It was working well on the earlier version.

11

Out of curiosity, does it work after manually copying secret?

12

Yep. It also works if I don’t use SecretsManager (and just use the Airbyte DB). Reviewing the code after I saw that issue https://github.com/airbytehq/airbyte/issues/43014 it really just appears to be that only GCP Persistence is implemented

https://github.com/airbytehq/airbyte-platform/blob/81b12e98fe0492284dab2943480f3740c1743ebf/airbyte-workload-launcher/src/main/kotlin/config/EnvVarConfigBeanFactory.kt#L356-L389

13

Hi <@U01HYQYV1BQ> seeing that this is now working in the latest version :pray:

One note in case it’s not on your radar: The global.secretsManager.awsSecretsManager.tags value is not working as expected. When tags are included per the docs (as a map), airbyte-workload-launcher throws this error:

https://github.com/airbytehq/airbyte-platform/blob/96f8c85cea2ff7f0c580f04a72a82850a92831f1/airbyte-config/config-secrets/src/main/kotlin/secrets/persistence/AwsSecretManagerPersistence.kt#L157-L160|https://github.com/airbytehq/airbyte-platform/blob/96f8c85cea2ff7f0c580f04a72a8285[…]/main/kotlin/secrets/persistence/AwsSecretManagerPersistence.kt

Removing global.secretsManager.awsSecretsManager.tags fixes the issue.

14

Specifically this:

  secretsManager:
    type: awsSecretManager
    awsSecretManager:
      region: us-east-1
      authenticationType: credentials
      tags:
        - key: AirbyteManaged
          value: true
      kms: arn:aws:kms:region:id:key/uuid```