Issue with OSS Helm Chart setup for Airbyte BigQuery source creation

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is facing a 502 error while testing a BigQuery source creation from the UI in Airbyte. The issue seems to be related to the webserver failing to find the airbyte-minio-svc. The user tried to reapply the chart with custom values, but encountered errors while trying to create a bucket with Minio. The logs show errors related to ‘no such host’ when trying to access airbyte-minio-svc.


Question

Hi all,

Question about the OSS helm chart setup. I am creating a bigquery source from the UI, but when I test it the progress bar eventually throws a 502 error. When I inspect the logs, I see that the webserver fails to find the airbyte-minio-svc. I tried to reapply the chart with my values and it hangs there trying to create a bucket with minio (not sure which bucket it’s refering to), but when I look at the create-bucket pod, I see these errors

...waiting...
mc: &lt;ERROR&gt; Unable to initialize new alias from the provided credentials. Get "<http://airbyte-minio-svc:9000/probe-bsign-fmthuefpcg3dsz1uoeozvknhm1lu4v/?location=>": dial tcp: lookup airbyte-minio-svc on 34.118.224.10:53: no such host.
...waiting...
mc: &lt;ERROR&gt; Unable to initialize new alias from the provided credentials. Get "<http://airbyte-minio-svc:9000/probe-bsign-gofcrij24b4em6gz13ih2bq5k3zwko/?location=>": dial tcp: lookup airbyte-minio-svc on 34.118.224.10:53: no such host.
...waiting...```
and I still don't see the minio-svc
```╰─ kubectl get svc                                                                                                                                  ─╯
NAME                                           TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE
airbyte-airbyte-connector-builder-server-svc   NodePort       34.118.239.32    &lt;none&gt;         80:32219/TCP   2d16h
airbyte-airbyte-server-svc                     ClusterIP      34.118.235.62    &lt;none&gt;         8001/TCP       2d16h
airbyte-airbyte-webapp-svc                     LoadBalancer   34.118.233.33    &lt;external_ip&gt;  80:32351/TCP   2d16h
airbyte-temporal                               ClusterIP      34.118.239.188   &lt;none&gt;         7233/TCP       2d16h
airbyte-workload-api-server-svc                ClusterIP      34.118.225.22    &lt;none&gt;         8007/TCP       2d16h```
This is the log from the helm upgrade command
```╰─ helm upgrade --debug --install airbyte airbyte/airbyte --values values.yaml                                                                      ─╯
history.go:56: [debug] getting history for release airbyte
upgrade.go:158: [debug] preparing upgrade for airbyte
upgrade.go:166: [debug] performing update for airbyte
upgrade.go:364: [debug] creating upgraded release for airbyte
...
wait.go:104: [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: [debug] creating 1 resource(s)
client.go:712: [debug] Watching for changes to Pod airbyte-minio-create-bucket with timeout of 5m0s
client.go:740: [debug] Add/Modify event for airbyte-minio-create-bucket: ADDED
client.go:799: [debug] Pod airbyte-minio-create-bucket pending
client.go:740: [debug] Add/Modify event for airbyte-minio-create-bucket: MODIFIED
client.go:801: [debug] Pod airbyte-minio-create-bucket running
upgrade.go:476: [debug] warning: Upgrade "airbyte" failed: post-upgrade hooks failed: 1 error occurred:
	* timed out waiting for the condition


Error: UPGRADE FAILED: post-upgrade hooks failed: 1 error occurred:
	* timed out waiting for the condition```
Would appreciate any pointers

cc - <@U07C8CCC68Y> :pray:

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1723822709201969) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["oss-helm-chart", "airbyte-bigquery-source", "502-error", "airbyte-minio-svc", "minio-create-bucket", "post-upgrade-hooks-failed"]
</sub>
2

Were you planning on using GCS or was that just to work around the Minio issue?

3

Was planning to use it eventually, so all good. That being said, I found some bugs / misinformation in the chart, cause the GCS storage secret name was being ignored, and the service was looking for a hardcoded secret name instead, even though I specified my own, and that’s how I ended up using credentialsJson as opposed to credentialsPath Anyways, I will collect more evidence and submit an issue, but I’m unblocked now

4

ok I will take a look at that

5

Can you share your values.yaml file?

6

sure

  enabled: false

externaldatabase:
  host: &lt;ip&gt;
  user: &lt;user&gt;
  database: &lt;db&gt;
  password: ""
  existingsecret: airbyte-config-secrets
  existingsecretpasswordkey: database-password
  port: &lt;port&gt;
  jdbcurl: ""

global:
  airbyteurl: &lt;url&gt;
  auth:
    cookiesecuresetting: "false"
    enabled: true
    instanceadmin:
      # known bug: oss airbyte is using the default `airbyte-auth-secrets`
      # as opposed to this one.
      secretname: airbyte-config-secrets
      firstname: ""
      lastname: ""
      emailsecretkey: instance-admin-email
      passwordsecretkey: instance-admin-password
  database:
    type: external
    user: &lt;user&gt;
    secretname: airbyte-config-secrets
    passwordsecretkey: database-password
    host: &lt;ip&gt;
    database: &lt;db&gt;
    port: &lt;port&gt;

webapp:
  service:
    type: loadbalancer

workload-api-server:
  enabled: true
  ingress:
    enabled: true

worker:
  enabled: true
  replicacount: 3```
7

not sure why the copy-paste messed with the camel casing, but they are named correctly as per the chart

8

update: was able to workaround the issue using GCP as the storage backend. For reference, this is my updated global section of values.yaml

    type: gcs
    #storageSecretName: gcp-cred-secrets # bug, this secrets name is not honored, hence the use of `credentialsJson` instead of `credentialsPath`
    bucket:
      log: &lt;custom-bucket&gt;
      state: &lt;custom-bucket&gt;
      workloadOutput: &lt;custom-bucket&gt;
    gcs:
      authenticationType: credentials
      projectId: &lt;project&gt;
      credentialsJson: &lt;base64 key&gt;```
Note: I had to remove the `secretsManager` portion that's recommended in <https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#migration-steps-2|this guide> as it was causing some weird errors