Summary
The user is facing an error when trying to install Airbyte locally using secrets.yaml and values.yaml files. The error message indicates a problem with patching the deployment in the Helm chart.
Question
Hi,
I have successfully installed Airbyte using abctl with the --low-resource-mode option.
The issue arises when I try to use secrets.yaml and values.yaml.
Attached are the formats for each file :
secrets.yaml :
kind: Secret
metadata:
name: airbyte-config-secrets
type: Opaque
stringData:
# AWS S3 Secrets
s3-access-key-id: SOME_ACCESS_KEY_ID
s3-secret-access-key: SOME_SECRET_ACCESS_KEY
## database-host: ## e.g. database.internal
## database-port: ## e.g. 5432
## database-name: ## e.g. airbyte
## database-user: ## e.g. airbyte
database-password: "some-password"```
`values.yaml` :
```postgresql:
enabled: false
global:
storage:
type: "S3"
storageSecretName: airbyte-config-secrets # Name of your Kubernetes secret.
bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
log: some.bucket
state: some.bucket
workloadOutput: some.bucket
s3:
region: "ap-southeast-1"
authenticationType: credentials ## Use "credentials" or "instanceProfile"
database:
# -- Secret name where database credentials are stored
secretName: "airbyte-config-secrets" # e.g. "airbyte-config-secrets"
# -- The database host
host: "some-host"
# -- The key within `secretName` where host is stored
#hostSecretKey: "" # e.g. "database-host"
# -- The database port
port: "5432"
# -- The key within `secretName` where port is stored
#portSecretKey: "" # e.g. "database-port"
# -- The database name
database: "some-db"
# -- The key within `secretName` where the database name is stored
#databaseSecretKey: "" # e.g. "database-name"
# -- The database user
user: "some-user" # -- The key within `secretName` where the user is stored
#userSecretKey: "" # e.g. "database-user"
# -- The key within `secretName` where password is stored
passwordSecretKey: "database-password" # e.g."database-password"
server:
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: "airbyte-config-secrets"
key: "s3-access-key-id"
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: "airbyte-config-secrets"
key: "s3-secret-access-key"
- name: STATE_STORAGE_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: "airbyte-config-secrets"
key: "s3-access-key-id"
- name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: "airbyte-config-secrets"
key: "s3-secret-access-key"
- name: STATE_STORAGE_S3_BUCKET_NAME
value: "bucket-log"
- name: STATE_STORAGE_S3_REGION
value: "ap-southeast-1"```
The error that appears when running `abctl local install --values values.yaml --low-resource-mode` is:
```ERROR Failed to install airbyte/airbyte Helm Chart
ERROR Unable to install Airbyte locally
ERROR unable to install airbyte chart: unable to install helm: cannot patch "airbyte-abctl-server" with kind Deployment: The order in patch list: XXXXXXX doesn't match $setElementOrder list:```
Thank you.
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1724125676639869) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["airbyte", "installation", "secrets.yaml", "values.yaml", "error", "helm chart", "local install"]
</sub>
The --secret is singular, we are working on making better error messages for issues like that.
Hi <@U07C8CCC68Y>, Thank you for the clarification. Does this mean we can still set values.yaml in --low-resource mode?
yeah your command would look something like: abctl local install --values values.yaml --secret secrets.yaml --low-resource-mode
I’m still facing the same error message with the command abctl local install --values values.yaml --secret secrets.yaml --low-resource-mode .
enabled: false
global:
database:
# -- Secret name where database credentials are stored
secretName: "airbyte-config-secrets" # e.g. "airbyte-config-secrets"
# -- The database host
host: "<http://rds-poc.mancingmaniamantap.ap-southeast-3.rds.amazonaws.com|rds-poc.mancingmaniamantap.ap-southeast-3.rds.amazonaws.com>"
# -- The key within `secretName` where host is stored
#hostSecretKey: "" # e.g. "database-host"
# -- The database port
port: "5432"
# -- The key within `secretName` where port is stored
#portSecretKey: "" # e.g. "database-port"
# -- The database name
database: "airbyte_db"
# -- The key within `secretName` where the database name is stored
#databaseSecretKey: "" # e.g. "database-name"
# -- The database user
user: "postgres" # -- The key within `secretName` where the user is stored
#userSecretKey: "" # e.g. "database-user"
# -- The key within `secretName` where password is stored
passwordSecretKey: "database-password" # e.g."database-password"```
is there any mistake on `values.yaml`?
I’ve checked using kubectl get pods -n airbyte-abctl temporal error:
airbyte-abctl-airbyte-bootloader 0/1 Completed 0 5m59s
airbyte-abctl-connector-builder-server-7b9c86f854-szmph 1/1 Running 0 5m20s
airbyte-abctl-cron-6bcfc88565-v7dvs 1/1 Running 0 5m20s
airbyte-abctl-pod-sweeper-pod-sweeper-86f7554668-26dqg 1/1 Running 0 5m20s
airbyte-abctl-server-7c78b6dcdc-5q25c 0/1 Running 3 (59s ago) 5m20s
airbyte-abctl-temporal-5d74ffdf47-k5np9 0/1 CrashLoopBackOff 5 (34s ago) 5m20s
airbyte-abctl-webapp-7675dcf766-499cv 1/1 Running 0 5m20s
airbyte-abctl-worker-6654c8d8cb-9tp9q 1/1 Running 0 5m20s
airbyte-abctl-workload-api-server-f9fc75d75-6k8hs 1/1 Running 0 5m20s
airbyte-abctl-workload-launcher-795f797898-w7jd2 0/1 Running 0 5m20s
airbyte-minio-0 1/1 Running 0 5m59s```
error message:
```[Fx] Error returned: received non-nil error from function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".ServerOptionsProvider
/home/builder/temporal/temporal/fx.go:180:
sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal_visibility error: pq: relation "schema_version" does not exist
[Fx] ERROR Failed to initialize custom logger: could not build arguments for function "<http://go.uber.org/fx|go.uber.org/fx>".(*module).constructCustomLogger.func2
/go/pkg/mod/go.uber.org/fx@v1.20.0/module.go:251:
failed to build fxevent.Logger:
could not build arguments for function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".init.func8
/home/builder/temporal/temporal/fx.go:1029:
failed to build log.Logger:
received non-nil error from function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".ServerOptionsProvider
/home/builder/temporal/temporal/fx.go:180:
sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal_visibility error: pq: relation "schema_version" does not exist
Unable to create server. Error: could not build arguments for function "<http://go.uber.org/fx|go.uber.org/fx>".(*module).constructCustomLogger.func2 (/go/pkg/mod/go.uber.org/fx@v1.20.0/module.go:251): failed to build fxevent.Logger: could not build arguments for function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".init.func8 (/home/builder/temporal/temporal/fx.go:1029): failed to build log.Logger: received non-nil error from function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".ServerOptionsProvider (/home/builder/temporal/temporal/fx.go:180): sql schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal_visibility error: pq: relation "schema_version" does not exist.```
Are you using an external dabatase? What version of Postgres are you using? This error means temporal couldn’t create the right schemas in your target database.
<@U01MMSDJGC9> I’m using postgres 16.
Ohh I’ll try using postgres 13. Sorry my mistake.
Postgres 15 is currently our target, but 16 should work as well.
tested with 13 and 15 seems not working as well.
currently test force_ssl is 0 and adding this on values.yaml
extraEnv:
- name: POSTGRES_TLS_ENABLED
value: "true"
- name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION
value: "true"
- name: SQL_TLS_ENABLED
value: "true"
- name: SQL_TLS_DISABLE_HOST_VERIFICATION
value: "true"```
checked on kubectl logs also in temporal
<@U0759B9026S> the database is <postgresql - RDS while connection error: no pg_hba.conf entry for host - Stack Overflow a file>. Can you confirm you have it configured properly? Maybe Postgres 13 this was automtically created and not anymore in Postgres 15.
Hi <@U07C8CCC68Y>,
is it possible for Temporal DB to use a JDBC URL in the values.yaml? I installed it using external database on PostgreSQL 15.5 and encountered the following error in temporal: Caused by: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "{our EC2's IP address}", user "xxx", database "xxx", no encryption. This error not occur in PostgreSQL 13.16.