Trouble setting up Airbyte on AWS EKS with external RDS database using Helm chart

slack-user-airbyte · September 18, 2024, 6:13am

Summary

User is facing issues setting up Airbyte on AWS EKS with an external RDS database using the Helm chart. Seeking guidance and help from experienced users.

Question

Hi Team , I am trying to install the open-source version of Airbyte on AWS EKS, using an external database (RDS). I’m encountering issues while setting it up via the Helm chart. If anyone has experience with this, your help and guidance would be greatly appreciated.

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

_{["install", "open-source", "airbyte", "aws-eks", "external-database", "rds", "helm-chart", "troubleshooting"]}

slack-user-airbyte · September 20, 2024, 6:20am

I am getting below error on temporal pod

TEMPORAL_ADDRESS is not set, setting it to 10.5.248.174:7233
PostgreSQL started.
Setup PostgreSQL schema.
2024-09-16T16:16:35.834Z        ERROR   Unable to create SQL database.  {"error": "unable to connect to DB, tried default DB names: postgres,defaultdb, errors: [pq: no pg_hba.conf entry for host \"10.5.248.174\", user \"postgres\", database \"postgres\", no encryption pq: no pg_hba.conf entry for host \"10.5.248.174\", user \"postgres\", database \"defaultdb\", no encryption]", "logging-call-at": "handler.go:94"}
2024/09/16 16:16:35 Loading config; env=docker,zone=,configDir=config
2024/09/16 16:16:35 Loading config files=[config/docker.yaml]
{"level":"info","ts":"2024-09-16T16:16:35.947Z","msg":"Build info.","git-time":"2024-03-22T16:43:28.000Z","git-revision":"92489dd75f17a2daa0a537278c8b6337f71fd704","git-modified":true,"go-arch":"amd64","go-os":"linux","go-version":"go1.22.1","cgo-enabled":false,"server-version":"1.23.0-rc16","debug-mode":false,"logging-call-at":"main.go:148"}
{"level":"info","ts":"2024-09-16T16:16:35.954Z","msg":"dynamic config changed for the key: limit.blobsize.error oldValue: nil newValue: { constraints: {} value: 15728640 }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.954Z","msg":"dynamic config changed for the key: frontend.historymgrnumconns oldValue: nil newValue: { constraints: {} value: 30 }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.954Z","msg":"dynamic config changed for the key: history.historymgrnumconns oldValue: nil newValue: { constraints: {} value: 50 }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: system.advancedvisibilitywritingmode oldValue: nil newValue: { constraints: {} value: off }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: frontend.enableupdateworkflowexecution oldValue: nil newValue: { constraints: {} value: true }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: frontend.enableclientversioncheck oldValue: nil newValue: { constraints: {} value: true }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: history.defaultactivityretrypolicy oldValue: nil newValue: { constraints: {} value: map[BackoffCoefficient:2 InitialIntervalInSeconds:1 MaximumAttempts:0 MaximumIntervalCoefficient:100] }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: history.defaultworkflowretrypolicy oldValue: nil newValue: { constraints: {} value: map[BackoffCoefficient:2 InitialIntervalInSeconds:1 MaximumAttempts:0 MaximumIntervalCoefficient:100] }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: limit.blobsize.warn oldValue: nil newValue: { constraints: {} value: 10485760 }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: frontend.enableupdateworkflowexecutionasyncaccepted oldValue: nil newValue: { constraints: {} value: true }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: frontend.persistencemaxqps oldValue: nil newValue: { constraints: {} value: 3000 }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: frontend.throttledlogrps oldValue: nil newValue: { constraints: {} value: 20 }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"dynamic config changed for the key: history.persistencemaxqps oldValue: nil newValue: { constraints: {} value: 3000 }","logging-call-at":"file_based_client.go:275"}
{"level":"info","ts":"2024-09-16T16:16:35.955Z","msg":"Updated dynamic config","logging-call-at":"file_based_client.go:195"}
{"level":"warn","ts":"2024-09-16T16:16:35.956Z","msg":"Not using any authorizer and flag `--allow-no-auth` not detected. Future versions will require using the flag `--allow-no-auth` if you do not want to set an authorizer.","logging-call-at":"main.go:178"}
[Fx] PROVIDE    *temporal.ServerImpl &lt;= <http://go.temporal.io/server/temporal.NewServerFxImpl()|go.temporal.io/server/temporal.NewServerFxImpl()>
[Fx] PROVIDE    *temporal.serverOptions &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    chan interface {} &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    temporal.synchronizationModeParams &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    *config.Config &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    *config.PProf &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    log.Config &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    resource.ServiceNames &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    resource.NamespaceLogger &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    resolver.ServiceResolver &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    client.AbstractDataStoreFactory &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    visibility.VisibilityStoreFactory &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    searchattribute.Mapper &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    []grpc.UnaryServerInterceptor &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    authorization.Authorizer &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    authorization.ClaimMapper &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    authorization.JWTAudienceMapper &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    log.Logger &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    client.FactoryProvider &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    dynamicconfig.Client &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    encryption.TLSConfigProvider &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    *client.Config &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    client.Client &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    metrics.Handler &lt;= <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] PROVIDE    *dynamicconfig.Collection &lt;= <http://go.temporal.io/server/common/dynamicconfig.NewCollection()|go.temporal.io/server/common/dynamicconfig.NewCollection()>
[Fx] PROVIDE    archiver.ArchivalMetadata &lt;= <http://go.temporal.io/server/common/resource.ArchivalMetadataProvider()|go.temporal.io/server/common/resource.ArchivalMetadataProvider()>
[Fx] PROVIDE    tasks.TaskCategoryRegistry &lt;= <http://go.temporal.io/server/temporal.TaskCategoryRegistryProvider()|go.temporal.io/server/temporal.TaskCategoryRegistryProvider()>
[Fx] PROVIDE    client.FactoryProviderFn &lt;= <http://go.temporal.io/server/temporal.PersistenceFactoryProvider()|go.temporal.io/server/temporal.PersistenceFactoryProvider()>
[Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] &lt;= <http://go.temporal.io/server/temporal.HistoryServiceProvider()|go.temporal.io/server/temporal.HistoryServiceProvider()>
[Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] &lt;= <http://go.temporal.io/server/temporal.MatchingServiceProvider()|go.temporal.io/server/temporal.MatchingServiceProvider()>
[Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] &lt;= <http://go.temporal.io/server/temporal.FrontendServiceProvider()|go.temporal.io/server/temporal.FrontendServiceProvider()>
[Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] &lt;= <http://go.temporal.io/server/temporal.InternalFrontendServiceProvider()|go.temporal.io/server/temporal.InternalFrontendServiceProvider()>
[Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] &lt;= <http://go.temporal.io/server/temporal.WorkerServiceProvider()|go.temporal.io/server/temporal.WorkerServiceProvider()>
[Fx] PROVIDE    *cluster.Config &lt;= <http://go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()|go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()>
[Fx] PROVIDE    config.Persistence &lt;= <http://go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()|go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()>
[Fx] PROVIDE    *pprof.PProfInitializerImpl &lt;= <http://go.temporal.io/server/common/pprof.NewInitializer()|go.temporal.io/server/common/pprof.NewInitializer()>
[Fx] PROVIDE    []trace.SpanExporter &lt;= <http://go.temporal.io/server/temporal.init.func2()|go.temporal.io/server/temporal.init.func2()>
[Fx] SUPPLY     []temporal.ServerOption
[Fx] PROVIDE    fx.Lifecycle &lt;= <http://go.uber.org/fx.New.func1()|go.uber.org/fx.New.func1()>
[Fx] PROVIDE    fx.Shutdowner &lt;= <http://go.uber.org/fx.(*App).shutdowner-fm()|go.uber.org/fx.(*App).shutdowner-fm()>
[Fx] PROVIDE    fx.DotGraph &lt;= <http://go.uber.org/fx.(*App).dotGraph-fm()|go.uber.org/fx.(*App).dotGraph-fm()>
[Fx] RUN        supply: stub([]temporal.ServerOption)
[Fx] RUN        provide: <http://go.temporal.io/server/temporal.ServerOptionsProvider()|go.temporal.io/server/temporal.ServerOptionsProvider()>
[Fx] Error returned: received non-nil error from function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".ServerOptionsProvider
        /home/builder/temporal/temporal/fx.go:180:
sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "10.5.248.174", user "postgres", database "temporal", no encryption
[Fx] ERROR              Failed to initialize custom logger: could not build arguments for function "<http://go.uber.org/fx|go.uber.org/fx>".(*module).constructCustomLogger.func2
        /go/pkg/mod/go.uber.org/fx@v1.20.0/module.go:251:
failed to build fxevent.Logger:
could not build arguments for function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".init.func8
        /home/builder/temporal/temporal/fx.go:1029:
failed to build log.Logger:
received non-nil error from function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".ServerOptionsProvider
        /home/builder/temporal/temporal/fx.go:180:
sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "10.5.248.174", user "postgres", database "temporal", no encryption
Unable to create server. Error: could not build arguments for function "<http://go.uber.org/fx|go.uber.org/fx>".(*module).constructCustomLogger.func2 (/go/pkg/mod/go.uber.org/fx@v1.20.0/module.go:251): failed to build fxevent.Logger: could not build arguments for function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".init.func8 (/home/builder/temporal/temporal/fx.go:1029): failed to build log.Logger: received non-nil error from function "<http://go.temporal.io/server/temporal|go.temporal.io/server/temporal>".ServerOptionsProvider (/home/builder/temporal/temporal/fx.go:180): sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "10.5.248.174", user "postgres", database "temporal", no encryption.```

slack-user-airbyte · September 20, 2024, 6:20am

ERROR Unable to create SQL database.

what do you get for SQL query?
SELECT * FROM pg_user;

slack-user-airbyte · September 20, 2024, 6:20am

Okay I need to check that you need from external RDS right ?

slack-user-airbyte · September 20, 2024, 6:20am

here we go

 usename  | usesysid | usecreatedb | usesuper | userepl | usebypassrls |  passwd  | valuntil |                                         
                                                                                                                                       
        useconfig                                                                                                                      
                                                                  
----------+----------+-------------+----------+---------+--------------+----------+----------+-----------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------
 postgres |    16413 | t           | f        | f       | f            | ******** | infinity | 
 rdsadmin |       10 | t           | t        | t       | t            | ******** | infinity | {TimeZone=utc,log_statement=all,log_min_
error_statement=debug5,log_min_messages=panic,exit_on_error=0,statement_timeout=0,role=rdsadmin,auto_explain.log_min_duration=-1,temp_f
ile_limit=-1,search_path=pg_catalog,synchronous_commit=local,default_tablespace=,stats_fetch_consistency=snapshot,idle_session_timeout=
0,pg_hint_plan.enable_hint=off,default_transaction_read_only=off}
(2 rows)```

slack-user-airbyte · September 20, 2024, 6:20am

Hmm, postgres user has usecreatedb set to t, so permissions shouldn’t be a problem.
I’ll check other settings in my setup and I’ll add more questions in a moment

slack-user-airbyte · September 20, 2024, 6:20am

Yeah I am passing default RDS credentails so it should have everything needed. Also if possible can you share your values.yaml file I can compare it with mine to ensure I am passing RDS parameters correctly. Thank you in advance

slack-user-airbyte · September 20, 2024, 6:20am

I found someone had similar issue https://airbytehq.slack.com/archives/C021JANJ6TY/p1718844732767449 does this mean i need to disable ssl on RDS if its enabled?

slack-user-airbyte · September 20, 2024, 6:20am

I have something like this when it comes to database configuration:

  database:
    type: "external"
    secretName: "airbyte-config-secrets"
    host: "${DATABASE_HOST}"
    port: "${DATABASE_PORT}"
    database: "${DATABASE_NAME}"
    userSecretKey: "DATABASE_USER"
    passwordSecretKey: "DATABASE_PASSWORD"```
do you enforce ssl or something?

slack-user-airbyte · September 20, 2024, 6:20am

I will check on RDS config and parameter group

slack-user-airbyte · September 20, 2024, 6:20am

Hi <@U07MQQ1J611> the ext db postgres version above 13?

slack-user-airbyte · September 20, 2024, 6:20am

<@U0759B9026S> Yes its 16.3 will that be an problem?

slack-user-airbyte · September 20, 2024, 6:20am

Yes seems like we have ssl enabled because I am using default parameter group

slack-user-airbyte · September 21, 2024, 6:18am

I’m facing same issues, set force_ssl to 0 can fix that. But for production is bad i think. Currently looking another option. I’ve tried install on 13.x no issues occurred.

slack-user-airbyte · September 21, 2024, 6:18am

<@U0759B9026S> if you have your RDS in private subnet, why force_ssl = 0 is bad for production?

slack-user-airbyte · September 21, 2024, 6:18am

<@U0759B9026S> So it worked for you with RDS Postgres 13.x is that correct?

slack-user-airbyte · September 21, 2024, 6:18am

it looks like temporal issue and for skynet2 this workaround worked
https://github.com/temporalio/temporal/issues/2293#issuecomment-1826304639

slack-user-airbyte · September 21, 2024, 6:18am

<@U05JENRCF7C> What version of helm chart are you running?

slack-user-airbyte · September 21, 2024, 6:18am

in my project I use 0.551.0 at this moment

slack-user-airbyte · September 21, 2024, 6:18am

And we also keep this flag write so it doesnt spin internal DB

  enabled: false```

Topic		Replies	Views
Stability of Helm chart versions for Airbyte on EKS cluster Platform Questions platform , airbyte , helm-chart , question , eks-cluster	2	7	July 4, 2024
Trouble deploying Airbyte with external database using Helm Platform Questions deployment , platform , helm , airbyte , external-database	3	71	June 20, 2024
Trouble deploying Airbyte on GKE using Helm charts Platform Questions deploy , platform , airbyte , errors , helm-charts	0	47	May 14, 2024
Trouble deploying Airbyte on AWS EKS using Helm Platform Questions kubernetes , platform , helm , question , aws-eks	2	144	June 23, 2024
Installing Airbyte using Helm with S3 storage type on EKS Platform Questions platform , helm , airbyte , eks , install	5	110	May 14, 2024

Trouble setting up Airbyte on AWS EKS with external RDS database using Helm chart

Summary

Question

Related Topics