Summary
User experiencing connection issues between S3 and Redshift via Airbyte, despite correct configuration. Error logs indicate failure during connection check with specific warning messages related to unknown keywords and a final failure message: ‘Connection Check Failed’.
Question
Hello Community,
I am struggling to resolve an issue.
I connect between s3(source) and redshift(destination) via airbyte.
I have granted airbyte_user access to Redshift, as the documentation says. I have to mention that Redshift is in a VPC, so I configured and tested the bastion host so I could connect to Redshift from the bastion host. I also added the following as jdbc parameters: sslmode=require.
Everything is well configured, but here is the log I get. I really need your help.
2024-12-06 23:35:54 platform INFO APPLY Stage: BUILD — (workloadId = f7a7d195-377f-cf5b-70a5-be6b819019dc_23492767_0_check) — (dataplaneId = prod-dataplane-gcp-us-west3-0)
2024-12-06 23:35:54 platform INFO APPLY Stage: MUTEX — (workloadId = f7a7d195-377f-cf5b-70a5-be6b819019dc_23492767_0_check) — (dataplaneId = prod-dataplane-gcp-us-west3-0)
2024-12-06 23:35:54 platform INFO No mutex key specified for workload: f7a7d195-377f-cf5b-70a5-be6b819019dc_23492767_0_check. Continuing...
2024-12-06 23:35:54 platform INFO APPLY Stage: LAUNCH — (workloadId = f7a7d195-377f-cf5b-70a5-be6b819019dc_23492767_0_check) — (dataplaneId = prod-dataplane-gcp-us-west3-0)
2024-12-06 23:35:54 platform INFO [initContainer] image: airbyte/workload-init-container:build-987f150ebc-20750-master resources: ResourceRequirements(claims=[], limits={memory=800Mi, cpu=4}, requests={memory=800Mi, cpu=4}, additionalProperties={})
2024-12-06 23:36:07 platform INFO Attempting to update workload: f7a7d195-377f-cf5b-70a5-be6b819019dc_23492767_0_check to LAUNCHED.
2024-12-06 23:36:07 platform INFO Pipeline completed for workload: f7a7d195-377f-cf5b-70a5-be6b819019dc_23492767_0_check.
2024-12-06 23:36:13 platform INFO
2024-12-06 23:36:13 platform INFO ----- START CHECK -----
2024-12-06 23:36:13 platform INFO
2024-12-06 23:37:24 platform INFO Connector exited, processing output
2024-12-06 23:37:24 platform INFO Output file jobOutput.json found
2024-12-06 23:37:24 platform INFO Connector exited with exit code 0
2024-12-06 23:37:24 platform INFO Reading messages from protocol version 0.2.0
2024-12-06 23:37:24 platform WARN MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient. Using a dummy client for now. Ignore this if Airbyte is configured to not publish any metrics.
2024-12-06 23:37:24 platform INFO main i.a.i.d.r.RedshiftDestination$Companion(main):518 starting destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-12-06 23:37:24 platform INFO main i.a.c.i.b.IntegrationCliParser$Companion(parseOptions):144 integration args: {check=null, config=/config/connectionConfiguration.json}
2024-12-06 23:37:24 platform INFO main i.a.c.i.b.IntegrationRunner(runInternal):130 Running integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedDestination
2024-12-06 23:37:24 platform INFO main i.a.c.i.b.IntegrationRunner(runInternal):131 Command: CHECK
2024-12-06 23:37:24 platform INFO main i.a.c.i.b.IntegrationRunner(runInternal):132 Integration config: IntegrationConfig{command=CHECK, configPath='/config/connectionConfiguration.json', catalogPath='null', statePath='null'}
2024-12-06 23:37:24 platform WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
2024-12-06 23:37:24 platform WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
2024-12-06 23:37:24 platform WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
2024-12-06 23:37:24 platform WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
2024-12-06 23:37:24 platform WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
2024-12-06 23:37:24 platform WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword groups - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword
2024-12-06 23:37:24 platform INFO main i.a.c.i.b.s.SshTunnel$Companion(getInstance):423 Starting connection with method: SSH_KEY_AUTH
2024-12-06 23:37:24 platform INFO main o.a.s.c.u.s.AbstractSecurityProviderRegistrar(getOrCreateProvider):112 getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
2024-12-06 23:37:24 platform INFO main o.a.s.c.i.DefaultIoServiceFactoryFactory(getIoServiceProvider):70 Using MinaServiceFactoryFactory
2024-12-06 23:37:24 platform INFO main o.a.s.c.c.h.ConfigFileHostEntryResolver(resolveEffectiveResolver):89 resolveEffectiveResolver(ec2-user@51.44.99.129:22/null) no configuration file at /root/.ssh/config
2024-12-06 23:37:24 platform INFO main i.a.c.i.b.IntegrationRunner(runInternal):224 Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedDestination
2024-12-06 23:37:24 platform INFO main i.a.i.d.r.RedshiftDestination$Companion(main):520 completed destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2024-12-06 23:37:24 platform INFO Checking for optional control message...
2024-12-06 23:37:24 platform INFO Writing output of f7a7d195-377f-cf5b-70a5-be6b819019dc_23492767_0_check to the doc store
2024-12-06 23:37:24 platform INFO Marking workload f7a7d195-377f-cf5b-70a5-be6b819019dc_23492767_0_check as successful
2024-12-06 23:37:24 platform INFO
2024-12-06 23:37:24 platform INFO Deliberately exiting process with code 0.
2024-12-06 23:37:24 platform INFO ----- END CHECK -----
2024-12-06 23:37:24 platform INFO
2024-12-06 23:37:25 platform INFO Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalPartialFailureLimit=20, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)
Backoff before next attempt: 10 seconds
2024-12-06 23:37:25 platform ERROR Failing job: 23492767, reason: Connection Check Failed 3ea68578-4d00-49da-a885-4ae9718039b7```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1733525811427159) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
['s3', 'redshift', 'connection-issue', 'airbyte', 'jdbc-parameters', 'bastion-host', 'sslmode', 'vpc', 'error-logs']
</sub>