Summary
The Postgres source connector on Airbyte is failing after upgrading the Postgres source DB to version 16 from version 13. The error indicates a failure in the source connection with HTTP 504 error code. The detailed logs show warnings and an exception related to replication slots. The issue seems to be related to the upgrade and configuration settings.
Question
Hi Everyone, we have recently upgraded our Postgres source DB to version 16 from version 13. Since then the source connector on Airbyte (version 0.64.1) is failing with below error:
An unknown error occurred. (HTTP 504)
The sync is failing with below error:
Failure in source: Checking source connection failed - please review this connection's configuration to prevent future syncs from failing
Detailed logs:
2024-09-13 02:30:18 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-13 02:30:18 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-13 02:30:18 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword min - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-13 02:30:18 WARN i.a.w.i.VersionedAirbyteStreamFactory(internalLog):305 - WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword max - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.i.b.s.SshTunnel$Companion(getInstance):423 Starting connection with method: NO_TUNNEL
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresUtils(isCdc):70 using CDC: true
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresSource(toSslJdbcParamInternal):924 VERIFY_CA toSslJdbcParam verify-ca
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main c.z.h.HikariDataSource(<init>):79 HikariPool-1 - Starting...
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main c.z.h.HikariDataSource(<init>):81 HikariPool-1 - Start completed.
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresUtils(isCdc):70 using CDC: true
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(initialize):24 Set initial fetch size: 10 rows
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(accept):33 Set new fetch size: 1304199 rows
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.d.j.s.TwoStageSizeEstimator$Companion(getTargetBufferByteSize):80 Max memory limit: 46204452864, JDBC buffer size: 27722671718
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresCatalogHelper(getPublicizedTables):156 For CDC, only tables in publication airbyte_publication_bookkeeper will be included in the sync: [public.farmer_loan_cash_repayment_claims, public.farmer_cash_loan_imports, public.collector_stats, public.cash_advance_imports, public.double_entry_line_checks, public.payments, public.await_and_notify_synchronizers, public.commission_taxes, public.cash_advance_requests, public.cash_deposit_claims, public.double_entry_lines, public.farmer_loan_repayments, public.loans, public.payment_steps, public.farmer_cash_loan_requests, public.mc_commissions, public.authorizations, public.outstanding_balances, public.procurements, public.settlements, public.locations, public.loan_repayments, public.settlement_adjustments, public.double_entry_account_balances, public.versions, public.farmer_cash_repayments, public.settlement_payable_imports, public.inbound_payments, public.mc_commission_rates, public.aging_reports, public.credit_limits, public.commission_allocations, public.verified_items, public.notifications, public.loan_adjustments, public.procurement_deposit_claims]
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresUtils(isCdc):70 using CDC: true
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresUtils(isXmin):190 using Xmin: false
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.i.s.j.AbstractJdbcSource(getCheckOperations$lambda$6):343 Attempting to get metadata from the database to see if we can connect.
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.i.s.j.AbstractJdbcSource(checkUserHasPrivileges):307 Checking if the user can perform select to any table in schema: public
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(initialize):24 Set initial fetch size: 10 rows
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(accept):33 Set new fetch size: 1230723 rows
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.d.j.s.TwoStageSizeEstimator$Companion(getTargetBufferByteSize):80 Max memory limit: 46204452864, JDBC buffer size: 27722671718
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresSource(lambda$getReplicationSlot$4):426 Attempting to find the named replication slot using the query: HikariProxyPreparedStatement@35864696 wrapping SELECT * FROM pg_replication_slots WHERE slot_name = ('airbyte_slot_bookkeeper') AND plugin = ('pgoutput') AND database = ('bookkeeper')
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(initialize):24 Set initial fetch size: 10 rows
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.i.s.r.AbstractDbSource(check):74 Exception while checking connection: io.airbyte.commons.exceptions.ConfigErrorException: Expected exactly one replication slot but found 0. Please read the docs and add a replication slot to your database.
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main c.z.h.HikariDataSource(close):349 HikariPool-1 - Shutdown initiated...
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main c.z.h.HikariDataSource(close):351 HikariPool-1 - Shutdown completed.
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.c.i.b.IntegrationRunner(runInternal):224 Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
2024-09-13 02:30:18 INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):308 - INFO main i.a.i.s.p.PostgresSource(main):700 completed source: class io.airbyte.integrations.source.postgres.PostgresSource
2024-09-13 02:30:18 INFO i.a.c.ConnectorMessageProcessor(updateConfigFromControlMessage):231 - Checking for optional control message...
2024-09-13 02:30:18 INFO i.a.c.ConnectorWatcher(run):134 - Writing output of decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check to the doc store
2024-09-13 02:30:19 INFO i.a.c.ConnectorWatcher(run):136 - Marking workload as successful
2024-09-13 02:30:19 INFO i.a.c.ConnectorWatcher(exitProperly):189 - Deliberately exiting process with code 0.
2024-09-13 02:30:19 INFO i.a.c.i.LineGobbler(voidCall):166 -
2024-09-13 02:30:19 INFO i.a.c.i.LineGobbler(voidCall):166 - ----- END CHECK -----
2024-09-13 02:30:19 INFO i.a.c.i.LineGobbler(voidCall):166 -
2024-09-13 02:30:20 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=20, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)
Backoff before next attempt: 10 seconds
2024-09-13 02:30:01 INFO i.a.w.l.c.WorkloadApiClient(claim):75 - Claimed: true for decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check via API for local
2024-09-13 02:30:01 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: CHECK_STATUS — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check) — (dataplaneId = local)
2024-09-13 02:30:01 INFO i.a.w.l.p.s.CheckStatusStage(applyStage):59 - No pod found running for workload decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check
2024-09-13 02:30:01 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: BUILD — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check) — (dataplaneId = local)
2024-09-13 02:30:01 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: MUTEX — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check) — (dataplaneId = local)
2024-09-13 02:30:01 INFO i.a.w.l.p.s.EnforceMutexStage(applyStage):50 - No mutex key specified for workload: decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check. Continuing...
2024-09-13 02:30:01 INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: LAUNCH — (workloadId = decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check) — (dataplaneId = local)
2024-09-13 02:30:05 INFO i.a.w.l.c.WorkloadApiClient(updateStatusToLaunched):60 - Attempting to update workload: decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check to LAUNCHED.
2024-09-13 02:30:05 INFO i.a.w.l.p.h.SuccessHandler(accept):60 - Pipeline completed for workload: decd338e-5647-4c0b-adf4-da0e75f5a750_518250_0_check.
2024-09-13 02:30:20 platform > Failing job: 518250, reason: Connection Check Failed a2a5e55d-bf70-46ce-9652-c8b23401ec74```
Can anyone help us understand why it's failing post upgrade? We have self hosted Airbyte on GCP via Helm Chart
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1726636370156029) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["postgres-source-connector", "upgrade", "version-16", "http-504", "replication-slots", "configuration-settings"]
</sub>