Summary
The user is facing issues with a MySQL connection in Airbyte, receiving a ‘Failed to obtain JDBC Connection’ error due to ‘remaining connection slots are reserved for non-replication superuser connections’. The source check failed, leading to a retry backoff period.
Question
Hello i have a problem with a mysql connection which was working norammly on airbyte :
2024-10-21 12:28:30 INFO i.a.w.t.s.ConnectionManagerWorkflowImpl(checkConnections):583 - SOURCE CHECK: Failed
2024-10-21 12:28:30 INFO i.a.w.t.s.ConnectionManagerWorkflowImpl(checkConnections):590 - DESTINATION CHECK: Skipped, source check failed
2024-10-21 12:28:30 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=4, totalCompleteFailures=4, successivePartialFailures=0, totalPartialFailures=0)
Backoff before next attempt: 4 minutes 30 seconds
2024-10-21 12:28:30 INFO i.a.w.t.s.ConnectionManagerWorkflowImpl(ensureCleanJobState):822 - This workflow is already attached to a job, so no need to clean job state.
2024-10-21 12:28:30 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=4, totalCompleteFailures=4, successivePartialFailures=0, totalPartialFailures=0)
2024-10-21 12:28:30 platform > Backing off for: 4 minutes 30 seconds.
2024-10-21 12:33:01 INFO i.a.w.t.s.ConnectionManagerWorkflowImpl(checkConnections):571 - SOURCE CHECK: Starting
2024-10-21 12:33:31 WARN i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):126 - Activity failure. ActivityId=031fb0b4-b63b-30ed-bcea-ec24e62c735c, activityType=RunWithJobOutput, attempt=1
io.micronaut.data.connection.jdbc.exceptions.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection
at io.micronaut.data.connection.jdbc.operations.DefaultDataSourceConnectionOperations.openConnection(DefaultDataSourceConnectionOperations.java:59) ~[micronaut-data-connection-jdbc-4.7.1.jar:4.7.1]
at io.micronaut.data.connection.jdbc.operations.DefaultDataSourceConnectionOperations.openConnection(DefaultDataSourceConnectionOperations.java:43) ~[micronaut-data-connection-jdbc-4.7.1.jar:4.7.1]
at io.micronaut.data.connection.support.AbstractConnectionOperations.executeWithNewConnection(AbstractConnectionOperations.java:137) ~[micronaut-data-connection-4.7.1.jar:4.7.1]
at io.micronaut.data.connection.support.AbstractConnectionOperations.execute(AbstractConnectionOperations.java:90) ~[micronaut-data-connection-4.7.1.jar:4.7.1]
at io.micronaut.transaction.support.AbstractTransactionOperations.doExecute(AbstractTransactionOperations.java:120) ~[micronaut-data-tx-4.7.1.jar:4.7.1]
at io.micronaut.transaction.support.AbstractPropagatedStatusTransactionOperations.execute(AbstractPropagatedStatusTransactionOperations.java:65) ~[micronaut-data-tx-4.7.1.jar:4.7.1]
at io.micronaut.transaction.interceptor.TransactionalInterceptor.intercept(TransactionalInterceptor.java:140) ~[micronaut-data-tx-4.7.1.jar:4.7.1]
at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:138) ~[micronaut-aop-4.4.10.jar:4.4.10]
at io.airbyte.config.secrets.persistence.$LocalTestingSecretPersistence$Definition$Intercepted.read(Unknown Source) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at io.airbyte.config.secrets.SecretsHelpers.getOrThrowSecretValue(SecretsHelpers.kt:284) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at io.airbyte.config.secrets.SecretsHelpers.combineConfig(SecretsHelpers.kt:173) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at io.airbyte.config.secrets.SecretsHelpers$combineConfig$1.invoke(SecretsHelpers.kt:183) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at io.airbyte.config.secrets.SecretsHelpers$combineConfig$1.invoke(SecretsHelpers.kt:177) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at io.airbyte.config.secrets.SecretsHelpers.combineConfig$lambda$2(SecretsHelpers.kt:177) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at io.airbyte.config.secrets.SecretsHelpers.combineConfig(SecretsHelpers.kt:177) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at io.airbyte.config.secrets.hydration.RealSecretsHydrator.hydrateFromDefaultSecretPersistence(RealSecretsHydrator.kt:21) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at io.airbyte.config.secrets.SecretsRepositoryReader.hydrateConfigFromDefaultSecretPersistence(SecretsRepositoryReader.kt:60) ~[io.airbyte.airbyte-config-config-secrets-0.62.1.jar:?]
at io.airbyte.workers.ConnectorSecretsHydrator.hydrateConfig(ConnectorSecretsHydrator.kt:33) ~[io.airbyte-airbyte-commons-worker-0.62.1.jar:?]
at io.airbyte.workers.CheckConnectionInputHydrator.getHydratedStandardCheckInput(CheckConnectionInputHydrator.kt:13) ~[io.airbyte-airbyte-commons-worker-0.62.1.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:206) ~[io.airbyte-airbyte-workers-0.62.1.jar:?]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.sql.SQLTransientConnectionException: config-pool - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=0)
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:686) ~[HikariCP-5.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:179) ~[HikariCP-5.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:144) ~[HikariCP-5.1.0.jar:?]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:99) ~[HikariCP-5.1.0.jar:?]
at io.micronaut.data.connection.jdbc.operations.DefaultDataSourceConnectionOperations.openConnection(DefaultDataSourceConnectionOperations.java:57) ~[micronaut-data-connection-jdbc-4.7.1.jar:4.7.1]
… 33 more
Caused by: org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725) ~[postgresql-42.7.3.jar:42.7.3]
at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2837) ~[postgresql-42.7.3.jar:42.7.3]
at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:175) ~[postgresql-42.7.3.jar:42.7.3]
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:317) ~[postgresql-42.7.3.jar:42.7.3]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) ~[postgresql-42.7.3.jar:42.7.3]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:273) ~[postgresql-42.7.3.jar:42.7.3]
at org.postgresql.Driver.makeConnection(Driver.java:446) ~[postgresql-42.7.3.jar:42.7.3]
at org.postgresql.Driver.connect(Driver.java:298) ~[postgresql-42.7.3.jar:42.7.3]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:137) ~[HikariCP-5.1.0.jar:?]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:360) ~[HikariCP-5.1.0.jar:?]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:202) ~[HikariCP-5.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:461) ~[HikariCP-5.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:724) ~[HikariCP-5.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:703) ~[HikariCP-5.1.0.jar:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
… 3 more
2024-10-21 12:33:31 ERROR i.a.w.t.FailureConverter(getFailureReason-Q2Q30fc):32 - exception classified as NOT_A_TIMEOUT
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.