Summary
The user is facing a MongoDB CDC replication issue where the source connector exits with a non-zero exit code 2, causing the replication to fail. The error message indicates a connection closure problem with PostgreSQL.
Question
Hi everyone, I am facing a very frustrating issue with MongoDB CDC replication. I define an integration, it works well, but then at some point it starts to fail. The log shows:
"failureOrigin" : "source",
"internalMessage" : "Source process exited with non-zero exit code 2",
"externalMessage" : "Something went wrong within the source connector",
"metadata" : {
"attemptNumber" : 2,
"jobId" : 71135,
"connector_command" : "read"
},
"stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 2\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:365)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:235)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1720356078976
} ]```
Usually reseting does the magic, but it's not a viable solution, and it doesn't always work.
I am monitoring resources etc, it's not a resource issue.
This is the exception on mongodb source:
```2024-07-03 06:08:58 ERROR i.a.s.a.ApiHelper(execute):49 - Unexpected Exception
org.jooq.exception.DataAccessException: SQL [null]; This connection has been closed.
at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
at org.jooq.impl.Tools.translate(Tools.java:3313) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.MetaDataFieldProvider.init(MetaDataFieldProvider.java:163) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.MetaDataFieldProvider.<init>(MetaDataFieldProvider.java:78) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.SQLResultQuery.getFields(SQLResultQuery.java:112) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:261) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:341) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:290) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetch(DefaultDSLContext.java:897) ~[jooq-3.17.8.jar:?]
at io.airbyte.persistence.job.DefaultJobPersistence.lambda$getLastSyncJobForConnections$52(DefaultJobPersistence.java:1105) ~[io.airbyte.airbyte-persistence-job-persistence-0.50.43.jar:?]
at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.50.43.jar:?]
at io.airbyte.db.ExceptionWrappingDatabase.query(ExceptionWrappingDatabase.java:31) ~[io.airbyte.airbyte-db-db-lib-0.50.43.jar:?]
at io.airbyte.persistence.job.DefaultJobPersistence.getLastSyncJobForConnections(DefaultJobPersistence.java:1104) ~[io.airbyte.airbyte-persistence-job-persistence-0.50.43.jar:?]
at io.airbyte.commons.server.handlers.JobHistoryHandler.getLatestSyncJobsForConnections(JobHistoryHandler.java:321) ~[io.airbyte-airbyte-commons-server-0.50.43.jar:?]
at io.airbyte.commons.server.handlers.WebBackendConnectionsHandler.getLatestJobByConnectionId(WebBackendConnectionsHandler.java:186) ~[io.airbyte-airbyte-commons-server-0.50.43.jar:?]
at io.airbyte.commons.server.handlers.WebBackendConnectionsHandler.webBackendListConnectionsForWorkspace(WebBackendConnectionsHandler.java:162) ~[io.airbyte-airbyte-commons-server-0.50.43.jar:?]
at io.airbyte.server.apis.WebBackendApiController.lambda$webBackendListConnectionsForWorkspace$4(WebBackendApiController.java:121) ~[io.airbyte-airbyte-server-0.50.43.jar:?]
at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:28) ~[io.airbyte-airbyte-server-0.50.43.jar:?]
at io.airbyte.server.apis.WebBackendApiController.webBackendListConnectionsForWorkspace(WebBackendApiController.java:119) ~[io.airbyte-airbyte-server-0.50.43.jar:?]
at io.airbyte.server.apis.$WebBackendApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.50.43.jar:?]
at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371) ~[micronaut-inject-3.10.1.jar:3.10.1]
at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.10.1.jar:3.10.1]
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303) ~[micronaut-router-3.10.1.jar:3.10.1]
at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.10.1.jar:3.10.1]
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.10.1.jar:3.10.1]
at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.10.1.jar:3.10.1]
at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.5.jar:3.5.5]
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) ~[reactor-core-3.5.5.jar:3.5.5]
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.5.jar:3.5.5]
at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.5.jar:3.5.5]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.5.jar:3.5.5]
at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53) ~[micronaut-context-3.10.1.jar:3.10.1]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:956) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1825) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:486) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgResultSetMetaData.fetchFieldMetaData(PgResultSetMetaData.java:247) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgResultSetMetaData.isAutoIncrement(PgResultSetMetaData.java:61) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgResultSetMetaData.getColumnTypeName(PgResultSetMetaData.java:326) ~[postgresql-42.5.4.jar:42.5.4]
at org.jooq.impl.MetaDataFieldProvider.init(MetaDataFieldProvider.java:131) ~[jooq-3.17.8.jar:?]```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1720356924505869) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["mongodb-cdc-replication", "source-connector", "non-zero-exit-code", "postgresql-connection-closure"]
</sub>