Summary
The user is facing an issue where the check connection pod errors out when trying to connect multiple sources in Airbyte. The logs show a NullPointerException and an error message indicating no output for checking connection status.
Question
Hello Airbyte Community,
We are facing an odd but interesting issue while using airbyte. While setting up a source (mailchimp) it’s check pod is created and connection is being succeeded. While I connect another source or destination i.e redshift/bigquery, the check connection pod errors out.
While airbyte worker logs are:
Log4j2Appender says:
2024-05-22 16:34:42 INFO i.a.c.i.LineGobbler(voidCall):149 -
Log4j2Appender says: ----- END CHECK -----
2024-05-22 16:34:42 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2024-05-22 16:34:42 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):330 - Temporal heartbeating stopped.
Log4j2Appender says:
2024-05-22 16:34:42 INFO i.a.c.i.LineGobbler(voidCall):149 -
2024-05-22 16:34:42 WARN i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=10c6cf05-ef8e-31fe-8796-3aa961395c4c, activityType=RunWithJobOutput, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:319) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:121) ~[io.airbyte-airbyte-workers-0.50.33.jar:?]
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]```
```Log4j2Appender says: (pod: dev / ion-redshift-check-4438e16d-6b8f-485b-9986-34668445e7fe-0-ujvfq) - Closed all resources for pod
2024-05-22 15:40:48 [32mINFO[m i.a.w.p.KubePodProcess(close):809 - (pod: dev / ion-redshift-check-4438e16d-6b8f-485b-9986-34668445e7fe-0-ujvfq) - Closed all resources for pod
Log4j2Appender says: Check connection job subprocess finished with exit code 3
2024-05-22 15:40:48 [33mWARN[m i.a.w.g.DefaultCheckConnectionWorker(run):110 - Check connection job subprocess finished with exit code 3
Log4j2Appender says: Unexpected error while checking connection:
2024-05-22 15:40:48 [1;31mERROR[m i.a.w.g.DefaultCheckConnectionWorker(run):133 - Unexpected error while checking connection:
io.airbyte.workers.exception.WorkerException: Error checking connection status: no status nor failure reason were outputted
at io.airbyte.workers.WorkerUtils.throwWorkerException(WorkerUtils.java:268) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:120) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.33.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:136) ~[io.airbyte-airbyte-workers-0.50.33.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:121) ~[io.airbyte-airbyte-workers-0.50.33.jar:?]
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
...
2024-05-22 15:40:48 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 -
Log4j2Appender says: ----- END CHECK -----
2024-05-22 15:40:48 [32mINFO[m i.a.c.t.TemporalUtils(withBackgroundHeartbeat):330 - Temporal heartbeating stopped.```
At the same time, logs of airbyte-server pod:
```2024-05-22 15:40:48 [1;31mERROR[m i.a.c.l.Exceptions(swallow):65 - Swallowed error.
java.lang.NullPointerException: Cannot invoke "io.airbyte.config.Metadata.getAdditionalProperties()" because the return value of "io.airbyte.config.FailureReason.getMetadata()" is null
at io.airbyte.persistence.job.errorreporter.JobErrorReporter.getFailureReasonMetadata(JobErrorReporter.java:283) ~[io.airbyte.airbyte-persistence-job-persistence-0.50.33.jar:?]
at io.airbyte.persistence.job.errorreporter.JobErrorReporter.reportJobFailureReason(JobErrorReporter.java:328) ~[io.airbyte.airbyte-persistence-job-persistence-0.50.33.jar:?]
at io.airbyte.persistence.job.errorreporter.JobErrorReporter.reportDestinationCheckJobFailure(JobErrorReporter.java:195) ~[io.airbyte.airbyte-persistence-job-persistence-0.50.33.jar:?]
at io.airbyte.commons.server.scheduler.DefaultSynchronousSchedulerClient.lambda$reportError$4(DefaultSynchronousSchedulerClient.java:293) ~[io.airbyte-airbyte-commons-server-0.50.33.jar:?]
at io.airbyte.commons.lang.Exceptions.swallow(Exceptions.java:63) ~[io.airbyte-airbyte-commons-0.50.33.jar:?]
at io.airbyte.commons.server.scheduler.DefaultSynchronousSchedulerClient.reportError(DefaultSynchronousSchedulerClient.java:286) ~[io.airbyte-airbyte-commons-server-0.50.33.jar:?]
at io.airbyte.commons.server.scheduler.DefaultSynchronousSchedulerClient.execute(DefaultSynchronousSchedulerClient.java:228) ~[io.airbyte-airbyte-commons-server-0.50.33.jar:?]
at io.airbyte.commons.server.scheduler.DefaultSynchronousSchedulerClient.createDestinationCheckConnectionJob(DefaultSynchronousSchedulerClient.java:143) ~[io.airbyte-airbyte-commons-server-0.50.33.jar:?]
at io.airbyte.commons.server.handlers.SchedulerHandler.checkDestinationConnectionFromDestinationCreate(SchedulerHandler.java:341) ~[io.airbyte-airbyte-commons-server-0.50.33.jar:?]
at io.airbyte.server.apis.SchedulerApiController.lambda$executeDestinationCheckConnection$0(SchedulerApiController.java:39) ~[io.airbyte-airbyte-server-0.50.33.jar:?]
at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:27) ~[io.airbyte-airbyte-server-0.50.33.jar:?]
at io.airbyte.server.apis.SchedulerApiController.executeDestinationCheckConnection(SchedulerApiController.java:39) ~[io.airbyte-airbyte-server-0.50.33.jar:?]
at io.airbyte.server.apis.$SchedulerApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.50.33.jar:?]
at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371) ~[micronaut-inject-3.10.1.jar:3.10.1]
at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.10.1.jar:3.10.1]
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303) ~[micronaut-router-3.10.1.jar:3.10.1]
at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.10.1.jar:3.10.1]
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.10.1.jar:3.10.1]
at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.10.1.jar:3.10.1]
at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.5.jar:3.5.5]
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) ~[reactor-core-3.5.5.jar:3.5.5]
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.5.jar:3.5.5]
at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.5.jar:3.5.5]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.5.jar:3.5.5]
at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53) ~[micronaut-context-3.10.1.jar:3.10.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2024-05-22 15:40:48 [32mINFO[m i.m.h.s.n.h.a.e.AccessLog(log):125 - 16-0-101-0.airbyte-airbyte-webapp-svc.dev.svc.cluster.local - - [22/May/2024:15:40:43 +0000] "POST /api/v1/scheduler/destinations/check_connection HTTP/1.0" 200 597```
While airbyte-temporal logs (no errrot at that time):
```{"level":"info","ts":"2024-05-22T15:38:30.264Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"1@airbyte-worker-79744d44ff-jzkqq:926328e8-4d8b-4426-a6ed-b389af190af9","wf-task-queue-type":"Workflow","wf-namespace":"default","lifecycle":"Started","logging-call-at":"taskQueueManager.go:292"}
{"level":"info","ts":"2024-05-22T15:38:30.286Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"1@airbyte-worker-79744d44ff-jzkqq:73310b14-0d08-4dc8-a4b0-6ddb59d02b4f","wf-task-queue-type":"Workflow","wf-namespace":"default","lifecycle":"Started","logging-call-at":"taskQueueManager.go:292"}```
---
To hightlight one error log, It is not telling the reason why it is failing the check connection:
`Caused by: io.airbyte.workers.exception.WorkerException: Error checking connection status: no ./ outputted`
Please help me out here how can I debug this further? Or if you have any resolutions for this.
Thanks in advance!!!
---
*Deployed On*: Kubernetes v1.28 (EKS)
*AIrbyte Version: _v0.50.33_*
*Helm Chart Version: _0.49.6_*
Deployed Via ArgoCD
FYI: <@U0333LTETKQ> <@U05BMQPJMGE> <@U03C20E10G6> <@U038BF2DY72> Team please add if I missed anything.
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1716396543014009) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["airbyte", "check-connection", "source", "destination", "error", "null-pointer-exception", "debugging"]
</sub>
how much data and how many streams do you have for connector