- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu
- Deployment: Kubernetes self deployment
- Airbyte Version: * 0.35.28-alpha
- Step: when creating the connection as well as trying to upgrade a connector
- Description: Im not able to create a source connector or upgrade a source connector. The below logs are being seen
2022-07-04 19:16:34 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):112 - Attempting to start pod = source-jira-sync-9a615e7e-2ed5-4eef-964f-d9404e2cd817-0-rndgf
2022-07-04 19:16:34 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):115 - source-jira-sync-9a615e7e-2ed5-4eef-964f-d9404e2cd817-0-rndgf stdoutLocalPort = 9004
2022-07-04 19:16:34 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):118 - source-jira-sync-9a615e7e-2ed5-4eef-964f-d9404e2cd817-0-rndgf stderrLocalPort = 9005
2022-07-04 19:16:34 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):521 - Creating stdout socket server...
2022-07-04 19:16:34 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):480 - Creating pod...
2022-07-04 19:16:34 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):532 - Creating stderr socket server...
2022-07-04 19:16:34 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):158 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Error while getting checking connection.
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:84) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.WorkerException: Operation: [create] for kind: [Pod] with name: [source-jira-sync-9a615e7e-2ed5-4eef-964f-d9404e2cd817-0-rndgf] in namespace: [airbyte] failed.
at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:149) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
... 3 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [source-jira-sync-9a615e7e-2ed5-4eef-964f-d9404e2cd817-0-rndgf] in namespace: [airbyte] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:373) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:53) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:411) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:394) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?]
at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:481) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:145) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
... 3 more
Caused by: java.io.FileNotFoundException: /root/.kube/config (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:?]
at java.io.FileInputStream.open(FileInputStream.java:216) ~[?:?]
at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:354) ~[jackson-dataformat-yaml-2.13.0.jar:2.13.0]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:15) ~[jackson-dataformat-yaml-2.13.0.jar:2.13.0]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3495) ~[jackson-databind-2.13.0.jar:2.13.0]
at io.fabric8.kubernetes.client.internal.KubeConfigUtils.parseConfig(KubeConfigUtils.java:42) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.intercept(TokenRefreshInterceptor.java:44) ~[kubernetes-client-5.3.1.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[okhttp-3.12.12.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[okhttp-3.12.12.jar:?]
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[kubernetes-client-5.3.1.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[okhttp-3.12.12.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[okhttp-3.12.12.jar:?]
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createApplicableInterceptors$6(HttpClientUtils.java:284) ~[kubernetes-client-5.3.1.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[okhttp-3.12.12.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[okhttp-3.12.12.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257) ~[okhttp-3.12.12.jar:?]
at okhttp3.RealCall.execute(RealCall.java:93) ~[okhttp-3.12.12.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:485) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:448) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:263) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:870) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:365) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:53) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:411) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:394) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?]
at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:481) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:145) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
... 3 more
2022-07-04 19:16:34 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-07-04 19:16:34 e[33mWARNe[m i.t.i.s.POJOActivityTaskHandler(activityFailureToResult):363 - Activity failure. ActivityId=187e072b-5cfe-3281-aba7-29da2e6b9c42, activityType=Run, attempt=1
java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Error while getting checking connection.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:129) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.run(CheckConnectionActivityImpl.java:78) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor454.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:286) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:252) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:209) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:193) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:151) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.6.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.WorkerException: Error while getting checking connection.
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:84) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
... 1 more
Caused by: io.airbyte.workers.WorkerException: Operation: [create] for kind: [Pod] with name: [source-jira-sync-9a615e7e-2ed5-4eef-964f-d9404e2cd817-0-rndgf] in namespace: [airbyte] failed.
at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:149) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
... 1 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [source-jira-sync-9a615e7e-2ed5-4eef-964f-d9404e2cd817-0-rndgf] in namespace: [airbyte] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:373) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:53) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:411) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:394) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?]
at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:481) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:145) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
... 1 more
Caused by: java.io.FileNotFoundException: /root/.kube/config (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:?]
at java.io.FileInputStream.open(FileInputStream.java:216) ~[?:?]
at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:354) ~[jackson-dataformat-yaml-2.13.0.jar:2.13.0]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:15) ~[jackson-dataformat-yaml-2.13.0.jar:2.13.0]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3495) ~[jackson-databind-2.13.0.jar:2.13.0]
at io.fabric8.kubernetes.client.internal.KubeConfigUtils.parseConfig(KubeConfigUtils.java:42) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.intercept(TokenRefreshInterceptor.java:44) ~[kubernetes-client-5.3.1.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[okhttp-3.12.12.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[okhttp-3.12.12.jar:?]
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68) ~[kubernetes-client-5.3.1.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[okhttp-3.12.12.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[okhttp-3.12.12.jar:?]
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createApplicableInterceptors$6(HttpClientUtils.java:284) ~[kubernetes-client-5.3.1.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[okhttp-3.12.12.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[okhttp-3.12.12.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257) ~[okhttp-3.12.12.jar:?]
at okhttp3.RealCall.execute(RealCall.java:93) ~[okhttp-3.12.12.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:485) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:448) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:263) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:870) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:365) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:53) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:411) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:394) ~[kubernetes-client-5.3.1.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:86) ~[kubernetes-client-5.3.1.jar:?]
at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:481) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:145) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.28-alpha.jar:?]
... 1 more
Hello Saif, there is an PR correcting this: https://github.com/airbytehq/airbyte/pull/11475/files
Could you try to change your files locally and see if solves the issue?
Could you try latest version too? Looks there were some updates in k8s trying to correct this.
@marcosmarxm that worked. Thank you!
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.