[Kubernetes Deployment] Airbyte's Services (add connectors, new source/destination) Produce Errors v0.40.8

bachdinh1442 · October 10, 2022, 11:01am

Is this your first time deploying Airbyte?: No
OS Version / Instance: Ubuntu 20.4
Memory / Disk:
Deployment: Kubernetes
Airbyte Version: 0.40.8
Source name/version:
Destination name/version:
Step: After deployment
Description:
- Services does not create job pods and produce error

2022-09-27 17:52:41	
2022-09-27 10:52:41 INFO i.a.w.p.KubePodProcess(<init>):520 - Creating pod source-twilio-spec-037cde12-631d-4b71-b610-5da9c9a827b2-0-ayhge...
2022-09-27 17:52:41	
Log4j2Appender says: Creating pod source-twilio-spec-037cde12-631d-4b71-b610-5da9c9a827b2-0-ayhge...
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG c.z.h.p.HikariPool(logPoolState):380 - HikariPool-2 - Before cleanup stats (total=1, active=0, idle=1, waiting=0)
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG c.z.h.p.HikariPool(logPoolState):380 - HikariPool-2 - After cleanup  stats (total=1, active=0, idle=1, waiting=0)
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.h.s.n.d.HttpRequestDecoder(decode):77 - Server main-ab-worker-57455cc8db-qbxvj:9000 Received Request: GET /
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.h.s.n.d.HttpRequestDecoder(decode):77 - Server main-ab-worker-57455cc8db-qbxvj:9000 Received Request: GET /
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.h.s.n.RoutingInBoundHandler(channelRead0):296 - Request GET /
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.h.s.n.RoutingInBoundHandler(channelRead0):296 - Request GET /
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.h.s.n.RoutingInBoundHandler(channelRead0):444 - Matched route GET - / to controller class io.airbyte.workers.controller.HeartbeatController
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.h.s.n.RoutingInBoundHandler(channelRead0):444 - Matched route GET - / to controller class io.airbyte.workers.controller.HeartbeatController
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.s.t.r.DefaultTokenResolver(resolveToken):67 - Request GET, /, no token found.
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.s.t.r.DefaultTokenResolver(resolveToken):67 - Request GET, /, no token found.
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.s.u.LoggingUtils(debug):42 - One or more of the IP patterns matched the host address [240.0.1.1]. Continuing request processing.
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.s.u.LoggingUtils(debug):42 - One or more of the IP patterns matched the host address [240.0.1.1]. Continuing request processing.
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.s.r.AbstractSecurityRule(compareRoles):84 - The given roles [[isAnonymous()]] matched one or more of the required roles [[isAnonymous()]]. Allowing the request
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.s.r.AbstractSecurityRule(compareRoles):84 - The given roles [[isAnonymous()]] matched one or more of the required roles [[isAnonymous()]]. Allowing the request
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.s.f.SecurityFilter(logResult):211 - Authorized request GET /. The rule provider io.micronaut.security.rules.ConfigurationInterceptUrlMapRule authorized the request.
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.s.f.SecurityFilter(logResult):211 - Authorized request GET /. The rule provider io.micronaut.security.rules.ConfigurationInterceptUrlMapRule authorized the request.
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.h.s.n.RoutingInBoundHandler(encodeBodyAsByteBuf):1433 - Encoding emitted response object [{up=true}] using codec: io.micronaut.json.codec.JsonMediaTypeCodec@3f725306
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.h.s.n.RoutingInBoundHandler(encodeBodyAsByteBuf):1433 - Encoding emitted response object [{up=true}] using codec: io.micronaut.json.codec.JsonMediaTypeCodec@3f725306
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.h.s.n.RoutingInBoundHandler(syncWriteAndFlushNettyResponse):1323 - Response 200 - GET /
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.h.s.n.RoutingInBoundHandler(syncWriteAndFlushNettyResponse):1323 - Response 200 - GET /
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestReceivedEvent[source=GET /]
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=GET /]
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.c.e.ApplicationEventPublisherFactory(notifyEventListeners):254 - Established event listeners [io.micronaut.runtime.http.scope.RequestCustomScope@4ffa9dc3] for event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=GET /]
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.c.e.ApplicationEventPublisherFactory(notifyEventListeners):260 - Invoking event listener [io.micronaut.runtime.http.scope.RequestCustomScope@4ffa9dc3] for event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=GET /]
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestReceivedEvent[source=GET /]
2022-09-27 17:52:41	
2022-09-27 10:52:41 DEBUG i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=GET /]
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.c.e.ApplicationEventPublisherFactory(notifyEventListeners):254 - Established event listeners [io.micronaut.runtime.http.scope.RequestCustomScope@4ffa9dc3] for event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=GET /]
2022-09-27 17:52:41	
2022-09-27 10:52:41 TRACE i.m.c.e.ApplicationEventPublisherFactory(notifyEventListeners):260 - Invoking event listener [io.micronaut.runtime.http.scope.RequestCustomScope@4ffa9dc3] for event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=GET /]
2022-09-27 17:52:41	
2022-09-27 10:52:41 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):162 - Completing future exceptionally...
2022-09-27 17:52:41	
java.lang.NoSuchMethodError: 'void okhttp3.internal.platform.Platform.log(int, java.lang.String, java.lang.Throwable)'
2022-09-27 17:52:41	
	at okhttp3.logging.HttpLoggingInterceptor$Logger$1.log(HttpLoggingInterceptor.java:114) ~[logging-interceptor-3.12.12.jar:?]
2022-09-27 17:52:41	
	at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:173) ~[logging-interceptor-3.12.12.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.send(OkHttpClientImpl.java:138) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:574) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:553) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:305) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:48) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:318) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:308) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:523) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.spec(AirbyteIntegrationLauncher.java:68) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:57) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:32) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2022-09-27 17:52:41	
Log4j2Appender says: Completing future exceptionally...
2022-09-27 17:52:41	
2022-09-27 10:52:41 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling...
2022-09-27 17:52:41	
Log4j2Appender says: Stopping cancellation check scheduling...
2022-09-27 17:52:41	
2022-09-27 10:52:41 WARN i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=83b97d62-e29f-395e-9d85-5b18451bbce0, activityType=Run, attempt=1
2022-09-27 17:52:41	
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: 'void okhttp3.internal.platform.Platform.log(int, java.lang.String, java.lang.Throwable)'
2022-09-27 17:52:41	
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
2022-09-27 17:52:41	
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:130) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.temporal.spec.SpecActivityImpl.run(SpecActivityImpl.java:74) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2022-09-27 17:52:41	
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
2022-09-27 17:52:41	
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2022-09-27 17:52:41	
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
2022-09-27 17:52:41	
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
2022-09-27 17:52:41	
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
2022-09-27 17:52:41	
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2022-09-27 17:52:41	
Caused by: java.lang.NoSuchMethodError: 'void okhttp3.internal.platform.Platform.log(int, java.lang.String, java.lang.Throwable)'
2022-09-27 17:52:41	
	at okhttp3.logging.HttpLoggingInterceptor$Logger$1.log(HttpLoggingInterceptor.java:114) ~[logging-interceptor-3.12.12.jar:?]
2022-09-27 17:52:41	
	at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:173) ~[logging-interceptor-3.12.12.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.send(OkHttpClientImpl.java:138) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:574) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:553) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:305) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:48) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:318) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:308) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:523) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.spec(AirbyteIntegrationLauncher.java:68) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:57) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:32) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	... 1 more
2022-09-27 17:52:41	
Log4j2Appender says: Activity failure. ActivityId=83b97d62-e29f-395e-9d85-5b18451bbce0, activityType=Run, attempt=1
2022-09-27 17:52:41	
2022-09-27 10:52:41 e[1;31mERRORe[m i.t.i.s.WorkflowExecuteRunnable(logWorkflowExecutionException):125 - Workflow execution failure WorkflowId=8f4996e4-eda5-4c07-9536-1ddbc2b428ec, RunId=029dd8d8-3dff-4e96-80b9-63983c169f43, WorkflowType=SpecWorkflow
2022-09-27 17:52:41	
io.temporal.failure.ActivityFailure: scheduledEventId=5, startedEventId=6, activityType='Run', activityId='83b97d62-e29f-395e-9d85-5b18451bbce0', identity='1@main-ab-worker-57455cc8db-rqnvv', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
2022-09-27 17:52:41	
	at java.lang.Thread.getStackTrace(Thread.java:1610) ~[?:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:48) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:77) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:70) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at jdk.proxy2.$Proxy79.run(Unknown Source) ~[?:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.temporal.spec.SpecWorkflowImpl.run(SpecWorkflowImpl.java:21) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at SpecWorkflowImplProxy.run$accessor$32B1Jyvw(Unknown Source) ~[?:?]
2022-09-27 17:52:41	
	at SpecWorkflowImplProxy$auxiliary$6rpS1kc0.call(Unknown Source) ~[?:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.temporal.support.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at SpecWorkflowImplProxy.run(Unknown Source) ~[?:?]
2022-09-27 17:52:41	
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2022-09-27 17:52:41	
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
2022-09-27 17:52:41	
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2022-09-27 17:52:41	
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:317) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:292) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.WorkflowExecuteRunnable.run(WorkflowExecuteRunnable.java:72) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:137) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:101) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:111) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
2022-09-27 17:52:41	
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
2022-09-27 17:52:41	
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
2022-09-27 17:52:41	
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
2022-09-27 17:52:41	
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2022-09-27 17:52:41	
Caused by: io.temporal.failure.ApplicationFailure: message='java.lang.NoSuchMethodError: 'void okhttp3.internal.platform.Platform.log(int, java.lang.String, java.lang.Throwable)'', type='java.util.concurrent.ExecutionException', nonRetryable=false
2022-09-27 17:52:41	
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
2022-09-27 17:52:41	
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:130) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at io.airbyte.workers.temporal.spec.SpecActivityImpl.run(SpecActivityImpl.java:74) ~[io.airbyte-airbyte-workers-0.40.8.jar:?]
2022-09-27 17:52:41	
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method:0) ~[?:?]
2022-09-27 17:52:41	
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
2022-09-27 17:52:41	
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2022-09-27 17:52:41	
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
2022-09-27 17:52:41	
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
2022-09-27 17:52:41	
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
2022-09-27 17:52:41	
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
2022-09-27 17:52:41	
	at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
2022-09-27 17:52:41	
Caused by: io.temporal.failure.ApplicationFailure: message=''void okhttp3.internal.platform.Platform.log(int, java.lang.String, java.lang.Throwable)'', type='java.lang.NoSuchMethodError', nonRetryable=false
2022-09-27 17:52:41	
	at okhttp3.logging.HttpLoggingInterceptor$Logger$1.log(HttpLoggingInterceptor.java:114) ~[logging-interceptor-3.12.12.jar:?]
2022-09-27 17:52:41	
	at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:173) ~[logging-interceptor-3.12.12.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2022-09-27 17:52:41	
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62) ~[kubernetes-client-5.12.2.jar:?]
2022-09-27 17:52:41	
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]

We are attempting to upgrade our Aibyte version to a newer one, our current version is 0.39.1 and is working somewhat ok. When we upgrade to version 0.40.9 and deployed, all the services produce errors similar to the logs above (adding connectors, create new source/destination)

Any help is appreciated. Thank you in advance!

marcosmarxm · October 10, 2022, 3:02pm

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

marcosmarxm · October 10, 2022, 7:21pm

Are you using Kustomize or Helm? Can you upload the complete server log file to debugging?

bachdinh1442 · October 11, 2022, 4:24am

Thank you for your response.
We deployed Airbyte using Helm Chart.
I have extracted the server log with period around the time I tested the services.
output_logs_airbyte_error.txt (77.7 KB)

bachdinh1442 · October 12, 2022, 8:47am

I suspect that there’s a version conflict between libraries:

okhttp-4.9.3
okhttp-3.12.12 from logging-interceptor-3.12.12
Are there any ways we could inject older version of okhttp before or after deployment?

bachdinh1442 · October 12, 2022, 10:45am

We recently redeployed Airbyte on our cluster and capture the server log in the meanwhile.

airbyte_server_log_when_deployed_Oct_12th.txt (175.6 KB)

marcosmarxm · October 12, 2022, 10:45am

We recently redeployed Airbyte on our cluster and capture the server log in the meanwhile.

airbyte_server_log_when_deployed_Oct_12th.txt (175.6 KB)

[Discourse post]

marcosmarxm · October 13, 2022, 4:49pm

I don’t see any errors in this server logs; Sorry to ask but could you try to upgrade your Helm Chart to latest version?

Topic		Replies	Views
Kubernetes Deploy: java.io.FileNotFoundException: /root/.kube/config (No such file or directory) Platform, Deploy & Infra Issues kubernetes , deploy	5	1024	July 14, 2022
Airbyte-worker pod failure with CreateContainerConfigError Platform, Deploy & Infra Issues deploy	3	544	November 16, 2022
Destination, orchestrator, and source pod all stuck in terminating Connector Questions & Issues kubernetes	9	806	September 14, 2022
Istio + Airbyte = Failures Platform, Deploy & Infra Issues kubernetes , deploy	3	867	July 25, 2022
Unable to add connectors: get spec job failed Connector Questions & Issues connectors , kubernetes	6	1630	August 12, 2022

[Kubernetes Deployment] Airbyte's Services (add connectors, new source/destination) Produce Errors v0.40.8

Related topics