Error creating pod for check step with kubectl cp failure

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is encountering an error while trying to create a pod for a check step, specifically with a failure related to ‘kubectl cp’. The error seems to be causing the job to fail after multiple retries.


Question

Hi, I am getting below error which is very random, i have also read many posts with similar error but not resolution there, can someone help?

2024-04-05 07:20:04 WARN i.a.w.t.s.a.AppendToAttemptLogActivityImpl(log):55 - Backing off for: 4 minutes 30 seconds.
2024-04-05 07:24:36 INFO i.a.w.t.TemporalAttemptExecution(get):126 - Cloud storage job log path: /workspace/3032/4/logs.log
2024-04-05 07:24:36 INFO i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.33
2024-04-05 07:24:36 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable AWS_ACCESS_KEY_ID: ''
2024-04-05 07:24:36 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable AWS_SECRET_ACCESS_KEY: ''
2024-04-05 07:24:36 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2024-04-05 07:24:36 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-04-05 07:24:36 INFO i.a.w.p.KubeProcessFactory(create):128 - Attempting to start pod = source-snowflake-check-3032-4-udkqt for airbyte/source-snowflake:0.2.2 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@15709af5[cpuRequest=2.0,cpuLimit=,memoryRequest=8Gi,memoryLimit=8Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@67aff193[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@19f26943[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@59c17fd5[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@59c17fd5[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@48f7e12a[hosts=[<http://ao22854.eu-west-1.snowflakecomputing.com|ao22854.eu-west-1.snowflakecomputing.com>, *.<http://datadoghq.com|datadoghq.com>, *.<http://datadoghq.eu|datadoghq.eu>, *.<http://sentry.io|sentry.io>],additionalProperties={}]
2024-04-05 07:24:36 INFO i.a.w.p.KubeProcessFactory(create):132 - source-snowflake-check-3032-4-udkqt stdoutLocalPort = 9024
2024-04-05 07:24:36 INFO i.a.w.p.KubeProcessFactory(create):135 - source-snowflake-check-3032-4-udkqt stderrLocalPort = 9025
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable AWS_ACCESS_KEY_ID: ''
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable AWS_SECRET_ACCESS_KEY: ''
2024-04-05 07:24:36 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable SYNC_JOB_INIT_RETRY_TIMEOUT_MINUTES: '5'
2024-04-05 07:24:36 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):668 - Creating stdout socket server...
2024-04-05 07:24:36 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$11):686 - Creating stderr socket server...
2024-04-05 07:24:36 INFO i.a.w.p.KubePodProcess(&lt;init&gt;):593 - Creating pod source-snowflake-check-3032-4-udkqt...
2024-04-05 07:24:37 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):350 - Waiting for init container to be ready before copying files...
2024-04-05 07:25:15 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):354 - Init container present..
2024-04-05 07:25:17 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):357 - Init container ready..
2024-04-05 07:25:17 INFO i.a.w.p.KubePodProcess(&lt;init&gt;):624 - Copying files...
2024-04-05 07:25:17 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):299 - Uploading file: source_config.json
2024-04-05 07:25:17 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):307 - kubectl cp /tmp/7da080ce-94d9-4aab-826c-81a1901b043d/source_config.json data-platform/source-snowflake-check-3032-4-udkqt:/config/source_config.json -c init
2024-04-05 07:25:17 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):310 - Waiting for kubectl cp to complete
2024-04-05 07:25:17 INFO i.a.w.p.KubePodProcess(cleanup):732 - (pod: data-platform / source-snowflake-check-3032-4-udkqt) - Destroying Kube process.
2024-04-05 07:25:17 INFO i.a.w.p.KubePodProcess(close):809 - (pod: data-platform / source-snowflake-check-3032-4-udkqt) - Closed all resources for pod
2024-04-05 07:25:17 INFO i.a.w.p.KubePodProcess(cleanup):738 - (pod: data-platform / source-snowflake-check-3032-4-udkqt) - Destroyed Kube process.
2024-04-05 07:25:17 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):133 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Failed to create pod for check step
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:188) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:143) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.33.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:136) ~[io.airbyte-airbyte-workers-0.50.33.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:121) ~[io.airbyte-airbyte-workers-0.50.33.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 1
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:326) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.airbyte.workers.process.KubePodProcess.&lt;init&gt;(KubePodProcess.java:625) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:184) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	... 20 more
Caused by: java.io.IOException: kubectl cp failed with exit code 1
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:320) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.airbyte.workers.process.KubePodProcess.&lt;init&gt;(KubePodProcess.java:625) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:184) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	... 20 more
2024-04-05 07:25:17 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2024-04-05 07:25:17 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----
2024-04-05 07:25:17 INFO i.a.c.i.LineGobbler(voidCall):149 - 
2024-04-05 07:25:18 INFO i.a.w.t.s.a.AppendToAttemptLogActivityImpl(log):56 - Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=5, totalCompleteFailures=5, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 13 minutes 30 seconds
2024-04-05 07:25:18 ERROR i.a.w.t.s.a.AppendToAttemptLogActivityImpl(log):54 - Failing job: 3032, reason: Job failed after too many retries for connection e9eac888-ce62-4eae-bc0e-60ca3e880428```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1712643916778419) if you want to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["error", "creating-pod", "check-step", "kubectl-cp", "job-failure"]
</sub>
2

hey Aakash! try to check wether your k9s is able to spin up a pod when you’re having this issue. check k8s events for any errors. this can happen when you have no available cpu/mem on your cluster or you’ve configured job resources too greedy (high resources.request)