Summary
The user is facing errors when trying to configure BigQuery or GCS sources/destinations in Airbyte on GKE. The error message indicates issues with creating pods for the check step, resulting in a WorkerException. The user is seeking help to resolve this issue.
Question
Hi All , I have installed airbyte using helm chart latest version on gke 1.26
on trying to configure a source or destination with bigquery or gcs both are failing with below error when the connection pod is trying to get up
Cloud storage job log path: /workspace/707bc687-68cb-4fd6-b47c-69bed2458e9e/0/logs.log
2024-03-06 06:36:13 platform > Executing worker wrapper. Airbyte version: 0.50.56
2024-03-06 06:36:13 platform > Attempt 0 to save workflow id for cancellation
2024-03-06 06:36:13 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: ‘2.0’
2024-03-06 06:36:13 platform >
2024-03-06 06:36:13 platform > ----- START CHECK -----
2024-03-06 06:36:13 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: ‘2.0’
2024-03-06 06:36:13 platform >
2024-03-06 06:36:13 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: ‘0.1’
2024-03-06 06:36:13 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: ‘0.1’
2024-03-06 06:36:13 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ‘’
2024-03-06 06:36:13 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ‘’
2024-03-06 06:36:13 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ‘’
2024-03-06 06:36:13 platform > Attempting to start pod = rce-bigquery-check-707bc687-68cb-4fd6-b47c-69bed2458e9e-0-mngwf for airbyte/source-bigquery:0.4.2 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@4c3f38bb[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@37dec96e[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@5cf52af0[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@72431ad4[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@72431ad4[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2024-03-06 06:36:13 platform > rce-bigquery-check-707bc687-68cb-4fd6-b47c-69bed2458e9e-0-mngwf stdoutLocalPort = 9026
2024-03-06 06:36:13 platform > rce-bigquery-check-707bc687-68cb-4fd6-b47c-69bed2458e9e-0-mngwf stderrLocalPort = 9027
2024-03-06 06:36:13 platform > Creating stdout socket server…
2024-03-06 06:36:13 platform > Creating stderr socket server…
2024-03-06 06:36:13 platform > Creating pod rce-bigquery-check-707bc687-68cb-4fd6-b47c-69bed2458e9e-0-mngwf…
java.net.SocketException: Socket closed
at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:682)
at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698)
at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585)
at java.base/java.net.ServerSocket.accept(ServerSocket.java:543)
at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$10(KubePodProcess.java:699)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
java.net.SocketException: Socket closed
at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:682)
at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698)
at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585)
at java.base/java.net.ServerSocket.accept(ServerSocket.java:543)
at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$11(KubePodProcess.java:717)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
2024-03-06 06:36:24 platform > Unexpected: PodDefinition is null when closing in KubePodProcess. Log an error to help debugging but not blocking the process.
2024-03-06 06:36:24 platform > Unexpected error while checking connection:
io.airbyte.workers.exception.WorkerException: Failed to create pod for check step
did anyone resolve this ?
Additionally I have referred this https://techblog.zozo.com/entry/airbyte-on-gke#terraform_service_account for the ksa ans gsa creation
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.