Error Configuring BigQuery and GCS Sources/Destinations on GKE

Summary

User encounters a SocketException when trying to configure BigQuery and GCS sources/destinations in Airbyte deployed on GKE using Helm. The error occurs during the pod creation process, indicating a failure in establishing socket connections.


Question

Hi All , I have installed airbyte using helm chart latest version on gke 1.26
on trying to configure a source or destination with bigquery or gcs both are failing with below error when the connection pod is trying to get up
Cloud storage job log path: /workspace/707bc687-68cb-4fd6-b47c-69bed2458e9e/0/logs.log
2024-03-06 06:36:13 platform > Executing worker wrapper. Airbyte version: 0.50.56
2024-03-06 06:36:13 platform > Attempt 0 to save workflow id for cancellation
2024-03-06 06:36:13 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: ‘2.0’
2024-03-06 06:36:13 platform >
2024-03-06 06:36:13 platform > ----- START CHECK -----
2024-03-06 06:36:13 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: ‘2.0’
2024-03-06 06:36:13 platform >
2024-03-06 06:36:13 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: ‘0.1’
2024-03-06 06:36:13 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: ‘0.1’
2024-03-06 06:36:13 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ‘’
2024-03-06 06:36:13 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ‘’
2024-03-06 06:36:13 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ‘’
2024-03-06 06:36:13 platform > Attempting to start pod = rce-bigquery-check-707bc687-68cb-4fd6-b47c-69bed2458e9e-0-mngwf for airbyte/source-bigquery:0.4.2 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@4c3f38bb[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@37dec96e[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@5cf52af0[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@72431ad4[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@72431ad4[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2024-03-06 06:36:13 platform > rce-bigquery-check-707bc687-68cb-4fd6-b47c-69bed2458e9e-0-mngwf stdoutLocalPort = 9026
2024-03-06 06:36:13 platform > rce-bigquery-check-707bc687-68cb-4fd6-b47c-69bed2458e9e-0-mngwf stderrLocalPort = 9027
2024-03-06 06:36:13 platform > Creating stdout socket server…
2024-03-06 06:36:13 platform > Creating stderr socket server…
2024-03-06 06:36:13 platform > Creating pod rce-bigquery-check-707bc687-68cb-4fd6-b47c-69bed2458e9e-0-mngwf…
java.net.SocketException: Socket closed
at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:682)
at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698)
at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585)
at java.base/java.net.ServerSocket.accept(ServerSocket.java:543)
at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$10(KubePodProcess.java:699)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
java.net.SocketException: Socket closed
at java.base/sun.nio.ch.NioSocketImpl.endAccept(NioSocketImpl.java:682)
at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698)
at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639)
at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585)
at java.base/java.net.ServerSocket.accept(ServerSocket.java:543)
at io.airbyte.workers.process.KubePodProcess.lambda$setupStdOutAndStdErrListeners$11(KubePodProcess.java:717)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
2024-03-06 06:36:24 platform > Unexpected: PodDefinition is null when closing in KubePodProcess. Log an error to help debugging but not blocking the process.
2024-03-06 06:36:24 platform > Unexpected error while checking connection:
io.airbyte.workers.exception.WorkerException: Failed to create pod for check step
did anyone resolve this ?
Additionally I have referred this https://techblog.zozo.com/entry/airbyte-on-gke#terraform_service_account for the ksa ans gsa creation



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['airbyte', 'gke', 'helm', 'bigquery', 'gcs', 'socket-exception', 'pod-creation']