File Source -> Snowflake Destination stuck

ssassi · April 20, 2022, 10:23pm

Is this your first time deploying Airbyte?: No
OS Version / Instance: AKS
Memory / Disk: NA
Deployment: Kubernetes
Airbyte Version: 0.35.63-alpha
Source name/version: airbyte/source-file 0.2.10
Destination name/version: airbyte/destination 0.4.24
Step: Sync

Trying to load https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv into a snowflake database.
Sometimes it runs, sometimes it get stuck forever.

Logs when stucked:

2022-04-20 21:51:28 e[32mINFOe[m i.a.w.w.WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.35.63-alpha
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):107 - Cloud storage job log path: /workspace/49/0/logs.log
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.63-alpha
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: Class path contains multiple SLF4J bindings.
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):104 - start sync worker. job id: 49 attempt id: 0
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):116 - configured sync modes: {null.nyt_covid_data=full_refresh - overwrite}
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteDestination(start):69 - Running destination...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):113 - Attempting to start pod = destination-snowflake-sync-49-0-pyanf
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):116 - destination-snowflake-sync-49-0-pyanf stdoutLocalPort = 9014
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):119 - destination-snowflake-sync-49-0-pyanf stderrLocalPort = 9015
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):565 - Creating stdout socket server...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):503 - Creating pod...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):583 - Creating stderr socket server...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):305 - Waiting for init container to be ready before copying files...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):318 - Init container present..
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):321 - Init container ready..
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):528 - Copying files...
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: destination_config.json
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/8df621c6-79ad-45ed-930c-6c3348f9819b/destination_config.json dcw-dev123/destination-snowflake-sync-49-0-pyanf:/config/destination_config.json -c init
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:32 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:32 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: destination_catalog.json
2022-04-20 21:51:32 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/09f31d5a-fa4f-4462-ba5c-694eea74a1cc/destination_catalog.json dcw-dev123/destination-snowflake-sync-49-0-pyanf:/config/destination_catalog.json -c init
2022-04-20 21:51:32 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:33 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:33 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: FINISHED_UPLOADING
2022-04-20 21:51:33 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/b4d0031d-1232-4966-a0df-2f07b2f01a4b/FINISHED_UPLOADING dcw-dev123/destination-snowflake-sync-49-0-pyanf:/config/FINISHED_UPLOADING -c init
2022-04-20 21:51:33 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:34 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:34 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):531 - Waiting until pod is ready...
2022-04-20 21:51:36 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):574 - Setting stdout...
2022-04-20 21:51:36 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):586 - Setting stderr...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):546 - Reading pod IP...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):548 - Pod IP: 10.244.3.36
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):551 - Creating stdin socket...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):113 - Attempting to start pod = source-file-sync-49-0-mfpcb
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):116 - source-file-sync-49-0-mfpcb stdoutLocalPort = 9012
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):119 - source-file-sync-49-0-mfpcb stderrLocalPort = 9013
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):565 - Creating stdout socket server...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):503 - Creating pod...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):583 - Creating stderr socket server...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):305 - Waiting for init container to be ready before copying files...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):318 - Init container present..
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):321 - Init container ready..
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):528 - Copying files...
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: source_config.json
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/c13b7035-8e41-448c-b987-7235d2fd618e/source_config.json dcw-dev123/source-file-sync-49-0-mfpcb:/config/source_config.json -c init
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:40 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:40 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: source_catalog.json
2022-04-20 21:51:40 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/a3fc9602-d278-4efe-9247-624545f7669b/source_catalog.json dcw-dev123/source-file-sync-49-0-mfpcb:/config/source_catalog.json -c init
2022-04-20 21:51:41 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:42 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:42 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: FINISHED_UPLOADING
2022-04-20 21:51:42 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/b5ff4be9-056c-4011-84cd-db8ce91cc828/FINISHED_UPLOADING dcw-dev123/source-file-sync-49-0-mfpcb:/config/FINISHED_UPLOADING -c init
2022-04-20 21:51:42 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:43 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:43 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):531 - Waiting until pod is ready...
2022-04-20 21:51:44 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):574 - Setting stdout...
2022-04-20 21:51:44 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):586 - Setting stderr...
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):546 - Reading pod IP...
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):548 - Pod IP: 10.244.3.37
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):555 - Using null stdin output stream...
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):339 - Destination output thread started.
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):158 - Waiting for source and destination threads to complete.
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):279 - Replication thread started.
2022-04-20 21:51:47 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 1000 (70 KB)
2022-04-20 21:51:48 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 2000 (143 KB)
2022-04-20 21:51:48 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 3000 (218 KB)
2022-04-20 21:51:49 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 4000 (293 KB)
2022-04-20 21:51:50 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 5000 (368 KB)
2022-04-20 21:51:50 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 6000 (444 KB)
2022-04-20 21:51:51 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 7000 (520 KB)
2022-04-20 21:51:52 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 8000 (596 KB)
2022-04-20 21:51:52 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 9000 (673 KB)
2022-04-20 21:51:53 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 10000 (749 KB)
2022-04-20 21:51:54 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 11000 (826 KB)
2022-04-20 21:51:54 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 12000 (903 KB)
2022-04-20 21:51:55 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 13000 (980 KB)
2022-04-20 21:51:56 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 14000 (1 MB)
2022-04-20 21:51:56 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 15000 (1 MB)
2022-04-20 21:51:57 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 16000 (1 MB)
2022-04-20 21:51:58 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 17000 (1 MB)
2022-04-20 21:51:59 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 18000 (1 MB)
2022-04-20 21:52:00 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 19000 (1 MB)
2022-04-20 21:52:01 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 20000 (1 MB)
2022-04-20 21:52:02 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 21000 (1 MB)
2022-04-20 21:52:03 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 22000 (1 MB)
2022-04-20 21:52:03 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 23000 (1 MB)
2022-04-20 21:52:04 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 24000 (1 MB)
2022-04-20 21:52:05 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 25000 (1 MB)
2022-04-20 21:52:05 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 26000 (1 MB)
2022-04-20 21:52:06 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 27000 (2 MB)
2022-04-20 21:52:06 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 28000 (2 MB)
2022-04-20 21:52:07 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 29000 (2 MB)
2022-04-20 21:52:08 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 30000 (2 MB)
2022-04-20 21:52:08 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 31000 (2 MB)
2022-04-20 21:52:09 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 32000 (2 MB)
2022-04-20 21:52:09 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 33000 (2 MB)
2022-04-20 21:52:10 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 34000 (2 MB)
2022-04-20 21:52:11 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 35000 (2 MB)
2022-04-20 21:52:11 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 36000 (2 MB)
2022-04-20 21:52:12 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 37000 (2 MB)
2022-04-20 21:52:12 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 38000 (2 MB)
2022-04-20 21:52:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 39000 (2 MB)
2022-04-20 21:52:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 40000 (3 MB)
2022-04-20 21:52:14 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 41000 (3 MB)
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationRunner(createSentryTransaction):339 - Sentry transaction event: 39aa68c6c6d647dc82ba22aeee328257
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):121 - Running integration: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):122 - Command: WRITE
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.d.j.c.SwitchingDestination(getConsumer):65 - Using destination type: INTERNAL_STAGING
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$toWriteConfig$0):96 - Write config: WriteConfig{streamName=nyt_covid_data, namespace=null, outputSchemaName=RAW, tmpTableName=_airbyte_tmp_gvd_nyt_covid_data, outputTableName=_airbyte_raw_nyt_covid_data, syncMode=overwrite}
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m i.a.i.d.b.BufferedStreamConsumer(startTracked):116 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$2):114 - Preparing tmp tables in destination started for 1 streams
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$2):122 - Preparing staging area in destination started for schema RAW stream nyt_covid_data: tmp table: _airbyte_tmp_gvd_nyt_covid_data, stage: 2022/04/20/21/58D96693-0CDC-406B-A0B6-4B39B8A80E9F/
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m c.z.h.HikariDataSource(getConnection):110 - HikariPool-1 - Starting...
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:42 e[32mINFOe[m c.z.h.p.HikariPool(checkFailFast):565 - HikariPool-1 - Added connection net.snowflake.client.jdbc.SnowflakeConnectionV1@ffd4cba
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:42 e[32mINFOe[m c.z.h.HikariDataSource(getConnection):123 - HikariPool-1 - Start completed.
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:43 e[32mINFOe[m i.a.d.j.DefaultJdbcDatabase(lambda$unsafeQuery$1):106 - closing connection
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:43 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$2):133 - Preparing staging area in destination completed for schema RAW stream nyt_covid_data
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:43 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$2):136 - Preparing tmp tables in destination completed.
2022-04-20 21:51:46 e[44msourcee[0m > Reading nyt_covid_data (https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv)...
2022-04-20 21:51:47 e[43mdestinatione[0m > 2022-04-20 21:51:47 e[32mINFOe[m i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):55 - Starting a new buffer for stream nyt_covid_data (current state: 0 bytes in 0 buffers)

harshith · April 21, 2022, 5:12am

Hey @ssassi can you help with more information on the resources like RAM, CPU?

ssassi · April 21, 2022, 2:21pm

This is running on a AKS cluster. We’re using the helm chart on the official airbyte repo. Nodes in that cluster are Standard_D2s_v3 (2 vcpu, 8gb ram).
Resource requests/limits are defined in the helm chart.

harshith · April 23, 2022, 12:41pm

Hey @ssassi can you try increasing the size of ram and CPU and try again

ssassi · April 23, 2022, 10:17pm

What do you mean by increasing cpu/ram?
We’re using aks autoscaler, it brings new nodes automatically to the cluster if needed, based on pods resources requests/limits definitions.
Do you mean we should try changing memory/cpu requests? On which service?
Thanks

harshith · April 25, 2022, 5:18am

Looks like when it is stuck it should be consuming more memory or something of that sort thus asking to increase the memory of the instance you are using. Also this is not related to auto-scaling because once the job is set it gonna only run in that instance

ssassi · April 26, 2022, 12:38pm

I upgraded nodes from 2vcpu/8gb ram to 4vcpu/16gb ram and the issue is still there.
I upgraded to airbyte 0.36.3 and now it recognizes that the sync process is “stuck” and retries after 2 hours. How could I set a shorter timeout so it retries earlier?

ssassi · April 26, 2022, 3:54pm

I’m attaching logs to review.
success_sync.txt (24.2 KB)
failed_sync.txt (32.2 KB)

Thanks.

harshith · May 5, 2022, 6:14am

Hey looks to me like there is some timeout happening over snowflake can you check snowflake logs and see if you can find something

marcosmarxm · July 13, 2022, 12:00am

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.

Topic		Replies	Views
Syncs getting stuck Connector Questions & Issues data-loading , connectors , kubernetes	3	331	February 4, 2023
RuntimeException during sync: "Cannot find pod while trying to retrieve exit code" Connector Questions & Issues data-loading , kubernetes	2	311	September 2, 2022
Sync worker failed Connector Questions & Issues normalization , data-loading , connector-development	5	999	July 14, 2022
Pagerduty to snowflake source failure Connector Questions & Issues connectors	1	164	August 15, 2022
Salesforce connections stuck and subsequent workflow runs failing Connector Questions & Issues source-salesforce , data-loading , connectors , kubernetes	5	423	July 14, 2022

File Source -> Snowflake Destination stuck

Related Topics