File Source -> Snowflake Destination stuck

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: AKS
  • Memory / Disk: NA
  • Deployment: Kubernetes
  • Airbyte Version: 0.35.63-alpha
  • Source name/version: airbyte/source-file 0.2.10
  • Destination name/version: airbyte/destination 0.4.24
  • Step: Sync

Trying to load https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv into a snowflake database.
Sometimes it runs, sometimes it get stuck forever.

Logs when stucked:

2022-04-20 21:51:28 e[32mINFOe[m i.a.w.w.WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.35.63-alpha
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):107 - Cloud storage job log path: /workspace/49/0/logs.log
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.63-alpha
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: Class path contains multiple SLF4J bindings.
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):104 - start sync worker. job id: 49 attempt id: 0
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):116 - configured sync modes: {null.nyt_covid_data=full_refresh - overwrite}
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteDestination(start):69 - Running destination...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):113 - Attempting to start pod = destination-snowflake-sync-49-0-pyanf
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):116 - destination-snowflake-sync-49-0-pyanf stdoutLocalPort = 9014
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):119 - destination-snowflake-sync-49-0-pyanf stderrLocalPort = 9015
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):565 - Creating stdout socket server...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):503 - Creating pod...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):583 - Creating stderr socket server...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):305 - Waiting for init container to be ready before copying files...
2022-04-20 21:51:29 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):318 - Init container present..
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):321 - Init container ready..
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):528 - Copying files...
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: destination_config.json
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/8df621c6-79ad-45ed-930c-6c3348f9819b/destination_config.json dcw-dev123/destination-snowflake-sync-49-0-pyanf:/config/destination_config.json -c init
2022-04-20 21:51:31 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:32 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:32 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: destination_catalog.json
2022-04-20 21:51:32 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/09f31d5a-fa4f-4462-ba5c-694eea74a1cc/destination_catalog.json dcw-dev123/destination-snowflake-sync-49-0-pyanf:/config/destination_catalog.json -c init
2022-04-20 21:51:32 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:33 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:33 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: FINISHED_UPLOADING
2022-04-20 21:51:33 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/b4d0031d-1232-4966-a0df-2f07b2f01a4b/FINISHED_UPLOADING dcw-dev123/destination-snowflake-sync-49-0-pyanf:/config/FINISHED_UPLOADING -c init
2022-04-20 21:51:33 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:34 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:34 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):531 - Waiting until pod is ready...
2022-04-20 21:51:36 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):574 - Setting stdout...
2022-04-20 21:51:36 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):586 - Setting stderr...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):546 - Reading pod IP...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):548 - Pod IP: 10.244.3.36
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):551 - Creating stdin socket...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):113 - Attempting to start pod = source-file-sync-49-0-mfpcb
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):116 - source-file-sync-49-0-mfpcb stdoutLocalPort = 9012
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):119 - source-file-sync-49-0-mfpcb stderrLocalPort = 9013
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):565 - Creating stdout socket server...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):503 - Creating pod...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):583 - Creating stderr socket server...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):305 - Waiting for init container to be ready before copying files...
2022-04-20 21:51:37 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):318 - Init container present..
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(waitForInitPodToRun):321 - Init container ready..
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):528 - Copying files...
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: source_config.json
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/c13b7035-8e41-448c-b987-7235d2fd618e/source_config.json dcw-dev123/source-file-sync-49-0-mfpcb:/config/source_config.json -c init
2022-04-20 21:51:39 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:40 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:40 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: source_catalog.json
2022-04-20 21:51:40 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/a3fc9602-d278-4efe-9247-624545f7669b/source_catalog.json dcw-dev123/source-file-sync-49-0-mfpcb:/config/source_catalog.json -c init
2022-04-20 21:51:41 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:42 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:42 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):254 - Uploading file: FINISHED_UPLOADING
2022-04-20 21:51:42 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):262 - kubectl cp /tmp/b5ff4be9-056c-4011-84cd-db8ce91cc828/FINISHED_UPLOADING dcw-dev123/source-file-sync-49-0-mfpcb:/config/FINISHED_UPLOADING -c init
2022-04-20 21:51:42 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - Waiting for kubectl cp to complete
2022-04-20 21:51:43 e[32mINFOe[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):279 - kubectl cp complete, closing process
2022-04-20 21:51:43 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):531 - Waiting until pod is ready...
2022-04-20 21:51:44 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):574 - Setting stdout...
2022-04-20 21:51:44 e[32mINFOe[m i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):586 - Setting stderr...
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):546 - Reading pod IP...
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):548 - Pod IP: 10.244.3.37
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.p.KubePodProcess(<init>):555 - Using null stdin output stream...
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):339 - Destination output thread started.
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):158 - Waiting for source and destination threads to complete.
2022-04-20 21:51:45 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):279 - Replication thread started.
2022-04-20 21:51:47 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 1000 (70 KB)
2022-04-20 21:51:48 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 2000 (143 KB)
2022-04-20 21:51:48 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 3000 (218 KB)
2022-04-20 21:51:49 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 4000 (293 KB)
2022-04-20 21:51:50 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 5000 (368 KB)
2022-04-20 21:51:50 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 6000 (444 KB)
2022-04-20 21:51:51 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 7000 (520 KB)
2022-04-20 21:51:52 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 8000 (596 KB)
2022-04-20 21:51:52 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 9000 (673 KB)
2022-04-20 21:51:53 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 10000 (749 KB)
2022-04-20 21:51:54 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 11000 (826 KB)
2022-04-20 21:51:54 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 12000 (903 KB)
2022-04-20 21:51:55 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 13000 (980 KB)
2022-04-20 21:51:56 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 14000 (1 MB)
2022-04-20 21:51:56 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 15000 (1 MB)
2022-04-20 21:51:57 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 16000 (1 MB)
2022-04-20 21:51:58 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 17000 (1 MB)
2022-04-20 21:51:59 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 18000 (1 MB)
2022-04-20 21:52:00 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 19000 (1 MB)
2022-04-20 21:52:01 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 20000 (1 MB)
2022-04-20 21:52:02 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 21000 (1 MB)
2022-04-20 21:52:03 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 22000 (1 MB)
2022-04-20 21:52:03 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 23000 (1 MB)
2022-04-20 21:52:04 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 24000 (1 MB)
2022-04-20 21:52:05 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 25000 (1 MB)
2022-04-20 21:52:05 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 26000 (1 MB)
2022-04-20 21:52:06 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 27000 (2 MB)
2022-04-20 21:52:06 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 28000 (2 MB)
2022-04-20 21:52:07 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 29000 (2 MB)
2022-04-20 21:52:08 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 30000 (2 MB)
2022-04-20 21:52:08 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 31000 (2 MB)
2022-04-20 21:52:09 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 32000 (2 MB)
2022-04-20 21:52:09 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 33000 (2 MB)
2022-04-20 21:52:10 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 34000 (2 MB)
2022-04-20 21:52:11 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 35000 (2 MB)
2022-04-20 21:52:11 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 36000 (2 MB)
2022-04-20 21:52:12 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 37000 (2 MB)
2022-04-20 21:52:12 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 38000 (2 MB)
2022-04-20 21:52:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 39000 (2 MB)
2022-04-20 21:52:13 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 40000 (3 MB)
2022-04-20 21:52:14 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 41000 (3 MB)
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2022-04-20 21:51:37 e[43mdestinatione[0m > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationRunner(createSentryTransaction):339 - Sentry transaction event: 39aa68c6c6d647dc82ba22aeee328257
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):121 - Running integration: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):122 - Command: WRITE
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:39 e[32mINFOe[m i.a.i.d.j.c.SwitchingDestination(getConsumer):65 - Using destination type: INTERNAL_STAGING
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$toWriteConfig$0):96 - Write config: WriteConfig{streamName=nyt_covid_data, namespace=null, outputSchemaName=RAW, tmpTableName=_airbyte_tmp_gvd_nyt_covid_data, outputTableName=_airbyte_raw_nyt_covid_data, syncMode=overwrite}
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m i.a.i.d.b.BufferedStreamConsumer(startTracked):116 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$2):114 - Preparing tmp tables in destination started for 1 streams
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$2):122 - Preparing staging area in destination started for schema RAW stream nyt_covid_data: tmp table: _airbyte_tmp_gvd_nyt_covid_data, stage: 2022/04/20/21/58D96693-0CDC-406B-A0B6-4B39B8A80E9F/
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:40 e[32mINFOe[m c.z.h.HikariDataSource(getConnection):110 - HikariPool-1 - Starting...
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:42 e[32mINFOe[m c.z.h.p.HikariPool(checkFailFast):565 - HikariPool-1 - Added connection net.snowflake.client.jdbc.SnowflakeConnectionV1@ffd4cba
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:42 e[32mINFOe[m c.z.h.HikariDataSource(getConnection):123 - HikariPool-1 - Start completed.
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:43 e[32mINFOe[m i.a.d.j.DefaultJdbcDatabase(lambda$unsafeQuery$1):106 - closing connection
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:43 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$2):133 - Preparing staging area in destination completed for schema RAW stream nyt_covid_data
2022-04-20 21:51:45 e[43mdestinatione[0m > 2022-04-20 21:51:43 e[32mINFOe[m i.a.i.d.s.StagingConsumerFactory(lambda$onStartFunction$2):136 - Preparing tmp tables in destination completed.
2022-04-20 21:51:46 e[44msourcee[0m > Reading nyt_covid_data (https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv)...
2022-04-20 21:51:47 e[43mdestinatione[0m > 2022-04-20 21:51:47 e[32mINFOe[m i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):55 - Starting a new buffer for stream nyt_covid_data (current state: 0 bytes in 0 buffers)

Hey @ssassi can you help with more information on the resources like RAM, CPU?

This is running on a AKS cluster. We’re using the helm chart on the official airbyte repo. Nodes in that cluster are Standard_D2s_v3 (2 vcpu, 8gb ram).
Resource requests/limits are defined in the helm chart.

Hey @ssassi can you try increasing the size of ram and CPU and try again

What do you mean by increasing cpu/ram?
We’re using aks autoscaler, it brings new nodes automatically to the cluster if needed, based on pods resources requests/limits definitions.
Do you mean we should try changing memory/cpu requests? On which service?
Thanks

Looks like when it is stuck it should be consuming more memory or something of that sort thus asking to increase the memory of the instance you are using. Also this is not related to auto-scaling because once the job is set it gonna only run in that instance

I upgraded nodes from 2vcpu/8gb ram to 4vcpu/16gb ram and the issue is still there.
I upgraded to airbyte 0.36.3 and now it recognizes that the sync process is “stuck” and retries after 2 hours. How could I set a shorter timeout so it retries earlier?

I’m attaching logs to review.
success_sync.txt (24.2 KB)
failed_sync.txt (32.2 KB)

Thanks.

Hey looks to me like there is some timeout happening over snowflake can you check snowflake logs and see if you can find something