Error syncing data from Snowflake to BQ using Airbyte OSS helm chart

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is facing an error when trying to sync data from Snowflake to BigQuery using the Airbyte OSS helm chart. The job hangs during a full sync, and the logs show specific messages in the source-snowflake-read, orchestrator, and destination-bigquery-write pods.


Question

Hi community,

I’m facing an error syncing data from snowflake to BQ. I’m using the OSS helm chart to deploy airbyte. When I manually (from the UI) create the snowflake source, it passes the set up check, but then when I try to do a full sync to BQ, the job hangs and the logs of the source-snowflake-read pod only show this

PARENT_PID: 1```
the orchestrator pod shows a ton of these
```2024-08-27 16:10:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 768 MB, allocated: 10 MB (10.0 MB), %% used: 0.013020833333333334 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-08-27 16:10:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0```
and the destination-bigquery-write shows something identical to the source
```Waiting on CHILD_PID 7
PARENT_PID: 1```
Any ideas?

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1724775262696229) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["error", "syncing-data", "snowflake", "bigquery", "airbyte-oss", "helm-chart", "logs"]
</sub>
2

This is how the hung job looks from the UI (stuck in starting)