Postgres to Snowflake only sync part of the data

Connector Questions & Issues
, , ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Official Docker images on AWS (CentOS)
  • Memory / Disk: you can use something like 8Gb / 100Gb
  • Deployment: Docker
  • Airbyte Version: 0.39.38-alpha
  • Source name/version: Posgtres 1.0.0
  • Destination name/version: Snowflake 0.4.34
  • Step: I setup a postgres source and snowflake destination as described in the documentation. The sync is setup to Full Overwrite, running manually, following the destination schema.
  • Description:

The sync works, but not all rows are synced. About 25% of the way there (3 million rows) the sync aborts with an error message saying “This should not happen” but then claims success. I tried different modes of synchronization (append, full, …) and get the same result each time.

3

Please disregard, I found the problem

4

Hey could you share the problem here?

5

Sure :slight_smile:

The instance on which I was running Airbyte had the WORKSPACE_ROOT set to /tmp/airbyte_workspace which was on the main partition, and only had 8GB storage total (including for the OS and all). So, Airbyte was running out of space. I setup WORKSPACE_ROOT to be on a larger partition, and the problem went away.