"Canceling Destination" on large local CSV file import to Postgres

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 32GB
  • Deployment: Are you using Docker or Kubernetes deployment? Docker
  • Airbyte Version: What version are you using now? 0.40.3
  • Source name/version: Local CSV file
  • Destination name/version: Postgres local install
  • Step: The issue is happening during sync, creating the connection or a new source? Sync

I am trying to sync a large (32M rows) CSV local file (/tmp/airbyte_local) into a Postgres table also on a local (non-docker) instance.

Postgres has been working fine for several other loads including both small CSV files as well as remote connections to Azure Synapse, Shopify and others.

But I cannot load this large local CSV file successfully. I’m seeing “Canceling Destination” message after several million records.

Any ideas ?

logs-345.txt (4.3 MB)

Hey could share the complete logs for the sync here as a file?