Bad performance while collecting data

  • OS Version / Instance: Ubuntu deployed on GCP
  • Memory / Disk: 4Gb / 30GB
  • Deployment: Docker
  • Airbyte Version: 0.36.2-alpha
  • Source name/version: Instagram
  • Destination name/version : Bigquery
  • Step: new source
  • Description: Hello, I’m trying to retrieve data from instagram and store it in google bigquery. While the job ran successfully, the performance is not that good.

As you can see below, it takes 2h46mins to commit 55965 records.

The number of sync workers is the default one which is 5 I believe.

Hey @yasmineboussaid do you mind sharing the logs for this sync here

logs-2.txt (144.2 KB)

Hey for how many months did you try pulling data ?