Bad performance while collecting data

Connector Questions & Issues
,
1
  • OS Version / Instance: Ubuntu deployed on GCP
  • Memory / Disk: 4Gb / 30GB
  • Deployment: Docker
  • Airbyte Version: 0.36.2-alpha
  • Source name/version: Instagram
  • Destination name/version : Bigquery
  • Step: new source
  • Description: Hello, I’m trying to retrieve data from instagram and store it in google bigquery. While the job ran successfully, the performance is not that good.

As you can see below, it takes 2h46mins to commit 55965 records.

image
image2650×208 43.1 KB

The number of sync workers is the default one which is 5 I believe.

2

Hey @yasmineboussaid do you mind sharing the logs for this sync here

4

Sure
logs-2.txt (144.2 KB)

5

Hey for how many months did you try pulling data ?

6

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.