Why airbyte is taking too much resources for the sync? I am currently using airbyte to sync data from postgres to bigquery and have deployed it in N2 compute engine with 4vCPU and 32 GB RAM and created 50 connections each having 300 tables, but data for those 300 tables is less than 2MB. I was only able to configure it with 8 workers and 20 submitters and it took 3 hours for the sync. If I am configuring more workers and submitters the sync are failing. And I want to make approximately 4000 connections with 300 tables each. So to achieve this either I need too much RAM and CPU or it will take a week or two for a sync. Is there any way around this?
Even if your data sources have little data volume, managing a lot of connections might create a lot of overhead on your instance.
Could you details which resources are exhausted (CPU or RAM)? Could you try to increase your number of CPUs and check if it leads to better performance?
Did you also try to tweak the following environment variables that are related to scaling?
### JOBS #### Relevant to scaling.SYNC_JOB_MAX_ATTEMPTS=3SYNC_JOB_MAX_TIMEOUT_DAYS=3JOB_MAIN_CONTAINER_CPU_REQUEST=JOB_MAIN_CONTAINER_CPU_LIMIT=JOB_MAIN_CONTAINER_MEMORY_REQUEST=JOB_MAIN_CONTAINER_MEMORY_LIMIT=
And finally, how did you host the Airbyte database? In an external instance or you are running it on docker? Running it in an external and well-provisionned instance might also help you gain a bit of performance.
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.