Airbyte taking too much resources and Time

Hello everyone,
Why airbyte is taking too much resources for the sync? I am currently using airbyte to sync data from postgres to bigquery and have deployed it in N2 compute engine with 4vCPU and 32 GB RAM and created 50 connections each having 300 tables, but data for those 300 tables is less than 2MB. I was only able to configure it with 8 workers and 20 submitters and it took 3 hours for the sync. If I am configuring more workers and submitters the sync are failing. And I want to make approximately 4000 connections with 300 tables each. So to achieve this either I need too much RAM and CPU or it will take a week or two for a sync. Is there any way around this?

Hey @SP271,
Even if your data sources have little data volume, managing a lot of connections might create a lot of overhead on your instance.
Could you details which resources are exhausted (CPU or RAM)? Could you try to increase your number of CPUs and check if it leads to better performance?

Did you also try to tweak the following environment variables that are related to scaling?


And finally, how did you host the Airbyte database? In an external instance or you are running it on docker? Running it in an external and well-provisionned instance might also help you gain a bit of performance.