Summary
The user is experiencing slower sync performance on an Airbyte deployment on a GCE instance compared to GKE. They have provided details about the configurations of both environments and the sync times for the same task. They are seeking help to identify and address the bottleneck causing the slower performance.
Question
Hi everyone,
Syncs for an Airbyte deployed on a GCE instance are much slower than an airbyte deployed on GKE.
I have a running airbyte on top of plural on GKE.
I wanted to have a more simple deployment with 1 GCE instance and a cloudSQL postgres instance. I managed to have my VM running but it seems much slower than the one on GKE.
To give you some context:
GKE:
• 1 cluster with 3 nodes: all are e2-standard-2 machines (I see that sometimes a couple of other instances are created and stoped shortly after)
GCE instance:
• e2 custom: 6 Vcpu (3 cores) with 8Gb of ram, on my monitoring interface, I hit the 50% usage on both memory and CPU
I have the same sync outputing to Bigquery, 300k records. On GKE deployed airbyte, it’s taking ~16-18 min. On my GCE instance it’s more like 35-42min
I really want to get close to the 18 min mark on my GCE instance and I am not sure how to do that. I checked a lot of answers. Using docker stats the bigquery destination connector doesn’t git the 2GB limit (1/4 of the total machine)
I am not sure where the bottleneck is ?!
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.