- Is this your first time deploying Airbyte?: Yes
- OS Version
/ Instance: Ubuntu / AWS on an EC2 instance type t2.large
- Memory / Disk: 8Gb / 30Gb
- Deployment: Docker
- Airbyte Version: What version are you using now? 0.39.41-alpha
- Source name/version: Intercom / 0.1.24
- Destination name/version: Mongodb / 0.1.5
- Step: The issue is happening during sync
We are having some performance issues when using Airbyte to retrieve contact and company data from Intercom and send them to our mongodb destination.
Let’s say we want to recover more than 300,000 objects from intercom through the companies and contacts streams.
At the beginning of the sync, everything is fine: the worker recovers about 7,000 objects per minute.
However, the performance starts to deteriorate from 130,000 objects recovered. Data recovery progressively slows down to 1,000 objects recovered every 10mins when reaching ~ 190,000 objects recovered.
We do not get any Intercom rate limit alert and everything seems to work normally but is getting slower and slower.
No matter how many syncs we launch, the problem remains the same and applies to the all the syncs that are launched.
The stack is hosted at AWS on an EC2 instance type t2.large.
It is configured with MAX_SYNC_WORKERS=3 and all sources and destinations are up to date.
Only 1 sync is runned for this performance test.
The CPU does not exceed 50% usage, the RAM is 2GB/ 8GO and only 16% of the hard disk is used.
Attached is an extract of the logs that we get from the sync
my logs.txt (41.8 KB)
Has anyone ever encountered such an issue?