Source Intercom - Performance issues when retrieving high volumes of contacts and companies

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version
    / Instance: Ubuntu / AWS on an EC2 instance type t2.large
  • Memory / Disk: 8Gb / 30Gb
  • Deployment: Docker
  • Airbyte Version: What version are you using now? 0.39.41-alpha
  • Source name/version: Intercom / 0.1.24
  • Destination name/version: Mongodb / 0.1.5
  • Step: The issue is happening during sync
  • Description:

Hi there :wave:

We are having some performance issues when using Airbyte to retrieve contact and company data from Intercom and send them to our mongodb destination.

Let’s say we want to recover more than 300,000 objects from intercom through the companies and contacts streams.

At the beginning of the sync, everything is fine: the worker recovers about 7,000 objects per minute.

However, the performance starts to deteriorate from 130,000 objects recovered. Data recovery progressively slows down to 1,000 objects recovered every 10mins when reaching ~ 190,000 objects recovered.

We do not get any Intercom rate limit alert and everything seems to work normally but is getting slower and slower.

No matter how many syncs we launch, the problem remains the same and applies to the all the syncs that are launched.

The stack is hosted at AWS on an EC2 instance type t2.large.
It is configured with MAX_SYNC_WORKERS=3 and all sources and destinations are up to date.
Only 1 sync is runned for this performance test.
The CPU does not exceed 50% usage, the RAM is 2GB/ 8GO and only 16% of the hard disk is used.
Attached is an extract of the logs that we get from the sync
my logs.txt (41.8 KB)
Has anyone ever encountered such an issue?

Thanks!

2

Hi @wande, welcome to the community and thanks for your detailed post (certainly makes my job easier!)

In the past, our connector team has investigated performance issues surrounding the Intercom connector and unfortunately it’s not a problem with the connector so much as it’s a characteristic of the API. Although we’ve designed the connector to respect the API policy and modulate the load based on the rate limits, the Intercom API is designed for targeted requests and not fetching all of the data associated with a particular account.

Unfortunately, a full refresh sync should take a while (in the magnitude of hours) but the connector is designed so that incremental syncs are much faster. Sorry, I wish I had a more concrete answer for you. Hope this helps!

1 Like
3

Thanks for your help @sajarin !