Hello! I’d like to ask for connector development advice 
I made a custom Intercom destination connector that performs reverse ETL (it syncs data from our database to certain objects in Intercom, such as the Company object in this case). It works okay: we have 100k+ records in the table and it’s able to sync them to Intercom.
However, even if it successfully syncs to Intercom; the connection itself doesn’t complete as a ‘success’; it always returns some form of error, such as ‘Sync worker failed’ or ‘Deadline exceeded’ (logs attached below). Notably, what happens is: after the connection syncs the whole table, the logs kind of ‘freeze’ (there’s no update for hours). Then afterwards, it either spontaneously errors out, or it only returns the sync worker/deadline exceeded error when we cancel the job.
This only happens for relatively big jobs (as mentioned, this table has 100k+ rows); I’ve tried using the same connector on 20 rows and it succeeds.
We’ve looked up the errors, and saw the Github Issues about similar deadline exceeded errors, where the recommendation was to increase the resources allocated to Airbyte (we run Airbyte on EKS). We’ve already increased the resources, but still run into this error.
Is there something else that may be causing the error? What other approaches can I try?
logs-160-attempt-2.txt (678.5 KB)
logs-160-attempt-3.txt (449.9 KB)
Hey, both of the errors can happen if resources are less. Kindly increase the resources (memory and CPU) and run it again and see if that helps
@kirstenhipolito would be possible to you run a local instance of Airbyte to isolate the running and see if works?
Hello Harshith! I tried increasing the resources, specifically:
- 4Gb minimum memory, burstable to 16Gb
- 1 vCPU unit per worker
- AWS EKS - 5 worker replicas
- Airbyte 0.35.27-alpha
However, I still run into the same freezing issue (‘freezing’: stops giving log outputs after the last record sync), and when I cancel the job, I eventually get a ‘Sync worker failed’ error (logs attached).
logs-256.txt (518.5 KB)
Do I need to allocate more resources for the connection?
Hey @kirstenhipolito my guess is you are running this in EKS thus sorry to clarify it before can you help me understand what is the configuration for the EKS setup like the instance type (memory/cpu)?
Hi there,
Allow me to jump in here, @kirstenhipolito and I are running airbyte in an EKS cluster running with t3.xlarge nodes at the moment.
1 Like
Do you mind trying 8 and 32 and see if the same problem happens?
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.