Intermittent sync failures from large Postgres table to Snowflake

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: JOB_MAIN_CONTAINER_MEMORY_REQUEST=2Gi, JOB_MAIN_CONTAINER_MEMORY_LIMIT=4Gi
  • Deployment: Amazon EKS
  • Airbyte Version: 0.35.30-alpha
  • Source name/version: Postgres 0.4.6
  • Destination name/version: Snowflake 0.4.11
  • Step: Sync
  • Description: I am syncing a table every two hours with incremental - deduped + history. The sync takes about 1hr 40 minutes because there can be up to 2 million new / updated records per sync and these have to be deduped against a snowflake table of 6.5 billion rows. Sometimes the sync fails, but it will eventually succeed after 2-3 tries. The problem is that I cannot view the logs; the server just shows ‘no logs available’. Do you have any idea what I could tweak in the Kubernetes settings to improve our sync success and maybe be able to view logs as well?

Screen Shot 2022-04-11 at 11.04.47 AM
Screen Shot 2022-04-11 at 11.04.47 AM1676×252 21.7 KB

2

Hello @andmo,
I have some additionnal questions to help you troubleshoot this problem:

  • Could you please upgrade Airbyte to its latest version and check if the problem persists?
  • Do you mind sharing your server logs? You can download these from the settings page in the UI.
  • How did you deploy Airbyte on Kubernetes? With the helm chart or via Kustomize?
  • Did you configure something specifically for logging (see log configuration documentation here)
  • Do you have logs for other connections / attempts?
3

Hello,

We’ve upgraded our Airbyte up to v0.39.17 and have fixed some of the issues along the way. We didn’t have logs because we are running our own log server and it was running out of space. We were able to increase the sync reliability by creating an index on the cursor field in the source. That sped up sync times and resulted in less failures. Of course, there have been some new issues we have run into, but I will open another ticket for those. Thanks for the suggestion to upgrade!

Kind regards,

Andrew

4

Thank you for reporting this! I’m glad you’re making progress.

5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.