Sync stuck in normalization step

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu 22.04
  • Memory / Disk: 16Gb / 500 Gb
  • Deployment: Docker
  • Airbyte Version: 40.14
  • Source name/version: Postgres
  • Destination name/version: Bigquery
  • Step: The issue is happening during normalization.
  • Description: We are syncing data every 15 minutes from Postgres(CDC) to Bigquery. The normalization container was stuck for almost 11 hrs today. Earlier it was running fine but recently we started facing this issue and we don’t encounter this issue with every sycn. We have to cancel the sync in order to start a new one. It is very hard to troubleshoot as there is not much information in the logs once it gets stuck

I was wondering how can fix this and also how can we detect this.

Logs :

image
image2094×682 206 KB

2

Can you sahre the complete log file?

3

@marcosmarxm here are the logs

b8aaa40c_1fef_4d54_8729_6f415c6ab171_logs_9712_txt.txt (325.5 KB)

4

@marcosmarxm any insights you can share ?

5

Prudhvi did you try using a smaller table? It work?

6

Hello.

Is it possible that the server docker is running on causes this issue on the normalization container? Do you have any logs you can check? Can you run a manual sync and profile your resources? Maybe you run out of RAM or disk.

I haven’t seen something similar, but from a first look it doesn’t seem like an issue with Airbyte but with the container itself.

Best regards.