OS Version / Instance: Ubuntu 18.04, EC2 t3.xlarge
Memory / Disk: 16Gb / 40GB SSD
Deployment: Kubernetes
Airbyte Version: 0.32.0-alpha
Source name/version: Postgres 0.3.7
Destination name/version: S3 0.1.13
Step: Setting new connection, source / On sync
Description: I’m trying to sync the data from Postgres to S3 and the process doesn’t finish and I start to get the attached error in server logs. Also the airbyte UI does not show the logs on UI for the same and connection job does not finish.
Hi @hokam,
I think there is a problem with the health of your deployment.
Could you please upgrade to the latest Airbyte version and redeploy Airbyte? And use Postgres connector version 0.4.4?
I have additional questions that could help us troubleshoot this:
How did you deploy Airbyte on Kubernetes, with our Helm Chart or Kustomize?
How many nodes do you have on your cluster?
Could please check if what does the memory consumption looks like when running this sync?
How did you deploy Airbyte on Kubernetes, with our Helm Chart or Kustomize?
With Helm Chart.
How many nodes do you have on your cluster?
Minimum 3 nodes and max 10 nodes of t3.xlarge.
Could please check if what does the memory consumption looks like when running this sync?
Checked the memory footprint on nodes, It is reaching 30 % when this job runs.
I will try by changing the Postgres version to 0.4.4 and see if that helps.
Could you please try to run all the Airbyte related pods in the same node? There is maybe a problem in sharing the volume data across nodes that could lead to failure in log retrieval.
Before doing this do you mind trying to set up another simple connection (e.g PokeAPI to S3) to check if the problem with the logs is a global one?
Let me know if the Postgres update makes it work better.
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.