Summary
The user is facing a Java heap issue while trying to sync a CSV file with 1 million rows from an SFTP source connector to a Postgresql destination connector. They provided details of their Kubernetes cluster setup and are seeking advice on whether they can make it work with the current hardware configuration or if they need to tune something in Airbyte.
Question
Hey team , I have a CSV file with 1 million rows and trying to sync with postgres
Source connector: SFTP
Destination connector: Postgresql
Issue: I’m facing java heap issue
K8 master : 4 cpu and 16GB ram
2 worker Nodes : 4 cpu and 32GB ram
Can anyone suggest is there a way i can make it work with above hardware or do i need to tune something at airbyte
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.