Redshift connection lost, job hanging

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: MacOS BigSur 11.6
  • Memory / Disk: 32Gb / 1 Tb
  • Deployment: Testing locally with docker-compose
  • Airbyte Version: 0.35.65
  • Source name/version: Redshift 0.3.9
  • Destination name/version: Clickhouse 0.1.4
  • Step: Sync
  • Description:
    I’m testing out Airbyte (awesome work!) and running into an issue ingesting data from Redshift. It looks like the connection is getting closed and the sync then hangs in the running state. It appears to do be doing this at around 37 million records (8GB), or could also be a timeout setting.

Anyone have any ideas on how to handle this? Trying to load billions in so I imagine there’s going to be lots of connection problems during the initial load. Would be great to be able to handle these and pick up where it left off.

2022-04-08 02:16:20 source > 2022-04-08 02:16:20 ERROR i.a.c.u.CompositeIterator(close):93 - exception while closing
2022-04-08 02:16:20 source > java.lang.RuntimeException: java.sql.SQLException: [Amazon](500150) Error setting/closing connection: Not Connected.
2

Hey @wohlgejm do you mind sharing the full sync log? Are you running on an Apple M1 computer?

4

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.