Redshift connection lost, job hanging

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: MacOS BigSur 11.6
  • Memory / Disk: 32Gb / 1 Tb
  • Deployment: Testing locally with docker-compose
  • Airbyte Version: 0.35.65
  • Source name/version: Redshift 0.3.9
  • Destination name/version: Clickhouse 0.1.4
  • Step: Sync
  • Description:
    I’m testing out Airbyte (awesome work!) and running into an issue ingesting data from Redshift. It looks like the connection is getting closed and the sync then hangs in the running state. It appears to do be doing this at around 37 million records (8GB), or could also be a timeout setting.

Anyone have any ideas on how to handle this? Trying to load billions in so I imagine there’s going to be lots of connection problems during the initial load. Would be great to be able to handle these and pick up where it left off.

2022-04-08 02:16:20 source > 2022-04-08 02:16:20 ERROR i.a.c.u.CompositeIterator(close):93 - exception while closing
2022-04-08 02:16:20 source > java.lang.RuntimeException: java.sql.SQLException: [Amazon](500150) Error setting/closing connection: Not Connected.

Hey @wohlgejm do you mind sharing the full sync log? Are you running on an Apple M1 computer?