SFTP Failing to Redshift

Troubleshooting
#1

Is this your first time deploying Airbyte : No
OS Version / Instance : EC2 t3a.xlarge
Memory / Disk : 16Gb
Deployment : Docker
Airbyte Version : 0.35.64-alpha
Source name/version : File 0.2.9
Destination name/version : Redshift 0.3.28
Step : Setting new connection, source / On sync
Description : Multiple sources from SFTP over to Redshift are failing on first sync. These do work just fine to a MySQL destination, but trying to move them over to Redshift now. Does not effect every SFTP connection, but a large number of them are now failing. Any ideas what might be wrong

logs-5132.txt (21.5 KB)

#2

It seems that if I delete the connection, and then set it up fresh, and instead of full refresh / overwrite, I choose to append, it seems to run. Any thoughts on what might cause that?

#3

Hi @krishagel,
Did you ran a refresh source schema after changing your destination?
According to this error in the logs:
java.lang.IllegalArgumentException: Message contained record from a stream that was not in the catalog. the schema used by the destination is not up to date with the source schema.
When deleting the connection you basically achieved the same thing.