- Is this your first time deploying Airbyte?: Yes
- Deployment: Using docker for deployment
- Airbyte Version: 0.39.39-alpha
- Source name/version: Redshift
- Destination name/version: Snowflake
- Step: The issue is happening during sync
- Description: Data not showing in snowflake from redshift.
Please update your topic with WHAT version are you using from the Redshift and Snowflake connector.
Are you syncing only one table, the stream is a view in redshift. Give more context about your problem.
Hi marc… thanks for replying to my topics
From the airbyte
AWS Redshift version: 1.0.40182
So basically am seeing the tables showing, but there is no data in the table. I have about 300+ tables. Also if I sync 1 to 20 tables instead of 300+ it works, not just working for my entire tables
Please share the complete log file to read what is causing the issue.
logs-32-new-db_sync_test copy.txt (3.4 MB)
This is the log of the complete sync log
From logs there are no errors but only a warning message saying the consumer was finished and after it’s possible to see the tmp table has 0 files. Not sure if it’s a limitation because of the number of tables or something else. As a workaround you can split the tables in multiple connections.
thanks, @marcosmarxm, for the feedback; yeah, that is what I see too, O bytes
So do you know or anyone knows how to sync multiple tables, we have over 250 tables to sync maybe that is too much to sync at a time
Airbyte has a limit based on the message size. So, if there are 250 tables with a lot of columns maybe this can break the discover step. My suggestion is to break the connection in multiple ones