Data not showing in snowflake from redshift

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: Yes
  • Deployment: Using docker for deployment
  • Airbyte Version: 0.39.39-alpha
  • Source name/version: Redshift
  • Destination name/version: Snowflake
  • Step: The issue is happening during sync
  • Description: Data not showing in snowflake from redshift.
2

Please update your topic with WHAT version are you using from the Redshift and Snowflake connector.
Are you syncing only one table, the stream is a view in redshift. Give more context about your problem.

3

Hi marc… thanks for replying to my topics
From the airbyte
Redshift: 0.3.11
Snowflake: 0.4.34

AWS Redshift version: 1.0.40182

5

So basically am seeing the tables showing, but there is no data in the table. I have about 300+ tables. Also if I sync 1 to 20 tables instead of 300+ it works, not just working for my entire tables

6

Please share the complete log file to read what is causing the issue.

7

logs-32-new-db_sync_test copy.txt (3.4 MB)
This is the log of the complete sync log

8

From logs there are no errors but only a warning message saying the consumer was finished and after it’s possible to see the tmp table has 0 files. Not sure if it’s a limitation because of the number of tables or something else. As a workaround you can split the tables in multiple connections.

9

thanks, @marcosmarxm, for the feedback; yeah, that is what I see too, O bytes
So do you know or anyone knows how to sync multiple tables, we have over 250 tables to sync maybe that is too much to sync at a time

10

Airbyte has a limit based on the message size. So, if there are 250 tables with a lot of columns maybe this can break the discover step. My suggestion is to break the connection in multiple ones

1 Like