Syncing historical data to Redshift with two connectors


The user wants to sync historical data to Redshift using two connectors, one for recent data and one for historical data. They are concerned about potential issues with overwriting new data.


I’ve got a connector for which I would like to sync a lot of historical data. Would it be possible to create two connectors that write to the same destination (redshift) so that one of them syncs the recent data and the other one syncs historical data? The connector uses the incremental append+dedup sync mode on the cursor updated_at field, so I believe there should be no issue that the older data would overwrite the new.
The historical sync would be a one time job but it may run for days or weeks. At the same time I’d like to receive fresh data updates.
Does this approach have a chance of success?

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["syncing-historical-data", "redshift-connector", "incremental-sync", "cursor-updated-at", "two-connectors"]