Merging and deduplicating messages from multiple Snowflake databases using Airbyte Slack connector

Summary

User is looking to merge and deduplicate messages from multiple Snowflake databases using the Airbyte Slack connector to create a sustainable sync destination.


Question

Airbyte folks, I have a bit of a crazy thing I’m trying to do and would appreciate any input/feedback.

Using the Slack source, I’ve been trying to export all the messages from a very large Slack community I run. (btw, I’ve been following the development of the new low-code Slack source closely and it’s a huge improvement!).

I’ve found myself in a place where I have 3 different syncs in 3 different Snowflake databases, and each one is missing messages, but I believe that combined they cover 99+% and are “good enough”. Particularly with regard to thread messages.

I’d like to merge and deduplicate them all into a single database while making it a sustainable sync destination moving forward. Has this ever been done? (plan in thread)



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["merge-messages", "deduplicate", "snowflake-database", "airbyte-slack-connector", "sync-destination"]