Dynamic Union of Monthly Transactional Files in Snowflake

Summary

User inquires about dynamically combining monthly transactional files into a single table in Snowflake, specifically addressing the challenge of syncing files that are generated in the future.


Question

Hi :wave: Has anyone tried combining/unioning monthly transactional files dynamically into one single file in the destination warehouse (Snowflake in our case)?

For example,

file_feb_2024,
file_mar_2024,
file_apr_2024,
file_may_2024,
...
...```
All of above should be unioned into one single file called `files_2024`. However, the files for *Mar 2024 onwards* have not been generated yet. When they are generated on the first day of month, they will start syncing. I d want the records to load into the `files_2024`  table in snowflake

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1708299009675709) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['snowflake', 'transactional-files', 'dynamic-union', 'data-sync']
</sub>