Summary
Advice on setting up data synchronization every 15-30 minutes from CDC Postgres to Snowflake on EKS
Question
hello everyone! I finally got my little ec2 installation over to an EKS rollout, and looking for advice on how to sync every 15-30 minutes. our data sets are not huge and mainly using cdc postgres -> snowflake. the perf looks much better now that I am on k8s thankfully
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["syncing-data", "cdc-postgres", "snowflake", "eks", "kubernetes", "data-synchronization"]
+1 to using cron scheduling… we run ours every 30 minutes (Community edition) on our k8s, in case that’s helpful
I run some connections every minute
saw some documentation that states:
Scheduling is limited to at maximum ever hour. If you have a use case for under 1 hour frequency, please reach out to our Sales team through this page.
(<https://reference.airbyte.com/reference/createconnection#scheduling-customization>)```
also:
```Syncs can run at most every 60 minutes in Airbyte Cloud. Reach out to Sales if you require replication more frequently than once per hour.
(<https://docs.airbyte.com/using-airbyte/core-concepts/sync-schedules#cron-scheduling>)```
i had it in my head that this was a limitation of the the SaaS Airbyte Cloud not of the self hosted community version. I will do more searching in slack to see!
so we can trigger the syncs from the api at whatever interval we want but airbyte community version only allows the minimum of 1 hour sync in the menu
You can use the Cron
scheduling option with an expression like 0 0/15 0 ? * * *
for every 15 minutes, or 0 0/30 0 ? * * *
for every 30.
If you Google around, just be mindful that Airbyte uses the Quartz syntax for Cron (which is different in some ways than the default unix crontab style)
<@U035912NS77> thats even better!! thank you for the help