Syncing Data from CDC Postgres to Snowflake on EKS

Summary

Advice on setting up data synchronization every 15-30 minutes from CDC Postgres to Snowflake on EKS


Question

hello everyone! I finally got my little ec2 installation over to an EKS rollout, and looking for advice on how to sync every 15-30 minutes. our data sets are not huge and mainly using cdc postgres -> snowflake. the perf looks much better now that I am on k8s thankfully



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["syncing-data", "cdc-postgres", "snowflake", "eks", "kubernetes", "data-synchronization"]

+1 to using cron scheduling… we run ours every 30 minutes (Community edition) on our k8s, in case that’s helpful

I run some connections every minute :sweat_smile:

saw some documentation that states:


Scheduling is limited to at maximum ever hour. If you have a use case for under 1 hour frequency, please reach out to our Sales team through this page.

(<https://reference.airbyte.com/reference/createconnection#scheduling-customization>)```
also:
```Syncs can run at most every 60 minutes in Airbyte Cloud. Reach out to Sales if you require replication more frequently than once per hour.

(<https://docs.airbyte.com/using-airbyte/core-concepts/sync-schedules#cron-scheduling>)```
i had it in my head that this was a limitation of the the SaaS Airbyte Cloud not of the self hosted community version.  I will do more searching in slack to see!

here we go! had to search more careful under “sync schedule”:

https://airbytehq.slack.com/archives/C039Y6EGTGV/p1695941005682119?thread_ts=1695868599.229819&amp;cid=C039Y6EGTGV

so we can trigger the syncs from the api at whatever interval we want but airbyte community version only allows the minimum of 1 hour sync in the menu

You can use the Cron scheduling option with an expression like 0 0/15 0 ? * * * for every 15 minutes, or 0 0/30 0 ? * * * for every 30.

If you Google around, just be mindful that Airbyte uses the Quartz syntax for Cron (which is different in some ways than the default unix crontab style)

<@U035912NS77> thats even better!! thank you for the help