Syncing more frequently than cron setting

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 64gb / 150gb
  • Deployment: Docker
  • Airbyte Version: 0.42.0
  • Source name/version: MySQL 2.0.0
  • Destination name/version: S3 0.3.21
  • Step: Sync frequency is ~1min rather than the 30 min cron setting
  • Description: We have our airbyte replication frequency set to cron = */30 * * * * ? UTC (every 30 min) but the replication actually pulls nearly every minute. Sometimes we run into networking source connection failures that eventually resolve themselves in time and I expect catchup jobs to run immediately in succession, but at this point we have had jobs running every minute for 12+ hours. Why might this happen?

image
image2420×416 38.6 KB

2

image
image1091×372 22.2 KB

3

I am experiencing the exact same problem. The cron job to set the sync to run every 30 minutes is just not working properly. Is there a catch to the cron format that Airbyte uses? Using the ones generated by Cron Guru gives Invalid Format error: Crontab.guru - The cron schedule expression editor

4

This cron expression sets the sync to run every 30 minutes: 0 0/30 * * * ?

Airbyte asks us to use the cron expressions format as documented on these docs, which in my opinion are not wholesome for devs working rarely with them.

The break down of cron expression 0 0/30 * * * ? is here:

  • The first field (0) represents Seconds. In this case, the job will execute when the seconds are at 0.
  • The second field (0/30) represents Minutes. Here, 0/30 means the job will start at the 0th minute and then execute every 30 minutes.
  • The third field (*) represents Hours. The asterisk (*) means the job will execute every hour of the day.
  • The fourth field (*) represents Day of the month. The asterisk (*) means the job will execute every day of the month.
  • The fifth field (*) represents Month. The asterisk (*) means the job will execute every month of the year.
  • The sixth field (?) represents Day of the week. The question mark (?) is used when there’s no specific value requirement, meaning it can be any day of the week.

So, in summary, 0 0/30 * * * ? can be read as “Execute the job every 30 minutes, at the start of the minute, every hour, every day of the month, every month, and every day of the week”. In other words, it sets the job to run every 30 minutes all day, every day.