You can adjust the load time to run as frequent as every five minutes and as infrequent as every 24 hours.
Additionally, you can configure syncs to only run manually. In this mode, syncs will only be performed if initiated through the UI, API, or an external orchestration tool like Airflow.
A follow up question may be:
Why wouldn’t I choose to load all of my data every five minutes?
While frequent data loads will give you more up-to-date data, there are a few reasons you wouldn’t want to load your data every five minutes, including:
- Higher API usage may cause you to hit a limit that could impact other systems that rely on that API.
- Higher cost of loading data into your warehouse.
- More frequent delays, resulting in increased delay notification emails. For instance, if the data source generally takes several hours to update but you choose five-minute increments, you may receive a delay notification every sync.
We generally recommended running incremental syncs to every hour to help limit API calls. Keep in mind that even though incremental syncs only retrieve new data, normalization will be performed on ALL data after a sync, which can be expensive.