Summary
Exploring the best method for setting up API-based ETL pipelines in Airbyte to periodically check an API endpoint, determine data refresh necessity, and perform ETL tasks.
Question
Hey everyone! I am new to Airbyte and have not started using it yet but instead feeling out the potentials. I am exploring the best methods to get my ETL pipelines setup and have already tried out Apache Airflow and Fivetran. The vast majority of my pipelines are API-based and essentially rely on checking a database against an API request and determining if data refresh is necessary as most of the APIs I work with at the moment don’t have any kind of webhook capabilities and so I have to do the checking myself. This was something I struggled to best workout with Airflow. Does anyone reckon Airbyte would be best used for such as use case? Put simply, I need to periodically check an API endpoint and determine whether to pull from that endpoint and perform ETL or simply wait for the next period and so on.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.