Summary
User inquires about the possibility of using the REST API with cursor-based pagination or the GraphQL API instead of the Bulk API for the Shopify connector in Airbyte, citing performance issues with the Bulk API for large datasets.
Question
Hi all, we’re looking into using Airbyte + its Shopify connector to pull all historical data from certain streams. Shopify offers the Bulk API which seems to be what the connector has been migrating to lately for most streams. Is it possible to switch over the REST API with cursor-based pagination or the GraphQL API and have the connector NOT use the Bulk API? Main reason being the Bulk API is often SLOW for shops with a lot of data. Syncs can take hours (due to Shopify generating the exports) when they would take <30 min via other methods.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.