Historical data sync issue with Shopify Connector in Airbyte


The user is facing an issue with the Shopify Connector in Airbyte where historical data is not syncing properly for a newly added stream. The user wants to sync all historical data and then append only the changes, but is encountering issues with the sync settings.


Hi hi, I have a Shopify Connector running on hosted Airbyte, it doesn’t sync historical data from a newly added stream.

Before I used to sync only orders (incremental, append + deduped). And yesterday I added one more stream: inventory_items on incremental, append + deduped. In the logs I see that it was reading a lot of items, but then in the datawarehouse I got only 13 of them (instead of 490)
Then I changed to “full refresh” and got all 490 that I wanted. However, now it gets loaded every time, which I don’t want. I switched again back to incremental, append and it again extracted only 13.
What am I doing wrong? I want to sync all data historically and then append only the changes.

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["shopify-connector", "historical-data", "sync-issue", "incremental-sync", "append-only"]

Hello <@U065S7D263F> a new version of Shopify was released which uses GraphQL API. Can you give a try? https://github.com/airbytehq/airbyte/pull/32345