Summary
The user is facing an issue with the Shopify Connector in Airbyte where historical data is not syncing properly for a newly added stream. The user wants to sync all historical data and then append only the changes, but is encountering issues with the sync settings.
Question
Hi hi, I have a Shopify Connector running on hosted Airbyte, it doesn’t sync historical data from a newly added stream.
Before I used to sync only orders (incremental, append + deduped). And yesterday I added one more stream: inventory_items on incremental, append + deduped. In the logs I see that it was reading a lot of items, but then in the datawarehouse I got only 13 of them (instead of 490)
Then I changed to “full refresh” and got all 490 that I wanted. However, now it gets loaded every time, which I don’t want. I switched again back to incremental, append and it again extracted only 13.
What am I doing wrong? I want to sync all data historically and then append only the changes.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.