Shopify to Postgres Sync Issue

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User is experiencing an issue with syncing Shopify Orders to Postgres 17 where the sync process hangs during the executeTypeAndDedupe step, preventing rows from being inserted into the final table despite logs indicating data is loaded into the intermediate table.


Question

Hey folks, I am new to Airbyte and using the hosted service. Wondering if anyone had problems syncing Shopify (source) Orders (table) into Postgres 17 (destination). the sync is just running forever. The logs show data is loaded and pushed into the intermediate table in postgres and the last step executeTypeAndDedupe never finishes (it hangs at the first query insert into). As I understand this steps should copy the intermediate table to the final table (with correct schema and types). My final table never gets any rows inserted. I can see the query running in pg_stat_activity for about 5 mins then it is “idle in transaction”. I ran the query manually and it does not error but it also does not insert any rows, it somehow does not read any rows from the "airbyte_internal"."public_raw__stream_orders". Am I missing something here?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['shopify', 'postgres-17', 'sync-issue', 'executeTypeAndDedupe', 'airbyte']
2

got any update on this ?
I am having the same issue with hubspot sync

3

hey, I did not get any response or support, what I did is that I tried removing a couple of columns from the schema and refreshing the stream. This worked. I first tried with a minimal schema (like primitive types and only an ID and value) then added more fields to it. I have not tried to isolate the specific type or field that was not working, so not sure which type is causing this or what the root cause is.

One thing: my api token does not have all possible permissions in shopify, so I am getting warnings about missing permissions in the logs. However, those are irrelevant (for my purposes) fields and I was expecting airbyte to simply keep those fields null or something. It could well be that those caused the problem but when I hand picked the relevant columns I omitted the ones with insufficient permissions

4

oh and as for the weird copy / select behaviour, I have not root caused this either. The whole trial and error cost me a day, I needed this for a demo and once I got into a state that worked for me, I was ok with it. However, I’d still be curious to know what is happening here.