Summary
Custom connector with pagination and incremental sync to Snowflake v2 destination is facing 504 timeout error, causing records not to be loaded and connection to be stuck. Need advice on how to handle this issue.
Question
Hi team !
Jusk aked the <#C01AHCD885S|ask-ai> channel but couldn’t find the answer so asking here instead .
I have a custom connector using the following parameters :
• Pagination with limit of 5000 records
• Incrementatil sync using an after param
The connector is binded with a snowflake v2 destination.
It happens that the source fails due to 504 time out . In such case , fetch records are not loaded into snowflake and thus state of the connection is not updated. In such case the connection is stuck in a never ending cycle , never commiting records that could be fetch and thus catch up .
Why should I do ?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.