Increased loaded records compared to extracted records in Snowflake connection

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is experiencing an issue where more records are being loaded than extracted in the Snowflake connection, despite running smoothly for the past 6 months. Additionally, a table is being deleted in Snowflake. The connector settings are set to Full Refresh and Overwrite.


Question

Hey All – Noticed since last Wednesday and now it happened again this Wednesday that we are having, seeing more records loaded than extracted in our snowflake connection. Was running completely smooth for the last 6 months with no issues. However, a table is being deleted every time in snowflake now. Table settings for connector is Full Refresh | Overwrite - see screenshots.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["increased-loaded-records", "extracted-records", "snowflake-connection", "full-refresh", "overwrite", "table-deleted"]
2

Some more context both these runs had 2 attempts to extract and load the connection and the first failed and it re-ran and succeeded, however one of our tables is being deleted when this happens

3

What source and destination are you using?

4

Source: Microsoft SQL Server (MSSQL) → Destination: Snowflake

5

I believe you misread this. There are 1,315,535 loaded and 1,325,221 extracted.

That’s ~9.7k less extracted records.

6

Correct I stated that above. Both times (this Wed and last Wed) where there is a mismatch in extracted and loaded is due to Attempt 1 failing for sync and Attempt 2 succeeding. However, for whatever reason it’s deleting one of the destination tables (in snowflake) when it does this, is my assumption.

7

Oh yes. This is expected because there was multiple attempts, so we probably re-read some data as we restarted from a checkpoint on the second attempt. Stats are not related to any Destination table deletes

8

So it’s deleting the table at the Destination - Snowflake ? Any suggestions to prevent that. Does not seem feasible to have to check every time this is happening and manually re-run. Also curious why it’s happening on Wed runs.