Issue with Incremental Sync in Custom Python Connector

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Custom Python connector with Incremental Sync feature performs full refresh sync twice, causing duplicate records.


Question

I created a custom connector using python http api source. I added a feature of Incremental Sync using IncrementalMixIn, however it performs full refresh sync twice causing duplicate records. See sequence and output and the attached code.

  1. first run - full refresh no state save - inserts all data
  2. second run - full refresh with state saved - inserts all data again causing duplicate
  3. third run - only inserts data that is new making the incremental sync work

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["custom-connector", "python", "http-api", "incremental-sync", "duplicate-records"]