Large Backfill with unreliable API

Connector Development
,
1

I am trying to backfill a large number of records from a source that consistently hits persistent retryable errors. I have increased the retry backoff and number of retires, but even with 10 retries with an exponential backoff the retryable errors continue.

With the current setup, an incremental sync using the get_updated_state method when I run into these errors it will retry, pulling the same records 3 times, and then commit whatever it had from the last sync. It seems like this will eventually sync all records, but it is not ideal, so I am trying to improve my connector code.

Reading the documentation it looks like using small stream slices could help with this, but it is not clear from the documentation exactly how. Would the stream slices mean that retries on error would not pull the same records over again?

I am also wondering if updating to use the Incremental Mixin would help. This was not available when I initially created the source.

2

Did you try to use checkpoint state? https://docs.airbyte.com/connector-development/cdk-python/incremental-stream#checkpointing-state

3

Yes, I am checkpointing state every 100 records.

4

You don’t receive any information from the API about the next retry or in the documentation?

5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.