Large Backfill with unreliable API

I am trying to backfill a large number of records from a source that consistently hits persistent retryable errors. I have increased the retry backoff and number of retires, but even with 10 retries with an exponential backoff the retryable errors continue.

With the current setup, an incremental sync using the get_updated_state method when I run into these errors it will retry, pulling the same records 3 times, and then commit whatever it had from the last sync. It seems like this will eventually sync all records, but it is not ideal, so I am trying to improve my connector code.

Reading the documentation it looks like using small stream slices could help with this, but it is not clear from the documentation exactly how. Would the stream slices mean that retries on error would not pull the same records over again?

I am also wondering if updating to use the Incremental Mixin would help. This was not available when I initially created the source.

Did you try to use checkpoint state?

Yes, I am checkpointing state every 100 records.

You don’t receive any information from the API about the next retry or in the documentation?