I’m developing a connector for a source with an unreliable (time-outs & rate-limits) API. I want to use this source for incremental syncs, but because of the unreliability of the API and the amount of data in the initial full sync, the duration of time required for that full sync is impractical. I have access to the full data of the underlying source (i.e. a db dump). Is it possible to load this data myself and only use Airbyte for the incremental updates? Are there any other strategies/methods that I can utilize for this scenario? Thank you in advance for any suggestions and ideas.
We don’t have a out of box solution for this? But you could load that in a different table and use out custom dbt to merge them after every sync