Support incremental sync "overwrite" mode

We have to index millions of records from Snowflake into an Elasticsearch index.

Our expectation is that an update of a record in Snowflake will update the correspondent document in the Elasticsearch index.

This behaviour is currently available for the fullrefresh-overwrite mode only (the table primary key is used as the document id).

We cannot afford to always do a full refresh of the index as it takes too much time (for our business use case) due to the volumetry.

We need to update the index incrementally, but the current types of Airbyte incremental sync are restricted to Append( and Deduped for some connectors), which will produce new documents in the Elasticsearch index, instead of updating the corresponding ones.

The new sync mode “Incremental - overwrite” will handle this use case (insert/update existing records in the destination).

When this mode will be available, it will be very easy to implement it in the Elasticsearch destination connector, as it only requires to pass the primary key as the document id (which is already done for the fullrefresh mode).

Hi @marc-mark, this sounds like a really useful feature. Could you please make a feature request on GitHub? Thank you!

Hi @natalyjazzviolin,

That’s done.