Summary
User is experiencing issues with the Clickhouse connector not allowing the Incremental - Append + Deduped mode, despite documentation indicating that deduplication should be supported.
Question
Hi, does anyone know why my clickhouse connection does not allow Incremental - Append + Deduped mode?
According to the https://docs.airbyte.com/integrations/destinations/clickhouse|documentation, the clickhouse connector should allow deduping
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
['clickhouse', 'incremental', 'append', 'deduped', 'connector']
Usually it’s the case when some conditions are not met for this mode. Check docs, something might be missing.
https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/
> Append Deduped: Write by first adding data to existing tables in the destination to keep a history of changes. The final table is produced by de-duplicating the intermediate ones using a primary key.
https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped
Thanks, i found that it has been removed. There is a discussion here around whether to bring it back as well as a workaround by downgrading: https://github.com/airbytehq/airbyte/discussions/35339