We use Kafka log compaction on our raw record topics. Kafka log compaction is a mechanism to provide finer-grained per-record retention instead of coarser-grained time-based retention. Records with the same key are selectively removed when there is a more recent update. This way the log is guaranteed to have at least the last record for each key.
This requires that we be able to configure the Kafka record key to be generated on a combination of static string and ingested record values.
I took a look at the Kafka Destination documentation. I see that the topic name pattern is configurable, but I’m not seeing that the record key pattern is configurable. Search doesn’t surface anything either. Are configurable Kafka record keys something that’s been explored or solved? I appreciate any insights.