Adding custom field on stream level for AWS Datalake destination with Athena and Iceberg

Summary

User is looking for a way to add a custom field on the stream level for AWS Datalake destination with Athena and Iceberg to partition data based on the source.


Question

Hi! i’m trying to extend the AWS-Datalake destination to use athena with Iceberg.
i have it working already, but one thing that i want to add is partitions based on the source.
Is there an easy way to add a custom field on the stream level, similar to the primary key but that the user can choose from, maybe a simply text field?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["aws-datalake-destination", "athena", "iceberg", "custom-field", "stream-level", "partition"]

i decided to go with the cursor_field as the one used for partitioning