Summary
User reports unexpected empty state messages from the
destination-aws-datalake
connector during incremental syncs, leading to integration failures. They seek clarification on recent changes and request access to the changelog.
Question
hi all - did something change recently on how Airbyte emits state messages? We use the destination-aws-datalake
destination and it uses an empty state message for incremental streams as a means to recreate the table for a given stream (when a stream gets reset, the state is reset + an empty state msg is emitted and that is/was the only way to determine when a full drop and replace is needed). In the past few days we’ve increasingly seen incremental syncs emit empty state messages for what appears no reason which has caused some of our integrations to stop working because the destination will try to recreate the table and remove all glue partitions. If something has changed here can someone point me to the changelog or more information on what has changed? Any help here is greatly appreciated! Thanks
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.