Summary
User reports an issue with S3 Source syncs where the stream fails if the source file is updated during the sync period, while using full refresh to stream data to PostgreSQL.
Question
Has anyone encountered behavior in S3 Source syncs where the stream breaks if the relevant stream file is updated during the stream period (could range from 8-30min)? I’m utilizing full refresh currently and and have separate docker processes updating files in S3 that are then streamed to PostgreSQL
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.