Issue with S3 Source Syncs and Stream Breaks

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Users facing issues with S3 Source syncs where the stream breaks if the relevant stream file is updated during the stream period.


Question

Has anyone encountered behavior in S3 Source syncs where the stream breaks if the relevant stream file is updated during the stream period (could range from 8-30min)? I’m utilizing full refresh currently and and have separate docker processes updating files in S3 that are then streamed to PostgreSQL


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["s3-source-syncs", "stream-breaks", "full-refresh", "docker-processes", "postgresql"]