Summary
The user is facing an issue with the Google Ads source connector where it does not keep track of the current state (segments.date) during incremental append sync. The connector synchronizes all data within the defined date range every time, causing overhead. The user is looking for a solution to sync only the last N days of data. The user has observed that the connector emits a state message but the state is not persisted.
Question
hi guys! I’m trying to use google ads source connector. but I’ve noticed that it doesn’t keep track of current stated (segments.date). The problem for me is that every time connector runs incremental append sync, it synchronises all data for range defined in (Start Date/End Date) source parameters. This is a bit overhead for me, as my start date is 2019 and syncing this much data every time will take way too much time. ideally connector should sync last N days, but not entire range. is it possible to solve somehow? I clearly see that connector emits state message
{"type": "STATE", "state": {"type": "STREAM", "stream": {"stream_descriptor": {"name": "custom_ga_query", "namespace": null}, "stream_state": {"REDACTED": {"segments.date": "2024-03-28"}}}, "sourceStats": {"recordCount": 54}}}
but in the end state is not persisted
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.