Summary
Advice on handling missing data at the source and ensuring data integrity in Airbyte
Question
Does anyone have any advice for missing data at the source? I’ve reset the stream and made the same request via postman and everything looks good with the data at the source. If missing data is common, how am I supposed to use Airbyte at all? We can’t just have missing data all the time.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["missing-data", "source", "reset-stream", "postman", "data-integrity", "airbyte"]
I’ve had a similar issue, airbyte stopped polling at a specific date every time. Even after restarting the job. Postman confirmed that there was data, but airbyte did not. I modified the json settings and manually changed the last sync date to move up a few seconds. Had to do this a few times before it caught up. Unfortunately, did not have a solution but was the only thing I could do to get it to complete
If this is happening to a certified source to a certified destination is really not expected. <@U02UFRZRJHX> this is for the https://airbytehq.slack.com/archives/C021JANJ6TY/p1705684554007219|issue reported for Hubspot? If possible report as a Github issue will ping the connector team about the problem.
Yes it is, thank you- I will create an issue and let you know when that’s submitted, as this has been quite an urgent issue.
Thanks Madison, please add the following info the issue:
• Source version
• Destination version
• Platform version
• Is this affecting all streams or only a specific one (generally certified connectors are full tested but maybe there is one endpoint didn’t have full coverage)
<@U06C5BK9H9P> the same for you if you’re not using Hubspot :octavia-loves: