Hi guys,
I am conducting benchmarking to understand Airbyte capabilities.
I have a table with 20.000.000 rows in Snowflake (300 mb of size):
When synced, this becomes 600 mb:
While it is clear that the increase is due to additional metadata Arbyte needs, but it is not clear how I got 6.5 gigs in the UI:
Could somebody explain where it came from?
Thanks!
What sync mode are you using? Please read our documentation about how they work here: https://docs.airbyte.com/understanding-airbyte/connections/
Thanks for your reply @marcosmarxm
I am using full refresh | override. To my knowledge, it reads everything from the source and overrides the destination’s data, just like the name suggests
However, I found no explanation on how the sync size displayed by Airbyte becomes x10 of what is actually written to the destination
Btw, the size remains constant regardless of the sync being the first one for a connection or 1+
The size displayed in Airbyte is from the JSON record and it’s not comparable with the final result in your destination.
Because you’re using Full Refresh you’re going to pull all data again and probably the sync size continues very similar.
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.
