Handling table truncation in Airbyte when source is offline

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Airbyte is overwriting tables in BigQuery when the SQL server source is offline. User wants to know if there is a way to prevent table truncation in such scenarios.


Question

Hey there! I’m sure I’m doing something dumb so wanted to ask the community for help. I have a sql server source replicating to bigquery. It’s doing a full overwrite each time. Small tables. Over the weekend our SQL server instance was offline. Our refresh jobs in Airbyte failed. However, the tables in BigQuery are now empty.

Is there a way to not have Airbyte truncate the table if the source is broken? I saw an option to append vs overwrite, but the desired behavior is to leave the data in BigQuery alone if the sql server instance is offline.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["sql-server", "bigquery", "table-truncation", "source-offline", "append-vs-overwrite"]
2

Awesome. I wish the truncate would only happen if the initial source check succeeds

3

Here’s a screenshot of the failing sources and the job overview of one of the connections showing a timeout error occurring.

4

Sounds good. Thanks, Marcos. We’re good about upgrading airbyte so :raised_hands: if that feature is coming soon.

5

It’s coming soon! We are <https://docs.airbyte.com/operator-guides/refreshes|re-vamping the whole airbyte platform >to focus on not having data downtime like this.

6

Right now, if you are doing a full refresh sync, the destination empties itself first… which if there’s no data coming in from the source… will stay empty. We have built a smarter system that will make tmp tables and swap them into place for full-refresh syncs only when the stream succeeds.