Configuring Airbyte to not truncate destination table when source connection is offline

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Airbyte truncates destination tables when source connection is offline, is there a way to configure it not to truncate?


Question

hi there!

Is there any way to configure Airbyte to not truncate destination table if the source connection is offline?

It seems that one of our source went offline indefinitely and the target tables were truncated after they went offline :melting_face:


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["configure", "airbyte", "truncate", "destination-table", "source-connection", "offline"]
2

what source connector do you use?

3

I also believe that the changes coming in resuamable full refresh (as well as moving from Reset->Refresh) will prevent the destination from being cleared until the load is more complete than the current state. I don’t think this is fully rolled out yet though, but it’s something the Airbyte folks have been talking about

4

<@U05JENRCF7C> we use the custom connector with full-refresh

5

That would be neat, Justin!

6

Is there any logic in code of source connector detecting that source if offline?
Raising exception should stop connection after number of attempts