Destination MSSQL doesn't work with incremental sync mode

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Almalinux
  • Memory / Disk: 5Gb / 55GB
  • Deployment: Docker
  • Airbyte Version: 0.39.41-alpha
  • Source name/version: file 0.2.14
  • Destination name/version: MSSQL 0.4.11
  • Step: during/after sync
  • Description:

When using this source:
https://opendata.dwd.de/climate_environment/CDC/regional_averages_DE/annual/sunshine_duration/regional_averages_sd_year.txt
I set these reader options:
{"sep" : ";", "skiprows" : 1, "skipinitialspace": true, "engine": "python"}

then the result is an empty dataset. Even the originally raw table is empty:

Nevertheless, the names of the columns get created as they should, it seems.

Log file is attached.
logs-9.txt (26.4 KB)

I suspect I would need to better set reader options or so, in order to get Airbyte to actually use the data… but what should be set in there?

Best Regards

ha, now it succeeded in getting data to the table with these reader options:
{"sep" : ";", "skiprows" : 1, "skipinitialspace": true, "usecols": [0,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]}

But now it’s not possible to get more than the first time sync working. Any subsequent sync attempt fails. Log is attached. Sync mode is set to ‘Full refresh | Overwrite’.

logs-30.txt (220.6 KB)
logs-26_first-sync.txt (30.2 KB)

May this happen due to this bug? 🐛 Destination MSSQL: Incremental normalization bug · Issue #8806 · airbytehq/airbyte · GitHub
Like this how does one use Airbyte with MSSQL at all?

Best Regards

1 Like

Sorry to hear that, the workaround for now its to use only full refresh sync mode.

Hello, that would be great if there is a workaround - how to get it to work with the workaround?

As described, I set it to ‘Full refresh | Overwrite’, but this leads to failure after initial sync.
Indeed, that is what I want in this case - the table should be completely overwritten.
By the way, it also fails when selecting to ‘Reset your data’.

Best Regards

Oh I thought the full refresh worked.
There are two ways to follow here: disable normalization OR run the first time, export normalization and edit to start working with MSSQL.

many thanks, I will have a look at that!