Destination MSSQL doesn't work with incremental sync mode

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Almalinux
  • Memory / Disk: 5Gb / 55GB
  • Deployment: Docker
  • Airbyte Version: 0.39.41-alpha
  • Source name/version: file 0.2.14
  • Destination name/version: MSSQL 0.4.11
  • Step: during/after sync
  • Description:

When using this source:
I set these reader options:
{"sep" : ";", "skiprows" : 1, "skipinitialspace": true, "engine": "python"}

then the result is an empty dataset. Even the originally raw table is empty:

Nevertheless, the names of the columns get created as they should, it seems.

Log file is attached.
logs-9.txt (26.4 KB)

I suspect I would need to better set reader options or so, in order to get Airbyte to actually use the data… but what should be set in there?

Best Regards

ha, now it succeeded in getting data to the table with these reader options:
{"sep" : ";", "skiprows" : 1, "skipinitialspace": true, "usecols": [0,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]}

But now it’s not possible to get more than the first time sync working. Any subsequent sync attempt fails. Log is attached. Sync mode is set to ‘Full refresh | Overwrite’.

logs-30.txt (220.6 KB)
logs-26_first-sync.txt (30.2 KB)

May this happen due to this bug? 🐛 Destination MSSQL: Incremental normalization bug · Issue #8806 · airbytehq/airbyte · GitHub
Like this how does one use Airbyte with MSSQL at all?

Best Regards

1 Like

Sorry to hear that, the workaround for now its to use only full refresh sync mode.

Hello, that would be great if there is a workaround - how to get it to work with the workaround?

As described, I set it to ‘Full refresh | Overwrite’, but this leads to failure after initial sync.
Indeed, that is what I want in this case - the table should be completely overwritten.
By the way, it also fails when selecting to ‘Reset your data’.

Best Regards

Oh I thought the full refresh worked.
There are two ways to follow here: disable normalization OR run the first time, export normalization and edit to start working with MSSQL.

many thanks, I will have a look at that!

@mmoole can you check the comment below:

I deleted and re-created the database, but this time with all letters in lower case for database name, and schema name.
And voilà: now it’s also working the second and more time to run a sync (Full refresh|Overwrite) :+1::star_struck:
So the issue seems to be that normalization to lowercase letters is handled differently at some stages during all those process steps, and using lower case letters from the start seems to circumvent the possible problems that arise because of that…