MSSQL CDC sync fails to convert numeric values to double

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • Deployment: Kubernetes
  • Airbyte Version: What version are you using now? 0.39.34-alpha
  • Source name/version: MSSQL 0.4.20
  • Destination name/version: S3 0.3.16 → parquet, snappy
  • Step: Incremental Sync
  • Description:

trying to sync data from MSSQL (CDC) to S3. When I set the sync to be a full sync, it works fine, but as soon as I put it to incremental, it fails with the following message: 2022-10-12 09:53:41 - Additional Failure Information: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field Location is expected to be one of these: NULL, DOUBLE. If this is a complex type, check if offending field (path: Location) adheres to schema: 9.447

It fails on a number of tables, all with the same type of data (double) that looks numeric.

2

Hi @Huib! First things first - could you please upgrade your Airbyte version? We get updates so often that just doing that might fix the issue. :slight_smile:

A few ideas:

  1. Have you seen the docs on Avro - JSON conversion?
    https://docs.airbyte.com/understanding-airbyte/json-avro-conversion/

  2. As a workaround, could you try using JSON or CSV while this gets resolved?

I’ve found a related issue on GitHub:
https://github.com/airbytehq/airbyte/issues/12888

If upgrading Airbyte doesn’t resolve this, I would suggest using the workaround for now as I can’t tell you where this is on the roadmap!

3

Thanks! I’ll try updating first, I’ll report back with the results.

4

Great, looking forward to it!

5

Updating Airbyte and moving to the Databricks destination connector fixed the issue - I am now able to sync the large tables reliably to our data lake :slight_smile: Thanks!

6

That is wonderful to hear! Thank you for the update :smiley: