Handling MongoDB _id Field Type Issue in Airbyte Connector for BigQuery

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Airbyte is throwing an error when ingesting data from MongoDB to BigQuery due to unsupported _id type. User seeks guidance on modifying the Airbyte MongoDB source connector to handle _id fields of type double and integer by converting them to strings.


Question

Hi everyone,

I am currently facing an issue with ingesting data from a MongoDB source to BigQuery using Airbyte. Some of the _id values in my MongoDB collections are of type double and some are integer. Unfortunately, Airbyte is throwing an error: Failure in source: Unsupported _id type Double

The source system is connected to third-party software, so I cannot change the datatype of the _id field. So, I am considering modifying the source connector code to address this issue.
Has anyone encountered a similar problem and found a solution? Specifically, I am looking for guidance on:

  1. How to modify the Airbyte MongoDB source connector to handle _id fields of type double and integer by converting them to strings.
  2. Any potential side effects or best practices to follow while making these modifications.
    I appreciate any help. Thank you!

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["mongodb", "bigquery", "airbyte", "connector", "id-field", "data-ingestion", "datatype-conversion"]