Airbyte added a suffix to columns from MongoDB Atlas and made it all null

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 16GB
  • Deployment: Docker
  • Airbyte Version: 0.30.34-alpha
  • Source name/version: MongoDB Atlas
  • Destination name/version: PostgresQL
  • Step: After sync
  • Description:

I have transactions table in MongoDB which contains columns _id, total, discount, and several other columns such as price. During syncing and before any custom dbt transformation could be applied, I saw that columns total and discount were renamed by Airbyte into total_aibyte_transform and discount_aibyte_transform. In addition, those columns are filled with null values even though most of them should have some numeric values.

Only columns total and discount are affected. Other columns such as price are transformed as intended. I’m not sure what’s different about those two columns, and I’d like some help in diagnosing the problem.

schema.yml.txt (206 Bytes)

airbyte-server:/tmp/workspace/${NORMALIZE_WORKSPACE}/build/run/airbyte_utils/models/generated/transactions.sql.txt (1.9 KB)

logs-11619-0.txt (100.6 KB)

2

Hey @Sang_Agung,
Could you please check the value of these columns in the raw table?
You’re running quite an “old” version of Airbyte, upgrading Airbyte to its latest version will allow you to benefit from a newer version of our normalization which might have fixed these problems.

3

Hi @alafanechere, following up this question, I’ve upgraded my Airbyte version to 0.39.7-alpha and I’m still seeing the same problem.

Here’s an example of the raw table:

{
  "_id": "123456789",
  "buyer": "appleseed",
  "notes": "lorem",
  "phone": "55512345",
  "price": 10000,
  "total": "10000",
  "discount": "9000"
}

Column total and discount are still being transformed into total_aibyte_transform and discount_aibyte_transform, and all of them only contain null values

Any idea how to fix this?

4

Hi @Sang_Agung,
I think this might be related to the schema inferred by the MongoDB connector : total and discount are considered as strings. Could you please share your catalog (instructions here on how to extract it)?
Could you also share the list of the other normalized tables? Do you have a table with the price column?

5

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.