Failed to fetch MongoDB schema

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Google Kubernetes Engine default OS
  • Memory / Disk: 4 x e2-medium
  • Deployment: Kubernetes
  • Airbyte Version: 0.38.1-alpha
  • Source name/version: source-mongodb-v2:v0.1.14
  • Destination name/version: destination-bigquery:v1.1.0
  • Step: The issue is happening creating the connection
  • Description:

After having successfully registered my mongodb source (atlas cluster) and my bigquery destination as well (with gcs stating), I try to connect both but it keep telling me the same issue:

Failed to fetch schema. Please try again.

I have multiple databases on this cluster, some of them connect well (so it’s not a connection issue), but other not and throw me this error. You will find in attachment the logs of the worker, but in summary, the logs hit me with the following error :

Exception in thread "main" com.mongodb.MongoCommandException: Command failed with error 17276 (Location17276): 'Invalid $project :: caused by :: Use of undefined variable: inc' on server XXX.mongodb.net:27017.

I can’t find why the connection works on some database but not on others and also where does this error comes from. Do you have any idea about how to solve my problem ?

logs.txt (53.9 KB)

Capture d’écran 2022-07-13 à 17.31.52
Capture d’écran 2022-07-13 à 17.31.522562×544 24.5 KB

2

Hi @Clovis, do you know if the user has permissions for all the collections being fetched? If you are not currently fetching authorized collections only, take a look at this GitHub comment: https://github.com/airbytehq/airbyte/issues/8752#issuecomment-1019887350 . Let me know if this helps!

3

Hi @natalyjazzviolin ! Thanks for the reply :slight_smile:

It could indeed have been a good lead ! But after further tests using small scripts, the user is perfectly able to list collections from the relevant databases and access data.

4

Interesting! Thanks for the update and I’m glad it got resolved :slight_smile:

5

My bad, I didn’t make myself clear in my previous post :grimacing:

What I was saying was that, on the contrary, as my user had the correct permissions to access data using other methods, it means that my Airbyte problem did not come from there.

I’m unfortunately still facing the issue and cannot fetch my databases schemas.

6

Oh gotcha! Could you please try upgrade the MongoDB connector to the latest version? I see you’re on 0.1.14, but the latest is 0.1.15. Here’s a guide:
https://docs.airbyte.com/operator-guides/upgrading-airbyte/

If that doesn’t help I will link this post to a GitHub issue, as I have found one open for the same problem.

7

I’ve just tested again after upgrading my whole environment with a fresh new install :

  • airbyte version: 0.39.37-alpha
  • source: source-mongodb-v2:v0.1.15
  • destination: destination-bigquery:v1.1.11

Sadly, I’m facing the exact same error :

com.mongodb.MongoCommandException: Command failed with error 17276 (Location17276): ‘Invalid $project :: caused by :: Use of undefined variable: inc’ on server XXX.mongodb.net:27017.

8

Hi @natalyjazzviolin ! Just to be able to track it as it is a major blocker for our team, could you please link me the github issue ?

9

Sure! Here is the issue, I am also linking your post to it internally.
https://github.com/airbytehq/airbyte/issues/8564

10

@Clovis on second thought it looks like we have several issues with MongoDB failing to fetch the schema, but they are all slightly different. Are you using nginx or a proxy in your configuration?

11

Hi @natalyjazzviolin ! Sorry for the delay. Indeed, this issue was not actually related to the ticket.

It appears that the fetch issue was related to some mongoDB documents comporting keys containing special characters (like document keys beginning with $ for example).

I had to find a workaround as it was critical for our team to extract data quickly and I did not try since then to test again with airbyte.