Issue with Microsoft Sharepoint Connector Configuration

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User is experiencing a configuration error while setting up a Microsoft Sharepoint source connector on Airbyte Cloud and Open Source. The error indicates a failure in listing files, suggesting potential issues with the provided credentials or permissions.


Question

Hi Community,

I’m trying to setup a Microsoft Sharepoint connector, and I’ve been pulling my hair for 3 days now.

Let’s say I have a Microsoft Sharepoint site: https://my-tenant.sharepoint.com/sites/my-site
I have 3 shared folder

  • /sites/my-site/Shared%20Documents/images
  • /sites/my-site/Shared%20Documents/reports
  • /sites/my-site/Shared%20Documents/shared
    I have 2 files: demo-csv-file.csv and demo-json-file.json in /sites/my-site/Shared%20Documents/reports
    I have Global admin access on the tenant
    I set up an App registration in Microsoft Entra ID, to which I gave Microsoft Graph
  • Files.Read.All and
  • Sites.Read.All permissions
  • with admin consent granted for my organization

I tested access to my Sharepoint using my app registration credentials on Postman and was able to access everything I needed to.

Now I want to setup a Microsoft Sharepoint source connector on Airbyte Cloud and Airbyte Opensource.
From the documentation I see that:

  • Airbyte Cloud needs Microsoft OAuth2.0 authentication (we provide the tenant_id and click on authenticate button to provide Entra ID credentials)
  • Airbyte Opensource uses Service Key Authentication (we provide tenant_id, client_id, client_secret and User Principal Name)

Connector Version: 0.5.2

I set up a stream to filter out csv files:

  • Delimiter: ;
  • Encoding: cp1252
  • Escape Character: \
  • Name: demo-stream
  • Globs: demo-csv*
  • Search Scope: ALL
  • Folder Path: empty

Both return:

Configuration check failed
[‘Traceback (most recent call last):\n File “/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py”, line 81, in _check_list_files\n file = next(iter(stream.get_files()))\n File “/airbyte/integration_code/source_microsoft_sharepoint/stream_reader.py”, line 274, in get_matching_files\n raise AirbyteTracedException(\nairbyte_cdk.utils.traced_exception.AirbyteTracedException: None\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File “/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py”, line 62, in check_availability_and_parsability\n file = self._check_list_files(stream)\n File “/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py”, line 87, in _check_list_files\n raise CheckAvailabilityError(FileBasedSourceError.ERROR_LISTING_FILES, stream=stream.name) from exc\nairbyte_cdk.sources.file_based.exceptions.CheckAvailabilityError: Error listing files. Please check the credentials provided in the config and verify that they provide permission to list files. Contact Support if you need assistance.\nstream=demo-stream\n’]

I’ve been trying to make this work for days now without success. I’m convinced there’s a configuration issue somewhere.

What am I missing here? Any insights?

Please help!

Thanks


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['microsoft-sharepoint', 'connector-configuration', 'airbyte-cloud', 'airbyte-opensource', 'oauth2', 'service-key-authentication', 'file-listing-error']
2

Globs parameter doesn’t look good for me. I’d try something like **/demo-csv*.csv, reports/demo-csv*.csv

3

Tried it, and keep getting the same error message

4

You can experiment with folder path as well.
If that doesn’t work, I guess more debugging is necessary.

What you can do is to clone repository https://github.com/airbytehq/airbyte
go to airbyte-integrations/connectors/source-microsoft-sharepoint
install prerequisites and dependencies, then those commands will be useful

poetry run source-microsoft-sharepoint check --config secrets/config.json
poetry run source-microsoft-sharepoint discover --config secrets/config.json
poetry run source-microsoft-sharepoint read --config secrets/config.json --catalog sample_files/configured_catalog.json```
my favourite approach in Python is to install IPython `pip install IPython` and add snippet
```from IPython import embed
embed()
from sys import exit
exit(0)```
somewhere in a code, to stop and interactively check result or execute some extra code
5

<@U080H4GQ10E> were you able to handle this? <@U081JSA6135> and I have been facing the same problem…

6

Globs parameter doesn’t look good for me. I’d try something like **/demo-csv*.csv, reports/demo-csv*.csv

7

Tried it, and keep getting the same error message

8

You can experiment with folder path as well.
If that doesn’t work, I guess more debugging is necessary.

What you can do is to clone repository https://github.com/airbytehq/airbyte
go to airbyte-integrations/connectors/source-microsoft-sharepoint
install prerequisites and dependencies, then those commands will be useful

poetry run source-microsoft-sharepoint check --config secrets/config.json
poetry run source-microsoft-sharepoint discover --config secrets/config.json
poetry run source-microsoft-sharepoint read --config secrets/config.json --catalog sample_files/configured_catalog.json```
my favourite approach in Python is to install IPython `pip install IPython` and add snippet
```from IPython import embed
embed()
from sys import exit
exit(0)```
somewhere in a code, to stop and interactively check result or execute some extra code
9

<@U080H4GQ10E> were you able to handle this? <@U081JSA6135> and I have been facing the same problem…