Summary
The user is facing an issue with the Confluence source connector where the body extraction appears as ‘null’ in the destination. They are seeking guidance on how to debug this problem.
Question
Hi I am trying to use the Confluence source connector to pull contents from my pages. I see there is a body that I can extract in the stream but it always shows up as null in my destination. Anyone know how I can get started in debugging this?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["confluence-source-connector", "body-extraction", "debugging"]
I’ve confirmed that the API token that I configured has access to the pages and I’m able to get the pages contents using:
curl -H "Authorization: Basic <token>" <https://credalai.atlassian.net/wiki/rest/api/space/CS/content/page/?expand=body.view> -H "Content-Type: application/json"
<@U05UW7N773R> saw you recently contributed to the Confluence source. Any ideas here on how to debug this?
Hi <@U06863ML00P>, I’m sure I’ve seen exactly this issue before but getting the connector to work was a maze of debugging so I can’t recall the exact solution. What I’d recommend you do is to run the connector locally with your credentials and drop a breakpoint in the airbyte CDK installed in the virtualenv you create. It’s ugly but it should give you the exact URL airbyte is using and other info to understand why it’s not working.
FYI if you create a virtualenv as .venv, the file you’ll want to add a breakpoint to is .venv/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py
Give me a shout if you need some pointers on running it locally. You’ll need to clone the airbyte repo for starters 
Awesome thanks yeah will try running locally to see if I can get to the bottom of it - good to hear someone else has also run into this before
<@U05UW7N773R> is there a way to run a debugger for a connector locally while running the rest of the Airbyte platform using ./run-ab-platform ? Any documentation around that would be much appreciated. Found this: https://docs.airbyte.com/contributing-to-airbyte/resources/developing-locally which doesn’t seem to mention anything around running the connectors in a debugger
So I’m just copying the below from my local version of the confluence connector:
From this connector directory, create a virtual environment:
python -m venv .venv
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
development environment of choice. To activate it from the terminal, run:
source .venv/bin/activate
pip install -r requirements.txt
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
#### Create credentials
Create a file `secrets/config.json` conforming to the `spec` field of the `source_confluence/manifest.yaml` file.
Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
### Locally running the connector
python main.py spec
python main.py check --config secrets/config.json
python main.py discover --config secrets/config.json
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --state integration_tests/sample_state.json
The python bit at the end is relevant. You'll first want to create a file `secrets/config.json` within `airbyte/airbyte-integrations/connectors/source-confluence` which contains your API key etc
Once you've got your secrets and a `.venv` setup with the airbyte cdk installed following the readme, you can edit that `http_requester.py` file and dump a `breakpoint()` in the `send_request` method. Then you can inspect the URL, headers, etc.
Hope that helps :slightly_smiling_face:
Will give it a shot. Thanks a ton!