Summary
User is looking to replicate a custom space in Confluence connector by applying a filter.
Question
The bot is not able to help me I think
I see in the Confluence connector doc that Space is a supported stream. Is there a way, in my connexion, to only replicate a custom space ?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["filtering", "custom-space", "confluence-connector", "replication"]
I’m trying to setup a Confluence to PInecone connection. It’s working, but it’s importing everything from my Confluence when I only need a particular space
Ok I’ve found the space, but still struggle to filter by space name
I don’t see a setting related to this, but maybe you could create a user/API key that only has access to that one Space as a way to lock down what it’s pulling?
I believe this one was migrated to low-code, so you could grab that YAML and drop it in builder and add a user input to filter the space ID on all the streams pretty easily.
I don’t get that part “I believe this one was migrated to low-code, so you could grab that YAML and drop it in builder and add a user input to filter the space ID on all the streams pretty easily.”
so with the recent changes to the no-code Builder, it makes it a lot easier to work with low-code connectors without having to set up a dev environment and actually build the docker images and such.
Basically what you can do is grab the YAML (you can <https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-confluence/source_confluence/manifest.yaml|download it straight from GitHub here>) and then go to Builder > New Custom Connector > Import a YAML Manifest and select that file.
Then from there you can use the no-code UI to either hard-code the specific space ID into the request for each stream (in whatever format their API docs say, e.g. query parameter or such), or add a user input field that will show in the source connector config that lets you set the ID (or give a list of them), then add the code in each stream to handle that appropriately.
Basically it means that for anything that’s already a low-code connector, it’s really easy to make simple tweaks in the no-code builder now.
And in most cases low-code connectors that use features that builder doesn’t directly implement will now cram that under the “additional fields” at the bottom (or toggle that one section to YAML mode). But unlike before, it doesn’t break (and you don’t need to worry about knowing how to edit the YAML unless you happen to need to modify one of those non-UI sections).
Once you’re happy with your change, you can test it right in Builder id you set up your Testing Values (so it has a way to connect), and then hit Publish to Workspace. And it’ll show up as a new Custom connector you can use instead of the built-in one.
Again, a dedicated user/API key might be easier, but this is really handy when that isn’t an option or you need to add a custom stream or such.
Thanks a lot for your advices. I was indeed able to hard code that. And like that I will be able to make the same work on others spaces.