Summary
User wants to add a new stream to an existing ‘community’ connector, specifically the ‘Shopify’ connector. They are looking to leverage the existing connector and focus on adding stream configurations. They are also inquiring about the possibility of this functionality being included in the OSS version.
Question
Hi Airbyte team,
Is it possible to add a new stream to an existing community connector? e.g. I want to add a stream to Shopify connector that doesn’t already exist as of now. It’d be helpful If I could leverage the already available Shopify connector and just worry about adding the stream configs correctly.
If it doesn’t exist, then is there a possibility of such a functionality being included in the OSS version?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["adding-new-stream", "existing-connector", "community-connector", "shopify", "stream-configurations", "oss-version"]
Hey Justin, Thank you so much for this answer. It’ll surely help me
The source code for those connectors is available in GitHub, and you can take them and create your own custom version.
This is easier with low-code connectors, as you can often just load the YAML into Connector Builder and adjust it as needed to make a new version. But in CDK connectors you’ll need to use whatever language they did (usually Python). Or build a new one.
Adding a stream is usually pretty easy, especially if it’s similar to another stream as you can just copy that and rename it, then add it to the manifests/schemas. It gets a little more involved if the stream is wildly different (e.g. uses a different type of pagination that isn’t already implemented, is a child stream and the connection doesn’t already implement one of those, etc.)
If you’re not comfortable working with the CDK, another option (if the API uses features compatible with it) is to make a new Builder connector for just the additional stream and set it to the same destination. While it will be a separate connection, the end result will be that you’ll have the same output tables in the end, and is sometimes easier for those not comfortable with diving into the CDK.
And if you add a stream that others might need, be sure to contribute your changes back so they can end up in the community connector for everyone!