Deploying CDK Connector in Cloud Instance with Custom Authentication Tools


Exploring the possibility of deploying a CDK connector in a Cloud Instance with custom authentication tools that may not fit in the UI builder.


How do you deploy a CDK connector in Cloud Instance , is it possible to, or we are focusing and promoting UI builder only , the concern is I have few custom authentication tools which won’t fit in UI builder so curious on how it will work

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["deploying", "cdk-connector", "cloud-instance", "custom-authentication-tools", "ui-builder"]

Yes, you can deploy a CDK connector in a Cloud instance. For complex custom authentication schemes that involve signing requests or doing multiple requests to authenticate, you would need to use the low-code CDK or Python CDK. This is especially useful when the authentication methods required by some APIs are not supported by the UI builder source. Once your connector is developed, it can be distributed as a Docker image and made part of the regular Airbyte connector catalog. The process to publish your connector can be found on the overview page source. If you have developed a Docker-based custom connector, you can upload it following the guide here. This guide assumes that you have followed other guides and tutorials about connector development, have your connector running locally on an Airbyte development instance, and want to deploy this connector to a production Airbyte instance running on a VM with docker-compose or on a Kubernetes cluster source. Please note that while Airbyte users can use custom connectors on Cloud, Airbyte cannot provide support for those connectors as if they were GA or Beta connectors. Proper monitoring, alerting, and instructions to revert or upgrade the CDK used by these connectors should be in place source.

Hey Siddu! I’m not confident (to my embarrassment ) that you can deploy a custom Python CDK image to a cloud workspace. And we indeed want to enable as many connectors as possible to be expressed in the Builder UI.

Tell us more about the custom auth tools. We have a lot of things scheduled for the Builder to improve in Q1, maybe we can squeeze what you want in.

<@U069EMNRPA4> I have another query , what happens to the custom connectors I build in cloud instance will that be shared with other customers . Can I use external credential stores for storing secrets and pass to airbyte rather than keeping them in airbyte cloud

They won’t be shared with other customers. If you want them to be available to others, you can grab the declaration and contribute it back on GitHub, but that’s up to you. They’re private to your workspace by default.

As for external credential storage — not entirely sure, I don’t think we have a standard mandated way of doing this yet. Hope I’m wrong! <@U02T7NVJ6A3>

Let me ask the relevant teams about these requests and I will update this thread when I learn more

> the concern is I have few custom authentication tools which won’t fit in UI builder
<@U067F0ZASG7> could you elaborate on this? Which auth tools won’t fit in the builder UI?