User is seeking help with using CDK for Airbyte connectors and outsourcing development of new connectors.
Anyone here have decent experience with using the CDK for Airbyte? I’m looking to outsource the development of building a couple of new connectors that we need.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Take a look at this: https://github.com/airbytehq/airbyte/blob/543c5353688944925d0388d34408d2da33015cff/airbyte-integrations/connectors/source-zoom/source_zoom/components.py#L37|https://github.com/airbytehq/airbyte/blob/543c5353688944925d0388d34408d2da33015cff[…]e-integrations/connectors/source-zoom/source_zoom/components.py —this is a custom component that implements one quirky oauth2 flow for Zoom.
It has a manifest.yaml that refers to that custom component.
Here’s what you can do:
There are a couple of docs scattered on how to run Builder with custom components, but I don’t think it will be easy to find and get to —it’s experimental.
Point me to where you would look for them —I would add a writeup there, and I’d make a guide for you.
<@U06PL8174LX> pretty recently had to deal with that.
<@U06JU9ANMAT> I’m hooked now —can you point me to a specific API that I could fetch data from, and perhaps a test account that would have minimal sandbox data?
DM is fine —I want to try it out and get you a video guide
I’m setting up a new developer account, but an example would be: https://developer.ebay.com/api-docs/sell/finances/resources/payout/methods/getPayouts
I think a lot of us do, but if at all possible I would highly recommend using low-code connectors (including through the Connection Builder UI, which just generates low-code connectors under the hood).
For most REST APIs with standard authentication, a non-developer can easily tackle this task and it makes maintaining the resulting connection far easier.
If you’re needing to integrate with APIs that are VERY complex, don’t support standard authentication methods, or aren’t RESTful—that’s where CDK comes in really handy as it means you still aren’t having to start from scratch. You’ll notice even Airbyte is moving as many of their certified connectors as possible to low-code.
Yeah, there are 2 specific APIs that auth flows that are not supported in the UI builder. Those are ones I’d like to have built with the CDK.
Walmart & eBay - both work the same with a typical oauth 2 flow, but also require a token in the auth header
You mean there are two things going on — both a static secret token in http request headers, AND additionally an oauth2 token?
Correct. OAuth 2 flow for auth while also having the client id/secret in the Auth header: Basic <encoded_credentials>
As a workaround, on each stream level you can add a Request Headers block, and in that block just add add the header with name Auth and value Basic {token}.
Yeah. I believe that is what I need to do when specifically during the auth flow, set the basic auth and do oauth 2 at the same time. Both ebay and walmart work that way, which is weird, but feels like I’d have to go down to the cdk to do that?
Did you guys get this to work? I’m in the same boat — need to connect to walmart and ebay with airbyte
Happy to help. A HUGE recommendation is to use lowcode manifests with a tiny custom component to implement just the auth piece. There are a few examples in the repo, happy to post to them. source-zoom is one I think.
But secondly, <@U063M9WHWTU> recently added JWT with support to the CDK. It’s not yet in the UI, but you can already use it in the lowcode manifest itself, even via the builder ui “yaml view”.
So, what authentication flow do you need? Chances are, building it into the cdk is easier than contracting out two full Python cdk connectors.
I think you can do that in builder today, let me check
An ideal solution would be something like a “combo authenticator” or something that tells all streams to always send that header (well, sounds awfully like combo auth).
Ahhhhhhh, so you’re not at stream level yet. Hmmm.
Well, no. Give me a second.