Ideas to facilitate direct OAuth access tokens?

Connector Development
1

Requirements

We need the ability to provide OAuth access tokens to connections at sync time instead of allowing connections to retrieve their own access tokens via the third-party’s OAuth endpoints (using client id, client secret, etc.) Since access tokens are extremely short-lived, it doesn’t make sense to put them in the connection configuration.

Background

We currently have an OAuth implementation that allows our clients to connect third-party services to their accounts in our system. The OAuth authorization code flow results in access tokens and refresh tokens stored in our database (non-Airbyte system).

We are looking to use Airbyte to ELT our clients’ data from third-party systems into our database for use by our user-facing analysis tools. We will have a separate Airbyte connection for each user’s third-party integration.

We cannot provide our OAuth client id, client secrets, refresh tokens etc. to Airbyte since we will end up with a “split-brain” scenario. Thanks to the fact that refresh tokens are often invalidated once used to obtain an access token, one system will lose access when the other one requests a new access token.

Potential Solutions

Here are some of the approaches we are considering:

  1. Update connectors to add the ability to override the OAuth URLs used. This would allow us to host our own OAuth endpoint that can provide access tokens from our internal systems.

  2. Use Docker’s http(s) proxy capability, pass all traffic through a custom proxy server that will intercept OAuth token requests and respond with access tokens obtained from our internal systems.

  3. Create a new connector that wraps the target connector’s docker image. When our connector is started, it first obtains a fresh access token from our internal system and passes it into the target connector image (environment variable, config setting?)

  4. Fork the connector in question and change the OAuth access token retrieval code to contact our system instead of the third-party system.

I’d love to hear thoughts on these or other potential approaches. Thanks all.

2

Hi Jay,
This is a very interesting question.
In my opinion the approach 2 (Using Docker HTTP proxy) might sound a bit hacky be would be the best one as you won’t have to create your own connector image and custom patches, you’ll be able to track updates on new connectors version and avoid maintenance on it.

I find the other approaches (1, 3, 4) extremely similar, as they require you to build your own connector image from ours, deciding on which one to take would depend of the current implementation of the connector you would like to use. It’s probable that each connector has a slightly different complexity in term of Oauth complexity.

I’m going to reach out to our connector team to get additional suggestions as they worked on connectors’ OAuth for our Cloud platform and might have a better idea of bottlenecks about the approaches you suggest.

1 Like
3

Hello again. Thanks for your thoughts on this. I just wanted to clarify: option number 1 differs from 3 and 4 in that I was not planning on building a connector image from yours. My intention, if other users are interested in similar functionality, is to submit a PR updating the various OAuth connectors in the Airbyte repo adding the ability to override the OAuth endpoint. This was listed as our number one choice, since it eliminates the hacky approach of option number 2 (with its potential bottlenecks and additional points of failure), while still obviating the need for us to maintain our own connectors wrapping yours.

4

Hey were you able to get a PR out?