When creating a new connector, how is HTTP 401 handled after a backoff?

Connector Development
,
1

I’m developing a new connector in Python for one of our client’s REST API. We authenticate via OAuth2 so I’ve extended Airbyte’s airbyte_cdk.sources.streams.http.requests_native_auth.Oauth2Authenticator and had to re-implement get_refresh_request_body() and refresh_access_token() to suit my needs and to pass custom headers needed by the API.

When syncing, I sometimes get HTTP 401 errors after a custom backoff handling HTTP 408.

    def should_retry(self, response: requests.Response) -> bool:
        return response.status_code in [401, 408] or super().should_retry(response)

    def backoff_time(self, response: requests.Response) -> Optional[float]:
        if response.status_code == 401:
            return 10
        if response.status_code == 408:
            return randint(120, 240)
        return super().backoff_time(response)

When retrying for a HTTP 401, how can I reset the requests authentication header?

Token is set to expire 1 hours after being issued. The problem is that I can get a backoff at, say 58 minutes of the token life’s (2 minutes remaining), and when the backoff retries, the token is no longer valid (60+ minutes).

I want to recall the Oauth2Authenticator magic method so that it can reset get a new access token:

    def __call__(self, request):
        request.headers.update(self.get_auth_header())
        return request
2

Phillippe can I recommend to you take a look in the Bing Ads implementation? Maybe too complex and you don’t need it:
https://github.com/airbytehq/airbyte/blob/d7cffe4ed716035d53f3d83dbeb009cac9060515/airbyte-integrations/connectors/source-bing-ads/source_bing_ads/client.py#L131-L151
But there is one example to get refresh Oauth token there.

Maybe you can add the logic inside the should_retryfunction, something like:

if status_code == 401: self.oauth = self.oauth.get_new_token()
3

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.