Random 401 errors when the job is running for a large amount of time

vcalmic · August 12, 2022, 7:42am

Hello Team,
I am developing a custom source connector for Amazon Ads API, which will be a long running job. I am almost at the finish line, but I have an issue. After the job runs for a quite long time, I receive 401 errors from Amazon, however the job runs for more than an hour and the token is refreshed accordingly. However, sometimes I get this 401 error.
I tried to hard refresh the token in should_retry function: tokens: Tuple[str, int] = self.authenticator.refresh_access_token(), but it fails with the following error:

File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 294, in _send
    if self.should_retry(response):
  File "/airbyte/integration_code/source_amazon_ads/streams/common.py", line 292, in should_retry
    tokens: Tuple[str, int] = self.authenticator.refresh_access_token()
AttributeError: 'NoAuth' object has no attribute 'refresh_access_token'

I do not uderstand why the authenticator property is of type NoAuth, because when I initialize the stream in source, I pass the Oauth authenticator. Could someone help or point into the right direction?

marcosmarxm · August 12, 2022, 1:09pm

@vcalmic did you check Amazon Ads API documentation explaining the reason of 401 errors? Maybe this can help you without doing random changes in your code.

vcalmic · August 12, 2022, 1:19pm

@marcosmarxm Yes. The 401 error is thrown when the token is expired.

marcosmarxm · August 16, 2022, 6:50pm

Can I recommend to you check the Source Kyriba, this sources does what you’re trying to achieve. Check the should_retry function:

github.com

airbytehq/airbyte/blob/f281f8f8b2c37e1a56e44fde83815531aad009bd/airbyte-integrations/connectors/source-kyriba/source_kyriba/source.py#L65-L76


      
          @backoff.on_exception(backoff.expo, requests.exceptions.RequestException, max_tries=3)
          def should_retry(self, response: requests.Response) -> bool:
              # Kyriba uses basic auth to generate an expiring bearer token
              # There is no refresh token, so users need to log in again when the token expires
              if response.status_code == 401:
                  self._session.auth = self.client.login()
                  response.request.headers["Authorization"] = f"Bearer {self.client.access_token}"
                  # change the response status code to 571, so should_give_up in rate_limiting.py
                  # does not evaluate to true
                  response.status_code = 571
                  return True
              return response.status_code == 429 or 500 <= response.status_code < 600

Topic		Replies	Views
When creating a new connector, how is HTTP 401 handled after a backoff? Connector Development source-http-request , connectors	3	756	July 14, 2022
HTTPError 400 Client Error: Bad Request when refreshing access token Connector Questions airbyte-connector , connector , bug , oauth-authentication , httperror	0	86	May 14, 2024
Amazon ADS source KeyError Connector Questions & Issues connectors	10	282	July 14, 2022
Airtable source connector - OAuth authentication problem Connector Questions & Issues connectors	2	614	March 28, 2023
Handle expiring refresh tokens with oauth Connector Development connectors , oauth	3	1454	December 15, 2022

Random 401 errors when the job is running for a large amount of time

Related topics