Check if the no-code builder supports following redirects for connectors.
Is it possible with the no-code builder to follow redirects?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Hmm, I don’t know enough to give you a confident answer yet. The provided extracts do not contain information about the no-code builder’s ability to follow redirects. I recommend checking the Airbyte documentation for more detailed information.
Any chance I could speak to a human, <@U04SE163WC8>?
Yes, you can definitely speak to a human for support. Airbyte provides several ways to get in touch with their team: 1. Airbyte Cloud Support: If you have questions about connector setup, error resolution, or want to report a bug, you can open a support ticket within the Airbyte Cloud platform by selecting the “Support” icon in the lower left navigation bar. Alternatively, you can submit a ticket through their Help Center by completing an Airbyte Cloud Support Request. Their team is online and available to assist from 7AM - 7PM Eastern. Source 2. Community Slack: You can join the Airbyte Slack community HERE. You can ask your questions first in the #ask-ai channel and if the bot cannot assist you, you can reach out to the community in the #ask-community-for-troubleshooting channel. Source 3. Airbyte Forum: Airbyte drives their community support from their forum on GitHub. Source 4. Office Hour: Airbyte provides a Daily Office Hour to discuss issues. It is a 45-minute meeting, the first 20 minutes are reserved for a weekly topic presentation about Airbyte concepts and the remaining 25 minutes are for general questions. Source 5. Sales Team: For account or credit-related inquiries, you can contact their sales team. Source Remember to sign up for Airbyte with your company email address, as they do not support personal accounts.
To be precise, I’m using the connector builder.
Huh, I actually don’t know yet, but seems useful. <@U02T7NVJ6A3> do you know if low code cdk / builder would follow redirects?
I just tested and yes, it does follow redirects
But with the authorisation headers?
I find that API requests fail with 401 unless I use the base URL that appears in the error message as the base URL of the connection (instead of the URL that forwards to it).
I just tested this locally and verified that the connector is properly passing my API Key header to the redirect URL. Are you sure that the API is returning a 302 Found request with the redirect URL set on the Location header? That is what seems to work as expected from my testing
Yes, it returns 302.
Handy script. Put this in dayforce.sh and make it executable.
curl --location --request GET '<https://test.dayforcehcm.com/api/ddn/v1/ClientMetadata>' --header 'Authorization: Bearer ${access_token}' -v```
This returns a 302 and finally a 401, just like *Airbyte*.And to get a 200 we need to replace --location with --location-trusted
(HTTP) Like -L, --location, but will allow sending the name
+ password to all hosts that the site may redirect to. This
may or may not introduce a security breach if the site
redirects you to a site to which you will send your
authentication info (which is plaintext in the case of HTTP
Basic authentication).
Providing --location-trusted multiple times has no extra
effect. Disable it again with --no-location-trusted.
Example:
curl --location-trusted -u user:password <https://example.com>```I can confirm that this fixes my issue with curl. How can I fix it in Airbyte?
The Airbyte CDK uses the https://requests.readthedocs.io/en/latest/|requests python library to execute HTTP requests, and a cursory google search seems to indicate that that library may not have a --location-trusted equivalent (see this pretty sparse SO post: https://stackoverflow.com/questions/34359529/python-requests-equivalent-for-curls-location-trusted)
So it may not be possible with the current implementation
The <https://requests.readthedocs.io/en/latest/api/|requests API> has a https://requests.readthedocs.io/en/latest/api/#requests.Session.rebuild_auth|rebuild_auth method that will reapply authentication after a redirect. However it doesn’t seem like it is exposed to users.
Here’s the solution, which is patching the rebuild_auth that comes with requests. It would be nice if users had the option of turning this on and off in Airbyte. I know it is a bad habit, but I have no control over the vendor I must connect to.
dayforce_redirect.py
import requests
from requests import Request, Session
import json
auth_url = "<https://dfidtst.np.dayforcehcm.com/connect/token>"
api_base = "test"
api_redirect = "ustest232-services"
api_url = f"https://{api_base}.dayforcehcm.com/api/ddn/v1/ClientMetadata"
class DayforceSession(requests.Session):
AUTH_DOMAINS = ['dayforcehcm.com']
def __init__(self):
"""Create Dayforce Session that preserves headers when redirecting"""
super(DayforceSession, self).__init__() # Python 2 and 3
def rebuild_auth(self, prepared_request, response):
"""Keep headers upon redirect as long as we are on any of self.AUTH_DOMAINS"""
headers = prepared_request.headers
url = prepared_request.url
if 'Authorization' in headers:
original_parsed = requests.utils.urlparse(response.request.url)
redirect_parsed = requests.utils.urlparse(url)
original_domain = '.'.join(original_parsed.hostname.split('.')[-2:])
redirect_domain = '.'.join(redirect_parsed.hostname.split('.')[-2:])
if (
original_domain != redirect_domain and
redirect_domain not in self.AUTH_DOMAINS and
original_domain not in self.AUTH_DOMAINS):
del headers['Authorization']
s = DayforceSession()
# Authentication
data = {
"grant_type" : "password",
"companyId" : "ddn",
"username" : "DFWSTest",
"password" : "DFWSTest",
"client_id" : "Dayforce.HCMAnywhere.Client"
}
token_resp = requests.post(auth_url, data=data)
access_token = json.loads(token_resp.text)["access_token"]
# Request data to REST API
s.headers['Authorization'] = f"Bearer {access_token}"
api_resp = s.get(api_url)
print(api_resp.history)
print(api_resp)
print(api_resp.text)
print(s.headers['User-Agent'])```I got the solution from <Issues · psf/requests · GitHub requests issue>.
I also opened a https://github.com/airbytehq/airbyte/discussions/35058|discussion.