Issue with Airbyte truncating request header value

Summary

Airbyte is truncating the value of a request header when setting it, specifically with the example provided in the message.


Question

Hopefully this is the right channel for it, but I’m having issues with Airbyte truncating the value of a request header I’m setting. Example:
Cookie: FIELD_1=abc; FIELD_2=123
When I set the below in the value of the header, it truncates FIELD_2
FIELD_1=abc; FIELD_2=123
Has anyone ever run into this?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["airbyte-platform", "request-header", "value-truncation", "bug"]

Hmm, I don’t know enough to give you a confident answer yet. However, if you’re encountering issues with how Airbyte handles request headers, especially when it comes to customizing them or dealing with specific formatting requirements, it might be helpful to look into the documentation or community discussions around custom parameter injection or handling API requirements that aren’t directly supported by the default configuration options. For instance, Airbyte’s documentation on Custom Parameter Injection discusses scenarios where the default methods for injecting parameters into requests might not suffice, such as when an API requires a prefix or suffix to be added to a value, or when values need to be combined in a specific way. While this section primarily focuses on query parameters, headers, and properties of the JSON body, the underlying principle of using Jinja expressions for customization could potentially be applied to your situation with request headers. Here’s a general approach based on the principles outlined in the documentation: 1. Disable Default Injection: First, ensure that any default injection methods provided by Airbyte that might be causing the truncation are disabled. 2. Use Generic Parameter Section: Utilize the generic parameter section at the bottom of the stream configuration form to manually configure the headers. 3. Apply Jinja Expressions: Employ Jinja expressions to precisely define the value of the Cookie header, ensuring that both FIELD_1 and FIELD_2 are included without truncation. While I can’t provide a direct solution or code snippet specific to your issue without more details, exploring the capabilities of Jinja expressions for custom parameter injection might offer a pathway to resolving the truncation problem. For more detailed guidance or to explore other users’ experiences with similar issues, the Airbyte GitHub issues and community forum could be valuable resources.

<@U06LXFHHRT4> this is odd, how are you building the connector? Are you using the connector builder, python cdk, etc?

Where are you setting the request header and where are you seeing the incorrect request header?

<@U047ANT3J84> I think I’ve narrowed it down to the cookie being overwritten by a Set-Cookie header from the previous response. The issue goes away when I remove a very specific field in the cookie. FWIW - I was creating via the Builder tab in the app. I think I’m good for now, I’ll just have to work around this particular service

Ok, thanks for the context! If it were a Builder bug I would want to get that info passed along. Is there anything you’re doing to deal with the Set-Cookie header that you can’t do in the builder?

Yeah, for the particular API I am working with, I’m using Basic Authentication. There’s an issue with the server of the API I’m using that forces me to include the session cookie in the header along with the basic authentication credentials. This is easily done in tools like Postman or cURL, but the builder does not work in this case.

If I were to set the cookie in the header like the above example via Postman, the request works successfully, but Builder does not. It mutates the cookie header and strips out the second value after the ;.

This may be difficult to reproduce since I’m not 100% sure it’s related directly with the builder. I just know the builder handles the response differently than cURL and Postman. If I were to change sap-usercontext=xxx to something else like foo-bar=x , then foo-bar does not get stripped out, which led me to believe it’s related to the Set-Cookie response from the server

Would you be willing to export the yaml so we can try it out? Is it an API that I could easily get my own account/token for?

Unfortunately it is not, it is a private, internally deployed API

I see, that makes it harder. I’ll admit I’m still a bit unclear about what the problem is, when you say its getting “Stripped out”.

The cookie you are sending in the request is not coming back in the response, because the server is sending you a set-cookie header telling you to change the value in your request - is that right?

Sure, I can clarify on the “stripped out” description. My cookie has the following content structure:

Cookie: SAP_SESSIONID_S4D_130=&lt;redacted&gt;; sap-usercontext=sap-client=130
When I inspect the request data from the test run via the Request tab on the right-hand side of the UI, you can see that the cookie in the request is completely missing the sap-usercontext value:

After some online digging I think this is a really weird requests library behavior. <https://mattslifebytes.com/2022/11/15/passing-cookies-as-a-predefined-header-in-python-requests/|This doc goes into it a bit>, but searching for “python requests cookie missing” pulls up a bunch of stuff. What this means functionally is that under the hood, we’ll probably have to set cookies separately from other headers, which means that we’d need to add the functionality to the builder to separate out cookies from other request headers. I’ll make an issue for it, but I’m not sure when that work will get picked up.

What’s the cookie doing for your connector?