Summary
The user is facing an issue with referencing a timestamp field in an API URL parameter for incremental data loading in a no-code integration platform. The API requires a specific format for the parameter, which is causing difficulties in setting it up through the platform’s UI.
Question
building using no code option in the UI against an API that is working well except for incremental… I have a field that I can use as a timestamp field, but the way that the API wants to reference it is something like this URL param
<https://api.url/resource?_filters=date_modified_greater_than_or_equal(><timestamp_value>)
This means that I can’t get the parameter name/value right because the standard low code interpolator wants the param to be name and then the value on the other side of the equals sign.
Anyone done anything like this on the low/no code side?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["timestamp-field", "api-url-parameter", "no-code", "incremental-data", "integration-platform"]
Based on the information provided in the extracts, it seems like you’re trying to configure incremental syncs for your API using Airbyte’s low-code framework. However, you’re facing an issue because your API requires a specific format for the timestamp parameter in the URL. In the Connector Builder UI documentation, it’s mentioned that the “Inject start/end time into outgoing HTTP request” option is typically used to add the start and end time as a query parameter or body parameter. However, it seems like your API has a unique requirement that can’t be handled this way. In such cases, you can disable the injection in the incremental sync form and use the generic parameter section at the bottom of the stream configuration form to freely configure query parameters, headers, and properties of the JSON body, as mentioned in the Custom Parameter Injection section. You can use jinja expressions and available variables to achieve this. However, the specific format you mentioned (_filters=date_modified_greater_than_or_equal(<timestamp_value>)) seems to be a bit complex and might not be directly supported by the low-code framework. If this is the case, you might need to consider building a custom connector using the Airbyte CDK (Connector Development Kit) to handle this specific requirement. If you need further assistance, I recommend reaching out to the Airbyte community on their Slack channel or forum.
It looks like I can go a low code route by overloading some functionality (https://docs.airbyte.com/connector-development/config-based/advanced-topics#custom-components) but I’m curious - should I just create a random “tools” connector that will never actually be a connector since I’m using the low code approach and then add the classes there? or is there a better code packaging / design approach for these extension points that are not really full connectors?
<@U06CC9J0T42> to inject the timestamp in a non-standard way, you can disable the Inject Start/End Time into outgoing HTTP Request fields, and instead manually add a query parameter with Key set to _filters, and Value set to date_modified_greater_than_or_equal({{ stream_interval.start_time }})
That {{ stream_interval.start_time }} piece is <Advanced Topics | Airbyte Documentation interpolation> which uses the https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/variables/stream_interval|stream_interval object to reference values from the incremental state
<@U02T7NVJ6A3> - fancy! I’ll give that a shot in a bit… very very cool… thanks
<@U02T7NVJ6A3> - this worked well, but do you know why I don’t have an incremental sync option on that entity in the connection configuration now? It still only lists both Full options but not any incremental options
Looks like I had to create a new/recreate a connection before incremental shows in the dropdown
So this happened because you originally created the connection when Incremental Sync was not configured in the builder? I can look into this
then the connection wouldn’t “see” that it was incremental ready - though, as previously reported, a new connection would
I can report that incremental is working like a champ today now that you helped me set it up correctly… thanks again, that is really neat
That’s great to hear, happy to help!