Summary
The user is looking to inject request parameters into the API response for a cursor field. They are unsure how to accomplish this.
Question
Hi, I have an API endpoint that takes a startDate and endDate and returns aggregate results within that window. Unfortunately, the response doesn’t have a viable cursor field in the response body, and a cursor field is required in order to configure the Stream.
Is it possible for me to inject the request startDate and endDate into the response body using transformations and use one of them for the required cursor field? If it is possible, it’s unclear to me how to accomplish this.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["api", "cursor-field", "transformations", "request-parameters"]
Based on the information provided in the extracts, it seems that you can’t directly inject the request startDate and endDate into the response body using transformations in Airbyte. The cursor field is typically a field in the response data that helps track which records are new or updated since the last sync source. However, you can specify a DatetimeBasedCursor on your Streams’s incremental_sync field to allow your stream to be configured so that only data that has changed since the prior sync is replicated to a destination source. In the case where the API endpoint takes a startDate and endDate, you can use the “Inject start/end time into outgoing HTTP request” option to define how to request records that got changed in the time range to sync source. Unfortunately, as per a forum post, it seems that using multiple fields as a cursor field is not currently supported in Airbyte source. For more detailed guidance, you may want to refer to the official Airbyte documentation or reach out to the Airbyte community for further assistance.