Summary
The user is setting up a new connector to make a daily post request to an external API with startDate and endDate keys. They are looking for a way to dynamically fill these keys based on the current day.
Question
Hello everyone,
I’m currently setting up a new connector that i would like to run once a day.
Its purpose is to make a post request to an external api, with in the body a “startDate” and a “endDate” key.
Is there a way to dynamically fill these 2 in? For example if the current day is 2024-01-25 i’d like it to fetch the data from 2024-01-24.
Thanks in advance!
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["dynamic-filling", "startdate", "enddate", "daily-connector-run", "post-request", "external-api"]
Yes, you can dynamically fill in the “startDate” and “endDate” keys in the body of your POST request using Airbyte’s incremental sync feature and custom parameter injection. Here’s how you can do it: 1. In the incremental sync form, disable the “Inject start time / end time into outgoing HTTP request” option. This is because you want to inject these values into the body of your POST request, not as query parameters or headers. 2. Use the generic parameter section at the bottom of the stream configuration form to configure properties of the JSON body. You can use jinja expressions and available variables to dynamically set the “startDate” and “endDate” keys. For example, you can set the “Field Name” to “startDate” and “Value” to “{{stream_interval.start_time}}”, and similarly for “endDate”. The stream_interval.start_time and stream_interval.end_time variables represent the start and end of the time window for the current sync, respectively. Here’s an example of how you can set it up: Field Name: startDate Value: {{stream_interval.start_time}} Field Name: endDate Value: {{stream_interval.end_time}} This will dynamically set the “startDate” and “endDate” keys in the body of your POST request to the start and end of the time window for the current sync. For more information, you can refer to the Airbyte documentation on custom parameter injection.
<@U04SE163WC8> Am i able to subtract a day from this though? As i want it to dynamically enter the current day - 1 day every time its ran.
Based on the information provided in the extracts, it seems that Airbyte does not directly support subtracting a day from the current date in its configuration. However, there are a few workarounds you might consider: 1. In the context of the Google Analytics v4 source, a timedelta subtraction of two days was added to ensure data is golden. This was done in the stream_slices method. You might be able to implement a similar solution in your own use case. Here is the link to the issue for reference. 2. Another user asked a similar question about configuring Airbyte data source’s “Replication Start Date” dynamically. The response suggested using the “Update Source” endpoint for the Airbyte API and programmatically passing in new replication start dates. You could potentially use this approach to pass in the current date minus one day. Here is the link to the forum post for reference. Please note that these are workarounds and might not work perfectly for your use case. You might need to adjust them according to your specific requirements.
So no way to do this with the normal builder?