Summary
When creating a source with the UI builder with incremental sync enabled, it fails to recognize a format even though multiple formats are defined for the cursor. The error message indicates that it is only picking up the first format specified and not attempting to use another one.
Question
Hey there, does anybody knows why when creating a source with the UI builder with incremental sync enabled it fails to recognize a format even there’s multiple formats defined for cursor
ValueError: No format in ['%Y-%m-%dT%H:%M:%S.%fZ'] matching 2023-12-13T05:41:18Z
Seems to be only picking up the first format it wanted and not even trying another one
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["error", "source", "UI-builder", "incremental-sync", "format", "cursor"]
<@U06U4DGD49F> What does the YAML look like for that section?
Also, if you delete the first one and re-add it to the end, does it switch which ones match/fail?
I tried switching the order of the formats, then click on release/publish new version with the changes then going back to my connection and no effect, same issue.
This is the yaml section for it
type: DatetimeBasedCursor
cursor_field: startedAtTime
is_data_feed: true
start_datetime:
type: MinMaxDatetime
datetime: '{{ config["start_date"] }}'
datetime_format: '%Y-%m-%dT%H:%M:%SZ'
datetime_format: '%Y-%m-%dT%H:%M:%S.%fZ'
cursor_datetime_formats:
- '%Y-%m-%dT%H:%M:%S.%fZ'```
hm, it looks like it doesn’t have the additional format listed. It should have an extra line at the bottom in the YAML cursor_datetime_formats list like this (note the additional line at the end):
type: DatetimeBasedCursor
cursor_field: startedAtTime
is_data_feed: true
start_datetime:
type: MinMaxDatetime
datetime: '{{ config["start_date"] }}'
datetime_format: '%Y-%m-%dT%H:%M:%SZ'
datetime_format: '%Y-%m-%dT%H:%M:%S.%fZ'
cursor_datetime_formats:
- '%Y-%m-%dT%H:%M:%S.%fZ'
- '%Y-%m-%dT%H:%M:%SZ'```
Maybe try adding that, and seeing if when you switch back to UI it works
If not, maybe flag what version of Airbyte you're on . . . possible there was some type of UI bug at some point. but I'm getting the expected behavior on both my instance and on Cloud.
Just for a bit more context the data seems to have a mix of both formats and when encountering a record that doesnt match the .%fZ part it breaks
You should be able to see it working just from using the test feature in Builder (just to save having to run a sync just to test). Sometimes you get a more informative message with that too.
I’ve had a couple APIs that produced multiple formats and this worked fine, including one that was running on v0.59.0 previously (although I’m not running that version anymore so can’t retest that).
When you toggle back in the Builder, are you seeing both formats listed in the YAML?
Also, it seems silly, but double-check that your existing connector is using the same version of the custom connector . . . sometimes you end up with two that are named similarly, and don’t realize until your updates aren’t appearing like this (or in our case, I did this with an old CDK connector that I rebuilt in Builder).
The test feature seems to pass, but the issue happens when running the sync only (as it starts finding records that don’t match).
Also how do you ensure that is using the right version? Can’t see any number that displays the connector version used
You can increase the number of records the test pulls if you don’t think it’s getting the mixed values.
You can check which connector type is being used by just looking at the source list and seeing if there are two with different names. (You could look at this either under Settings > Sources or when you go to Sources > New Source > only select Custom). You could also look at the source type IDs in the database and see which one matches the ID of the builder connector.
If there are two with the same/similar names, look at your connection and see which it’s using. Or change the name of your Builder connector so you can clearly see whether it’s changing on the Connection.
When it comes to the version, of the Builder connector, you’ll see this on the Builder screen in the Active Version column. If this isn’t what you’re expecting ,you can click that value to select a different version. The default when you click Release New Version is to set it as Active, so it should be updating your existing connectors.
Hi, I’m jumping into the conversation because it might relate to an issue I’m experiencing: https://airbytehq.slack.com/archives/C027KKE4BCZ/p1716967760576539. If you have any advice or suggestions, I would greatly appreciate it. Thanks!
Oh I think I sent the wrong one as I have 2 streams, I just added the formats on both of the streams, and re-released the connector, then ran again the sync for my connection, seems like it didn’t picked it up. Should I recreate the connection to be able to pickup the new source version?
Im using Airbyte v0.59.0