Trouble passing datetime in Sakari connector for Airbyte

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Difficulty passing datetime in Sakari connector for Airbyte, resulting in error message when ingested in BigQuery


Question

Hii, building a sakari connector for my company and im stuck with how to correctly pass the datetime. I am converting and returning the pandas dataframe as json, the datetime column is returned as epoch but when ingested in a connection with BQ I just get error message {“errors”:[“Problem with Date”]} on resulting table


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["sakari-connector", "datetime", "pandas-dataframe", "json", "epoch", "bigquery"]
2

Hi, it seems like you’re having trouble with date formatting when building a custom connector. From the information provided in the extracts, it appears that the issue might be related to how the datetime is formatted before it’s passed to BigQuery. In Airbyte, you can use the format_datetime function to convert a datetime or a datetime-string to the specified format. Here’s an example from the Airbyte documentation: python "{{ format_datetime(config['start_time'], '%Y-%m-%d') }}" "{{ format_datetime(config['start_date'], '%Y-%m-%dT%H:%M:%S.%fZ') }}" In these examples, config['start_time'] and config['start_date'] are the datetime objects or strings to convert, and '%Y-%m-%d' and '%Y-%m-%dT%H:%M:%S.%fZ' are the datetime formats. If you’re dealing with UNIX Epoch time in milliseconds, there’s a feature request for adding an integer format for this. In the meantime, you might need to convert the UNIX Epoch time to a format that BigQuery can understand. If you’re still having trouble, it might be helpful to check the error logs for more details about the issue. You can also refer to the Airbyte documentation or ask for help in the Airbyte community.

3

Im using format_datetime(today, ‘Y-MM-ddTH: m :ss.Z’) but still no luck :'c

4

Correct format was “YYYY-MM-dd HH:MM:ss UTC”

5

Did that work out for you in the end?