Maintain datetime/timestamp datatype in destination (IE Postgres & SQL Server sources)

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • Memory / Disk: you can use something like 4Gb / 1 Tb
  • Deployment: Kube
  • Airbyte Version: 40.15
  • Source name/version: Postgres 1.0.19 and SQL Server 0.4.23
  • Destination name/version: Snowflake 0.4.38
  • Step: Sync/read
  • Description:
    Both the Postgres and SQL Server source convert all date/time related datatypes into strings, currently.
    First question: why was this done this way?
    More importantly: how do we best work around this? Simple solution would be an additional transform (whether in Airbyte/DBT or after the fact); however this is just inefficient (additional compute cost, time, and storage space). I was wondering if there’s maybe an easy way to inject our own datatypes into the base normalization maybe, while still maintaining the rest of the built-in normalization logic (IE deduping), and ensuring any future changes to normalization are still propagated. Or maybe another way entirely that I’m not aware of?

Thanks!

3

Hey Luke,

Thanks for the post and apologies for the delay, there’s been a large influx of user questions and we’ve been trying hard to reply to everyone.

As for your question, this issue might provide some context: https://github.com/airbytehq/airbyte/issues/5129. It was in an effort to keep things relatively quick to iterate.

As far as I’m aware an additional transform is probably the method we can recommend, even if it is inefficient. You may be able to modify the normalization but that might require more in depth expertise. Please feel free to raise an issue about on Github.