Summary
Schema change detection issue in Postgres and BigQuery connections, inconsistency in detection, seeking detailed information on DiscoverSchema functionality
Question
Hi everyone !
Got an issue on the schema change detection. Got a connection on a Postgres and BigQuery.
Sometimes the schema change is detected (appear as a flag in the “replication” tab), sometimes it’s not.
May be expected depending on how DiscoverSchema works, but I did not find any detailed information on that (https://airbyte.com/blog/schema-propagation-feature)
Has anyone faced something similar ?
Many thanks in advance !
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["schema-change-detection", "postgres-connector", "bigquery-connector", "discoverschema", "replication", "schema-propagation"]
How quickly are you checking Airbyte after updating Postgres? Airbyte automatically checks for schema changes before a since, but at most once every 24 hours.
You can click the “Refresh Source Schema” button on the replication tab to check it manually sooner than that, though.
https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes|https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes
Hello !
Thanks for your reply !
I was checking Airbyte right after updating a postgres source (changing the schema). Got it for the “Refresh Source Schema” manually.
Actually I’m using the API to handle the change (got 100s of connections to update).
A bit on the side, but our problem was:
• we wanted to update the source of 100s of connections.
• we do that through the API
• the update was only caught from time to time. we realize that it was caught only when a “refresh sourced” was caught during a previous sync.
• to force the update, we are refreshing the cache for the connection before making the update (with ignore_cache)