- Is this your first time deploying Airbyte?: No
- Deployment: Kubernetes
- Airbyte Version: v0.39.36-alpha
- Source name/version: Postgres 11.16
- Destination name/version: Big Query
We are trying to set up a connection using logical replication between Postgres and Big Query. We managed to make it work on a small table. But when running on a bigger dataset, it fails after all the data have been uploaded to GCS for staging (which can take a couple of hours).
Here is the log file of the first failed run: https://storage.googleapis.com/zapper-fi-assets/misc/logs-15692.txt (I had to upload it to our bucket because it busts the 8mb limit)
A few things that stand out:
- JSON schema validator errors
org.postgresql.util.PSQLException: ERROR: snapshot too old
Sync worker failed, Source cannot be stopped!
At the end it states that the run was successful, but it actually fails and restarts from scratch in the second attempt.
I would appreciate any pointers you can provide