Performance Issues with Postgres Connector 2.4.0

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User reports decreased performance in the destination-postgres connector version 2.4.0 compared to 0.4.0, with longer synchronization times and high disk usage during data sync between BigQuery and Postgres on CrunchyBridge. Seeks insights on performance differences and disk space optimization.


Question

Hi everyone,
I’ve noticed a significant decrease in performance when using version 2.4.0 of the destination-postgres connector compared to version 0.4.0. The synchronization process seems to take much longer, and I’m trying to understand the reasons behind this change.
I’m syncing data between BigQuery and Postgres, my postgres instance is hosted on CrunchyBridge(gcp)

Additionally, I’m experiencing high disk usage during synchronization. Is there a recommended way to optimize disk space consumption in this version?
Any insights into the performance differences or tips for reducing disk usage would be greatly appreciated.
Thanks in advance!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['performance-issues', 'destination-postgres-connector', 'bigquery', 'disk-usage', 'sync-optimization']
2

<@U03JMLXLW3D>, could you please share both sync logs? With these, we can investigate which steps are taking more time and verify that both syncs are using the same method, such as overwrite.