Slow data extraction performance from Connector Builder to Postgres database in Kubernetes cluster

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is experiencing slow data extraction performance when using the Connector Builder to extract 70,000 records (31MB) to a Postgres database in a Kubernetes cluster, compared to faster performance with Postman. They are seeking advice on potential causes for this issue.


Question

Hi all!

What could be causes that it takes up to 15 minutes to extract 70.000 records (31MB in total) from a source built with the Connector Builder and store it into a Postgres database, whilst it takes 3 calls taking 10 seconds each when I use Postman? (Already asked AI, but buffer size etc. seems only to be relevant for databases.) The environment it runs on, is a Kubernetes cluster with a minimum of 4 nodes with 2 cores and 8GB of memory each.

Thanks!


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["slow-data-extraction", "connector-builder", "postgres-database", "kubernetes-cluster", "performance-issue"]
2

interesting to hear about status, if destination is changed to something else like s3