Slow data loading and sequential loading in Airbyte from S3 to Postgres

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is facing challenges with slow data loading from S3 to Postgres using Airbyte. They are looking for ways to decrease loading time and load data in parallel streams instead of sequentially.


Question

Hello,
I am using airbyte for moving data from s3 to postgres. There are 2 challenges that I am facing

  1. The process to load data is very slow. I have data about 60m records but the process takes so much time that i cannot bare for my operations. Is there any way to decrease the loading time.
  2. The data is loading in sequential manner, by this i mean that i have to load data in 2 3 tables but all stream work in sequential way, that one load works at a time making other streams to wait. How can i make in parallel manner?

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["slow-data-loading", "sequential-loading", "s3-connector", "postgres-connector", "parallel-loading"]
2

ad. 2
separate connections for different streams (or group of streams)