Source Postgres: source cannot be stopped!

Connector Questions & Issues
,
1

Hello airbyte. I am giving your product a first spin using the web interface. Data from a tiny schema transfers without errors, but a schema with more tables/data gets the following error:

“Additional Failure Information: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!”

I am doing a Postgres->Postgres replication. I have tried overwriting the destination data, appending, and incremental without dups. All get the same error. The source is an on-prem db and the destination is in AWS.
It sort of seems the data has transferred. I have not done exhaustive tests. What might this indicate? Fwiw my schema has about 300 tables, all identical in structure. My logs are attached.
logs-447490.txt (5.5 MB)

It seems other people ran into a similar issue: Sync job fails/retries itself after successfully transferring all the data. · Issue #5870 · airbytehq/airbyte · GitHub

I have about 300 tables, 17GB of records, 158 million records

I have a larger dataset that I have held off replicating. I don’t like the idea of the connection trying this more than once and failing even if the data is successfully copied

2

Hey @jeffburnett, thanks for giving Airbyte a try and welcome to the community!

Is any data being synced to your destination? If so, how much data is not being synced? What tables are missing?

Based on the logs, I’m see a “Terminating due to java.lang.OutOfMemoryError: Java heap space” a bit before the source cannot be stopped error. I’m wondering if this has something to do with the partial sync.

Could you share some more information about what your Kubernetes deployment looks like? How many resources did you allocate for the airbyte instance, etc.