Summary
The user is facing issues while syncing data from an MSSQL database to PostgreSQL, especially with large tables. The process stops in the middle with an error message indicating a source process still being alive. The sync summary shows a failed status with zero records and bytes synced.
Question
hello I am facing an issue while syncing data from an mssql db to postgresql. the data is quite big. I tried moving small tables first and it seems all is going well. but when i try syncing big tables the troubles begin. I made sure that I have enough space on the destination but the process keep stopping in the middle.
I have received this issue report:
this is the report
readFromSource: exception caught
749
java.lang.IllegalStateException: Source process is still alive, cannot retrieve exit value.
750
at com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[guava-31.1-jre.jar:?]
751
at io.airbyte.workers.internal.DefaultAirbyteSource.getExitValue(DefaultAirbyteSource.java:127) ~[io.airbyte-airbyte-commons-worker-0.50.45.jar:?]
752
at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:363) ~[io.airbyte-airbyte-commons-worker-0.50.45.jar:?]
753
at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:235) ~[io.airbyte-airbyte-commons-worker-0.50.45.jar:?]
754
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
755
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
756
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
757
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
758
2024-01-31 12:58:08 platform > readFromSource: done. (source.isFinished:false, fromSource.isClosed:true)
759
2024-01-31 12:58:35 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=3, totalCompleteFailures=3, successivePartialFailures=0, totalPartialFailures=0)
760
Backoff before next attempt: 1 minute 30 seconds
761
2024-01-31 12:59:08 platform > airbyte-source gobbler IOException: Stream closed. Typically happens when cancelling a job.
762
2024-01-31 12:59:08 platform > sync summary: {
763
“status” : “failed”,
764
“recordsSynced” : 0,
765
“bytesSynced” : 0,
766
“startTime” : 1706705706928,
767
“endTime” : 1706705948847,
768
“totalStats” : {
769
“bytesCommitted” : 0,
770
“bytesEmitted” : 1245466,
771
“destinationStateMessagesEmitted” : 0,
772
“destinationWriteEndTime” : 0,
773
“destinationWriteStartTime” : 1706705706929,
774
“meanSecondsBeforeSourceStateMessageEmitted” : 0,
775
“maxSecondsBeforeSourceStateMessageEmitted” : 0,
776
“maxSecondsBetweenStateMessageEmittedandCommitted” : 0,
777
“meanSecondsBetweenStateMessageEmittedandCommitted” : 0,
778
“recordsEmitted” : 1273,
779
“recordsCommitted” : 0,
780
“replicationEndTime” : 1706705948847,
781
“replicationStartTime” : 1706705706928,
782
“sourceReadEndTime” : 0,
783
“sourceReadStartTime” : 1706705706929,
784
“sourceStateMessagesEmitted” : 0
785
},
786
“streamStats” : [ {
787
“streamName” : “Billing_CDR_Main”,
788
“streamNamespace” : “dbo”,
789
“stats” : {
790
“bytesCommitted” : 0,
791
“bytesEmitted” : 1245466,
792
“recordsEmitted” : 1273,
793
“recordsCommitted” : 0
794
}
795
} ],
796
“performanceMetrics” : {
797
“processFromSource” : {
798
“elapsedTimeInNanos” : 11887411,
799
“executionCount” : 1273,
800
“avgExecTimeInNanos” : 9338.107619795757
801
},
802
“readFromSource” : {
803
it seems that the process is going this way :
it stopped the replication
started checking connections.
source failed
destination succ
started checking connection
source succ
destination succ
started replication
error
stopped replication
can someone help please.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.