Scaling Airbyte components for high availability

slack-user-airbyte · August 6, 2024, 6:13am

Summary

After a Kubernetes maintenance, connections were stuck and needed to be canceled/rerun. Looking to increase availability by scaling Airbyte components. Questioning the safety of scaling airbyte-workload-api-server, airbyte-workload-launcher, and airbyte-temporal after upgrading to version 0.64.3 ‘workload’ model.

Question

Hi guys,

I’ve had some problems recently where all of my connections were “stuck” after a kubernetes maintenance, so need to increase availability.
Everything was fine after canceling/rerunning them, but somehow the maintenance caused the connections to be in a state that couldn’t be “healed”
I’m reading all docs I can find (https://docs.airbyte.com/enterprise-setup/scaling-airbyte#high-availability)

TLDR: docs say to scale “airbyte-worker” and “airbyte-server”
but how about theses ones? are they safe to scale? Asking because I just upgraded to 0.64.3 “workload” model
airbyte-workload-api-server
airbyte-workload-launcher
airbyte-temporal

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

_{["scaling", "high-availability", "kubernetes", "airbyte-worker", "airbyte-server", "airbyte-workload-api-server", "airbyte-workload-launcher", "airbyte-temporal"]}

slack-user-airbyte · August 6, 2024, 6:18am

Can you provide more information about: couldn't be healed.
cc <@U01HYQYV1BQ> fyi

slack-user-airbyte · August 7, 2024, 6:18am

would be glad to!
I think maybe the issue is very specific to us

for context:

running on kubernetes cluster (GKE)
platform version: 0.64.4 OSS
have like 4 connections that need to be kept in sync (source salesforce, destination bigquery), so CRON every minute (know it’s not the ideal scenario)
my deployment has a flaw that I’m taking care of: I have 1 deployment of “cloud-sql-proxy” that acts as a “bridge” to my db instead of running as sidecar. All airbyte containers point to this pod -> fixing this right now
What happened:
Google made a routine maintenance on my nodes
Pods were redeployed
Cloud-sql-proxy pod was unavailable for some time

After everything came back on, all of my airbyte connections that were running got “stuck” (getting logs right now) for 20 hours+
They got back to normal when I cancelled and synced again

slack-user-airbyte · August 7, 2024, 6:18am

This went out exactly when my db connection went out
It seems it had “just finished” reading and was ready to commit

2024-08-04 05:29:11 destination > INFO sync-operations-1 i.a.i.d.b.o.BigQueryStorageOperation(overwriteFinalTable):140 Overwriting table `salesforce_v3`.`Describe` with `salesforce_v3`.`Describe_airbyte_tmp` 1498 2024-08-04 05:29:59 ERROR i.a.c.Application(run):82 - Killing orchestrator because of an Exception 1499 java.net.SocketTimeoutException: Connect timed out 1500 at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?] 1501 at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[?:?] 1502 at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?] 1503 at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?] 1504 at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[okhttp-4.12.0.jar:?] 1505 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[okhttp-4.12.0.jar:?] 1506 at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[okhttp-4.12.0.jar:?] 1507 at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.12.0.jar:?] 1508 at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.12.0.jar:?] 1509 at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.12.0.jar:?] 1510 at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.12.0.jar:?] 1511 at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.12.0.jar:?] 1512 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?] 1513 at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.12.0.jar:?] 1514 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?] 1515 at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.12.0.jar:?] 1516 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?] 1517 at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.12.0.jar:?] 1518 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?] 1519 at io.airbyte.api.client.UserAgentInterceptor.intercept(UserAgentInterceptor.kt:34) ~[io.airbyte.airbyte-api-commons-0.63.13.jar:?] 1520 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?] 1521 at io.airbyte.api.client.auth.AirbyteAuthHeaderInterceptor.intercept(AirbyteAuthHeaderInterceptor.kt:28) ~[io.airbyte.airbyte-api-commons-0.63.13.jar:?] 1522 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?] 1523 at io.airbyte.workload.api.client.auth.WorkloadApiAuthenticationInterceptor.intercept(WorkloadApiAuthenticationInterceptor.kt:36) ~[io.airbyte.airbyte-api-workload-api-0.63.13.jar:?] 1524 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?] 1525 at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.12.0.jar:?] 1526 at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.12.0.jar:?] 1527 at io.airbyte.workload.api.client.generated.WorkloadApi.workloadSuccessWithHttpInfo(WorkloadApi.kt:4869) ~[io.airbyte.airbyte-api-workload-api-0.63.13.jar:?] 1528 at io.airbyte.workload.api.client.generated.WorkloadApi.workloadSuccess(WorkloadApi.kt:927) ~[io.airbyte.airbyte-api-workload-api-0.63.13.jar:?] 1529 at io.airbyte.container_orchestrator.orchestrator.ReplicationJobOrchestrator.succeedWorkload(ReplicationJobOrchestrator.java:174) ~[io.airbyte-airbyte-container-orchestrator-0.63.13.jar:?] 1530 at io.airbyte.container_orchestrator.orchestrator.ReplicationJobOrchestrator.updateStatusInWorkloadApi(ReplicationJobOrchestrator.java:154) ~[io.airbyte-airbyte-container-orchestrator-0.63.13.jar:?] 1531 at io.airbyte.container_orchestrator.orchestrator.ReplicationJobOrchestrator.runJob(ReplicationJobOrchestrator.java:113) ~[io.airbyte-airbyte-container-orchestrator-0.63.13.jar:?] 1532 at io.airbyte.container_orchestrator.Application.run(Application.java:79) [io.airbyte-airbyte-container-orchestrator-0.63.13.jar:?] 1533 at io.airbyte.container_orchestrator.Application.main(Application.java:39) [io.airbyte-airbyte-container-orchestrator-0.63.13.jar:?]

slack-user-airbyte · August 7, 2024, 6:18am

Then:

The sync summary was printed (happens when sync runs alright):

2024-08-04 05:29:59 INFO i.a.a.SegmentAnalyticsClient(close):233 - Closing Segment analytics client...
1535
2024-08-04 05:29:59 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):289 - Waiting for Segment analytic client to flush enqueued messages...
1536
2024-08-04 05:29:59 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):301 - Segment analytic client flush complete.
1537
2024-08-04 05:29:59 INFO i.a.a.SegmentAnalyticsClient(close):237 - Segment analytics client closed.  No new events will be accepted.
1538
2024-08-04 05:29:17 destination &gt; INFO sync-operations-9 i.a.i.b.d.o.AbstractStreamOperation(finalizeTable):327 Not overwriting final table for direcional_salesforce_v3.PagamentosTransacoes__c. Truncate sync: false; stream success: true; final table suffix not blank: false
1539
2024-08-04 05:29:17 destination &gt; INFO main i.a.i.b.d.o.DefaultSyncOperation(finalizeStreams):150 Cleaning up sync operation thread pools
1540
2024-08-04 05:29:17 destination &gt; INFO main i.a.c.i.d.a.AsyncStreamConsumer(close):195 class io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer closed
1541
2024-08-04 05:29:17 destination &gt; INFO main i.a.c.i.b.IntegrationRunner(runInternal):269 Completed integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination
1542
2024-08-04 05:29:17 destination &gt; INFO main i.a.i.d.b.BigQueryDestinationKt(main):565 Completed Destination : class io.airbyte.integrations.destination.bigquery.BigQueryDestination
1543
2024-08-04 05:29:18 replication-orchestrator &gt; (pod: integration / destination-bigquery-write-366802-0-cfltk) - Closed all resources for pod
1544
2024-08-04 05:29:18 replication-orchestrator &gt; readFromDestination: done. (writeToDestFailed:false, dest.isFinished:true)
1545
2024-08-04 05:29:18 replication-orchestrator &gt; thread status... timeout thread: false , replication thread: true
1546
2024-08-04 05:29:28 replication-orchestrator &gt; Closing StateCheckSumCountEventHandler
1547
2024-08-04 05:29:28 replication-orchestrator &gt; No checksum errors were reported in the entire sync.
1548
2024-08-04 05:29:28 replication-orchestrator &gt; sync summary: {
1549
  "status" : "completed",```

slack-user-airbyte · August 8, 2024, 6:18am

Right after the summary, “replication-orchestrator” errored on the heartbeat

1756
2024-08-04 05:29:28 replication-orchestrator &gt; 
1757
2024-08-04 05:29:30 replication-orchestrator &gt; Error while trying to heartbeat, re-trying
1758
java.net.SocketTimeoutException: Connect timed out
1759
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
1760
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[?:?]
1761
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
1762
        at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
1763
        at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[okhttp-4.12.0.jar:?]
1764
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[okhttp-4.12.0.jar:?]
1765
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[okhttp-4.12.0.jar:?]
1766
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.12.0.jar:?]
1767
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.12.0.jar:?]
1768
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.12.0.jar:?]
1769
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.12.0.jar:?]
1770
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.12.0.jar:?]
1771
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
1772
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.12.0.jar:?]
1773
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
1774
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.12.0.jar:?]
1775
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
1776
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.12.0.jar:?]
1777
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
1778
        at io.airbyte.api.client.UserAgentInterceptor.intercept(UserAgentInterceptor.kt:34) ~[io.airbyte.airbyte-api-commons-0.63.13.jar:?]
1779
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
1780
        at io.airbyte.api.client.auth.AirbyteAuthHeaderInterceptor.intercept(AirbyteAuthHeaderInterceptor.kt:28) ~[io.airbyte.airbyte-api-commons-0.63.13.jar:?]
1781
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
1782
        at io.airbyte.workload.api.client.auth.WorkloadApiAuthenticationInterceptor.intercept(WorkloadApiAuthenticationInterceptor.kt:36) ~[io.airbyte.airbyte-api-workload-api-0.63.13.jar:?]
1783
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
1784
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.12.0.jar:?]
1785
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.12.0.jar:?]
1786
        at io.airbyte.workload.api.client.generated.WorkloadApi.workloadHeartbeatWithHttpInfo(WorkloadApi.kt:2748) ~[io.airbyte.airbyte-api-workload-api-0.63.13.jar:?]
1787
        at io.airbyte.workload.api.client.generated.WorkloadApi.workloadHeartbeat(WorkloadApi.kt:429) ~[io.airbyte.airbyte-api-workload-api-0.63.13.jar:?]
1788
        at io.airbyte.workers.general.ReplicationWorkerHelper.getWorkloadStatusHeartbeat$lambda$1(ReplicationWorkerHelper.kt:153) ~[io.airbyte-airbyte-commons-worker-0.63.13.jar:?]
1789
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
1790
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
1791
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
1792
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
1793
2024-08-04 05:30:20 platform &gt; Cloud storage job log path: /workspace/366802/0/logs.log
1794
2024-08-04 05:30:20 platform &gt; Executing worker wrapper. Airbyte version: 0.63.13
1795
2024-08-04 05:30:20 platform &gt; Creating workload badd9a5e-26ce-4fd3-b3b3-5a7476583d86_366802_0_sync
1796
2024-08-04 05:30:20 platform &gt; Workload badd9a5e-26ce-4fd3-b3b3-5a7476583d86_366802_0_sync is launched```

slack-user-airbyte · August 8, 2024, 6:18am

Then, it kepts logging this forever until canceled:

2024-08-04 05:30:20 platform &gt; Executing worker wrapper. Airbyte version: 0.63.13
1795
2024-08-04 05:30:20 platform &gt; Creating workload badd9a5e-26ce-4fd3-b3b3-5a7476583d86_366802_0_sync
1796
2024-08-04 05:30:20 platform &gt; Workload badd9a5e-26ce-4fd3-b3b3-5a7476583d86_366802_0_sync is launched
1797
2024-08-04 05:32:20 platform &gt; Workload badd9a5e-26ce-4fd3-b3b3-5a7476583d86_366802_0_sync is launched```

slack-user-airbyte · August 8, 2024, 6:18am

It seems to me that on the final moments of the sync, “something” I don’t know is saved to the airbyte db (maybe stats, connection status)
If somthing fails at this specific moment, the connection “cant heal”

Just logs this forever -> 2024-08-04 05:32:20 platform > Workload badd9a5e-26ce-4fd3-b3b3-5a7476583d86_366802_0_sync is launched
1798

slack-user-airbyte · August 8, 2024, 6:18am

additional detail:
the time between this log that goes on forever is exactly 2 minutes between each one

slack-user-airbyte · August 8, 2024, 6:18am

In the meantime, I’ll try to remedy this by using the sidecar pattern for my db connection
Expect it to remove the 1 point of failure by using the deployment
Maybe if the entire pod goes down it’ll be easier to recover

hope the logs help
and would be happy to share more details if needed

slack-user-airbyte · August 14, 2024, 6:19am

Interesting. How long was the DB unavailable? That would be my guess. Airbyte has strong db dependency today so if the db is down for sometime, weird things can happen

slack-user-airbyte · August 15, 2024, 6:18am

hey!
maybe something like 40s, 1 minute tops (time to restart the pod)
happened again this weekend
will try to mitigate this by deploying the db connection as sidecars instead of a single pod, so that the entire pod is unhealthy during maintenance

slack-user-airbyte · August 19, 2024, 6:18am

didn’t help
I think it’s during GKE maintenance time when it swaps the nodes in the pool
when a sync is running, it just hangs forever when the orchestrator fails to heartbeat the workload-api-server

slack-user-airbyte · August 19, 2024, 6:18am

logs before the sync summary:

java.net.SocketTimeoutException: Connect timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[?:?]
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
        at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
        at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at io.airbyte.api.client.UserAgentInterceptor.intercept(UserAgentInterceptor.kt:34) ~[io.airbyte.airbyte-api-commons-0.63.15.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at io.airbyte.api.client.auth.AirbyteAuthHeaderInterceptor.intercept(AirbyteAuthHeaderInterceptor.kt:28) ~[io.airbyte.airbyte-api-commons-0.63.15.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at io.airbyte.workload.api.client.auth.WorkloadApiAuthenticationInterceptor.intercept(WorkloadApiAuthenticationInterceptor.kt:36) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.12.0.jar:?]
        at io.airbyte.workload.api.client.generated.WorkloadApi.workloadHeartbeatWithHttpInfo(WorkloadApi.kt:2748) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
        at io.airbyte.workload.api.client.generated.WorkloadApi.workloadHeartbeat(WorkloadApi.kt:429) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
        at io.airbyte.workers.general.ReplicationWorkerHelper.getWorkloadStatusHeartbeat$lambda$1(ReplicationWorkerHelper.kt:153) ~[io.airbyte-airbyte-commons-worker-0.63.15.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
        Suppressed: java.io.IOException: unexpected end of stream on <http://airbyte-workload-api-server-svc:8007/>...
                at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:210) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
                at io.airbyte.api.client.UserAgentInterceptor.intercept(UserAgentInterceptor.kt:34) ~[io.airbyte.airbyte-api-commons-0.63.15.jar:?]
                at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
                at io.airbyte.api.client.auth.AirbyteAuthHeaderInterceptor.intercept(AirbyteAuthHeaderInterceptor.kt:28) ~[io.airbyte.airbyte-api-commons-0.63.15.jar:?]
                at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
                at io.airbyte.workload.api.client.auth.WorkloadApiAuthenticationInterceptor.intercept(WorkloadApiAuthenticationInterceptor.kt:36) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
                at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.12.0.jar:?]
                at io.airbyte.workload.api.client.generated.WorkloadApi.workloadHeartbeatWithHttpInfo(WorkloadApi.kt:2748) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
                at io.airbyte.workload.api.client.generated.WorkloadApi.workloadHeartbeat(WorkloadApi.kt:429) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
                at io.airbyte.workers.general.ReplicationWorkerHelper.getWorkloadStatusHeartbeat$lambda$1(ReplicationWorkerHelper.kt:153) ~[io.airbyte-airbyte-commons-worker-0.63.15.jar:?]
                at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
                at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
        Caused by: java.io.EOFException: \n not found: limit=0 content=…
                at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:335) ~[okio-jvm-3.6.0.jar:?]
                at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29) ~[okhttp-4.12.0.jar:?]
                at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180) ~[okhttp-4.12.0.jar:?]```

slack-user-airbyte · August 19, 2024, 6:18am

slack-user-airbyte · August 19, 2024, 6:18am

then just hangs until canceled

can open an issue on gh

2024-08-17 05:49:09 replication-orchestrator &gt; 
2024-08-17 05:49:09 replication-orchestrator &gt; ----- END REPLICATION -----
2024-08-17 05:49:09 replication-orchestrator &gt; 
2024-08-17 05:49:40 ERROR i.a.c.Application(run):82 - Killing orchestrator because of an Exception
java.net.SocketTimeoutException: Connect timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[?:?]
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
        at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
        at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at io.airbyte.api.client.UserAgentInterceptor.intercept(UserAgentInterceptor.kt:34) ~[io.airbyte.airbyte-api-commons-0.63.15.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at io.airbyte.api.client.auth.AirbyteAuthHeaderInterceptor.intercept(AirbyteAuthHeaderInterceptor.kt:28) ~[io.airbyte.airbyte-api-commons-0.63.15.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at io.airbyte.workload.api.client.auth.WorkloadApiAuthenticationInterceptor.intercept(WorkloadApiAuthenticationInterceptor.kt:36) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.12.0.jar:?]
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.12.0.jar:?]
        at io.airbyte.workload.api.client.generated.WorkloadApi.workloadSuccessWithHttpInfo(WorkloadApi.kt:4869) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
        at io.airbyte.workload.api.client.generated.WorkloadApi.workloadSuccess(WorkloadApi.kt:927) ~[io.airbyte.airbyte-api-workload-api-0.63.15.jar:?]
        at io.airbyte.container_orchestrator.orchestrator.ReplicationJobOrchestrator.succeedWorkload(ReplicationJobOrchestrator.java:174) ~[io.airbyte-airbyte-container-orchestrator-0.63.15.jar:?]
        at io.airbyte.container_orchestrator.orchestrator.ReplicationJobOrchestrator.updateStatusInWorkloadApi(ReplicationJobOrchestrator.java:154) ~[io.airbyte-airbyte-container-orchestrator-0.63.15.jar:?]
        at io.airbyte.container_orchestrator.orchestrator.ReplicationJobOrchestrator.runJob(ReplicationJobOrchestrator.java:113) ~[io.airbyte-airbyte-container-orchestrator-0.63.15.jar:?]
        at io.airbyte.container_orchestrator.Application.run(Application.java:79) [io.airbyte-airbyte-container-orchestrator-0.63.15.jar:?]
        at io.airbyte.container_orchestrator.Application.main(Application.java:39) [io.airbyte-airbyte-container-orchestrator-0.63.15.jar:?]
2024-08-17 05:49:40 INFO i.a.a.SegmentAnalyticsClient(close):233 - Closing Segment analytics client...
2024-08-17 05:49:40 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):289 - Waiting for Segment analytic client to flush enqueued messages...
2024-08-17 05:49:40 INFO i.a.a.BlockingShutdownAnalyticsPlugin(waitForFlush):301 - Segment analytic client flush complete.
2024-08-17 05:49:40 INFO i.a.a.SegmentAnalyticsClient(close):237 - Segment analytics client closed.  No new events will be accepted.
2024-08-17 05:50:19 platform &gt; Cloud storage job log path: /workspace/401366/0/logs.log
2024-08-17 05:50:19 platform &gt; Executing worker wrapper. Airbyte version: 0.63.15
2024-08-17 05:50:19 platform &gt; Creating workload badd9a5e-26ce-4fd3-b3b3-5a7476583d86_401366_0_sync```

Topic		Replies	Views
Adjusting CPU resources for specific Airbyte components in Kubernetes deployment Platform Questions kubernetes , platform , question , scaling , cpu-resources	4	62	June 25, 2024
Rebooting pods with high CPU utilization in Airbyte deployment on AKS cluster Platform Questions platform , question , airbyte-deployment , rebooting-pods , high-cpu-utilization	0	3	July 17, 2024
Scaling Airbyte Docker Deployment Without Kubernetes Platform Questions platform , performance , question , scaling , airbyte-docker-deployment	0	8	August 25, 2024
Scaling airbyte-worker pods for resource optimization Platform Questions platform , question , scaling , airbyte-worker , resource-optimization	0	35	June 13, 2024
Handling Severed Nodes/Pods in Airbyte on Kubernetes Cluster Platform Questions platform , airbyte , helm-chart , kubernetes-cluster , pods	2	0	September 13, 2024

Scaling Airbyte components for high availability

Summary

Question

Related Topics