Istio + Airbyte = Failures

Platform, Deploy & Infra Issues
,
1

Re-opening this for visibility:

  • Is this your first time deploying Airbyte: No
  • OS Version / Instance: Kubernetes on EKS
  • Memory / Disk: 64GB+ / gp3 volume of the manifests’ default sizes.
  • Deployment: Kubernetes
  • Airbyte Version: 0.35.62-alpha
  • Source name/version: N/A
  • Destination name/version: N/A
  • Step: Trying to add a source or destination, or update a connector.
  • Description: I’ve deployed to an EKS cluster that’s using the istio service mesh, and am running into problems when adding an initial source connector. The source connector pod that’s created errs, and the pod logs show:

Using existing AIRBYTE_ENTRYPOINT: /airbyte/base.sh
Waiting on CHILD_PID 7
PARENT_PID: 1
Heartbeat to worker failed, exiting…
received ABRT

–
I tried today without the Istio sidecars, and things worked without issue.

The manifests I used to deploy istio and then airbyte are linked in a comment in the original topic: Airbyte issues with Istio sidecar

I received a DM on Slack from another community member stating she was having the same issue, but I haven’t troubleshot further since posting.

2

Hi @bradfair, I am not very familiar with Istio sidecars, so I’ll go ahead and ask for input from my team. While I do that, would you be able to provide the full logs you’re getting? Is it possible to tag the other community member, so we can see what errors she is getting as well? Thank you!

3

Hello,
I’m the other member.

I’m running Airbyte v0.39.37-alpha on GKE.
If I deactivate istio, it works.

Here are some logs I had on my side for a BigQuery destination check test when istio is enabled.
Note that I had the same kind of error trying other checks for a mysql source.

Job logs:
connection_check_job.txt (10.0 KB)

Job pod logs by container:
main:
Started at 16:34:37, Ended at 16:34:43 (Error 1)

Using existing AIRBYTE_ENTRYPOINT: /airbyte/base.sh
Waiting on CHILD_PID 8
PARENT_PID: 1
Heartbeat to worker failed, exiting...
received ABRT

relay-stdout:
Started at 16:34:38, Ended at 16:34:40 (Error 1)

2022/07/22 14:34:39 socat[9] N reading from and writing to stdio
2022/07/22 14:34:39 socat[9] N opening connection to AF=2 10.208.3.92:9024
2022/07/22 14:34:39 socat[9] W recvmsg(): Resource temporarily unavailable
2022/07/22 14:34:39 socat[9] E connect(5, AF=2 10.208.3.92:9024, 16): Connection refused
2022/07/22 14:34:39 socat[9] N exit(1)

relay-stderr:
Started at 16:34:39, Ended at 16:34:39 (Error 1)

2022/07/22 14:34:39 socat[8] N reading from and writing to stdio
2022/07/22 14:34:39 socat[8] N opening connection to AF=2 10.208.3.92:9025
2022/07/22 14:34:39 socat[8] W recvmsg(): Resource temporarily unavailable
2022/07/22 14:34:39 socat[8] E connect(5, AF=2 10.208.3.92:9025, 16): Connection refused
2022/07/22 14:34:39 socat[8] N exit(1)

call-heartbeat-server: no logs
Started at 16:34:42, Ended at 16:34:42 (Error 7)

istio-proxy:
Started at 16:34:42, Running
istio_logs.txt (30.9 KB)

4

Hi @mcGIh, thank you for all the info! This looks like a long-running issue that hasn’t been resolved, so I’ve gone ahead and made a GitHub issue for it. I’ll triage it to correct team, and if you don’t mind let’s continue the discussion there:
https://github.com/airbytehq/airbyte/issues/15043