Issue with Airbyte deployment on EC2 instance

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

User encountered warning messages and issues with Airbyte deployment on an EC2 instance, including readiness probe failures and localhost access problem.


Question

Hello, I created a fresh amazon ami on ec2 and followed the instructions https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart
<@U07C8CCC68Y> I did not see these warning messages come up on your youtube video
when running
abctl local install
• I see a bunch of Warning messages, but it states it completes?
• abctl local install --host [HOSTNAME] I run this command but replace the hostname with the public A-name record created with the ec2 public ip. When I view the kubeconfig after running its always localhost.

            Provider: kind
            Kubeconfig: /home/ec2-user/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 25.0.6                                                                                                                                        
 SUCCESS  Port 8000 appears to be available                                                                                                                                                
  INFO    No existing cluster found, cluster 'airbyte-abctl' will be created                                                                                                               
 SUCCESS  Cluster 'airbyte-abctl' created                                                                                                                                                  
  INFO    Namespace 'airbyte-abctl' created                                                                                                                                                
  INFO    Persistent volume 'airbyte-minio-pv' created                                                                                                                                     
  INFO    Persistent volume 'airbyte-volume-db' created                                                                                                                                    
  INFO    Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' created                                                                                                         
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' created                                                                                                                 
  INFO    Starting Helm Chart installation of 'airbyte/airbyte' (version: 0.429.0)                                                                                                         
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-workload-launcher-6c46ffdfcb-7ljnx.17ebb99ff98cda22
            Reason: Unhealthy
            Message: Readiness probe failed: HTTP probe failed with statuscode: 503
            Count: 6
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-workload-launcher-6c46ffdfcb-7ljnx.17ebb99ff98cda22
            Reason: Unhealthy
            Message: Readiness probe failed: HTTP probe failed with statuscode: 503
            Count: 7
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-server-79585964c9-tpnlw.17ebb9a487ccf525
            Reason: Unhealthy
            Message: Readiness probe failed: Get "<http://10.244.0.12:8001/api/v1/health>": dial tcp 10.244.0.12:8001: connect: connection refused
            Count: 6
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-workload-launcher-6c46ffdfcb-7ljnx.17ebb99ff98cda22
            Reason: Unhealthy
            Message: Readiness probe failed: HTTP probe failed with statuscode: 503
            Count: 8
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-workload-launcher-6c46ffdfcb-7ljnx.17ebb99ff98cda22
            Reason: Unhealthy
            Message: Readiness probe failed: HTTP probe failed with statuscode: 503
            Count: 9
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-workload-launcher-6c46ffdfcb-7ljnx.17ebb99ff98cda22
            Reason: Unhealthy
            Message: Readiness probe failed: HTTP probe failed with statuscode: 503
            Count: 10
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-server-79585964c9-tpnlw.17ebb9a487ccf525
            Reason: Unhealthy
            Message: Readiness probe failed: Get "<http://10.244.0.12:8001/api/v1/health>": dial tcp 10.244.0.12:8001: connect: connection refused
            Count: 7
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-workload-launcher-6c46ffdfcb-7ljnx.17ebb99ff98cda22
            Reason: Unhealthy
            Message: Readiness probe failed: HTTP probe failed with statuscode: 503
            Count: 11
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-server-79585964c9-tpnlw.17ebb9a487ccf525
            Reason: Unhealthy
            Message: Readiness probe failed: Get "<http://10.244.0.12:8001/api/v1/health>": dial tcp 10.244.0.12:8001: connect: connection refused
            Count: 8
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-workload-launcher-6c46ffdfcb-7ljnx.17ebb99ff98cda22
            Reason: Unhealthy
            Message: Readiness probe failed: HTTP probe failed with statuscode: 503
            Count: 12
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-server-79585964c9-tpnlw.17ebb9a487ccf525
            Reason: Unhealthy
            Message: Readiness probe failed: Get "<http://10.244.0.12:8001/api/v1/health>": dial tcp 10.244.0.12:8001: connect: connection refused
            Count: 9
 WARNING  Encountered an issue deploying Airbyte:                                                                                                                                          
            Pod: airbyte-abctl-server-79585964c9-tpnlw.17ebb9a487ccf525
            Reason: Unhealthy
            Message: Readiness probe failed: Get "<http://10.244.0.12:8001/api/v1/health>": dial tcp 10.244.0.12:8001: connect: connection refused
            Count: 10
 SUCCESS  Installed Helm Chart airbyte/airbyte:                                                                                                                                            
            Name: airbyte-abctl
            Namespace: airbyte-abctl
            Version: 0.429.0
            AppVersion: 0.63.15
            Release: 1
  INFO    Starting Helm Chart installation of 'nginx/ingress-nginx' (version: 4.11.1)                                                                                                      
 SUCCESS  Installed Helm Chart nginx/ingress-nginx:                                                                                                                                        
            Name: ingress-nginx
            Namespace: ingress-nginx
            Version: 4.11.1
            AppVersion: 1.11.1
            Release: 1
  INFO    No existing Ingress found, creating one                                                                                                                                          
 SUCCESS  Ingress created                                                                                                                                                                  
 WARNING  Failed to launch web-browser.                                                                                                                                                    
          Please launch your web-browser to access <http://localhost:8000>
 SUCCESS  Airbyte installation complete.                                                                                                                                                   
            A password may be required to login. The password can by found by running
            the command abctl local credentials```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1723679135277509) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["amazon-ami", "ec2", "airbyte-deployment", "kubeconfig", "readiness-probe", "localhost-access"]
</sub>
2

Warning is not an Error :wink:
Warnings with Message: Readiness probe failed occur because some pods are starting earlier, some later and there are dependencies between them. If remember correctly, e.g. server waits for temporal to be running, and I think, workload-launcher also might wait for temporal.
Different EC2 instances have different cpu/memory available, but also different network bandwidth
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html
Those things combined affect how fast pods are started. Pulling gigabytes of Docker images even over fast network takes time. There are also limitations for Docker engine / Kind cluster how many Docker images are downloaded simultaneously. Even for simple docker pull ... can be noticed, that layers are downloaded separately in no predictable order.

WARNING Failed to launch web-browser. is expected when you are running it on ec2 machine.

I hope you get a point what might be the reason(s) behind those warnings :wink: