Running Airbyte on Linux VM within organizational network

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Running Airbyte on a Linux VM within an organizational network, facing issues accessing localhost:8000, tried different approaches including editing ingress and using Docker Compose. Looking for a secure alternative for continuous functioning.


Question

Hi all!

I’m trying to run AirByte on a linux VM within an organizational network. Since it is a server, there is no way I can access it’s localhost:8000 using a browser.

I have tried the following:

  1. In kubectl edit ingress -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig → Tried changing the host to airbyte.local which points to the public IPv4 address of the server. That didn’t work. Then tried to reference the same to 0.0.0.0 in /etc/hosts. That also didn’t work as I was unable to access the Airbyte GUI on the <PUBLIC_IP>:8000.
  2. Tried letting AirByte run on Localhost and set up port forwarding to my windows system. Failed miserably.
  3. Then installed Airbyte using Docker Compose (Deprecated) but this worked as I was able to access the AirByte GUI using <PUBLIC_IP>:8000 :white_check_mark:
    But the application running through Docker Compose is not secure and I don’t want it to run like that. Any alternative that retains security on continuous functioning of this setup?

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte", "linux-vm", "organizational-network", "localhost", "8000", "kubectl", "ingress", "docker-compose", "security"]
2

It gets started. Says AirByte is running on localhost:8000. But when I try to use Ngrok to forward, it says 404 Error, nginx not found. When I do curl localhost:8000, it returns some webpage data. But when I do curl 127.0.0.1:8000, it throws the same 404 error.

Not sure what the issue is! When I run using docker compose, it works fine on the public IPV4 address of the EC2 instance.

Oh! One more thing, in abctl.kubeconfig when I edit the host to the DNS of EC2 instance, it is working again!

But the issue here is that the production sever to which I am planning to deploy AirByte doesn’t have any public DNS. It only has an IP Address.

3

you can check this thread https://airbytehq.slack.com/archives/C021JANJ6TY/p1722848599322599?thread_ts=1722845492.200389&cid=C021JANJ6TY

4

Thank you for the info!

5

But it says creating-cluster and keeps rotating. It’s been more 7 minutes. Is it normal?

6

ERROR Cluster ‘airbyte-abctl’ could not be created
ERROR unable to create kind cluster: failed to init node with kubeadm: command “docker exec --privileged airbyte-abctl-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6” failed with error: exit status 1

7

pulling Docker images takes time, on my local machines it took sometimes more than 15 minutes

8

how much memory and cpu has linux VM that you’re using?