Hello,
We are installing Airbyte on a private GKE using kustomize. We have put the webapp service behind an ingress and set up IAP in front of the ingress to secure the access. The UI is accessible via the domain we have set up but once we want to interact with Airbyte (for instance to create a new connection), we get the following error: " Cannot reach server. The server may still be starting up." And If we go in the chrome developer console, here are the logs:
Thank you for any hints.
Hi @charles1104 could you share the content of these environment variables on the server:
INTERNAL_API_HOST
WEBAPP_URL
API_URL
I think they should be consistent with your ingress setup. You can try to set the INTERNAL_API_HOST to your ingress domain.
These are the default values:
INTERNAL_API_HOST=airbyte-server:8001
WEBAPP_URL=http://localhost:8000/
# Although not present as an env var, required for webapp configuration.
API_URL=/api/v1/
Thank you @alafanechere
Here are the values that come from the “.env” file:
WEBAPP_URL=airbyte-webapp-svc:80
API_URL=/api/v1/
INTERNAL_API_HOST=airbyte-server-svc:8001
We tried to replace INTERNAL_API_HOST with our domain but we continue to have the same error. We are using DBT images “0.36.0-alpha”.
Our GKE deployment seems to be fine with all the services up and running. We can see in the logs errors related to this issue but according to the thread, it shouldn’t impact the functionality of Airbyte.
Our cluster is fully private and we cannot create a Google NAT in fornt of it and the only way to give it access to internet is through a proxy but it doesn’t seem that Airbyte helm chart supports the use of a proxy. And we first want to be sure that the current issue is due to lack of internet connection.
Thank you for your help.
Could you please try to set WEPAPP_URL to your ingress URL?
Do you mind sharing the server logs too?
Hi @alafanechere ,
Same results. I have put some additional screenshot.
The server seems to time out while trying to access this “list_latest” endpoint on internet.
I have also put you on the following link the code we use to deploy Airbyte on GKE. The overlay we use is the “sbx” one.
Basically, we don’t touch the official airbyte resources but we are deployign additional resources as per our requirements:
Those are located in the folder “resources-specific-ric”.
In the overlay folder, we are also applying some patches to have node affinity to a specific node pool, to define a secret to pull the images from our container registry SaaS ‘Jfrog’ (our cluster doesn’t have internet access so we pulled the Airbyte image on a containre registry that our cluster has access to).
Thank you for your help on this. We are confident we are getting close to get this setup working fully privately and once fully working we will be glad to write a blog post and share it with the community.
Airbyte indeeds need to access to internet to retrieve the following file that list the latest existing sources and destinations:
And we first want to be sure that the current issue is due to lack of internet connection.
I think that now it’s clear, but feel free to share the server logs too to make sure this is why you see Cannot reach server.
The only way to give it access to internet is through a proxy but it doesn’t seem that Airbyte helm chart supports the use of a proxy.
Do you have an idea how to make this work?
Thank you for your help on this. We are confident we are getting close to get this setup working fully privately and once fully working we will be glad to write a blog post and share it with the community.
Amazing!
I have uploaded a file (link below) with the server logs where we can see the “httpConnectTimeoutException”.
We have no idea how to make the proxy work within Airbyte helm charts and we are investigating if we can instead set up a Cloud NAT which will make things easier.
Yes, a NAT with DNS filtering could be easier to set up
Following-up on @charles1104 description, just wanted to give an update.
We have setup a custom solution to make both airbyte-server and airbyte-worker use a redsocks proxy for all http/https connections, as well as containerd daemon config to pull new source focker images dynamically going through our corporate proxy.
this fixed the issue with source retrieval and launching new docker upon new source seems to work properly as well.
Unfortunately, the UI still reports a failure, and it is not clear where is the issue:
Thank you in advance for your help
Thank you @Tobbey for the update. I’m glad you’re making headways.
It looks like the response you get from the server while running a check_connection is invalid.
Could you please monitor the network call in your browser inspector and send a screenshot of the response you receive for this specific call?
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.
