NGINX reverse proxy behind port forward to local Airbyte webserver

jordan · August 31, 2022, 1:55am

Unable to connect to web server through NGINX reverse proxy. Works at localhost:nginx_port ‘sometimes’, ie I can reload the page multiple times and it eventually loads, local_network_ip:nginx_port spins, public_ip:nginx_port spins, and localhost:8000 works fine.

Is this your first time deploying Airbyte?: Yes
OS Version / Instance: Windows
Memory / Disk: 128GB / 4TB
Deployment: Docker
Airbyte Version: 0.39.35-alpha
Step: The issue is happening when connecting to the web UI through NGINX reverse proxy
Description: Unauthorized access to manifest.json and timeout when loading static js files

(timeout: x.x.x.x:200/static/js/107.9e468796.chunk.js) ```

``` Uncaught ChunkLoadError: Loading chunk 107 failed.
(timeout: x.x.x.x:200/static/js/107.9e468796.chunk.js) ```

``` Source map error: Error: request failed with status 401
Resource URL: 192.168.0.10:200/static/js/107.9e468796.chunk.js
Source Map URL: 107.9e468796.chunk.js.map ```

``` Failed to load resource: the server responded with a status of 401 (Unauthorized)
manifest.json:1 
        
  Manifest: Line: 1, column: 1, Syntax error.
manifest.json:1 
Failed to load resource: the server responded with a status of 401 (Unauthorized) ```


```server {
    listen        200;
    listen        [::]:200;
    server_name localhost;
    location / {
        proxy_pass         http://localhost:8000/;
        proxy_pass_header Accept;
        proxy_pass_header Server;
        proxy_pass_header Authorization;
        proxy_http_version 1.1;        
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "upgrade";
        proxy_set_header   Host $host;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for; 
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-User $http_authorization;
        proxy_set_header   Authorization   $http_authorization;
        proxy_set_header   ns_server-ui yes;
        proxy_set_header   X-Forwarded-Proto   $scheme;
        proxy_set_header    X-Forwarded-Server      $host;
        proxy_set_header   X-Forwarded-Host   $host:$server_port;
        proxy_hide_header   Referer;
        proxy_hide_header   Origin;
        proxy_set_header    Referer           '';
        proxy_set_header    Origin            '';
        proxy_cache_bypass $http_upgrade;
        auth_basic "Administrator’s Area";
        auth_basic_user_file /etc/apache2/.htpasswd;
        proxy_connect_timeout   1200;
        proxy_send_timeout   1200;
        proxy_read_timeout   1200;
        fastcgi_send_timeout   1200;
        fastcgi_read_timeout   1200;
        send_timeout   1800;   
    }
}

2022/08/30 19:59:01 [error] 1970#1970: *3936 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.0.10, server: localhost, request: "POST /api/v1/destination_definitions/list_latest HTTP/1.1", upstream: "http://127.0.0.1:8000/api/v1/destination_definitions/list_latest", host: "192.168.0.10:200", referrer: "http://192.168.0.10:200/workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/connections"

2022/08/30 18:36:45 [error] 1970#1970: *3440 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.0.10, server: localhost, request: "POST /api/v1/destination_definitions/list HTTP/1.1", upstream: "http://127.0.0.1:8000/api/v1/destination_definitions/list", host: "192.168.0.10:200", referrer: "http://192.168.0.10:200/workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/connections"

If I look at a successful connection on localhost:8000 vs localhost:200 (the port I’m reverse proxying from) I can see in the airbyte-webapp that its not receiving an address:

localhost:8000 successful:

airbyte-webapp      | 172.18.0.1 - - [31/Aug/2022:02:17:55 +0000] "POST /api/v1/workspaces/get HTTP/1.1" 200 377 "http://localhost:8000/workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "-"
airbyte-server      | 2022-08-31 02:17:55 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/source_definitions/list
airbyte-webapp      | 172.18.0.1 - - [31/Aug/2022:02:17:55 +0000] "POST /api/v1/source_definitions/list HTTP/1.1" 200 2006023 "http://localhost:8000/workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "-"
airbyte-server      | 2022-08-31 02:17:55 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/web_backend/connections/list - {"workspaceId":"639e6d1c-e1a4-4292-933e-eadcf1a126f0"}
airbyte-webapp      | 172.18.0.1 - - [31/Aug/2022:02:17:55 +0000] "POST /api/v1/web_backend/connections/list HTTP/1.1" 200 52743 "http://localhost:8000/workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/connections" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "-"
airbyte-server      | 2022-08-31 02:17:55 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/source_definitions/list_latest
airbyte-webapp      | 172.18.0.1 - - [31/Aug/2022:02:17:55 +0000] "POST /api/v1/source_definitions/list_latest HTTP/1.1" 200 2016935 "http://localhost:8000/workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "-"
airbyte-server      | 2022-08-31 02:17:55 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/destination_definitions/list
airbyte-webapp      | 172.18.0.1 - - [31/Aug/2022:02:17:55 +0000] "POST /api/v1/destination_definitions/list HTTP/1.1" 200 204174 "**http://localhost:8000/workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/connections**" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "-"
airbyte-server      | 2022-08-31 02:17:55 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/destination_definitions/list_latest
airbyte-webapp      | 172.18.0.1 - - [31/Aug/2022:02:17:55 +0000] "POST /api/v1/destination_definitions/list_latest HTTP/1.1" 200 204465 "**http://localhost:8000/workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/connections**" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "-"

However, the localhost:200 is not receiving a proper address:

airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:15:31 +0000] "GET /workspaces/639e6d1c-e1a4-4292-933e-eadcf1a126f0/connections HTTP/1.1" 200 1211 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"
airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:15:32 +0000] "GET /static/css/main.a272a452.css.map HTTP/1.1" 200 4785 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"
airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:15:32 +0000] "GET /static/css/224.e564b584.chunk.css.map HTTP/1.1" 200 31954 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"
airbyte-server      | 2022-08-31 02:15:32 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/workspaces/get - {"workspaceId":"639e6d1c-e1a4-4292-933e-eadcf1a126f0"}
airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:15:32 +0000] "POST /api/v1/workspaces/get HTTP/1.1" 200 377 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"
airbyte-server      | 2022-08-31 02:15:32 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/source_definitions/list_latest
airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:15:32 +0000] "POST /api/v1/source_definitions/list_latest HTTP/1.1" 200 2016879 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"
airbyte-server      | 2022-08-31 02:15:32 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/source_definitions/list
airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:15:32 +0000] "POST /api/v1/source_definitions/list HTTP/1.1" 200 2006127 **"-"** "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"
airbyte-server      | 2022-08-31 02:15:32 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 POST 200 /api/v1/web_backend/connections/list - {"workspaceId":"639e6d1c-e1a4-4292-933e-eadcf1a126f0"}
airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:15:32 +0000] "POST /api/v1/web_backend/connections/list HTTP/1.1" 200 52758 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"
airbyte-server      | 2022-08-31 02:15:52 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 GET 200 /api/v1/health
airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:15:52 +0000] "GET /api/v1/health HTTP/1.1" 200 18 **"-"** "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"
airbyte-server      | 2022-08-31 02:16:12 INFO i.a.s.RequestLogger(filter):112 - REQ 172.18.0.3 GET 200 /api/v1/health
airbyte-webapp      | 172.18.0.1 - admin [31/Aug/2022:02:16:12 +0000] "GET /api/v1/health HTTP/1.1" 200 18 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.70" "::1"

When attempting to load the web UI on localhost:200 it’ll either spin and give me timeouts and unauthorized errors, or it’ll let me in just fine. On average, it takes ~ 2-3 refreshes to get in. on public_ip:200 I only get unauthorized and timeouts.

marcosmarxm · August 31, 2022, 8:54pm

Did you check the post https://shadabshaukat.medium.com/deploy-and-secure-airbyte-with-nginx-reverse-proxy-basic-authentication-lets-encrypt-ssl-72bee223a4d9 explaining how to do a reverse proxy with Airbyte?

jordan · September 1, 2022, 3:07pm

I did, I have similar settings as him in my deployment, I even deployed an additional setup exactly as stated and I receive the same issue. That’s why I’m here, there’s no document discussing this error. My server config is currently:

server {
listen 200;
listen [::]:200;
server_name localhost;
location / {
proxy_pass http://localhost:8000/;
proxy_pass_header Accept;
proxy_pass_header Server;
proxy_pass_header Authorization;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
proxy_set_header Connection “”;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-User $http_authorization;
proxy_set_header Authorization $http_authorization;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_hide_header Referer;
proxy_hide_header Origin;
proxy_set_header Referer ‘’;
proxy_set_header Origin ‘’;
#proxy_set_header ns_server-ui yes;
proxy_cache_bypass $http_upgrade;
#auth_basic “Administrator’s Area”;
#auth_basic_user_file /etc/apache2/.htpasswd;
proxy_connect_timeout 1200;
proxy_send_timeout 1200;
proxy_read_timeout 1200;
fastcgi_send_timeout 1200;
fastcgi_read_timeout 1200;
send_timeout 1800;
}
}

Changing to https/ssl from http doesn’t make a difference, and shouldn’t when I’m accessing localhost:200. My deployment is local docker on prem, on a windows PC with nginx deployed through wsl.

marcosmarxm · September 5, 2022, 10:12pm

Hello Jordan I asked to our frontend team to check your issue. Hope to return with some help in a couple of days

jordan · September 7, 2022, 12:08am

Thanks Marcos. In the meantime, I’m going to spin up a few more machines and test separately on a different network and see if I can isolate what the issue is.

Topic		Replies	Views
Nginx reverse proxy tells 502 until containers restarted Platform, Deploy & Infra Issues docker , deploy	4	1143	December 1, 2022
Airbyte Proxy - Nginx 504 Gateway Time-out EC2 Docker deployment Platform, Deploy & Infra Issues docker , deploy , temporal	4	901	December 8, 2022
Host not found in upstream "airbyte-server:8001" in /etc/nginx/conf.d/default.conf Platform, Deploy & Infra Issues deploy	6	2832	November 25, 2022
Airbyte giving "localhost refused to connect error" while visiting http://localhost:8000/ Platform, Deploy & Infra Issues deploy	3	1504	August 9, 2022
Cannot access webapp as call to list_latest times out Platform, Deploy & Infra Issues kubernetes , deploy	3	409	September 20, 2022

NGINX reverse proxy behind port forward to local Airbyte webserver

Related Topics