Migrating Data from Docker Compose to Kubernetes with Airbyte Helm Charts

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

User is facing an error 502 bad gateway when trying to migrate data from Docker Compose to Kubernetes using Airbyte Helm Charts. Assistance is needed to debug and gracefully migrate the data.


Question

Hi everyone. I previously hosted Airbyte via Docker Compose. Now I’m trying to set up via Kubernetes with Helm Charts. I can get it to run locally on port 8080, but I’m trying to load my data from Docker Compose into the Kubernetes one. I managed to do a pg dump in the VM where I have the Docker Compose version and ingested this pg dump file into DigitalOcean’s Postgres database. I connected to this database from Tableplus - no issues viewing all the data. Then I tried to run helm install again but this time with the values.yaml file containing the Postgres credentials. However, this time, my local port 8080 returns error 502 with http error bad gateway.

Need some assistance on debugging why is this happening, and how can I gracefully migrate all my data from Docker Compose to Kubernetes.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["migrating-data", "docker-compose", "kubernetes", "helm-charts", "502-error", "bad-gateway", "postgres", "values.yaml"]
2

This is the full error message:

  "url": "<http://127.0.0.1:8080/workspaces/>&lt;workspace-id&gt;/connections",
  "airbyteVersion": "0.63.19",
  "errorType": "HttpError",
  "errorConstructor": "gd",
  "error": {
    "i18nKey": "errors.http.badGateway",
    "i18nParams": {
      "status": 502
    },
    "name": "HttpError",
    "requestId": "&lt;request-id&gt;",
    "request": {
      "url": "/api/v1/instance_configuration",
      "method": "get"
    },
    "status": 502,
    "response": "<!DOCTYPE html>\n&lt;html&gt;\n&lt;head&gt;\n&lt;title&gt;Error&lt;/title&gt;\n&lt;style&gt;\nhtml { color-scheme: light dark; }\nbody { width: 35em; margin: 0 auto;\nfont-family: Tahoma, Verdana, Arial, sans-serif; }\n&lt;/style&gt;\n&lt;/head&gt;\n&lt;body&gt;\n&lt;h1&gt;An error occurred.&lt;/h1&gt;\n&lt;p&gt;Sorry, the page you are looking for is currently unavailable.&lt;br/&gt;\nPlease try again later.&lt;/p&gt;\n&lt;p&gt;If you are the system administrator of this resource then you should check\nthe error log for details.&lt;/p&gt;\n&lt;p&gt;&lt;em&gt;Faithfully yours, nginx.&lt;/em&gt;&lt;/p&gt;\n&lt;/body&gt;\n&lt;/html&gt;\n"
  },
  "stacktrace": "HttpError: errors.http.badGateway\n    at Rve (<http://127.0.0.1:8080/assets/core-du4n84dfbx.js:119:10478>)",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0",
  "featureFlags": {}
}```
3

So you are using local port 8080 to communicate with your remote kubernetes cluster and postgres database? Are you using kubernetes port forwarding then?

4

Yup I should be using port forwarding. I was just following this guide (https://docs.airbyte.com/deploying-airbyte/#6-set-up-port-forward-for-ui-access)