HTTP 504 error when adding custom destination connector

Summary

User is facing HTTP 504 error when trying to add a custom destination connector in Airbyte. The connector was successfully added on MacOS but not on Linux VMs. User has tried different VM types, allowed ports in firewall, and checked logs using k9s.


Question

Hi,

I haven’t been able to add a custom destination connector from the UI because it shows an HTTP 504 error.

I developed a Google AppSheet custom destination connector using Python CDK, built it locally, pushed it to a <https://hub.docker.com/r/vetatech/airbyte-destination-appsheet|public registry in Docker Hub>, and then successfully added it through the UI in my local machine by pulling from the public registry (no need to use the kind load docker-image command). My Airbyte instance was installed in my local machine (MacOS) using abctl.

Then, several times, I replicated the same steps in a fresh GCP VM of different types, including an n1-standard-8 (8 vCPUs, 30 GB memory), with no success; it always threw a 504 HTTP error. Even after using the kind load docker-image command, it kept failing and not showing the connector in the UI). I allowed all the ports, protocols, and IP addresses in the firewall of the VM and installed Airbyte using abctl with and without --insecure-cookies and --low-resource-mode flags.

I followed the suggestion of <@U05JENRCF7C> of using k9s to check the pods (thanks for taking the time :+1:), and the logs are in the screenshots, but I can’t figure out what they mean and how to tackle it.

In short, adding a custom destination connector works in abctl-MacOS but not in abctl-Linux (tested on Debian and Ubuntu).

Has anyone encountered this issue recently? Would you happen to have any advice on how to fix it? Thanks community.



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["http-504-error", "custom-destination-connector", "docker-hub", "abctl", "gcp-vm", "k9s", "linux"]

This error looks like architecture mismatch

I’d recommend building images for multiple architectures
https://docs.docker.com/build/building/multi-platform/

https://hub.docker.com/r/vetatech/airbyte-destination-appsheet/tags
https://cloudprice.net/gcp/compute/instances/n1-standard-8

<@U05JENRCF7C>, I overlooked that! Now it makes sense; thank you very much for helping me out. I will investigate how to build the Docker image for multiple architectures and see how it goes :octavia-wave:

Following up on this, for the record, that was it. I used this command to generate the image for the required architecture:
airbyte-ci connectors --name destination-appsheet build -a linux/amd64

Then, I added it successfully through the UI.

Later, I used this command to generate a multi-platform image:
docker buildx build --platform linux/amd64,linux/arm64 -t vetatech/airbyte-destination-appsheet:0.0.30 .

But I couldn’t add this multi-platform image because of HTTP 500 error. I didn’t investigate further because it is ok for now, but if I get more information, I will add it here for anyone who may find it helpful.

Thanks again for pointing me in the right direction <@U05JENRCF7C>.