Summary
The user is encountering a 504 error on the Select streams step when setting up a connector with a postgres target and mysql source. They have tried adding connectTimeout=0 to each connection to troubleshoot. They are seeking advice to determine if this is a configuration issue or a connection problem.
Question
Hi I’m trying out airbyte open source locally for the first time. I’ve set up a postgres target and a mysql source and keep running into a 504 error on the Select streams step when setting up the Connector. I’ve tried adding connectTimeout=0 to each to see if it was a timeout issue. I’m trying to determine if this is a configuration issue or something with my connection. Any advice would be very appreciated! Thanks!
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["504-error", "select-streams", "postgres-target", "mysql-source", "connecttimeout"]
More than likely this is a timeout in the networking stack . . . most often I see this with load balancer configuration (which often have low default timeouts), but it really depends on how you’re deploying.
If you share a little more about how you’re deployed (e.g. Docker vs. Kubernetes, and local or cloud—and if so which cloud/platform/tool) I might be able to help you narrow things down a bit more.
Hi Justin, this is all local right now on my machine testing it for a POC
What Airbyte version is this?
airbyte-abctl version 0.163.0
I’m seeing a lot of chatter on this from people who upgraded to 0.61.x, but may affect new installs as well:
https://github.com/airbytehq/airbyte/issues/38854
Some say downgrading worked for them. but might check that thread and see if it matches your issue
Are you otherwise able to connect from that local machine to the source/destination systems? (just to rule out any network or firewall type issues)
yep I just transfered data from the mysql source to postgres destination using dlt this morning 
ah okay I see the airbyte running version is 0.63.0. Sorry haha taking me a minute to understand the versioning
If basic auth is enabled (which is it by default), there’s an nginx proxy that sits in front of the other containers . . . in your .env file, setting BASIC_AUTH_PROXY_TIMEOUT=600 (or some other high value) my resolve this for you
That’s at least the most common issue I’ve seen, since some tasks just take a while in this type of system (depending on the sources involved)
I would start there, and we can hunt for better answers if that doesn’t work
if it’s already 600, you could try increasing it . . . but usually if something is taking that long to run, it probably means there aren’t enough resources for it for Docker to run the sync/discovery container
ahh okay yeah there are at least 800 tables in the mysql source
could be related, but you could always try connecting to a DB with only a small number of tables and see if that’s the issue. (if so, there’s got to be some environment variables or such that related to those values.yaml options from the Helm deployment)
but discover tasks do tend to be slower in general, so my first guess is still that this is related to the auth proxy timeout.
I used brew to install abctl and I’m kinda confused on where the actual docker configuration is stored. I did the brew install in my root directory but am not sure what I should be looking for. All I see is the .airbyte hidden directory with the abdtl folder with abctl.kubeconfig file
ah okay I’m going with the docker-compose installation. the quick start didn’t seem to allow as much config
Nice! I think abctl reads the environment variables documented https://docs.airbyte.com/operator-guides/configuring-airbyte|here, but to be honest I haven’t played around with it at all yet since our main deployments are via Helm/Kubernetes to GKE on Google Cloud Platform. Glad to hear it’s working for you via Docker though!