Summary
There seems to be an issue with viewing Airbyte logs from the UI, possibly related to a size limitation. User is unable to view logs for some sync and is seeking assistance on how to resolve this problem.
Question
Hello @everyone , I am currentely having a issue with viewing Airbyte logs from UI. Is there like a size limitation to the size of log that can be viewed from UI. I am uncble to view logs for some sync. can someone please help how to go about this issue?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["issue", "viewing-logs", "airbyte", "ui", "size-limitation", "sync"]
Could you provide version details, how you’re deploying, and what you’re using for logging (e.g. minio, GCS, or S3)?
Do you get any specific errors? (especially in the net panel of DevTools)
My first guess would be that potentially there’s either a config issue or some type of timeout happening. If it’s a timeout, you’ll likely see some type of 504 status on the log requests in the network panel.
I am running 0.50.40 on EC2 docker. I am able to view logs when the sync is started and after about 30 minutes the logs will just stop loading. For smaller tables, i can view logs even after sync. We are not logging to s3 yet.
Are you getting any errors in the DevTools Network panel or in the logs?
yes, i see 504 timed out error. but for smaller los are able to load
Very likely that this is just a timeout because the logs are taking too long to load. That’s the symptom, anyway—it could also be the cause with large logs, or that could be a sign of resource contention on the pod serving or retrieving the log file that’s leading to the delay.
I would start with increasing the timeout on your frontend (whether you’re accessing the exposed instance directly or have it behind a load balancer or such). I’m not as familiar day to day with AWS (we’re a GCP shop), but if I recall correctly the default timeouts are similar, either 30 or 60 seconds. One of the recommendations I’ve seen around https://discuss.airbyte.io/t/airbyte-proxy-nginx-504-gateway-time-out-ec2-docker-deployment/3305/4|here is setting this in your .env:
BASIC_AUTH_PROXY_TIMEOUT=600
Can I get a API key ,if I am hosting Airbyte on my localhost using docker compose <@U04SE163WC8>
<@U070VN61E1E> Kapa isn’t in this channel. But no, you don’t use API keys when hosting locally or deploying OSS—you just control access however you want to. The API can either be accessed on the API pod directly, or on the webapp service (with a prefix of /api/v1/).
Many of us use other forms of access control, either a proxy or IAM roles or such when deploying in the Cloud. If you’re running locally, you probably don’t need any access control at all if it’s just for other services running locally to access.