Summary
Error message related to a server error after upgrading docker compose setup and BigQuery destination, causing connection issues. Looking for troubleshooting steps.
Question
After upgrading a docker compose setup to abctl and upgrading the BigQuery destination we are seeing this error breaking all our connections:
Caused by: io.temporal.failure.ApplicationFailure: message='Server error : 500 Internal Server Error {"message":"Internal Server Error","logref":null,"path":null,"_links":{"self":{"href":"/api/v1/workload/create","templated":false,"profile":null,"deprecation":null,"title":null,"hreflang":null,"type":null,"name":null}},"_embedded":{"errors":[{"message":"Internal Server Error: Error executing PERSIST: ERROR: could not read block 8 in file \"base/16385/17447\": read only 0 of 8192 bytes","logref":null,"path":null,"_links":{},"_embedded":{}}]}}', type='io.airbyte.workload.api.client.generated.infrastructure.ServerException', nonRetryable=false
Would be happy if anyone had any idea what is causing this or how to troubleshoot. I haven’t found anyone with a similar problem on Github issues.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["docker-compose", "abctl", "bigquery-destination", "server-error", "internal-server-error", "troubleshoot"]
Hello Thomas, do you mind sharing what happened here and what needs to be fixed? :octavia-thanks: !
Hi, so eventually I was able to fix this one myself by searching the web for similar could not read block 8 in file errors and there were just very few hits but they all talked about postgres. In the end it turned out the abctl local migrate step somehow corrupted the postgres DB but luckily a REINDEX schema public; followed by VACUUM FULL; was able to recover it
The error I am trying to solve in replication is this: Warning from replication: Something went wrong during replication
message='io.airbyte.workload.api.client.generated.infrastructure.ServerException: Server error : 500 Internal Server Error {"message":"Internal Server Error","logref":null,"path":null,"_links":{"self":{"href":"/api/v1/workload/create","templated":false,"profile":null,"deprecation":null,"title":null,"hreflang":null,"type":null,"name":null}},"_embedded":{"errors":[{"message":"Internal Server Error: Error executing PERSIST: ERROR: could not read block 4 in file \"base/16385/127275\": read only 0 of 8192 bytes","logref":null,"path":null,"_links":{},"_embedded":{}}]}}', type='java.lang.RuntimeException', nonRetryable=false
Samuel the error you’re getting is because the cluster is not running or the config you’re using is not valid.
<@U01MMSDJGC9>
when I execute that command I get this error:
kubectl exec --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig --namespace airbyte-abctl -it airbyte-db-0 -- psql -U airbyte -d db-airbyte
E0902 14:29:04.500744 36948 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"<http://localhost:8080/api?timeout=32s>\": dial tcp [::1]:8080: connect: connection refused"
E0902 14:29:04.502268 36948 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"<http://localhost:8080/api?timeout=32s>\": dial tcp [::1]:8080: connect: connection refused"
E0902 14:29:04.503754 36948 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"<http://localhost:8080/api?timeout=32s>\": dial tcp [::1]:8080: connect: connection refused"
The connection to the server localhost:8080 was refused - did you specify the right host or port?```
Hello Thomas. Could you please suggest how you can access the internal PG DB to run reindex/vacuum? I have the same issue after migration from docker-compose to abctl
<@U077LQ2PSP6> sure, install kubectl and then execute the following: kubectl exec --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig --namespace airbyte-abctl -it airbyte-db-0 -- psql -U airbyte -d db-airbyte
Maybe report back if this fixes it for you as well
i’ll try. Thank you for your fast response. I am quite new to kubectl so it a bit confusing
Cheers
Yeah, this was the fix for my could not read block 8 in file
Thank you one again
Great that it also worked for you Dmytro. <@U01MMSDJGC9> maybe the abctl tool should execute these queries at the end of a migration. Or of course not corrupt the DB in the first place but I’m not sure how easy it will be to fix that.
Helo Thomas and Dmytro, I created <Issues · airbytehq/airbyte · GitHub issue> to implement a better mechanism in the future. The problem is, today we can’t reproduce the issue. Can you add as much information you have about your env/deployment? (number of connections, type of connectors, cpu/memory, version etc)
Did you stop the docker compose during the migration?