Airbyte Server Not Healthy Exception with prefect-airbyte starting (or api call) with 0.40.28

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: you can use something like 4Gb / 1 Tb
  • Deployment: Kubernetes
  • Airbyte Version: 0.40.28
  • Source name/version: Postgres 1.0.42
  • Destination name/version: 0.2.14
  • Step: sync
  • Description:

I use prefect and the prefect-airbyte package GitHub - PrefectHQ/prefect-airbyte and everything was doing fine before upgrading to 0.40.28 (same for latest 0.40.32) and now I got an AirbyteServerNotHealthyException.
In the log server, it says that a sync is already running but It is not and when I revert to 0.40.27 it works well again.

This issue happens with prefect but I guess it would the same with an API call on the airbyte server.

Thanks

2

Hey @lucienfregosi, thanks for all the details! I’m not very familiar with Prefect, so I need to look into that. My first thought is too look through the changelog and try to discern what might have broken:
https://github.com/airbytehq/airbyte/releases/tag/v0.40.28

And it looks like that repo is maintained outside of Airbyte and the latest merge was in December. Do you know anything about their roadmap?

3

I’ve already looked at the chagelog and I didn’t see a change which could be related.

I don’t know about their roadmap but first I ensure that the issue was not on their side and this package is super trivial basically it wraps an HTTP request like this

I’m pretty sure that it could be reproducible outside prefect-airbyte context

4

@lucienfregosi most recent version of Airbyte added the login/password which maybe the Prefect didn’t update the code yet. Also there a lot of work in Airbyte API which can change some endpoints or payloads.

1 Like
5

Hi @marcosmarxm

It seems that prefect team already manage login/password.

From the airbyte server logs I got this error message

java.lang.NullPointerException: Cannot invoke "io.airbyte.config.storage.CloudStorageConfigs.getType()" because the return value of "io.airbyte.config.helpers.LogConfigs.getStorageConfigs()" is null
6

I’ve seen this thread (my case is sligthly different as the error appears in the server image)

But I have already these variable set for the worker (but not for the server indeed but I got an other error if I set them the way I did for the worker; - name: WORKER_LOGS_STORAGE_TYPE
value: “MINIO”)

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: minio

TBH I don’t even use MINIO

7

I updated these 2 variables :

configs_database_minimum_flyway_migration_version: "0.40.23.002"
jobs_database_minimum_flyway_migration_version : "0.40.26.001"

And now I have this issue
java.lang.IllegalArgumentException: null

I should have missed something… but I don’t know what variable is missing

8

Hello lucienfregosi, it’s been a while without an update from us. Are you still having problems or did you find a solution?

9

@marcosmarxm no sadly don’t have a solution

for now I’m stuck with airbyte 0.40.27 which is not that annoying but I can’t upgrade to 0.40.28

10

@marcosmarxm I git into this issue again

I looked at the server and spot some more precised errors

2023-02-21 16:32:18 ERROR i.a.s.a.ApiHelper(execute):28 - Unexpected Exception
software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.

Caused by: java.net.UnknownHostException: minio

The thing is I don’t even use minio or aws …
I log everything into GCP cloud storage

If you have an idea it would be awesoooooome

11

Solved

I had to replace
WORKER_LOGS_STORAGE_TYPE=MINIO
WORKER_STATE_STORAGE_TYPE=MINIO

by
WORKER_LOGS_STORAGE_TYPE=GCS
WORKER_STATE_STORAGE_TYPE=GCS

But I didn’t know why it worked before aha