Upgrade fails to version 0.39.28-alpha

Platform, Deploy & Infra Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu 18
  • Memory / Disk: you can use something like 64Gb / 180 Gb
  • Deployment: Kubernetes deployment
  • Airbyte Version: 0.35.67-alpha
  • Source name/version: -
  • Destination name/version: -
  • Step: Upgrade
  • Description: bootloader pod fails and bocks whole upgrade

I’m running Airbyte in k8s deployed via Helm and unable to upgrade from version 0.35.67-alpha to latest release 0.39.28-alpha because of failing bootloader pod - at first it seemed like an issue with Postgres not being ready / timeout which I’ve worked-around as described here: Unable to upgrade to v0.39.4-alpha or later via Helm chart

But then I ran into this error which fails / blocks whole upgrade via Helm again:

Exception in thread “main” java.lang.IllegalArgumentException: malformed JsonSchema array type, must have items field in {“type”:“array”,“title”:“Channel name filter”,“default”:,“examples”:[“channel_one”,“channel_two”],“description”:“A channel name list (without leading ‘#’ char) which limit the channels from which you’d like to sync. Empty list means no filter.”}

Could you help me how to resolve it?

2

Hi @donatas.jocas,
Could you please share the full logs of the process in which you see this error happens?
It looks like an error related to a Slack connector configuration. It’s a bit strange that this error is raised during an upgrade?Could you try to deactivate this connection before the upgrade?

3

Hi @alafanechere thank you very much for reply! Unfortunately I do not have full logs of the failed container anymore…but I could save and share full logs next time I’ll try to run the upgrade if the problem persists.

Could you elaborate how in particular I should “deactivate” connector before giving another run to upgrade? Thanks!

4

@donatas.jocas you can disable a connection on its setting page by clicking on this button:


You can also try to connect to Airbyte database by connecting to the airbyte-db pod:
kubectl exec -it airbyte-db psql -U docker -d airbyte
and run something like DELETE FROM connection WHERE id = 'your-connection-id';

5

Hi @alafanechere - thanks for the tip, however I’m a bit confused as we are not using Slack connector anywhere…

Any other thoughts? I didn’t yet had a chance to re-run the deployment, but as soon as I do would share the full logs!

6

Could you please share the full bootloader logs so that I can better understand the context in which this error happens? Did you set up some slack notifications on your current version? If you did do you mind trying to disable these before upgrading?

7

Hi, I’m back to this topic a again - please find attached full bootloader log
airbyte-bootloader-20220725.log (59.7 KB)

8

@alafanechere we used Slack source connector quite some time ago perhaps in older versions of airbyte for testing purposes, but the source and connection itself has been deleted since then. However it looks like bootloader migration takes into account those “deleted” sources - here is the the message in particular from the log:

2022-07-25 12:34:08 e[32mINFOe[m i.a.c.p.DatabaseConfigPersistence(updateConfigsFromSeed):1631 - Connectors in use: [airbyte/source-file, airbyte/source-slack, airbyte/source-mysql, airbyte/source-s3, airbyte/source-pipedrive, airbyte/source-google-analytics-v4, airbyte/destination-snowflake, airbyte/destination-local-json, airbyte/source-google-ads, airbyte/destination-postgres, airbyte/source-salesforce, airbyte/source-zenloop, airbyte/source-jira, airbyte/destination-redshift, airbyte/source-redshift, airbyte/source-postgres, airbyte/source-google-sheets, airbyte/source-mssql]

We don’t use half of the connectors mentioned above.

Could this be a bug? Like those old dummy connections, probably even old versions, stopping the upgrade. And any ideas how to get around this?

Very appreciated your help.

9

Okey I think I’e made it work :slight_smile: downgraded Slack connector version to 0.1.12 before an upgrade and the next bootloader run completed successfully along with all Helm upgrade proccess - log attached.

Hope this thread will help if someone else faces similar issue - though still not sure on what was the problem in the first place and if there are chance we face it again next time upgrading having newer connector versions…

airbyte-bootloader-success.log (53.1 KB)

10

Thanks for the update, and we appreciate your sharing the solution with us.