Airtbyte in a crashloop on fresh helm install

rotemtam · September 19, 2022, 3:59am

Is this your first time deploying Airbyte?: Yes
OS Version / Instance: Linux
Deployment: K8S
Airbyte Version: helm chart v0.40.8
Description:

Installing the chart for the first time.
Multiple containers in a crashloop.

Getting:


    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
        : airbyte-workers :

  Micronaut (v3.6.3)

2022-09-18 13:40:31 e[32mINFOe[m i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [k8s, cloud, ec2]
2022-09-18 13:40:32 e[32mINFOe[m i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):107 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2022-09-18 13:40:33 e[1;31mERRORe[m i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Failed to inject value for parameter [dslContext] of method [configDatabase] of class: io.airbyte.db.Database

Message: No bean of type [org.jooq.DSLContext] exists for the given qualifier: @Named('config'). Make sure the bean is not disabled by bean requirements (enable trace logging for 'io.micronaut.context.condition' to check) and if the bean is enabled then ensure the class is declared a bean and annotation processing is enabled (for Java and Kotlin the 'micronaut-inject-java' dependency should be configured as an annotation processor).
Path Taken: new ApplicationInitializer() --> ApplicationInitializer.checkConnectionActivities --> List.checkConnectionActivities([CheckConnectionActivity checkConnectionActivity]) --> CheckConnectionActivityImpl.secretsHydrator --> SecretsHydrator.secretsHydrator([SecretPersistence secretPersistence]) --> SecretPersistence.localTestingSecretPersistence([Database configDatabase]) --> Database.configDatabase([DSLContext dslContext])
io.micronaut.context.exceptions.DependencyInjectionException: Failed to inject value for parameter [dslContext] of method [configDatabase] of class: io.airbyte.db.Database

Message: No bean of type [org.jooq.DSLContext] exists for the given qualifier: @Named('config'). Make sure the bean is not disabled by bean requirements (enable trace logging for 'io.micronaut.context.condition' to check) and if the bean is enabled then ensure the class is declared a bean and annotation processing is enabled (for Java and Kotlin the 'micronaut-inject-java' dependency should be configured as an annotation processor).
Path Taken: new ApplicationInitializer() --> ApplicationInitializer.checkConnectionActivities --> List.checkConnectionActivities([CheckConnectionActivity checkConnectionActivity]) --> CheckConnectionActivityImpl.secretsHydrator --> SecretsHydrator.secretsHydrator([SecretPersistence secretPersistence]) --> SecretPersistence.localTestingSecretPersistence([Database configDatabase]) --> Database.configDatabase([DSLContext dslContext])
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2087) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1297) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.airbyte.workers.config.$DatabaseBeanFactory$ConfigDatabase0$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1297) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.airbyte.workers.config.$SecretPersistenceBeanFactory$LocalTestingSecretPersistence1$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1297) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.airbyte.workers.config.$SecretPersistenceBeanFactory$SecretsHydrator4$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForField(AbstractInitializableBeanDefinition.java:1603) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.airbyte.workers.temporal.check.connection.$CheckConnectionActivityImpl$Definition.injectBean(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
	at io.airbyte.workers.temporal.check.connection.$CheckConnectionActivityImpl$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1297) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.airbyte.workers.config.$ActivityBeanFactory$CheckConnectionActivities0$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:1680) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractBeanResolutionContext.findBean(AbstractBeanResolutionContext.java:84) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveOptionalBean(AbstractInitializableBeanDefinition.java:2178) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.findBeanForField(AbstractInitializableBeanDefinition.java:1843) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.airbyte.workers.$ApplicationInitializer$Definition.injectBean(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
	at io.airbyte.workers.$ApplicationInitializer$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2853) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.addCandidateToList(DefaultBeanContext.java:3511) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistrations(DefaultBeanContext.java:3457) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBeanRegistrations(DefaultBeanContext.java:3427) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:1381) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:903) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:893) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.lambda$$0(ApplicationEventPublisherFactory.java:217) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.core.util.SupplierUtil$2.initialize(SupplierUtil.java:77) ~[micronaut-core-3.6.3.jar:3.6.3]
	at io.micronaut.core.util.SupplierUtil$2.get(SupplierUtil.java:72) ~[micronaut-core-3.6.3.jar:3.6.3]
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:574) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:568) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:297) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
	at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.6.3.jar:3.6.3]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.6.3.jar:3.6.3]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.6.3.jar:3.6.3]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.6.3.jar:3.6.3]
	at io.airbyte.workers.Application.main(Application.java:12) ~[io.airbyte-airbyte-workers-0.40.7.jar:?]
Caused by: io.micronaut.context.exceptions.NoSuchBeanException: No bean of type [org.jooq.DSLContext] exists for the given qualifier: @Named('config'). Make sure the bean is not disabled by bean requirements (enable trace logging for 'io.micronaut.context.condition' to check) and if the bean is enabled then ensure the class is declared a bean and annotation processing is enabled (for Java and Kotlin the 'micronaut-inject-java' dependency should be configured as an annotation processor).
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2805) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.6.3.jar:3.6.3]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.6.3.jar:3.6.3]
	... 98 more

Any guidance?

rotemtam · September 19, 2022, 4:15am

Downgrading to v0.40.6, getting a different crash.

│ java.lang.IllegalArgumentException: 'INTERNAL_API_HOST' environment variable cannot be null                            │
│     at com.google.common.base.Preconditions.checkArgument(Preconditions.java:220) ~[guava-31.0.1-jre.jar:?]            │
│     at io.airbyte.config.EnvConfigs.getEnsureEnv(EnvConfigs.java:1107) ~[io.airbyte.airbyte-config-config-models-0.40. │
│     at io.airbyte.config.EnvConfigs.getAirbyteApiHost(EnvConfigs.java:490) ~[io.airbyte.airbyte-config-config-models-0 │
│     at io.airbyte.workers.WorkerApiClientFactoryImpl.<init>(WorkerApiClientFactoryImpl.java:35) ~[io.airbyte-airbyte-w │
│     at io.airbyte.workers.WorkerApp.initializeCommonDependencies(WorkerApp.java:442) ~[io.airbyte-airbyte-workers-0.40 │
│     at io.airbyte.workers.WorkerApp.main(WorkerApp.java:578) [io.airbyte-airbyte-workers-0.40.6.jar:?]                 │

Are there integration tests for this chart? What version is stable?

marcosmarxm · September 19, 2022, 3:02pm

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

jyates-om1 · September 19, 2022, 3:13pm

I am getting the same bean error on the worker with 0.40.3. This is strange because it was actually running properly on this version for a week before this issue started popping up.

jyates-om1 · September 19, 2022, 3:26pm

Scratch that - I misunderstood our deployment pipeline and we actually encountered this error on 0.43.7. I am currently troubleshooting, but @rotemtam we figured out recently that the docker-compose.yaml is sometimes tied to the version you are running, so you may have to pull the docker-compose.yaml from the same github tag as the version you are running (environment variable bindings can change from version to version)

jyates-om1 · September 19, 2022, 3:45pm

Given that we are both experiencing the same issue and I haven’t seen a github bug yet, I opened one here: Airbyte 0.40.7 worker is failing on startup with an exception, while the same environment worked fine for 0.40.3 · Issue #16893 · airbytehq/airbyte · GitHub

alex-swell · September 19, 2022, 3:45pm

I am getting the same errors as @rotemtam on the helm versions 0.40.6 and 0.40.8. Pretty frustrating… I don’t see a matching chart value for INTERNAL_API_HOST from airbyte/charts/airbyte at master · airbytehq/airbyte · GitHub either. Some help here would be much appreciated.

alex-swell · September 19, 2022, 4:31pm

For what’s it worth the environment configmap (i.e. INTERNAL_API_HOST: airbyte-8fde24ea-airbyte-server-svc:8001) has this variable set. I modified the k8s deployment to pull this value as an environment variable from the configmap and the pod will at least run but eventually the container crashes. There aren’t any logged errors or anything.

Last few lines in log:

2022-09-19 16:25:04 INFO o.e.j.s.h.ContextHandler(doStart):860 - Started o.e.j.s.ServletContextHandler@2429e0a9{/,null,AVAILABLE}
2022-09-19 16:25:04 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="CONNECTION_UPDATER", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:04 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="CONNECTION_UPDATER", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:04 INFO o.e.j.s.AbstractConnector(doStart):331 - Started ServerConnector@561050b7{HTTP/1.1, (http/1.1)}{0.0.0.0:9000}
2022-09-19 16:25:04 INFO o.e.j.s.Server(doStart):400 - Started @13255ms
2022-09-19 16:25:04 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="CHECK_CONNECTION", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="CHECK_CONNECTION", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="CHECK_CONNECTION", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="SYNC", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="SYNC", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="SYNC", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Workflow Poller taskQueue="GET_SPEC", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Local Activity Poller taskQueue="GET_SPEC", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Activity Poller taskQueue="GET_SPEC", namespace="default", identity=1@airbyte-8fde24ea-server-6dc57b7976-rsx5d}
2022-09-19 16:25:05 INFO i.t.i.w.Poller(start):91 - start: Poller{name=Host Local Workflow Poller, identity=7033c802-dfa6-4275-a07c-c94aec1ecaf2}
2022-09-19 16:25:41 INFO c.z.h.HikariDataSource(close):347 - HikariPool-1 - Shutdown initiated...
2022-09-19 16:25:41 INFO c.z.h.p.HikariPool(shutdown):204 - HikariPool-1 - Close initiated...
2022-09-19 16:25:41 INFO c.z.h.p.HikariPool(shutdown):238 - HikariPool-1 - Closed.
2022-09-19 16:25:41 INFO c.z.h.HikariDataSource(close):349 - HikariPool-1 - Shutdown completed.
2022-09-19 16:25:41 INFO c.z.h.HikariDataSource(close):347 - HikariPool-2 - Shutdown initiated...
2022-09-19 16:25:41 INFO c.z.h.p.HikariPool(shutdown):204 - HikariPool-2 - Close initiated...
2022-09-19 16:25:41 INFO c.z.h.p.HikariPool(shutdown):238 - HikariPool-2 - Closed.
2022-09-19 16:25:41 INFO c.z.h.HikariDataSource(close):349 - HikariPool-2 - Shutdown completed.

cdgz · September 19, 2022, 5:24pm

After long hours I managed to bootstrap latest(?) Airbyte OSS stack. INTERNAL_API_HOST issue seems related to wrong image used in chart (fix merged 6 hours ago). Micronaut issue was introduced in 0.40.7 of airbyte-worker. So my fix was to pull latest chart changes, reference local paths for dependencies and force worker version to 0.40.6.

charts/airbyte/Chart.yaml

apiVersion: v2
name: airbyte
description: Helm chart to deploy airbyte

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.45.2

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "0.40.7"

dependencies:
  - name: common
    repository: https://charts.bitnami.com/bitnami
    tags:
      - bitnami-common
    version: 1.x.x
  - condition: postgresql.enabled
    name: postgresql
    version: 10.x.x
    repository: https://charts.bitnami.com/bitnami
  - condition: global.logs.minio.enabled
    name: minio
    version: 11.x.x
    repository: https://charts.bitnami.com/bitnami
  - condition: airbyte-bootloader.enabled
    name: airbyte-bootloader
    repository: "file://../airbyte-bootloader"
  - condition: temporal.enabled
    name: temporal
    repository: "file://../airbyte-temporal"
  - condition: webapp.enabled
    name: webapp
    repository: "file://../airbyte-webapp"
  - condition: server.enabled
    name: server
    repository: "file://../airbyte-server"
  - condition: worker.enabled
    name: worker
    repository: "file://../airbyte-worker"
  - condition: pod-sweeper.enabled
    name: pod-sweeper
    repository: "file://../airbyte-pod-sweeper"
  - condition: metrics.enabled
    name: metrics
    repository: "file://../airbyte-metrics"

Helm install command

cd <repo_root>/charts
helm upgrade --install --namespace airbyte airbyte ./airbyte --set worker.image.tag=0.40.6

Then port forward and access UI.

alex-swell · September 23, 2022, 8:14pm

FWIW I was able to deploy everything through helm onto EKS successfully with version 0.41.2. Hopefully a nonissue going forward.

rotemtam · September 25, 2022, 3:54am

@alex-swell thanks for sharing. Still completely broken for me:

github.com/airbytehq/airbyte

[Helm] Helm chart broken on latest 0.41.2: INSTALLATION FAILED: deployments.apps "airbyte-minio" already exists

opened 02:31PM - 24 Sep 22 UTC

rotemtam

type/bug community needs-triage team/use autoteam

Error: ## Environment - **Airbyte version**: chart v0.41.2 - **OS Version / Instance**:EKS - **Deployment**: Kubernetes ## Current Behavior INSTALLATION FAILED: deployments.apps "airbyte-minio" already exists ## Expected Behavior *Tell us what should happen.* Installation on default options should succeed. ## Logs ## Steps to Reproduce 1. kubectl create ns data-eng 2. helm install -n data-eng airbyte airbyte/airbyte ## More info Installing with `minio.enabled=false` works. Appears that minio is installed from both the airbyte chart and as a subchart.

Topic		Replies	Views
Cannot upgrade to latest airbyte: Error starting Micronaut server in airbyte-cron Connector Questions & Issues deployment	1	636	December 30, 2022
Deploy Kubernetes(EKS): Unable to launch airbyte using helm charts Platform, Deploy & Infra Issues getting-started , deploy	5	959	November 7, 2022
Helm Chart From 0.45.1 --> 0.42.0 Causes template Error Connector Questions & Issues kubernetes	7	425	September 20, 2022
Deploy Kubernetes: Unable to launch airbyte using helm charts Platform, Deploy & Infra Issues getting-started , kubernetes , deploy	7	1643	July 14, 2022
Nil pointer error when deploying Helm chart Platform, Deploy & Infra Issues deploy	3	11679	July 14, 2022

Airtbyte in a crashloop on fresh helm install

Related Topics