- Is this your first time deploying Airbyte?: No
- OS Version / Instance: EC2 AL2_x86_64 on EKS
- Memory / Disk: you can use something like 4Gb per node
- Deployment: EKS
- Airbyte Version: v0.40.14
- Source name/version: n/a
- Destination name/version: n/a
- Step: The issue is happening during upgrade to v0.40.14
We run Airbyte on EKS cluster. Airbyte is configured to use S3 bucket for logs as described in the guide docs.airbyte.com/deploying-airbyte/on-kubernetes/
that is
S3_LOG_BUCKET=<your_s3_bucket_to_write_logs_in>
S3_LOG_BUCKET_REGION=<your_s3_bucket_region>
S3_MINIO_ENDPOINT=
S3_PATH_STYLE_ACCESS=
This worked for us when running on Airbyte version 0.39.37
When attempting to upgrade to v0.40.9 (and v0.40.14) we encountered issue with airbyte-worker
service and airbyte-cron
When configuring Airbyte deployment for v0.40.14, we kept the S3_*
env settings as described above (and in the doc).
We also noticed that in v 0.40.9 (14) new env variables were introduced
WORKER_LOGS_STORAGE_TYPE=MINIO
WORKER_STATE_STORAGE_TYPE=MINIO
Since we want to use S3 bucket for logs and state storage, we set the WORKER_*
env vars to S3
(value found in the code)
Deploying Airbyte to the EKS cluster resulted in airbyte-worker
service going into perpetual crash loop with the log
2022-10-07 23:19:43,718 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml
___ _ __ __
/ | (_)____/ /_ __ __/ /____
/ /| | / / ___/ __ \/ / / / __/ _ \
/ ___ |/ / / / /_/ / /_/ / /_/ __/
/_/ |_/_/_/ /_.___/\__, /\__/\___/
/____/
: airbyte-workers :
Micronaut (v3.7.1)
2022-10-07 23:19:46 INFO i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [k8s, cloud, ec2, control-plane]
2022-10-07 23:19:47 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-1 - Starting...
2022-10-07 23:19:48 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-1 - Start completed.
2022-10-07 23:19:48 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-2 - Starting...
2022-10-07 23:19:49 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-2 - Start completed.
2022-10-07 23:19:49 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):107 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2022-10-07 23:19:52 INFO i.a.w.c.DatabaseBeanFactory(configsDatabaseMigrationCheck):129 - Configs database configuration: 0.35.15.001 60000
2022-10-07 23:19:53 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):221 - Waiting for temporal server...
2022-10-07 23:19:53 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for namespace default to be initialized in temporal...
2022-10-07 23:19:56 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-10-07 23:20:01 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):249 - Temporal namespace default initialized!
2022-10-07 23:20:01 WARN i.a.m.l.MetricClientFactory(getMetricClient):46 - MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient. Using a dummy client for now. Ignore this if Airbyte is configured to not publish any metrics.
2022-10-07 23:20:02 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Error instantiating bean of type [io.airbyte.workers.temporal.sync.ReplicationActivityImpl]
Path Taken: new ApplicationInitializer() --> ApplicationInitializer.syncActivities --> List.syncActivities([ReplicationActivity replicationActivity],NormalizationActivity normalizationActivity,DbtTransformationActivity dbtTransformationActivity,PersistStateActivity persistStateActivity,NormalizationSummaryCheckActivity normalizationSummaryCheckActivity) --> new ReplicationActivityImpl([Optional containerOrchestratorConfig],ProcessFactory processFactory,SecretsHydrator secretsHydrator,Path workspaceRoot,WorkerEnvironment workerEnvironment,LogConfigs logConfigs,String airbyteVersion,FeatureFlags featureFlags,Integer serverPort,AirbyteConfigValidator airbyteConfigValidator,TemporalUtils temporalUtils,AirbyteApiClient airbyteApiClient)
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.airbyte.workers.temporal.sync.ReplicationActivityImpl]
Path Taken: new ApplicationInitializer() --> ApplicationInitializer.syncActivities --> List.syncActivities([ReplicationActivity replicationActivity],NormalizationActivity normalizationActivity,DbtTransformationActivity dbtTransformationActivity,PersistStateActivity persistStateActivity,NormalizationSummaryCheckActivity normalizationSummaryCheckActivity) --> new ReplicationActivityImpl([Optional containerOrchestratorConfig],ProcessFactory processFactory,SecretsHydrator secretsHydrator,Path workspaceRoot,WorkerEnvironment workerEnvironment,LogConfigs logConfigs,String airbyteVersion,FeatureFlags featureFlags,Integer serverPort,AirbyteConfigValidator airbyteConfigValidator,TemporalUtils temporalUtils,AirbyteApiClient airbyteApiClient)
at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2367) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:1680) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractBeanResolutionContext.findBean(AbstractBeanResolutionContext.java:84) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractInitializableBeanDefinition.resolveOptionalBean(AbstractInitializableBeanDefinition.java:2178) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractInitializableBeanDefinition.findBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1581) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.airbyte.workers.temporal.sync.$ReplicationActivityImpl$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1297) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.airbyte.workers.config.$ActivityBeanFactory$SyncActivities4$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:1680) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractBeanResolutionContext.findBean(AbstractBeanResolutionContext.java:84) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractInitializableBeanDefinition.resolveOptionalBean(AbstractInitializableBeanDefinition.java:2178) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.AbstractInitializableBeanDefinition.findBeanForField(AbstractInitializableBeanDefinition.java:1843) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.airbyte.workers.$ApplicationInitializer$Definition.injectBean(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.airbyte.workers.$ApplicationInitializer$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2853) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.addCandidateToList(DefaultBeanContext.java:3511) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.resolveBeanRegistrations(DefaultBeanContext.java:3457) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.getBeanRegistrations(DefaultBeanContext.java:3427) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:1381) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:903) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:893) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.event.ApplicationEventPublisherFactory$2.lambda$$0(ApplicationEventPublisherFactory.java:217) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.core.util.SupplierUtil$2.initialize(SupplierUtil.java:77) ~[micronaut-core-3.7.1.jar:3.7.1]
at io.micronaut.core.util.SupplierUtil$2.get(SupplierUtil.java:72) ~[micronaut-core-3.7.1.jar:3.7.1]
at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.7.1.jar:3.7.1]
at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.7.1.jar:3.7.1]
at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.7.1.jar:3.7.1]
at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.7.1.jar:3.7.1]
at io.airbyte.workers.Application.main(Application.java:12) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
Caused by: java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
at io.airbyte.config.storage.DefaultS3ClientFactory.validate(DefaultS3ClientFactory.java:31) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
at io.airbyte.config.storage.DefaultS3ClientFactory.<init>(DefaultS3ClientFactory.java:24) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
at io.airbyte.workers.storage.S3DocumentStoreClient.s3(S3DocumentStoreClient.java:46) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.airbyte.workers.storage.StateClients.create(StateClients.java:18) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.airbyte.workers.config.ContainerOrchestratorConfigBeanFactory.kubernetesContainerOrchestratorConfig(ContainerOrchestratorConfigBeanFactory.java:48) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.airbyte.workers.config.$ContainerOrchestratorConfigBeanFactory$KubernetesContainerOrchestratorConfig0$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.7.1.jar:3.7.1]
... 68 more
Then we change value of the WORKER_ env vars to MINIO and that also resulted in the worker service to fail with a different error:
2022-10-08 05:29:48,548 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml
___ _ __ __
/ | (_)____/ /_ __ __/ /____
/ /| | / / ___/ __ \/ / / / __/ _ \
/ ___ |/ / / / /_/ / /_/ / /_/ __/
/_/ |_/_/_/ /_.___/\__, /\__/\___/
/____/
: airbyte-workers :
Micronaut (v3.7.1)
2022-10-08 05:29:51 INFO i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [k8s, cloud, ec2, control-plane]
2022-10-08 05:29:52 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-1 - Starting...
2022-10-08 05:29:53 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-1 - Start completed.
2022-10-08 05:29:54 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-2 - Starting...
2022-10-08 05:29:54 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-2 - Start completed.
2022-10-08 05:29:54 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):107 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2022-10-08 05:29:57 INFO i.a.w.c.DatabaseBeanFactory(configsDatabaseMigrationCheck):129 - Configs database configuration: 0.35.15.001 60000
2022-10-08 05:29:58 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):221 - Waiting for temporal server...
2022-10-08 05:29:58 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for namespace default to be initialized in temporal...
2022-10-08 05:30:02 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-10-08 05:30:07 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):249 - Temporal namespace default initialized!
2022-10-08 05:30:07 WARN i.a.m.l.MetricClientFactory(getMetricClient):46 - MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient. Using a dummy client for now. Ignore this if Airbyte is configured to not publish any metrics.
2022-10-08 05:30:09 INFO i.a.w.ApplicationInitializer(initializeCommonDependencies):157 - Initializing common worker dependencies.
2022-10-08 05:30:09 INFO i.a.c.EnvConfigs(getEnvOrDefault):1091 - Using default value for environment variable METRIC_CLIENT: ''
2022-10-08 05:30:09 INFO i.a.c.EnvConfigs(getEnvOrDefault):1091 - Using default value for environment variable METRIC_CLIENT: ''
2022-10-08 05:30:09 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`.
2022-10-08 05:30:09 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:34) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:48) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:164) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:151) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:163) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:136) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:59) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.7.1.jar:3.7.1]
at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.7.1.jar:3.7.1]
at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.7.1.jar:3.7.1]
at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.7.1.jar:3.7.1]
at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.7.1.jar:3.7.1]
at io.airbyte.workers.Application.main(Application.java:12) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
This second error can be explained as we configured our deployment to use custom S3 bucket and had to keep S3_MINIO_ENDPOINT
empty. On the other hand, the error above is due to this check in the code github.com/airbytehq/airbyte/blob/master/airbyte-config/config-models/src/main/java/io/airbyte/config/storage/MinioS3ClientFactory.java#L34
which requires a non-empty MINIO endpoint.
Note that the new WORKER_*
env vars and config were introduced since 0.40.6 version. In the versions before that, to store the state of the worker jobs MINIO was used and the config was controlled with parameters
STATE_STORAGE_MINIO_BUCKET_NAME=airbyte-dev-logs
STATE_STORAGE_MINIO_ENDPOINT=http://airbyte-minio-svc:9000
which were pointing to MINIO service in our config.
Note since the v0.40 the STATE_*
parameters are no longer used by the code (What are the State Storage Configuration used for? - #3 by sajarin)
When we reverted S3 configuration to default (MINIO) settings, i.e.,
S3_LOG_BUCKET=airbyte-dev-logs
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=http://airbyte-minio-svc:9000
S3_PATH_STYLE_ACCESS=true
WORKER_LOGS_STORAGE_TYPE=MINIO
WORKER_STATE_STORAGE_TYPE=MINIO
the worker process started successfully.
Separately from the issue with the worker process, cron service fails to start on any of the 3 configurations we tried. It complains about unresolved placeholder ${DATABASE_PASSWORD}
however we do set this env var in the secrets and other processes (e.g., server) successfully use it.
To sum up, we see the following issues when upgrading to the new version:
-
Configuration suggested by the doc On Kubernetes (Beta) | Airbyte Documentation is outdated with the introduction of the variables WORKER_LOGS_STORAGE_TYPE and WORKER_STATE_STORAGE_TYPE=MINIO.
-
Worker process fail to work with the custom (non-MINIO) S3 configuration
-
Cron process fails to start due to unresolved config parameter.