Worker process of Airbyte > v0.40.9 fails to start on custom S3 config

Platform, Deploy & Infra Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: EC2 AL2_x86_64 on EKS
  • Memory / Disk: you can use something like 4Gb per node
  • Deployment: EKS
  • Airbyte Version: v0.40.14
  • Source name/version: n/a
  • Destination name/version: n/a
  • Step: The issue is happening during upgrade to v0.40.14

We run Airbyte on EKS cluster. Airbyte is configured to use S3 bucket for logs as described in the guide docs.airbyte.com/deploying-airbyte/on-kubernetes/

that is

S3_LOG_BUCKET=<your_s3_bucket_to_write_logs_in>
S3_LOG_BUCKET_REGION=<your_s3_bucket_region>
S3_MINIO_ENDPOINT=
S3_PATH_STYLE_ACCESS=

This worked for us when running on Airbyte version 0.39.37

When attempting to upgrade to v0.40.9 (and v0.40.14) we encountered issue with airbyte-worker service and airbyte-cron

When configuring Airbyte deployment for v0.40.14, we kept the S3_* env settings as described above (and in the doc).

We also noticed that in v 0.40.9 (14) new env variables were introduced

WORKER_LOGS_STORAGE_TYPE=MINIO
WORKER_STATE_STORAGE_TYPE=MINIO

Since we want to use S3 bucket for logs and state storage, we set the WORKER_* env vars to S3 (value found in the code)

Deploying Airbyte to the EKS cluster resulted in airbyte-worker service going into perpetual crash loop with the log

2022-10-07 23:19:43,718 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
        : airbyte-workers :

  Micronaut (v3.7.1)

2022-10-07 23:19:46 INFO i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [k8s, cloud, ec2, control-plane]
2022-10-07 23:19:47 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-1 - Starting...
2022-10-07 23:19:48 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-1 - Start completed.
2022-10-07 23:19:48 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-2 - Starting...
2022-10-07 23:19:49 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-2 - Start completed.
2022-10-07 23:19:49 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):107 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2022-10-07 23:19:52 INFO i.a.w.c.DatabaseBeanFactory(configsDatabaseMigrationCheck):129 - Configs database configuration: 0.35.15.001 60000
2022-10-07 23:19:53 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):221 - Waiting for temporal server...
2022-10-07 23:19:53 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for namespace default to be initialized in temporal...
2022-10-07 23:19:56 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-10-07 23:20:01 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):249 - Temporal namespace default initialized!
2022-10-07 23:20:01 WARN i.a.m.l.MetricClientFactory(getMetricClient):46 - MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient. Using a dummy client for now. Ignore this if Airbyte is configured to not publish any metrics.
2022-10-07 23:20:02 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Error instantiating bean of type  [io.airbyte.workers.temporal.sync.ReplicationActivityImpl]

Path Taken: new ApplicationInitializer() --> ApplicationInitializer.syncActivities --> List.syncActivities([ReplicationActivity replicationActivity],NormalizationActivity normalizationActivity,DbtTransformationActivity dbtTransformationActivity,PersistStateActivity persistStateActivity,NormalizationSummaryCheckActivity normalizationSummaryCheckActivity) --> new ReplicationActivityImpl([Optional containerOrchestratorConfig],ProcessFactory processFactory,SecretsHydrator secretsHydrator,Path workspaceRoot,WorkerEnvironment workerEnvironment,LogConfigs logConfigs,String airbyteVersion,FeatureFlags featureFlags,Integer serverPort,AirbyteConfigValidator airbyteConfigValidator,TemporalUtils temporalUtils,AirbyteApiClient airbyteApiClient)
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.airbyte.workers.temporal.sync.ReplicationActivityImpl]

Path Taken: new ApplicationInitializer() --> ApplicationInitializer.syncActivities --> List.syncActivities([ReplicationActivity replicationActivity],NormalizationActivity normalizationActivity,DbtTransformationActivity dbtTransformationActivity,PersistStateActivity persistStateActivity,NormalizationSummaryCheckActivity normalizationSummaryCheckActivity) --> new ReplicationActivityImpl([Optional containerOrchestratorConfig],ProcessFactory processFactory,SecretsHydrator secretsHydrator,Path workspaceRoot,WorkerEnvironment workerEnvironment,LogConfigs logConfigs,String airbyteVersion,FeatureFlags featureFlags,Integer serverPort,AirbyteConfigValidator airbyteConfigValidator,TemporalUtils temporalUtils,AirbyteApiClient airbyteApiClient)
    at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2367) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:1680) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractBeanResolutionContext.findBean(AbstractBeanResolutionContext.java:84) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractInitializableBeanDefinition.resolveOptionalBean(AbstractInitializableBeanDefinition.java:2178) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractInitializableBeanDefinition.findBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1581) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.airbyte.workers.temporal.sync.$ReplicationActivityImpl$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1297) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.airbyte.workers.config.$ActivityBeanFactory$SyncActivities4$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:1680) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractBeanResolutionContext.findBean(AbstractBeanResolutionContext.java:84) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractInitializableBeanDefinition.resolveOptionalBean(AbstractInitializableBeanDefinition.java:2178) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.AbstractInitializableBeanDefinition.findBeanForField(AbstractInitializableBeanDefinition.java:1843) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.airbyte.workers.$ApplicationInitializer$Definition.injectBean(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.airbyte.workers.$ApplicationInitializer$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2853) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.addCandidateToList(DefaultBeanContext.java:3511) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.resolveBeanRegistrations(DefaultBeanContext.java:3457) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.getBeanRegistrations(DefaultBeanContext.java:3427) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:1381) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:903) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:893) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.lambda$$0(ApplicationEventPublisherFactory.java:217) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.core.util.SupplierUtil$2.initialize(SupplierUtil.java:77) ~[micronaut-core-3.7.1.jar:3.7.1]
    at io.micronaut.core.util.SupplierUtil$2.get(SupplierUtil.java:72) ~[micronaut-core-3.7.1.jar:3.7.1]
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
    at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
    at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
    at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.7.1.jar:3.7.1]
    at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
    at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.7.1.jar:3.7.1]
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.7.1.jar:3.7.1]
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.7.1.jar:3.7.1]
    at io.airbyte.workers.Application.main(Application.java:12) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
Caused by: java.lang.IllegalArgumentException
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
    at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
    at io.airbyte.config.storage.DefaultS3ClientFactory.validate(DefaultS3ClientFactory.java:31) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
    at io.airbyte.config.storage.DefaultS3ClientFactory.<init>(DefaultS3ClientFactory.java:24) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
    at io.airbyte.workers.storage.S3DocumentStoreClient.s3(S3DocumentStoreClient.java:46) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.airbyte.workers.storage.StateClients.create(StateClients.java:18) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.airbyte.workers.config.ContainerOrchestratorConfigBeanFactory.kubernetesContainerOrchestratorConfig(ContainerOrchestratorConfigBeanFactory.java:48) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.airbyte.workers.config.$ContainerOrchestratorConfigBeanFactory$KubernetesContainerOrchestratorConfig0$Definition.build(Unknown Source) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.7.1.jar:3.7.1]
    ... 68 more

Then we change value of the WORKER_ env vars to MINIO and that also resulted in the worker service to fail with a different error:

2022-10-08 05:29:48,548 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
        : airbyte-workers :

  Micronaut (v3.7.1)

2022-10-08 05:29:51 INFO i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [k8s, cloud, ec2, control-plane]
2022-10-08 05:29:52 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-1 - Starting...
2022-10-08 05:29:53 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-1 - Start completed.
2022-10-08 05:29:54 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-2 - Starting...
2022-10-08 05:29:54 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-2 - Start completed.
2022-10-08 05:29:54 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):107 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2022-10-08 05:29:57 INFO i.a.w.c.DatabaseBeanFactory(configsDatabaseMigrationCheck):129 - Configs database configuration: 0.35.15.001 60000
2022-10-08 05:29:58 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):221 - Waiting for temporal server...
2022-10-08 05:29:58 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for namespace default to be initialized in temporal...
2022-10-08 05:30:02 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-10-08 05:30:07 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):249 - Temporal namespace default initialized!
2022-10-08 05:30:07 WARN i.a.m.l.MetricClientFactory(getMetricClient):46 - MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient. Using a dummy client for now. Ignore this if Airbyte is configured to not publish any metrics.
2022-10-08 05:30:09 INFO i.a.w.ApplicationInitializer(initializeCommonDependencies):157 - Initializing common worker dependencies.
2022-10-08 05:30:09 INFO i.a.c.EnvConfigs(getEnvOrDefault):1091 - Using default value for environment variable METRIC_CLIENT: ''
2022-10-08 05:30:09 INFO i.a.c.EnvConfigs(getEnvOrDefault):1091 - Using default value for environment variable METRIC_CLIENT: ''
2022-10-08 05:30:09 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`. 
2022-10-08 05:30:09 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
    at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:34) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
    at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
    at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:48) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
    at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:164) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
    at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:151) ~[io.airbyte.airbyte-config-config-models-0.40.14.jar:?]
    at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:163) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:136) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:59) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
    at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.7.1.jar:3.7.1]
    at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
    at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
    at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
    at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1]
    at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.7.1.jar:3.7.1]
    at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
    at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.7.1.jar:3.7.1]
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.7.1.jar:3.7.1]
    at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.7.1.jar:3.7.1]
    at io.airbyte.workers.Application.main(Application.java:12) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]

This second error can be explained as we configured our deployment to use custom S3 bucket and had to keep S3_MINIO_ENDPOINT empty. On the other hand, the error above is due to this check in the code github.com/airbytehq/airbyte/blob/master/airbyte-config/config-models/src/main/java/io/airbyte/config/storage/MinioS3ClientFactory.java#L34 which requires a non-empty MINIO endpoint.

Note that the new WORKER_* env vars and config were introduced since 0.40.6 version. In the versions before that, to store the state of the worker jobs MINIO was used and the config was controlled with parameters

STATE_STORAGE_MINIO_BUCKET_NAME=airbyte-dev-logs
STATE_STORAGE_MINIO_ENDPOINT=http://airbyte-minio-svc:9000

which were pointing to MINIO service in our config.

Note since the v0.40 the STATE_* parameters are no longer used by the code (What are the State Storage Configuration used for? - #3 by sajarin)

When we reverted S3 configuration to default (MINIO) settings, i.e.,

S3_LOG_BUCKET=airbyte-dev-logs
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=http://airbyte-minio-svc:9000
S3_PATH_STYLE_ACCESS=true

WORKER_LOGS_STORAGE_TYPE=MINIO
WORKER_STATE_STORAGE_TYPE=MINIO

the worker process started successfully.

Separately from the issue with the worker process, cron service fails to start on any of the 3 configurations we tried. It complains about unresolved placeholder ${DATABASE_PASSWORD} however we do set this env var in the secrets and other processes (e.g., server) successfully use it.

To sum up, we see the following issues when upgrading to the new version:

  1. Configuration suggested by the doc (Deprecated) Deploy Airbyte on Kubernetes using Kustomize | Airbyte Documentation is outdated with the introduction of the variables WORKER_LOGS_STORAGE_TYPE and WORKER_STATE_STORAGE_TYPE=MINIO.

  2. Worker process fail to work with the custom (non-MINIO) S3 configuration

  3. Cron process fails to start due to unresolved config parameter.

1 Like
2

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

1 Like
3

+1

I’m getting the exact same error with a very similar deployment on AWS EKS, using custom S3 logging. This was working with our previous version of Airbyte, v0.38.3-alpha, and started failing after the upgrade to v0.40.14. A resolution that does not involve reverting back to MINIO would be well appreciated here.

1 Like
4

Hey there, I previously created an issue requesting better documentation on these config options, please add a thumbs up and comment any other info you’d like to add: https://github.com/airbytehq/airbyte/issues/17649

Are you deploying with helm or kustomize?

I am seeing here that S3 should be an acceptable config option for WORKER_STATE_STORAGE_TYPE:
https://github.com/airbytehq/airbyte/blob/master/airbyte-workers/src/main/java/io/airbyte/workers/config/CloudStorageBeanFactory.java#L84

Created https://github.com/airbytehq/airbyte/issues/18016 to track this issue

5

Hello, I see that there has been some updates.

Please check to make sure you have these envs filled out (example in Helm): https://github.com/airbytehq/airbyte/blob/master/charts/airbyte/values.yaml#L23

state:
## state.storage.type Determines which state storage will be utilized. One of "MINIO", "S3" or "GCS"
storage:
type: "S3"
logs:
## logs.accessKey.password Logs Access Key
## logs.accessKey.existingSecret
## logs.accessKey.existingSecretKey
accessKey:
password: ""
existingSecret: ""
existingSecretKey: ""
## logs.secretKey.password Logs Secret Key
## logs.secretKey.existingSecret
## logs.secretKey.existingSecretKey
secretKey:
password: ""
existingSecret: ""
existingSecretKey: ""
## logs.storage.type Determines which log storage  will be utilized.  One of "MINIO", "S3" or "GCS"
##                   Used in conjunction with logs.minio.*, logs.s3.* or logs.gcs.*
storage:
  type: "s3"

##  logs.minio.enabled Switch to enable or disable the Minio helm chart
minio:
  enabled: false

##  logs.externalMinio.enabled Switch to enable or disable an external Minio instance
##  logs.externalMinio.host External Minio Host
##  logs.externalMinio.port External Minio Port
##  logs.externalMinio.endpoint Fully qualified hostname for s3-compatible storage
externalMinio:
  enabled: false
  host: localhost
  port: 9000

##  logs.s3.enabled Switch to enable or disable custom S3 Log location
##  logs.s3.bucket Bucket name where logs should be stored
##  logs.s3.bucketRegion Region of the bucket (must be empty if using minio)
s3:
  enabled: false
  bucket: airbyte-dev-logs
  bucketRegion: ""

6

Are you deploying with helm or kustomize?

We are deploying it with kustomize - I provided link above to the airbyte documentation which discusses the environment variables in context of the kustomize config files. Unfortunately, the example with Helm variables would not apply to us. Could you provide an example with kustomize configs here https://github.com/airbytehq/airbyte/tree/master/kube/overlays/stable. Thank you!

2 Likes
7

I’m in the same situation and would like to know how to get this working with kustomize.

2 Likes
8

Hey all!

I was having the same issue. I’m using helm and am not super familiar with kustomize, but hopefully this helps. I had to set a couple more values in my values.yaml file to get it to work.

global:
  # ...
  logs:
    accessKey:
      password: <access_key_id>
      # Downstream charts don't use the secret created by the password above, so we need to pass in the secret info ourselves
      existingSecret: <helm_release_name>-airbyte-secrets
      existingSecretKey: AWS_ACCESS_KEY_ID
    secretKey:
      password: <secret_access_key>
      # Downstream charts don't use the secret created by the password above, so we need to pass in the secret info ourselves
      existingSecret: <helm_release_name>-airbyte-secrets
      existingSecretKey: AWS_SECRET_ACCESS_KEY

Dug in to the code and found out that basically the airbyte-worker and airbyte-server deployment.yaml files only set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables if the existingSecret and existingSecretKey are set, or if minio or externalMinio is enabled. There’s nothing there if I’m just passing in the password myself.

For your situation, I assume the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY aren’t being set properly on the worker/server for some reason. Hope that helps!

1 Like
9

We’re also attempting to upgrade to 0.40.22 with kustomize and run into the exact same problem with the worker as stated here. We’ve been using S3 for logging instead of Minio as well.

What should be the course of action here? Stay stuck to a version before the WORKER_* vars were introduced like 0.40.6? @sh4sh any clue?

1 Like
10

Hello Oleg Gusak, it’s been a while without an update from us. Are you still having problems or did you find a solution?

11

I am also stuck on this same problem.
Is there any update on any solution?

12

This is still an issue with using kustomization overlays for version 0.40.26. Oddly, the helm chart works correctly (for this, there are other things that are broken which is why I’m trying kustomization) so there’s probably a workaround.

13

This is still an issue with using kustomization overlays for version 0.40.26. Oddly, the helm chart works correctly (for this, there are other things that are broken which is why I’m trying kustomization) so there’s probably a workaround.

[Discourse post]

14

I have confirmed a workaround to get this fixed in version 0.40.26. In order to configure S3 logs correctly using the kustomization overlays, you need to follow the instructions found here as well as set WORKER_LOGS_STORAGE_TYPE=S3. Note that WORKER_STATE_STORAGE_TYPE needs to remain unchanged.

15

We are using Kustomize and our airbyte version is 0.40.23. The issue we are seeing is that we failed to set custom s3 as a state storage bucket. The workaround right now is to turn on mino back just for state information.

I put a fix before based on the limited knowledge I have.

16

Hi - I’m still having some trouble with this and wondered if you could confirm your set up

Env overlay

S3_LOG_BUCKET=<your_s3_bucket_to_write_logs_in>
S3_LOG_BUCKET_REGION=<your_s3_bucket_region>
# Set this to empty.
S3_MINIO_ENDPOINT=
# Set this to empty.
S3_PATH_STYLE_ACCESS=
WORKER_LOGS_STORAGE_TYPE=S3
# leave as is, for me, defaults to MINIO
# WORKER_STATE_STORAGE_TYPE=

Secrets overlay

AWS_ACCESS_KEY_ID=<your_aws_access_key_id>
AWS_SECRET_ACCESS_KEY=<your_aws_secret_access_key>

And that’s it? I’ve tried this on v0.40.28 and v0.40.26 but I’m still getting the same issue as the original post.

17

Thanks @rcheatham-q - your suggestion to set vars as

WORKER_LOGS_STORAGE_TYPE=S3
WORKER_STATE_STORAGE_TYPE=MINIO

worked for us too.

18

anyone has figured out this for GCS logs?
I’m not convinced that I should put minio related values if I have only gcs logs activated

19

Yes; we encountered a similar problem with GCS.

These configuration changes solved the issue for us (note that we are using the k8s manifests directly, not the helm chart):

  1. In .env, the env var GCS_LOG_BUCKET needs to be set to the log bucket and the additional variable called STATE_STORAGE_GCS_BUCKET_NAME needs to be set to the state storage bucket. As far as I can tell, STATE_STORAGE_GCS_BUCKET_NAME isn’t documented, but you can see that it is part of the GCS configuration block for the workers: airbyte/application.yml at 7676af5f5fb53542ebaff18a415f9c89db417055 · airbytehq/airbyte · GitHub . The Minio/S3 variables for us are mostly nulled out, so the config variables for logs and storage largely look like so:
# S3/Minio Log Configuration
S3_LOG_BUCKET=
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=
S3_PATH_STYLE_ACCESS=

# GCS Log Configuration
GCS_LOG_BUCKET=<log bucket>
STATE_STORAGE_GCS_BUCKET_NAME=<state bucket>

# State Storage Configuration
STATE_STORAGE_MINIO_BUCKET_NAME=
STATE_STORAGE_MINIO_ENDPOINT=

# Cloud Storage Configuration
WORKER_LOGS_STORAGE_TYPE=gcs
WORKER_STATE_STORAGE_TYPE=gcs
  1. Secondly, the manifests for the workers need to be modified to actually pass the GCS state bucket variables as they currently do not. In the airbyte-worker deployment (airbyte/worker.yaml at master · airbytehq/airbyte · GitHub), we added the following vars (note that GOOGLE_APPLICATION_CREDENTIALS are reused here, but it is probably better to have a separate SA credentials for writing state):
- name: STATE_STORAGE_GCS_BUCKET_NAME
  valueFrom:
    configMapKeyRef:
      name: airbyte-env
      key: STATE_STORAGE_GCS_BUCKET_NAME
- name: STATE_STORAGE_GCS_APPLICATION_CREDENTIALS
  valueFrom:
    secretKeyRef:
      name: airbyte-secrets
      key: GOOGLE_APPLICATION_CREDENTIALS

Hope this helps.

20

thanks a lot ! it works !