Summary
Orchestrator-repl pod in Airbyte on Kubernetes is throwing an error during sync from MySQL to S3 despite setting service account and AWS keys on the connector through UI. Orchestrator seems to be ignoring the service account that is working on the rest of the stack. Logs indicate orchestrator is unable to publish logs.
Question
Hello! We have installed airbyte on Kubernetes, using the helmchart v0.50.35. All was running fine, but when we trigger a sync mysql -> s3, the pod orchestrator-repl is throwing an error. Despite we set the service account and even AWS keys on the connector through UI. It’s like the orchestrator is just ignoring the service account that is already working on the rest of the stack. I think, from the attached logs, the orchestrator is not been able to publish the logs. Please advise.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["error", "orchestrator-repl", "sync", "mysql", "s3", "kubernetes", "helmchart", "service-account", "AWS-keys", "logs"]
Logs from crashing orchestrator-repl
2024-01-03 16:31:18 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable AWS_ACCESS_KEY_ID: ''
2024-01-03 16:31:18 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable AWS_SECRET_ACCESS_KEY: ''
2024-01-03 16:31:18 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable STATE_STORAGE_S3_ACCESS_KEY: ''
2024-01-03 16:31:18 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable STATE_STORAGE_S3_SECRET_ACCESS_KEY: ''
2024-01-03 16:31:18 INFO i.a.c.EventListeners(setLogging):81 - started logging
2024-01-03 16:31:18,579 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml
Using cache monitor: TimePeriodBasedBufferMonitor(periodInSeconds: 60)
Registering AWS S3 publish helper -> S3 configuration (airbyte-logs-awsregion:job-logging/workspace/3/0/logs.log in region aws-region; compressed: false)
Log4j2Appender says: settings env vars
2024-01-03 16:31:18 INFO i.a.c.EventListeners(setEnvVars):67 - settings env vars
Collecting content into /tmp/toBePublished16580814654487246246.tmp before uploading.
Log4j2Appender says: Startup completed in 4128ms. Server Running: <http://orchestrator-repl-job-3-attempt-0:9000>
2024-01-03 16:31:19 INFO i.m.r.Micronaut(lambda$start$2):98 - Startup completed in 4128ms. Server Running: <http://orchestrator-repl-job-3-attempt-0:9000>
Log4j2Appender says: path /flags does not exist, will return default flag values
2024-01-03 16:31:19 INFO i.a.f.ConfigFileClient(<init>):96 - path /flags does not exist, will return default flag values
Publishing to S3 (bucket=airbyte-logs-awsregion; key=job-logging/workspace/3/0/logs.log/20240103163119_orchestrator-repl-job-3-attempt-0_c97dbcf406ce4edb8a81796850ff043d):
2024-01-03 16:31:20 WARN c.a.l.CommonsLog(warn):113 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant.If you are using Java 9+, you will need to include javax.xml.bind:jaxb-api as a dependency.
java.lang.RuntimeException: Cannot publish to S3: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 2AE3WQHX0NJ1ZKZC; S3 Extended Request ID: A7PcQBOJF6DXcoGc68My2RVFDwckXhJ/2HYz9R3tmn2aoLAOSggxZko2ONmRqWfC0hoS7/9/1LU=; Proxy: null)```
orchestrator-repl pod environment
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: aws-region
AWS_REGION: aws-region
AWS_ROLE_ARN: arn:aws:iam::XXXXXXXXXXX:role/airbyte-service-development
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/config from airbyte-config (rw)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-scdq4 (ro)
Containers:
main:
Container ID: <containerd://09f2bce031a90aff2ec6dd52feae3b734ee1abff3913570ffa35a7fcab16390>c
Image: airbyte/container-orchestrator:0.50.20
Image ID: <http://docker.io/airbyte/container-orchestrator@sha256:deff9ab48add45985aba7e5a1d4c0c770568414201c47204638fe559b80cc545|docker.io/airbyte/container-orchestrator@sha256:deff9ab48add45985aba7e5a1d4c0c770568414201c47204638fe559b80cc545>
Ports: 9878/TCP, 9877/TCP, 9880/TCP, 9000/TCP, 9879/TCP, 9000/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 03 Jan 2024 10:41:07 -0600
Finished: Wed, 03 Jan 2024 10:41:17 -0600
Ready: False
Restart Count: 0
Environment:
PUBLISH_METRICS: false
ACCEPTANCE_TEST_ENABLED: false
STATE_STORAGE_MINIO_ACCESS_KEY: minio
AIRBYTE_API_AUTH_HEADER_NAME:
JAVA_OPTS:
JOB_KUBE_NAMESPACE: airbyte
S3_MINIO_ENDPOINT:
DD_INTEGRATION_GOOGLE_HTTP_CLIENT_ENABLED: false
MICRONAUT_ENVIRONMENTS: control-plane
TRACKING_STRATEGY: segment
APPLY_FIELD_SELECTION: false
JOB_MAIN_CONTAINER_MEMORY_REQUEST:
DD_SERVICE: airbyte-container-orchestrator
WORKSPACE_ROOT: /workspace
SEGMENT_WRITE_KEY: 7UDdp5K55CyiGgsauOr2pNNujGvmhaeu
STATE_STORAGE_MINIO_ENDPOINT:
STATE_STORAGE_MINIO_BUCKET_NAME: state-storage
LOG_LEVEL: INFO
INTERNAL_API_HOST: airbyte-airbyte-server-svc:8001
DD_DOGSTATSD_PORT:
S3_LOG_BUCKET_REGION: aws-region
STATE_STORAGE_S3_BUCKET_NAME: airbyte-logs-awsregion
AUTO_DETECT_SCHEMA: true
GCS_LOG_BUCKET:
SOCAT_KUBE_CPU_REQUEST: 0.1
DD_INTEGRATION_NETTY_ENABLED: false
AIRBYTE_API_AUTH_HEADER_VALUE:
FIELD_SELECTION_WORKSPACES:
LOCAL_ROOT: /tmp/airbyte_local
WORKSPACE_DOCKER_MOUNT: workspace
JOB_MAIN_CONTAINER_CPU_REQUEST:
AIRBYTE_VERSION: 0.50.20
S3_LOG_BUCKET: airbyte-logs-awsregion
JOB_MAIN_CONTAINER_CPU_LIMIT:
STATE_STORAGE_S3_REGION: aws-region
GOOGLE_APPLICATION_CREDENTIALS:
WORKER_ENVIRONMENT: KUBERNETES
METRIC_CLIENT:
STATE_STORAGE_MINIO_SECRET_ACCESS_KEY: minio123
DATA_PLANE_SERVICE_ACCOUNT_EMAIL:
FEATURE_FLAG_CLIENT:
DD_INTEGRATION_NETTY_4_1_ENABLED: false
DD_INTEGRATION_HTTPURLCONNECTION_ENABLED: false
DD_INTEGRATION_URLCONNECTION_ENABLED: false
JOB_MAIN_CONTAINER_MEMORY_LIMIT:
DD_AGENT_HOST:
LAUNCHDARKLY_KEY:
DD_INTEGRATION_GRPC_ENABLED: false
DD_INTEGRATION_GRPC_CLIENT_ENABLED: false
S3_PATH_STYLE_ACCESS:
SOCAT_KUBE_CPU_LIMIT: 2.0
DATA_PLANE_SERVICE_ACCOUNT_CREDENTIALS_PATH:
OTEL_COLLECTOR_ENDPOINT:
DD_INTEGRATION_GRPC_SERVER_ENABLED: false
CONTROL_PLANE_AUTH_ENDPOINT:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: aws-region
AWS_REGION: aws-region
AWS_ROLE_ARN: arn:aws:iam::XXXXXXXXXXX:role/airbyte-service-development
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/config from airbyte-config (rw)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-scdq4 (ro)```
Does that container use the correct service account with the role mapping annotation?
fwiw; i always used the AWS_ACCESS_KEY directly for airbyte logs — I don’t think I was able able to get it to work early on; although they seems to have fixed it some months ago; certainly fixed by the version you are on. https://github.com/airbytehq/airbyte-platform/commit/37ba07b5001f825abaa2191422be7b04f72fff4d
and if its working for other components; chances are they are maybe not using the correct service account; could be a bug
Yes, I see the container is having the right SA, with the role mapping annotation. I am also thinking this could be a bug 
So maybe as a workaround I can overwrite the AWS_ACCESS_KEY in that container?
After I have overwritten the AWS Keys on the orchestrator-repl pod, now I was having a different error.
Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods \"destination-s3-write-6-1-xwqdz\" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount \"airbyte-admin\" not found
So it seems the service account must be named airbyte-admin. Despite I have set a custom name through global.serviceAccountName and global.serviceAccount.name.
After I updated the serviceAccount.name to airbyte-admin everything worked like a charm!
Now orchestrator-repl is using the right service account/role instead of static AWS keys!