Summary
User reports that the Airbyte MySql connector is still using S3 for logging despite disabling S3 logs in the helm chart. They seek clarification on whether this is intended behavior or a bug.
Question
Hi Team,
I have been using Airbyte MySql v3.3.0 connector as a source and Airbyte BQ v2.3.25 as a destination connector. Deploying using helm chart v0.50.11 in K8s cluster of GCP env. I noticed, it’s using s3 bucket even I disabled the s3 logs from chart. Please let me know if it is a intended behaviour or a bug.
Publishing to S3 (bucket=airbyte-dev-logs; key=job-logging/workspace/1728/0/logs.log/20231226134520_airbyte-worker-6cbc957bb6-zhmt8_93d8bd1b303e4471afc98799113903ba):
2023-12-26 13:45:20 DEBUG c.a.h.AmazonHttpClient$RequestExecutor(executeOneRequest):1288 - Sending Request: PUT <http://airbyte-minio-svc:9000> /airbyte-dev-logs/job-logging/workspace/1728/0/logs.log/20231226134520_airbyte-worker-6cbc957bb6-zhmt8_93d8bd1b303e4471afc98799113903ba
2023-12-26 13:45:20 DEBUG c.a.l.CommonsLog(debug):33 - AWS4 Canonical Request: '"PUT
/airbyte-dev-logs/job-logging/workspace/1728/0/logs.log/20231226134520_airbyte-worker-6cbc957bb6-zhmt8_93d8bd1b303e4471afc98799113903ba
amz-sdk-invocation-id:e2e070f8-1f84-5dd4-4a2a-c8c22691c8f4
amz-sdk-request:attempt=1;max=4
amz-sdk-retry:0/0/500
content-length:1218377
content-md5:vHQp45Vq7ghhLEOCYOPLZg==
content-type:application/octet-stream
host:airbyte-minio-svc:9000
user-agent:aws-sdk-java/1.12.477 Linux/5.10.176+ OpenJDK_64-Bit_Server_VM/19.0.2+7-FR java/19.0.2 kotlin/1.8.20 vendor/Amazon.com_Inc. cfg/retry-mode/legacy
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20231226T134520Z
x-amz-decoded-content-length:1217392
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD"
2023-12-26 13:45:20 DEBUG c.a.l.CommonsLog(debug):33 - AWS4 String to Sign: '"AWS4-HMAC-SHA256
20231226T134520Z
20231226/us-east-1/s3/aws4_request
7baee23f4a2fb91ba8bd4677abd372741ebd9400d084ee9e7d55944a6031070e"
2023-12-26 13:45:20 DEBUG c.a.r.ClockSkewAdjuster(getServerDate):179 - Reported server date (from 'Date' header): Tue, 26 Dec 2023 13:45:20 GMT
2023-12-26 13:45:20 DEBUG c.a.h.r.AwsResponseHandlerAdapter(handle):86 - Received successful response: 200, AWS Request ID: 17A465A590CA74A1
2023-12-26 13:45:20 DEBUG c.a.h.r.AwsResponseHandlerAdapter(logHeaderRequestId):138 - x-amzn-RequestId: not available
2023-12-26 13:45:20 DEBUG c.a.h.r.AwsResponseHandlerAdapter(logResponseRequestId):158 - AWS Request ID: 17A465A590CA74A1
2023-12-26 13:45:20 DEBUG c.a.h.r.AwsResponseHandlerAdapter(logExtendedRequestId):179 - AWS Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
2023-12-26 13:45:20 DEBUG i.m.h.s.n.RoutingInBoundHandler(channelRead0):296 - Request GET /
2023-12-26 13:45:20 DEBUG i.m.h.s.n.RoutingInBoundHandler(syncWriteAndFlushNettyResponse):1323 - Response 200 - GET /
2023-12-26 13:45:20 DEBUG i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestReceivedEvent[source=GET /]
2023-12-26 13:45:20 DEBUG i.m.c.e.ApplicationEventPublisherFactory$2(publishEvent):227 - Publishing event: io.micronaut.http.context.event.HttpRequestTerminatedEvent[source=GET /] ```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1703600097519859) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
['airbyte', 'mysql-connector', 'bigquery-connector', 'helm', 'kubernetes', 's3-logs', 'gcp']
</sub>