Setting Environment Variables for S3 Destination Pod

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is inquiring about how to set environment variables for a job pod using the S3 destination image in Airbyte, specifically to address a known bug in S3Proxy when interfacing with Azure Blob Storage.


Question

Hello,
Im trying out the S3 destination against S3Proxy on Azure Blob Storage (https://github.com/gaul/s3proxy)
There seem to be a know bug in S3Proxy which could be solved by adding a system property.

As I understood it I can add the system property by setting an environment variable. I guess that env var need to be set on the job pod which are using the s3 destination image.
How do I mount env vars to that pod? Or do you have any other solution for this?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['s3-destination', 's3proxy', 'azure-blob-storage', 'environment-variables', 'kubernetes']
2

link to issue: https://github.com/gaul/s3proxy/issues/96

3

tried setting adding the env var to global.env_vars

    JAVA_TOOL_OPTIONS: "-Dcom.amazonaws.services.s3.disablePutObjectMD5Validation=1"```
but that didnt seem to be added to the job pod :confused:
4

solved this by creating a new connector image, inheriting from the s3 destination and added the system property as environment variable in the docker file.

5

link to issue: https://github.com/gaul/s3proxy/issues/96

6

tried setting adding the env var to global.env_vars

    JAVA_TOOL_OPTIONS: "-Dcom.amazonaws.services.s3.disablePutObjectMD5Validation=1"```
but that didnt seem to be added to the job pod :confused:
7

solved this by creating a new connector image, inheriting from the s3 destination and added the system property as environment variable in the docker file.