Airbyte in Kubernetes with S3 Log Storage

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Using Airbyte in Kubernetes with S3 as log storage and encountering error 500 when configuring sources or destinations.


Question

Hello friends, does anyone use airbyte in Kubernetes using S3 as log storage? Have you ever faced any problems? When I configure it, I face error 500 when trying to enter any source or destination.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte", "kubernetes", "s3", "log-storage", "error-500", "source", "destination"]
2

<@U03FNJGED7D>, Hello again.

I’m working to deploy Airbyte on Kubernetes using EKS, but I’m having a lot of difficulty finding documentation informing about the possible values.yaml configurations.
Could you give me your values.yaml, removing all the information you don’t want to share?

3

i’d recommend just using the chart itself as a source. https://github.com/airbytehq/airbyte-platform/tree/main/charts. i usually start with the main values.yaml file, but sometimes there are values supported that aren’t documented there. my usual strategy is to start with a template for the item i’m trying to change and see if there are any explicit references to .Values.___ that are the missing clues. good luck!

4

Thank you, I’m going to need luck, you’re helping me a lot

5

yes, we do. i recall docs being incomplete, but you can get it to work

6

can you help me? According to the documentation, an error always occurs!

7

always the same error

8

if you’re able to share the relevant section of your helm values file, I can compare to ours and see if there’s something obviously off.

9

https://pastebin.com/kxLRFNMj

10

what version of the helm chart are you using? if its something recent, there’s quite a bit you’re missing. (it’s a lot simpler though - replaces all the extraEnv nonsense you used to need!)

11

take a look at this sample values.yaml - you’ve got a lot more to populate in your global.storage section https://github.com/airbytehq/airbyte-platform/blob/7eea81d81ff3c39c540f9511596a40333ec0491c/configs/airbyte.sample.yml#L27-L39

12

if you’re only trying to get logs working, then you only need to configure bucket.log I’d imagine. we only have log and state there, not all 4

13

Could you give me the relevant part of your values?

14 
  image:
    tag: ${airbyte_version}
  storage:
    type: s3
    bucket:
      log: ${s3_logs_bucket}
      state: ${state_bucket}
    s3:
      authenticationType: instanceProfile
      region: ${aws_region}```
15

the first example I sent was setup for that from the look of it.

16

i’m a user like you - i don’t know about the state of documentation.

17

When I started using Airbyte, I thought the documentation would be magic :joy:

18

I’m going to do some tests here, thank you very much, you’re helping me a lot!

19

hey, have you setup an external db? I no longer trust the documentation. If you can pass it on to me I would appreciate it

20

i’m not on the latest helm chart and I think I saw that it changed recently, so I wouldn’t use anything I have for that. I tend to use the helm chart itself as the source of truth for these sorts of questions