How can I monitor Airbyte OSS on Kubernetes using Prometheus?

Guides
,
#1

We use the prom stack and are deploying Airbyte on our Kubernetes cluster. It looks like https://github.com/airbytehq/airbyte/pull/6529/files added a framework for emitting events. Are there any metrics currently being emitted to prometheus? If there are, what metrics are there and how can we enable them?

Thanks!

2 Likes
#2

Hi @miguel-firebolt,
Airbyte does not yet exposes Prometheus metrics. We are internally working with Datadog for Airbyte Cloud. If you are using DataDog too you can set the following environment variables on your Airbyte pods:

  • PUBLISH_METRICS: true
  • DD_AGENT_HOST: your datadog agent host
  • DD_DOGSTATSD_PORT: your datadog statsd port
#3

Thanks @alafanechere! Unfortunately, we are using Prometheus instead of Datadog.

#4

We know Prometheus is an industry standard and will eventually expose prometheus metrics but I can’t share an ETA for this at the moment.

1 Like
#5

@alafanechere do i have to deploy airbyte-metric reporter for this or just having those env values in scheduler is enough?

#6

Hey @Bikram Dhoju,

do i have to deploy airbyte-metric reporter for this or just having those env values in scheduler is enough?

Do you mean you want to report metrics with Datadog? If this is what you want to do please open a new topic in the Guide section and we will try to explain how to set this up.

#7

This thread contains more details about how to setup Airbyte monitoring with Datadog

#8

Hi we are super interested with monitoring airbyte in Prometheus as we don’t have datadog.
Do you have a workaround/hack in mind to be able to have at least the metric for Job successs/failure before you release the prometheus exporter ? :slight_smile:
Thanks

#9

Hey @lucienfregosi,
I think I have good news! We recently added OpenTelemetry that can help you transfer the metrics to Prometheus and other monitoring solution. You will find more details in this documentation.

2 Likes
#10

Awesome @alafanechere i will have a look

Thanks a lot “ça régale” like we say in french

1 Like
#11

Great! Let me know if it works! If “ça roule” like we say in French.

#12

@alafanechere I tried locally with Docker, followed the tutorial steps and nothing happened …
http://localhost:9090/api/v1/label/__name__/values don’t show these metrics airbyte/OssMetricsRegistry.java at master · airbytehq/airbyte · GitHub

#13

Hey @lucienfregosi,
Which version of Airbyte are you running? I think a missing part of the puzzle can be found here.
You need to use Airbyte > 0.39.19 and deploy an airbyte-metrics service along with the other services declared in the docker-compose.
Something like:

airbyte-metrics: image: airbyte/metrics-reporter:${VERSION} container_name: airbyte-metrics environment: - METRIC_CLIENT=${METRIC_CLIENT} - OTEL_COLLECTOR_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT}

This feature is quite fresh and still not really well documented, sorry about that!

#14

Yeah @alafanechere it works way better with the metrics-reporter container :slight_smile: Thanks !
One “weird” thing is that in Prometheus the metric is called promexample_job_succeeded_by_release_stage instead of job_succeeded_by_release_stage (the pattern is for all the metrics) but it’s not a big issue …
Now I will try to get the same result in my production/kubernetes airbyte env