Issues with metrics-exporter on GKE using managed Prometheus

Summary

User encounters errors with the metrics-exporter pod in GKE when configured for managed Prometheus. The logs indicate it’s trying to connect to Datadog endpoints, resulting in a runtime exception related to port reachability.


Question

Hello!

I’m trying to get the metrics-exporter to work on GKE with managed Prometheus, I have this enabled:

  metrics:
    metricClient: "otel"
    otelCollectorEndpoint: "<http://otel-collector.opentelemetry.svc:4317>"

metrics:
  enabled: true```
However, the logs from the metrics pod are throwing those errors, it seems it is still trying to connecto to datadog endpoints. Any suggestions? Appreciate it!

```2024-12-06 01:00:12 WARN i.m.s.r.u.Loggers$Slf4JLogger(warn):299 - [3ea1e23f, L:/127.0.0.1:50344 - R:localhost/127.0.0.1:8125] An exception has been observed post termination, use DEBUG level to see the full stack: java.net.PortUnreachableException: recvAddress(..) failed: Connection refused
2024-12-06 01:00:12 WARN i.m.s.r.u.Loggers$Slf4JLogger(warn):299 - [3ea1e23f, L:/127.0.0.1:50344 - R:localhost/127.0.0.1:8125] An exception has been observed post termination, use DEBUG level to see the full stack: java.net.PortUnreachableException: recvAddress(..) failed: Connection refused
2024-12-06 01:00:22 WARN i.m.s.r.u.Loggers$Slf4JLogger(warn):299 - [3ea1e23f, L:/127.0.0.1:50344 - R:localhost/127.0.0.1:8125] An exception has been observed post termination, use DEBUG level to see the full stack: java.net.PortUnreachableException: recvAddress(..) failed: Connection refused```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1733487951665159) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['metrics-exporter', 'gke', 'prometheus', 'connection-refused', 'datadog-endpoints']
</sub>

I noticed that other users had some trouble configuring OTEL metrics last week. I’ll take a look at this next week and get back to you, <@U082UBBU412>.

awesome, thank you <@U01MMSDJGC9>!