Statsd exporter raises a 500 error

In reaction of this alertmanager alert:

<s​whprombot> Alert WARNING resolved - staging/archive-staging-rke2 - TargetDown - 100% of the prometheus-statsd-exporter/prometheus-statsd-exporter targets in swh-cassandra namespace are down.

swh@swh-toolbox-85fd4c7d87-mkcf6:~$ curl -v http://10.42.151.53:9102/metrics
*   Trying 10.42.151.53:9102...
* Connected to 10.42.151.53 (10.42.151.53) port 9102 (#0)
> GET /metrics HTTP/1.1
> Host: 10.42.151.53:9102
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 24 May 2024 13:31:16 GMT
< Content-Length: 585
<
An error has occurred while serving metrics:

6 error(s) occurred:
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_batch_duration_seconds"

EDIT: It seems the issue is not present in production The production has the same issue in the swh-cassandra namespace:

swh@swh-toolbox-6876cf865f-6trqr:~$ curl http://10.42.52.109:9102/metrics
An error has occurred while serving metrics:

10 error(s) occurred:
* "__hosts_" is not a valid label name for metric "swh_scrubber_objects_hashed_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_batch_duration_seconds"
* "__hosts_" is not a valid label name for metric "swh_scrubber_batch_duration_seconds"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_missing_object_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_hash_mismatch_total"
* "__hosts_" is not a valid label name for metric "swh_scrubber_batch_duration_seconds_error_count"