[alertmanager] [Blocked] Update the configuration to be able to receive alert from other namespace than cattle-monitoring-system
Prometheus operator add by default a matcher on the main AlertManager(AM) rule to limit the alerts to the same namespace where the AlermanagerConfig is deployed.
In our case, the AMConfig is deployed on the cattle-monitoring-system
so the alerts must have the label namespace=cattle-monitoring-system
to match the routing.
routes:
- receiver: cattle-monitoring-system/irc-relay-config/ircrelay
group_by:
- '...'
matchers:
- namespace="cattle-monitoring-system" <--- automagically added by the operator
An option to disable this behavior was added in the operator: https://github.com/prometheus-operator/prometheus-operator/pull/5084 and release in version v0.61.0
Unfortunately, we are using the rancher-monitoring helm chart to deploy the monitoring stack and the v102.0.1
version is highly coupled with the v0.59.1
version of the operator.
We should wait for a new release of the rancher-monitoring application to be able to disable this behavior.
In the meantime, we can configure our alerts to force the namespace: example
but we lose all the default alerts declared with the monitoring stack