components/alerting: Add KEDA scaledobjects alert

Related to swh/infra/sysadm-environment#5362 (closed)

These modifications will create an alert if a scaledobject shows errors within the last 15 minutes in staging and production environment.

Helm-diff

[cluster-components] Comparing changes between branches production and scaled_object_alert...
Your branch is up to date with 'origin/production'.
[cluster-components] Generate config in production branch for cluster-components/values/admin-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/archive-production-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/archive-staging-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/default.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/gitlab-production.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/gitlab-staging.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/local-cluster.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/rancher.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/test-staging-rke2.yaml...
Your branch is up to date with 'origin/scaled_object_alert'.
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/admin-rke2.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/archive-production-rke2.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/archive-staging-rke2.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/default.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/gitlab-production.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/gitlab-staging.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/local-cluster.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/rancher.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/test-staging-rke2.yaml...


------------- diff for cluster-components/values/admin-rke2.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/admin-rke2.yaml.before, 32 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/admin-rke2.yaml.after, 32 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for cluster-components/values/archive-production-rke2.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/archive-production-rke2.yaml.before, 23 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/archive-production-rke2.yaml.after, 23 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned one difference
        |___/

spec.groups.swh-production.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/cattle-monitoring-system/swh-production.rules)
  + one list entry added:
    - alert: KEDA_Scaled_Object_Errors_In_Production
      annotations:
        description: "Scaled object {{ $labels.exported_namespace }}/{{ $labels.scaledObject }} have encountered {{ $value }} errors within the last 15 minutes."
        summary: "KEDA scaled object {{ $labels.scaledObject }} in error."
      expr: "sum by(scaledObject) (rate(keda_scaled_object_errors_total{exported_namespace!="swh-cassandra-next-version"}[15m])) > 0"
      for: 15m
      labels:
        severity: warning
        namespace: cattle-monitoring-system
    
  



------------- diff for cluster-components/values/archive-staging-rke2.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/archive-staging-rke2.yaml.before, 55 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/archive-staging-rke2.yaml.after, 55 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned one difference
        |___/

spec.groups.swh-staging.rules.rules  (monitoring.coreos.com/v1/PrometheusRule/cattle-monitoring-system/swh-staging.rules)
  + one list entry added:
    - alert: KEDA_Scaled_Object_Errors_In_Staging
      annotations:
        description: "Scaled object {{ $labels.exported_namespace }}/{{ $labels.scaledObject }} have encountered {{ $value }} errors within the last 15 minutes."
        summary: "KEDA scaled object {{ $labels.scaledObject }} in error."
      expr: "sum by(scaledObject) (rate(keda_scaled_object_errors_total{exported_namespace!="swh-cassandra-next-version"}[15m])) > 0"
      for: 15m
      labels:
        severity: warning
        namespace: cattle-monitoring-system
    
  



------------- diff for cluster-components/values/default.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/default.yaml.before, six documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/default.yaml.after, six documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for cluster-components/values/gitlab-production.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/gitlab-production.yaml.before, seven documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/gitlab-production.yaml.after, seven documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for cluster-components/values/gitlab-staging.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/gitlab-staging.yaml.before, seven documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/gitlab-staging.yaml.after, seven documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for cluster-components/values/local-cluster.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/local-cluster.yaml.before, eight documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/local-cluster.yaml.after, eight documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for cluster-components/values/rancher.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/rancher.yaml.before, seven documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/rancher.yaml.after, seven documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for cluster-components/values/test-staging-rke2.yaml -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.cluster-components.MXBdQbxx/test-staging-rke2.yaml.before, 18 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.cluster-components.MXBdQbxx/test-staging-rke2.yaml.after, 18 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/

Check the rule files

~/_swh_src/sysadm-environment/swh-charts/cluster-components (scaled_object_alert ✔) ᐅ helm template -f values.yaml -f values/archive-production-rke2.yaml alerting . | \
grep "^[[:space:]]*groups:" -A 72 | promtool check rules
Checking standard input
  SUCCESS: 7 rules found