Add metrics for swh-scrubber using the sql exporter
Thanks to @vlorentz for providing the initial query.
Ref. swh/devel/swh-scrubber#4684 (closed)
octocatalog-diff (patched with https://github.com/github/octocatalog-diff/issues/261#issuecomment-1334980032) output:
belvedere and db1.staging
Found host belvedere.internal.softwareheritage.org Found host db1.internal.staging.swh.network Cloning into '/tmp/swh-ocd.zVnjkR15/swh-site'... done. Switched to a new branch 'mr/swh-scrubber-metrics' branch 'mr/swh-scrubber-metrics' set up to track 'origin/mr/swh-scrubber-metrics'. Switched to a new branch 'mr/puppet-7-compat' branch 'mr/puppet-7-compat' set up to track 'origin/mr/puppet-7-compat'. Switched to branch 'production' Your branch is up to date with 'origin/production'. [production 63ac75e4] WIP: Update Puppetfile for compatibility with Puppet 7 Date: Mon Nov 28 14:17:01 2022 +0100 1 file changed, 7 insertions(+), 1 deletion(-) Switched to branch 'mr/swh-scrubber-metrics' Your branch is up to date with 'origin/mr/swh-scrubber-metrics'. [mr/swh-scrubber-metrics b0b5f9d8] WIP: Update Puppetfile for compatibility with Puppet 7 Date: Mon Nov 28 14:17:01 2022 +0100 1 file changed, 7 insertions(+), 1 deletion(-) WARN -> Branch mr/puppet-7-compat filtered out by ignore_branch_prefixes ["mr/puppet-7-compat"] WARN -> Environment "mr/swh-scrubber-metrics" contained non-word characters, correcting name to mr_swh_scrubber_metrics WARN -> Branch mr/puppet-7-compat filtered out by ignore_branch_prefixes ["mr/puppet-7-compat"] WARN -> Environment "mr/swh-scrubber-metrics" contained non-word characters, correcting name to mr_swh_scrubber_metrics Cloning into '/tmp/swh-ocd.zVnjkR15/environments/production/data/private'... done. Cloning into '/tmp/swh-ocd.zVnjkR15/environments/mr_swh_scrubber_metrics/data/private'... done. *** Running octocatalog-diff on host belvedere.internal.softwareheritage.org I, [2023-03-28T17:00:57.513543 #1506303] INFO -- : Catalogs compiled for belvedere.internal.softwareheritage.org I, [2023-03-28T17:00:57.753757 #1506303] INFO -- : Diffs computed for belvedere.internal.softwareheritage.org diff origin/production/belvedere.internal.softwareheritage.org current/belvedere.internal.softwareheritage.org ******************************************* + File[/etc/prometheus-sql-exporter/swh-scrubber.yml] => parameters => "ensure": "present" "group": "root" "mode": "0644" "notify": "Service[prometheus-sql-exporter]" "owner": "root" "content": >>> - name: swh_scrubber_coverage scope: database database: ^(swh|softwareheritage)-scrubber$ interval: '1h' help: "Software Heritage Scheduler scrubber coverage" query: | SELECT datastore.package AS datastore_package, datastore.class AS datastore_class, datastore.instance AS datastore_instance, checked_partition.object_type AS object_type, checked_partition.nb_partitions AS partitions_total, COUNT(*) AS partitions_checked_total, EXTRACT (EPOCH FROM MIN(last_date)) AS oldest_partition_check_timestamp_seconds, EXTRACT (EPOCH FROM MAX(last_date)) AS newest_partition_check_timestamp_seconds FROM checked_partition INNER JOIN datastore ON (datastore.id=checked_partition.datastore) GROUP BY 1,2,3,4,5; labels: - datastore_package - datastore_class - datastore_instance - object_type values: - partitions_total - partitions_checked_total - oldest_partition_check_timestamp_seconds - newest_partition_check_timestamp_seconds <<< ******************************************* Profile::Prometheus::Export_scrape_config[sql] => parameters => metric_relabel_configs => - [{"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scheduler_origins;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scheduler_origins_${1}"}] + [{"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scheduler_origins;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scheduler_origins_${1}"}, {"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scrubber_coverage;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scrubber_coverage_${1}"}] ******************************************* *** End octocatalog-diff on belvedere.internal.softwareheritage.org *** Running octocatalog-diff on host db1.internal.staging.swh.network I, [2023-03-28T17:01:02.390662 #1506402] INFO -- : Catalogs compiled for db1.internal.staging.swh.network I, [2023-03-28T17:01:02.738085 #1506402] INFO -- : Diffs computed for db1.internal.staging.swh.network diff origin/production/db1.internal.staging.swh.network current/db1.internal.staging.swh.network ******************************************* + File[/etc/prometheus-sql-exporter/swh-scrubber.yml] => parameters => "ensure": "present" "group": "root" "mode": "0644" "notify": "Service[prometheus-sql-exporter]" "owner": "root" "content": >>> - name: swh_scrubber_coverage scope: database database: ^(swh|softwareheritage)-scrubber$ interval: '1h' help: "Software Heritage Scheduler scrubber coverage" query: | SELECT datastore.package AS datastore_package, datastore.class AS datastore_class, datastore.instance AS datastore_instance, checked_partition.object_type AS object_type, checked_partition.nb_partitions AS partitions_total, COUNT(*) AS partitions_checked_total, EXTRACT (EPOCH FROM MIN(last_date)) AS oldest_partition_check_timestamp_seconds, EXTRACT (EPOCH FROM MAX(last_date)) AS newest_partition_check_timestamp_seconds FROM checked_partition INNER JOIN datastore ON (datastore.id=checked_partition.datastore) GROUP BY 1,2,3,4,5; labels: - datastore_package - datastore_class - datastore_instance - object_type values: - partitions_total - partitions_checked_total - oldest_partition_check_timestamp_seconds - newest_partition_check_timestamp_seconds <<< ******************************************* Profile::Prometheus::Export_scrape_config[sql] => parameters => metric_relabel_configs => - [{"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scheduler_origins;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scheduler_origins_${1}"}] + [{"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scheduler_origins;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scheduler_origins_${1}"}, {"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scrubber_coverage;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scrubber_coverage_${1}"}] ******************************************* *** End octocatalog-diff on db1.internal.staging.swh.network
Edited by Antoine R. Dumont