counters: Configure staging and webapp1 to use swh-counters

• Move the swh-counters's history configuration to a top level configuration as it need to be deployed on all the environments
• Configure staging's webapp and webapp1.production to use swh-counters as the source of the counters
• Add a history file resulting of the merge of the old history-counters-munin.json and prometheus data issued from the sql_swh_archive_object_count collection until 2021-04-15
• Schedule the regularly refresh of the history file in production. it will be initially used by webapp1 only

Related to T3250

Test Plan

• Moma:

*******************************************
  File[/etc/softwareheritage/web/web.yml] =>
   parameters =>
     content =>
      @@ -17,4 +17,8 @@
         cls: remote
         url: http://saatchi.internal.softwareheritage.org:5008/
      +counters:
      +  cls: remote
      +  url: http://counters1.internal.softwareheritage.org:5011/
      +counters_backend: swh-storage        <------------------------ this one is important
       log_dir: "/var/log/softwareheritage/webapp"
       secret_key: swh::deploy::webapp::conf::secret_key
*******************************************
*** End octocatalog-diff on moma.softwareheritage.org

• webapp1.production

diff origin/production/webapp1.internal.softwareheritage.org current/webapp1.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/web/web.yml] =>
   parameters =>
     content =>
      @@ -17,4 +17,8 @@
         cls: remote
         url: http://saatchi.internal.softwareheritage.org:5008/
      +counters:
      +  cls: remote
      +  url: http://counters1.internal.softwareheritage.org:5011/
      +counters_backend: swh-counters
       log_dir: "/var/log/softwareheritage/webapp"
       secret_key: swh::deploy::webapp::conf::secret_key
      @@ -81,5 +85,5 @@
         server_url: https://auth.softwareheritage.org/auth/
         realm_name: SoftwareHeritage
      -history_counters_url: https://stats.export.softwareheritage.org/history_counters.json
      +history_counters_url: http://counters1.internal.softwareheritage.org:5011/counters_history/history.json
       allowed_hosts:
       - webapp1.internal.softwareheritage.org
*******************************************

• webapp0.staging

diff origin/production/webapp.internal.staging.swh.network current/webapp.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/web/web.yml] =>
   parameters =>
     content =>
      @@ -19,4 +19,8 @@
         args:
           url: http://scheduler0.internal.staging.swh.network:5008/
      +counters:
      +  cls: remote
      +  url: http://counters0.internal.staging.swh.network:5011/
      +counters_backend: swh-counters
       log_dir: "/var/log/softwareheritage/webapp"
       secret_key: swh::deploy::webapp::conf::secret_key
*******************************************

• counters0.staging

diff origin/production/counters0.internal.staging.swh.network current/counters0.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/counters/server.yml] =>
   parameters =>
     content =>
      @@ -9,5 +9,5 @@
         live_data_start: 1609462861
         cache_base_directory: "/srv/softwareheritage/counters"
      -  interval: 24h
      +  interval: 12h
         labels:
           environment: staging
*******************************************
*** End octocatalog-diff on counters0.internal.staging.swh.network

• counters1.production

diff origin/production/counters1.internal.softwareheritage.org current/counters1.internal.softwareheritage.org
*******************************************
+ Concat_file[profile::cron::default] =>
   parameters =>
     "group": "root",
     "mode": "0644",
     "owner": "root",
     "path": "/etc/puppet-cron.d/default",
     "tag": "profile::cron::default"
*******************************************
+ Concat_fragment[profile::cron::default::_header] =>
   parameters =>
     "content": "# Managed by puppet (module profile::cron), manual changes will ...
     "order": "00",
     "tag": "profile::cron::default",
     "target": "profile::cron::default"
*******************************************
+ Concat_fragment[profile::cron::refresh_counters_cache] =>
   parameters =>
     "content": "# Cron snippet refresh_counters_cache\n0 */4 * * * swhstorage ch...
     "order": "10",
     "tag": "profile::cron::default",
     "target": "profile::cron::default"
*******************************************
+ File[/etc/cron.d/puppet-default] =>
   parameters =>
     "ensure": "link",
     "target": "/etc/puppet-cron.d/default"
*******************************************
  File[/etc/softwareheritage/counters/server.yml] =>
   parameters =>
     content =>
      @@ -3,3 +3,12 @@
         cls: redis
         host: localhost:6379
      +history:
      +  cls: prometheus
      +  prometheus_host: pergamon.internal.softwareheritage.org
      +  prometheus_port: 9090
      +  live_data_start: 1618415227
      +  cache_base_directory: "/srv/softwareheritage/counters"
      +  interval: 12h
      +  labels:
      +    environment: production
      _
*******************************************
+ File[/srv/softwareheritage/counters/static_history.json] =>
   parameters =>
     "content": "{\"content\": [[1441065600000, 0], [1444348800000, 220307136.07]...
     "ensure": "present",
     "group": "swhstorage",
     "mode": "0755",
     "owner": "swhstorage"
*******************************************
+ Profile::Cron::D[refresh_counters_cache] =>
   parameters =>
     "command": "chronic /usr/local/bin/refresh_counters_cache.sh history.json st...
     "hour": "*/4",
     "minute": 0,
     "target": "default",
     "unique_tag": "refresh_counters_cache",
     "user": "swhstorage"
*******************************************
+ Profile::Cron::File[default] =>
   parameters =>
     "target": "default"
*******************************************
*** End octocatalog-diff on counters1.internal.softwareheritage.org

---

Migrated from D5531 (view on Phabricator)