Migrate remaining *storage services to staging dynamic infra

Plan:

• Make storage1.staging a rancher node [2] (so we can deploy rpc services there [1])

  • swh/infra/puppet/puppet-swh-site!660 (merged): Adapt role for that node
  • swh/infra/puppet/puppet-swh-site!661 (merged): Further adaptation for ^
  • Register rancher node (cf. the azure admin registration console)
  • Unstuck agent registering (it's currently stuck for failing to pull image for some reason)
  • Add label swh/objstorage=true

• Migrate storage1.staging's objstorage service to elastic infra

  • objstorage: Adapt docker image to use uid 1005 (swhstorage to avoid permission issue)
  • Migrate objstorage service to dynamic infra (persistent volume & claim inside)
    • swh/infra/ci-cd/swh-charts!246 (merged): swh-charts: Add objstorage template which must be multi-deployment (e.g. read-only, read-write, ...)
    • Deploy changes ^ (ongoing storage1.staging is taking forever and a half to deploy it...)

• Migrate storage1.staging's *storage services to elastic infra

  • swh/infra/ci-cd/swh-charts!236 (merged): swh-charts: Deploy storage postgres to dynamic infra (with ingress)
  • swh/infra/ci-cd/swh-charts!249 (merged): Make the storage template multi-instance
  • Deploy changes
  • swh/infra/ci-cd/swh-charts!251 (merged): Migrate storage-postgresql deployment to use the http://objstorage-read-write-rpc-ingress (storage ^)
  • swh/infra/ci-cd/swh-charts!251 (merged): Migrate writing services (loader) to http://storage-postgresql-rpc-ingress
  • swh/infra/ci-cd/swh-charts!251 (merged): Deploy read-only storage instance (which uses the read-only objstorage)
  • swh/infra/ci-cd/swh-charts!251 (merged): Migrate reading services (indexer, ...) to read-only storage
  • Checks
    • webapp postgres
    • webapp staging (cassandra)
    • icinga checks (some failure due to periodic node slowness when a backup routine is triggering)
    • swh/infra/puppet/puppet-swh-site@391c8a41: Db backup was no longer relevant so it got deactivated on storage1.staging
    • icinga checks back to normal
  • Migrate swh storage create-object-reference-partitions cron to a cronjob
    • swh/infra/ci-cd/swh-charts!258 (merged): swh-charts: Add new cronjob template
    • Deploy for staging instance
    • Checks (can't really do it without actually impacting the db so we'll check next month if the cron is ok or not)

• swh/infra/puppet/puppet-swh-site!668 (merged): Decommission storage1.staging static services (gunicorn-swh-storage, gunicorn-swh-objstorage)

  • storage1.staging: clean up old services (~ systemctl disable gunicorn-swh-*storage)
  • storage1.staging: clean up old packages (~ apt remove python3-swh.*storage; apt autoremove)

[1] Specifically the pathslicing objstorage requires a specific mount point as a volume on that node (to avoid a migration of TB of data)

Refs. #4780 (closed)