production: Test swh.graph workload in kubernetes

requisite:

• graph must run on rancher-node-metal05 (old "met" with sufficient memory and disks)

plan:

• Add swh/graph=true label to rancher-node-metal05
• Transfer graph dataset from maxxi to rancher-node-metal05 (zfs snapshot)
• Reference graph.i.s.o & graph-grpc.i.s.o as CNAME (for internal use purposes)
• swh/infra/ci-cd/swh-charts!486 (merged): swh-charts: Add values to start the graph deployment in kube (only prepare the volumes and start nothing)
• Configure kubernetes volumes
  • Declare in the chart the persistent volume (and start the service so proper pv gets created)
  • Manually mount the zfs dataset in that persistent volume's path mounted in the host machine
• swh-charts: Incrementally start the deployment (prepareMemoryVolume: true, startService: true)
• Allow reindexation to happen from the chart
• Investigate further issue ~> old dataset used
• Transfer most recent dataset from granet
• Deploy new graph using the most recent dataset (1 grpc, 1 rpc)
• Checks
  • grpc? ok
  • rpc? ok
• Test graph api through webapp-postgresql instance
• monitoring
  • grafana board
    • workload: https://grafana.softwareheritage.org/goto/wgG5ONiHz?orgId=1
    • nginx: https://grafana.softwareheritage.org/goto/xt9eONmNz?orgId=1
  • Add blackbox probe
• go for production deployment

Cleanup:

• Clean up old and wrong dataset from the first tryout