[terraform/rancher] Migrate monitoring app deployment from terraform to swh-chart
Deploying the monitoring app from terraform often generate errors and force to manually deploy a cluster in several steps.
For example:
terraform apply /tmp/terraform.out 10:49:43
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring]
module.rancher-node-test-rke2-worker1.proxmox_vm_qemu.node: Destroying... [id=uffizi/qemu/146]
module.rancher-node-test-rke2-worker2.proxmox_vm_qemu.node: Destroying... [id=uffizi/qemu/147]
module.rancher-node-test-rke2-mgmt1.proxmox_vm_qemu.node: Destroying... [id=uffizi/qemu/143]
module.rancher-node-test-rke2-mgmt1.proxmox_vm_qemu.node: Destruction complete after 6s
module.rancher-node-test-rke2-worker1.proxmox_vm_qemu.node: Destruction complete after 8s
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 10s elapsed]
module.rancher-node-test-rke2-worker2.proxmox_vm_qemu.node: Still destroying... [id=uffizi/qemu/147, 10s elapsed]
module.rancher-node-test-rke2-worker2.proxmox_vm_qemu.node: Destruction complete after 10s
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 20s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 30s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 40s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 50s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 1m0s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 1m10s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 1m20s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 1m30s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 1m40s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 1m50s elapsed]
rancher2_app_v2.archive-test-rke2-rancher-monitoring: Still destroying... [id=c-m-v4vn2gdh.cattle-monitoring-system/rancher-monitoring, 2m0s elapsed]
╷
│ Warning: Applied changes may be incomplete
│
│ The plan was created with the -target option in effect, so some changes requested in the configuration may have been ignored and the output values may not be fully updated. Run the following command to verify that no other changes are pending:
│ terraform plan
│
│ Note that the -target option is not suitable for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an error message.
╵
╷
│ Error: Getting App V2: Timeout getting Catalog V2 Client at cluster ID c-m-v4vn2gdh: Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [lost connection to cluster: failed to find Session for client stv-cluster-c-m-v4vn2gdh] from [https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-v4vn2gdh/v1]
This is due of dependencies issues, the nodes are removed from the cluster before the app is removed. Rancher doesn't allow any actions after the etcd node is removed.
The same kind of issue exists during the creation. The app can't be deployed before the cluster is in a valid state, which can take a long time. And terraform fails in timeout.