[dynamic infra] Some loader git get stuck during the repositoring cloning step

It seems the production loader git (and only the loader git) are subject to hang during the cloning of the git repositories.

Most of them remain stuck on the clone phase:

2023-09-25T16:28:01.223213729Z loaders Total 22 (delta 2), reused 0 (delta 0), pack-reused 16
2023-09-25T16:28:07.721862059Z loaders Total 47 (delta 19), reused 25 (delta 12), pack-reused 0
2023-09-25T16:28:07.722616244Z loaders {"asctime": "2023-09-25 16:28:07,722", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/loader.py", "lineno": 381, "funcName": "fetch_data", "task_name": null, "task_id": null, "name": "swh.loader.git.loader", "levelname": "INFO", "message": "Listed 56 refs for repo https://github.com/guilledk/fc", "swh_type": "git_repo_list_refs", "swh_repo": "https://github.com/guilledk/fc", "swh_num_refs": 56}
2023-09-25T16:28:11.575682944Z loaders {"asctime": "2023-09-25 16:28:11,575", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/base.py", "lineno": 132, "funcName": "store_data", "task_name": null, "task_id": null, "name": "swh.loader.git.loader.GitLoader", "levelname": "INFO", "message": "Fetched 48 objects; 30 are new"}
2023-09-25T16:28:11.916624009Z loaders {"asctime": "2023-09-25 16:28:11,916", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateGitRepository[aeaac044-f676-4b11-9708-684c400af5b8] succeeded in 6.611545452848077s: {'status': 'eventful'}", "data": {"id": "aeaac044-f676-4b11-9708-684c400af5b8", "name": "swh.loader.git.tasks.UpdateGitRepository", "return_value": "{'status': 'eventful'}", "runtime": 6.611545452848077, "args": "()", "kwargs": "{'url': 'https://github.com/guilledk/fc', 'lister_name': 'github', 'lister_instance_name': 'github'}"}}
2023-09-25T16:28:11.919533848Z loaders {"asctime": "2023-09-25 16:28:11,919", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateGitRepository[1488f8bf-2808-4b57-a438-ea7a23bebd85] received", "data": {"id": "1488f8bf-2808-4b57-a438-ea7a23bebd85", "name": "swh.loader.git.tasks.UpdateGitRepository", "args": "()", "kwargs": "{'url': 'https://github.com/paruljain14/brokejaan', 'lister_name': 'github', 'lister_instance_name': 'github'}", "eta": null}}
2023-09-25T16:28:12.064283634Z loaders {"asctime": "2023-09-25 16:28:12,063", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 396, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.git.loader.GitLoader", "levelname": "INFO", "message": "Load origin 'https://github.com/zky001/jynew' with type 'git'"}
2023-09-25T16:28:12.932710077Z loaders Enumerating objects: 69652, done.


2023-09-25T16:10:36.325016542Z loaders {"asctime": "2023-09-25 16:10:36,324", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/loader.py", "lineno": 381, "funcName": "fetch_data", "task_name": null, "task_id": null, "name": "swh.loader.git.loader", "levelname": "INFO", "message": "Listed 2 refs for repo https://github.com/PeterZhouSZ/FluidX12", "swh_type": "git_repo_list_refs", "swh_repo": "https://github.com/PeterZhouSZ/FluidX12", "swh_num_refs": 2}
2023-09-25T16:10:36.447077603Z loaders {"asctime": "2023-09-25 16:10:36,446", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/base.py", "lineno": 132, "funcName": "store_data", "task_name": null, "task_id": null, "name": "swh.loader.git.loader.GitLoader", "levelname": "INFO", "message": "Fetched 1 objects; 1 are new"}
2023-09-25T16:10:36.498382520Z loaders {"asctime": "2023-09-25 16:10:36,498", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateGitRepository[57f19d88-e80e-41e6-81f0-4e03a2f526ff] succeeded in 1.2892945911735296s: {'status': 'eventful'}", "data": {"id": "57f19d88-e80e-41e6-81f0-4e03a2f526ff", "name": "swh.loader.git.tasks.UpdateGitRepository", "return_value": "{'status': 'eventful'}", "runtime": 1.2892945911735296, "args": "()", "kwargs": "{'url': 'https://github.com/PeterZhouSZ/FluidX12', 'lister_name': 'github', 'lister_instance_name': 'github'}"}}
2023-09-25T16:10:36.500840688Z loaders {"asctime": "2023-09-25 16:10:36,500", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateGitRepository[a8e3c9f9-e2d5-4aaf-98b8-cd87cfc2f14a] received", "data": {"id": "a8e3c9f9-e2d5-4aaf-98b8-cd87cfc2f14a", "name": "swh.loader.git.tasks.UpdateGitRepository", "args": "()", "kwargs": "{'url': 'https://github.com/eikun79/ps4-payload-guest-languages', 'lister_name': 'github', 'lister_instance_name': 'github'}", "eta": null}}
2023-09-25T16:10:36.631267517Z loaders {"asctime": "2023-09-25 16:10:36,630", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 396, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.git.loader.GitLoader", "levelname": "INFO", "message": "Load origin 'https://github.com/RVP346346/SupportArticles-docs' with type 'git'"}
2023-09-25T16:10:38.358058617Z loaders Enumerating objects: 74860, done.
2023-09-25T16:10:38.517143147Z loaders Counting objects:   0% (1/19245)
...
2023-09-25T16:10:38.358058617Z Counting objects: 100% (19245/19245), done.
2023-09-25T16:10:38.711895343Z loaders Compressing objects:   0% (1/1465)
...
Compressing objects: 100% (1465/1465)
2023-09-25T16:10:38.711895343Z Compressing objects: 100% (1465/1465), done.



2023-09-25T19:06:12.261798181Z loaders Total 8887 (delta 0), reused 0 (delta 0), pack-reused 8887
2023-09-25T19:06:12.262404521Z loaders {"asctime": "2023-09-25 19:06:12,262", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/loader.py", "lineno": 381, "funcName": "fetch_data", "task_name": null, "task_id": null, "name": "swh.loader.git.loader", "levelname": "INFO", "message": "Listed 23 refs for repo https://github.com/Longgood-Anny/MediaPipeUnityPlugin", "swh_type": "git_repo_list_refs", "swh_repo": "https://github.com/Longgood-Anny/MediaPipeUnityPlugin", "swh_num_refs": 23}
2023-09-25T19:06:23.904090217Z loaders {"asctime": "2023-09-25 19:06:23,903", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/base.py", "lineno": 132, "funcName": "store_data", "task_name": null, "task_id": null, "name": "swh.loader.git.loader.GitLoader", "levelname": "INFO", "message": "Fetched 8888 objects; 1 are new"}
2023-09-25T19:06:23.947243457Z loaders {"asctime": "2023-09-25 19:06:23,946", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateGitRepository[a62113bd-f8d8-4835-8ac0-d346c6cc9852] succeeded in 13.853397749364376s: {'status': 'eventful'}", "data": {"id": "a62113bd-f8d8-4835-8ac0-d346c6cc9852", "name": "swh.loader.git.tasks.UpdateGitRepository", "return_value": "{'status': 'eventful'}", "runtime": 13.853397749364376, "args": "()", "kwargs": "{'url': 'https://github.com/Longgood-Anny/MediaPipeUnityPlugin', 'lister_name': 'github', 'lister_instance_name': 'github'}"}}
2023-09-25T19:06:23.949716846Z loaders {"asctime": "2023-09-25 19:06:23,949", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateGitRepository[ef2e0d27-1106-4502-967d-f0f8ae3dcb86] received", "data": {"id": "ef2e0d27-1106-4502-967d-f0f8ae3dcb86", "name": "swh.loader.git.tasks.UpdateGitRepository", "args": "()", "kwargs": "{'url': 'https://github.com/Akshay-Achuthan/QuizApp', 'lister_name': 'github', 'lister_instance_name': 'github'}", "eta": null}}
2023-09-25T19:06:24.071936703Z loaders {"asctime": "2023-09-25 19:06:24,071", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 396, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.git.loader.GitLoader", "levelname": "INFO", "message": "Load origin 'https://github.com/Serhatcck/Cervusdama' with type 'git'"}
2023-09-25T19:06:24.791057275Z loaders Enumerating objects: 868, done.

There is no move after several hours, even for quite small repositories.

It's not clear yet if it a regression of the dynamic infra or something that was hidden by the systemd watchdog that was deployed in the static infrastructure