Staging instance, all changes can be removed at any time

Skip to content

Deploy latest 'directory' loaders

Contains:

  • New 'directory' Loaders were inconsistently named with visit types and task names mismatching, which resulted in origins not being scheduled (without any errors and lots of questions)
    • svn: v1.8.1: SvnDirectoryLoader -> SvnExportLoader
    • hg: v3.4.0: HgDirectoryLoader -> HgCheckoutLoader
    • git: v2.5.0: GitDirectoryLoader -> GitCheckoutLoader

Impacts (staging):

  • scheduler: New task types must be registered with their backend names matching the changes ^
  • swh-charts: Queue names must be renamed in the 'directory' loaders

Plan:

  • Build images
  • staging: scheduler0/rabbitmq: Purge old queues from old messages
    • swh.loader.git.tasks.LoadGitDirectory
    • swh.loader.mercurial.tasks.LoadHgDirectory
    • swh.loader.svn.tasks.LoadSvnDirectory
    • swh.loader.core.tasks.LoadTarballDirectory
  • swh-charts: Adapt impacted loader configurations (task name change so queue should change too)
  • staging: scheduler0: clean up unused old types [2]
  • staging: scheduler0: Register task types [3]
  • swh-charts: Update image versions to deploy
  • Deploy
  • staging: scheduler0/rabbitmq: Delete old queues
    • swh.loader.git.tasks.LoadGitDirectory
    • swh.loader.mercurial.tasks.LoadHgDirectory
    • swh.loader.hg.tasks.LoadHgDirectory
    • swh.loader.svn.tasks.LoadSvnDirectory
  • Reschedule origins in new queues [4]
  • Checks
    • all loaders ok (impacted and non impacted) [5]
    • ingestion on those origins is ok
  • Deploy to production [1]

[1] It won't impact production as this only concerns new directory loaders not deployed there yet.

[2]

2023-06-12 15:29:38 swh-scheduler@db1:5432 λ select type, backend_name from task_type where type in ('load-hg-directory', 'load-svn-directory', 'load-git-directory');
+--------------------+---------------------------------------------------+
|        type        |                   backend_name                    |
+--------------------+---------------------------------------------------+
| load-hg-directory  | swh.loader.mercurial.tasks.LoadMercurialDirectory |
| load-git-directory | swh.loader.git.tasks.LoadGitDirectory             |
| load-svn-directory | swh.loader.svn.tasks.LoadSvnDirectory             |
| load-directory     | swh.loader.core.tasks.LoadDirectory               |
+--------------------+---------------------------------------------------+
(3 rows)

Time: 4.271 ms
2023-06-12 15:29:44 swh-scheduler@db1:5432 λ delete from task_type where type in ('load-hg-directory', 'load-svn-directory', 'load-git-directory', 'load-directory');
DELETE 4
Time: 1932.327 ms (00:01.932)

[3]


2023-06-12 15:27:24 swh-scheduler@db1:5432 λ update task_type set backend_name='swh.loader.svn.tasks.LoadSvnExport' where type='load-svn-export';
UPDATE 1
Time: 52.221 ms
2023-06-12 15:28:33 swh-scheduler@db1:5432 λ select type, backend_name from task_type where type in ('load-hg-checkout', 'load-git-checkout', 'load-svn-export')  ;
+-------------------+---------------------------------------------------+
|       type        |                   backend_name                    |
+-------------------+---------------------------------------------------+
| load-git-checkout | swh.loader.git.tasks.LoadGitDirectory             |
| load-hg-checkout  | swh.loader.mercurial.tasks.LoadMercurialDirectory |
| load-svn-export   | swh.loader.svn.tasks.LoadSvnExport                |
+-------------------+---------------------------------------------------+
(3 rows)

Time: 4.720 ms

Note: svn-export got done manually since the debian package is still red.

[4]

swhscheduler@scheduler0:~$ lister_name='nixguix'; type='svn-export'; swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery $type --lister-name $lister_name
10000 slots available in celery queue
103 visits to send to celery
swhscheduler@scheduler0:~$ lister_name='nixguix'; type='hg-checkout'; swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery $type --lister-name $lister_name
10000 slots available in celery queue
26 visits to send to celery
swhscheduler@scheduler0:~$ lister_name='nixguix'; type='git-checkout'; swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery $type --lister-name $lister_name
10000 slots available in celery queue
6813 visits to send to celery
swhscheduler@scheduler0:~$ lister_name='nixguix'; type='tarball-directory'; swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery $type --lister-name $lister_name
10000 slots available in celery queue
10000 visits to send to celery

[5] a tad verbose but eventful visits can be seen through the matrix (of logs)

loaders {"asctime": "2023-06-12 13:45:59,783", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 396, "funcName": "load", "task_name": null, "task_id": null, "name":
 "swh.loader.svn.directory.SvnExportLoader", "levelname": "INFO", "message": "Load origin 'svn://www.tug.org/texlive/tags/texlive-2021.3/Master/texmf-dist/source/latex/upquote' with type 'svn-export'"}
loaders {"asctime": "2023-06-12 13:46:01,517", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "cele
ry.app.trace", "levelname": "INFO", "message": "Task swh.loader.svn.tasks.LoadSvnExport[a704e955-5be4-45cb-9f8c-2d48c534c8bf] succeeded in 2.114382553845644s: {'status': 'eventful'}", "data": {"id": "a704e955-5be4-45cb-9f8c-2d48c534c8bf",
"name": "swh.loader.svn.tasks.LoadSvnExport", "return_value": "{'status': 'eventful'}", "runtime": 2.114382553845644, "args": "[]", "kwargs": "{'url': 'svn://www.tug.org/texlive/tags/texlive-2021.3/Master/texmf-dist/source/latex/upquote',
'lister_name': 'nixguix', 'lister_instance_name': 'guix.gnu.org', 'ref': 59745, 'checksums': {'sha256': 'b747fed41bc5e8c2356e574ed8e8718ed87d68f1f376e05fec968f9322282034'}, 'fallback_urls': [], 'checksum_layout': 'nar'}"}}
loaders {"asctime": "2023-06-12 13:46:01,520", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id
": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.core.tasks.LoadTarballDirectory[5a846dab-2fce-4a02-b5dc-9e8f5427d177] received", "data": {"id": "5a846dab-2fce-4a02-b5dc-9e8f5427d177", "name": "sw
h.loader.core.tasks.LoadTarballDirectory", "args": "[]", "kwargs": "{'url': 'http://downloads.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'lister_name': 'nixguix', 'lister_instance_name': 'guix.gnu.org', 'checksum
s': {'sha256': '9a6c6b02ea91e9f836448ccc5a614caa514a9ba17e94f1d6c0babc72a4395b09'}, 'fallback_urls': ['http://ufpr.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://freefr.dl.sourceforge.net/project/pstoedit/
pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://internode.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://jaist.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://liquidtelecom.dl
.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://nchc.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://netcologne.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77
.tar.gz', 'http://netix.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://pilotfiber.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://tenet.dl.sourceforge.net/project/pstoedit/
pstoedit/3.77/pstoedit-3.77.tar.gz', 'https://bordeaux.guix.gnu.org/file/pstoedit-3.77.tar.gz/sha256/02av76j75g5sq3bg353yl6dlllda9ihmmk4c8hvgiscix816nv4s', 'https://ci.guix.gnu.org/file/pstoedit-3.77.tar.gz/sha256/02av76j75g5sq3bg353yl6dll
lda9ihmmk4c8hvgiscix816nv4s', 'https://tarballs.nixos.org/sha256/02av76j75g5sq3bg353yl6dlllda9ihmmk4c8hvgiscix816nv4s'], 'checksum_layout': 'standard'}"}}
loaders {"asctime": "2023-06-12 13:46:01,835", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 396, "funcName": "load", "task_name": null, "task_id": null, "name":
 "swh.loader.svn.directory.SvnExportLoader", "levelname": "INFO", "message": "Load origin 'https://code.call-cc.org/svn/chicken-eggs/release/5/iset/tags/2.2' with type 'svn-export'"}
loaders {"asctime": "2023-06-12 13:46:16,956", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "cele
ry.app.trace", "levelname": "INFO", "message": "Task swh.loader.svn.tasks.LoadSvnExport[ed884d70-0841-4a04-bd63-10245799223a] succeeded in 15.436203259974718s: {'status': 'eventful'}", "data": {"id": "ed884d70-0841-4a04-bd63-10245799223a",
 "name": "swh.loader.svn.tasks.LoadSvnExport", "return_value": "{'status': 'eventful'}", "runtime": 15.436203259974718, "args": "[]", "kwargs": "{'url': 'https://code.call-cc.org/svn/chicken-eggs/release/5/iset/tags/2.2', 'lister_name': 'n
ixguix', 'lister_instance_name': 'guix.gnu.org', 'ref': 39057, 'checksums': {'sha256': 'eb5b68e3a7c70165b0a26c151713ce6e99df33053ee79ba62e3698128b05e53f'}, 'fallback_urls': [], 'checksum_layout': 'nar'}"}}
loaders {"asctime": "2023-06-12 13:46:16,963", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id
": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.svn.tasks.LoadSvnExport[914a07bf-9cac-476f-a314-f183759a6e30] received", "data": {"id": "914a07bf-9cac-476f-a314-f183759a6e30", "name": "swh.loader
.svn.tasks.LoadSvnExport", "args": "[]", "kwargs": "{'url': 'https://code.call-cc.org/svn/chicken-eggs/release/5/test/tags/1.1', 'lister_name': 'nixguix', 'lister_instance_name': 'guix.gnu.org', 'ref': 39263, 'checksums': {'sha256': 'fd5b3
74bef771ff46095c1cc28c3e16c9cc9bc4ef1383c4d9f121aab3b0b2992'}, 'fallback_urls': [], 'checksum_layout': 'nar'}"}}
loaders {"asctime": "2023-06-12 13:46:17,166", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 396, "funcName": "load", "task_name": null, "task_id": null, "name":
 "swh.loader.core.loader.TarballDirectoryLoader", "levelname": "INFO", "message": "Load origin 'http://downloads.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz' with type 'tarball-directory'"}
loaders {"asctime": "2023-06-12 13:46:22,518", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "cele
ry.app.trace", "levelname": "INFO", "message": "Task swh.loader.core.tasks.LoadTarballDirectory[5a846dab-2fce-4a02-b5dc-9e8f5427d177] succeeded in 5.555736476555467s: {'status': 'eventful'}", "data": {"id": "5a846dab-2fce-4a02-b5dc-9e8f542
7d177", "name": "swh.loader.core.tasks.LoadTarballDirectory", "return_value": "{'status': 'eventful'}", "runtime": 5.555736476555467, "args": "[]", "kwargs": "{'url': 'http://downloads.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedi
t-3.77.tar.gz', 'lister_name': 'nixguix', 'lister_instance_name': 'guix.gnu.org', 'checksums': {'sha256': '9a6c6b02ea91e9f836448ccc5a614caa514a9ba17e94f1d6c0babc72a4395b09'}, 'fallback_urls': ['http://ufpr.dl.sourceforge.net/project/pstoed
it/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://freefr.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://internode.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://jaist.dl.sou
rceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://liquidtelecom.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://nchc.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.
tar.gz', 'http://netcologne.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://netix.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://pilotfiber.dl.sourceforge.net/project/pstoe
dit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'http://tenet.dl.sourceforge.net/project/pstoedit/pstoedit/3.77/pstoedit-3.77.tar.gz', 'https://bordeaux.guix.gnu.org/file/pstoedit-3.77.tar.gz/sha256/02av76j75g5sq3bg353yl6dlllda9ihmmk4c8hvgiscix81
6nv4s', 'https://ci.guix.gnu.org/file/pstoedit-3.77.tar.gz/sha256/02av76j75g5sq3bg353yl6dlllda9ihmmk4c8hvgiscix816nv4s', 'https://tarballs.nixos.org/sha256/02av76j75g5sq3bg353yl6dlllda9ihmmk4c8hvgiscix816nv4s'], 'checksum_layout': 'standar
d'}"}}
loaders {"asctime": "2023-06-12 13:46:22,521", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id
": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.svn.tasks.LoadSvnExport[f7c7ed0c-8931-47f6-85ab-f84a33dbc47d] received", "data": {"id": "f7c7ed0c-8931-47f6-85ab-f84a33dbc47d", "name": "swh.loader
.svn.tasks.LoadSvnExport", "args": "[]", "kwargs": "{'url': 'svn://www.tug.org/texlive/tags/texlive-2021.3/Master/texmf-dist/source/latex/xmpincl', 'lister_name': 'nixguix', 'lister_instance_name': 'guix.gnu.org', 'ref': 59745, 'checksums'
: {'sha256': '649c13c297e219b5d7a59518b56bc19101f0d89996522e5f29896b47966b0353'}, 'fallback_urls': [], 'checksum_layout': 'nar'}"}}
loaders {"asctime": "2023-06-12 13:46:22,761", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 396, "funcName": "load", "task_name": null, "task_id": null, "name":
 "swh.loader.svn.directory.SvnExportLoader", "levelname": "INFO", "message": "Load origin 'https://code.call-cc.org/svn/chicken-eggs/release/5/test/tags/1.1' with type 'svn-export'"}

Refs. swh/meta#4979 (closed)

Edited by Antoine R. Dumont