Deploy pagure lister (with lister v5.7.1)

• Package lister v5.7.1
• Build docker image [1]

staging:

• swh/infra/ci-cd/swh-charts!75 (merged): swh-charts: Reference new lister version
• swh/infra/ci-cd/swh-charts!76 (merged): swh-charts: Add pagure lister pod to consume new lister task
• Let argocd deploy
• scheduler0.staging: Update swh.lister & register new task type [2]
• Schedule forge for listing [3]
• Listing Checks [4]
• Schedule some origins for ingestion [5]
• Checks
• Queue status [6]

production:

• swh-charts: Add lister to production
• saatchi: Register new lister task type in scheduler db
• Schedule forges for listing
• Listing Checks

Potential Instances (subset tested in staging with success):

• pagure.io
• git.centos.org
• src.fedoraproject.org
• code.opensuse.org

[1] https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-images/6/console

[2]

root@scheduler0:~# apt list --upgradable | grep lister
python3-swh.lister/unknown 5.7.1-1~swh1~bpo10+1 all [upgradable from: 5.6.3-1~swh1~bpo10+1]
root@scheduler0:~# apt install -y python3-swh.lister
...
Preparing to unpack .../python3-swh.lister_5.7.1-1~swh1~bpo10+1_all.deb ...
Unpacking python3-swh.lister (5.7.1-1~swh1~bpo10+1) over (5.6.3-1~swh1~bpo10+1) ...
Setting up python3-swh.lister (5.7.1-1~swh1~bpo10+1) ...
...
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register
...
INFO:swh.scheduler.cli.task_type:Create task type list-pagure in scheduler
...

[3]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/   add-forge-now --preset staging     register-lister pagure       instance=pagure.io
Created 1 tasks

Task 33421701
  Next run: today (2023-06-26T13:51:38.744666+00:00)
  Interval: 1 day, 0:00:00
  Type: list-pagure
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    instance: 'pagure.io'
    max_origins_per_page: 10
    max_pages: 3

[4]

listers {"asctime": "2023-06-26 13:52:48,777", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "cele
ry.app.trace", "levelname": "INFO", "message": "Task swh.lister.pagure.tasks.PagureListerTask[c719584c-ed3e-4b05-8d31-6d255f82c173] succeeded in 7.068502369336784s: {'pages': 3, 'origins': 30}", "data": {"id": "c719584c-ed3e-4b05-8d31-6d25
5f82c173", "name": "swh.lister.pagure.tasks.PagureListerTask", "return_value": "{'pages': 3, 'origins': 30}", "runtime": 7.068502369336784, "args": "[]", "kwargs": "{'instance': 'pagure.io', 'max_pages': 3, 'enable_origins': False, 'max_or
igins_per_page': 10}"}}

[5]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/   add-forge-now --preset staging   schedule-first-visits     --type-name git     --lister-name pagure     --lister-instance-name pagure.io
97 slots available in celery queue
30 visits to send to celery

[6]

loaders {"asctime": "2023-06-26 14:00:08,662", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id
": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateGitRepository[91c9738e-e904-49a3-86cd-4e3b04b4ad1a] received", "data": {"id": "91c9738e-e904-49a3-86cd-4e3b04b4ad1a", "name": "swh.
loader.git.tasks.UpdateGitRepository", "args": "[]", "kwargs": "{'url': 'https://pagure.io/fork/obudai/fedora-infra/ansible', 'lister_name': 'pagure', 'lister_instance_name': 'pagure.io'}"}}

[6] https://grafana.softwareheritage.org/goto/9jpTQqXVz?orgId=1