Staging instance, all changes can be removed at any time

Skip to content

Unstuck no longer working opam stack stuck in the past

mclovin collaborator user reported that our opam ingestion is out of sync (from irc 2 days or so ago) [1]

Investigate and fix.

tl; dr:

  • The service to manage the shared state in the static workers was no longer refreshing appropriately the shared state for workers (and lister)
  • Commands used in opam lister and loader was not throwing issues when a problem happened
  • swh/devel/swh-lister!482 (merged): the opam lister command used within the lister was displaying a warning after a switch not being set, this lead to actually understand this out-of-sync issue [2]
  • swh/devel/swh-loader-core!484 (merged) the opam loader could no longer work since various changes happened in most recent opam version (it used to rely on old opam file structure) [3]
  • lastly, the opam origins (which don't have a last update timestamp) are never scheduled according to [4] (default policy does not configure any origins without a last update)
  • Deploy all the things

Once the last points are fixed (merged and released), it's going to be deployed.

[5] https://gitlab.softwareheritage.org/swh/devel/swh-scheduler/-/blob/master/swh/scheduler/celery_backend/recurrent_visits.py#L279-287

[4] https://gitlab.softwareheritage.org/swh/devel/swh-scheduler/-/blob/master/swh/scheduler/celery_backend/recurrent_visits.py#L52

18:58 <mclovin> hi
18:59 <mclovin> It seems to me that the opam lister/loader is not archiving "recent" packages
19:10 <mclovin> I didn't bisected completely, but no package seems to have been archived since around march 2022
19:12 <mclovin> https://github.com/ocaml/opam-repository/pull/20715 can be found (10 february 2022)
19:12 <+swhbot> ^ Add: note.0.0.2 by dbuenzli · Pull Request #20715 · ocaml/opam-repository · GitHub
19:12 <mclovin> but this one can't https://github.com/ocaml/opam-repository/pull/20760 (18 february 2022)
19:12 <+swhbot> ^ Package ppx_monad.0.1.0 by Niols · Pull Request #20760 · ocaml/opam-repository · GitHub
19:15 <mclovin> I believe an opam update is missing somewhere in the lister/loader code. Or something wrong happened but I would need to see some logs to try debugging.
19:45 <+vlorentz> hmm
19:46 <+vlorentz> the latter had a broken visit: https://archive.softwareheritage.org/browse/origin/directory/?origin_url=opam%2Bhttps://opam.ocaml.org/packages/ppx_monadic/&timestamp=2021-09-24T19:21:39Z
19:46 <+swhbot> ^ Directory - / - None - origin: opam+https://opam.ocaml.org/packages/ppx_monadic/ – Software Heritage archive
19:46 <+vlorentz> it might have corrupted our state somehow?
19:47 <+vlorentz> seems unlikely though
19:50 <+vlorentz> 'opam show ppx_monadic' show exactly the set of versions SWH has
19:50 <+vlorentz> was ppx_monad 0.1.0 still on opam by 2021-11-10?
...
11:28 <mclovin> opam show ppx_monad has 0.1.0 and 0.2.0 and swh has none of them
11:28 <mclovin> I believe we need to run an opam update somewhere in swh, it never seems to be called
11:26 <mclovin> @vlorentz, ppx_monadic and ppx_monad are not the same package
Edited by Antoine R. Dumont