Save code now: prevent huge ingestion tasks from starving regular ones
On June 17th 2023 we noticed more than 2000 tasks waiting in the save code now queue (see )
It seems that this may come from a few huge ingestion tasks clogging the save code now pipeline (see https://gitlab.alpinelinux.org/messense/aports running 142,738 Commits in this screen capture)
A possible mitigation path would be to dispatch save code now requests on different queues based on criterias that take into account the size of the packfile/nb of commits and the fact that an origin is known or not. Large repositories from not known origins should not be allowed to clog the regular pipelin operations :-)
Edited by Antoine R. Dumont