cpan: Improve listing process by querying the metacpan release endpoint

Review changes
Download
Patches
Plain diff

Closed cpan: Improve listing process by querying the metacpan release endpoint

generated-differential-D8615-source into generated-differential-D8615-target

Overview 19
Commits 1
Pipelines 0
Changes 8

Closed Antoine Lambert requested to merge generated-differential-D8615-source into generated-differential-D8615-target 2 years ago

Overview 19
Commits 1
Pipelines 0
Changes 8

Instead of querying the metacpan distribution endpoint to list origins, prefer to use the release endpoint instead enabling to list all artifacts associated to CPAN packages by scrolling results.

Compared to previous implementation, it enables to compute a last_update date for all CPAN packages but also to obtain artifact sha256 checksums that will be used by the CPAN loader to check downloads integrity.

It also enables to save a call to metacapan Web API in cpan loader as all needed info about package artifacts are now provided as extra loader arguments.

Related to #2833

When testing this in docker, I could list all CPAN packages and their artifacts in less than 4 minutes.

Migrated from D8615 (view on Phabricator)