Staging instance, all changes can be removed at any time

Skip to content

Implement a loader for npm

Antoine Lambert requested to merge generated-differential-D1254-source into generated-differential-D1254-target

First diff showcasing the work done so far in order to ingest into the archive the numerous source packages hosted on the npm registry.

This work is greatly inspired from the PyPI loader written by @ardumont. Apart some slight differences between PyPI and npm, the process to load their content into the archive is almost the same: read package metatada from a JSON endpoint then download tarballs and load them into the archive.

This clearly calls for a future refactoring to remove some code duplication and establish a loading worflow to ingest content from other package managers.

The loader can be tested through the following command line:

$ python3 -m swh.loader.npm.loader <npm_package_name>

To correctly browse the results with swh-web, you will need to use its development version as current package published on PyPI does not handle browsing this new type of origin yet.

Related swh-loader-core#1379 (closed)

Migrated from D1254 (view on Phabricator)

Merge request reports