Automate the bulk loading of origins in a kubernetes job
The bulk loading of origins can be quite long.
The commands to launch [1] are in the toolbox so it could be possible to automate the process in a kubernetes job not have to launch everything manually.
The idea is to have a part in the helm chart that can handle that.
The information to have are:
- an url of a file containing the origins to load
- the task type
- possibly the origin batch size
Positive side effect:
- Job configuration is committed
- Logs are ingested in kibana
[1] https://docs.softwareheritage.org/sysadm/deployment/howto-bulk-ingest.html
Tasks:
-
Update the chart -
Update the documentation
Edited by Vincent Sellier