Staging instance, all changes can be removed at any time

Skip to content

loader: Rename loaders to more meaningful names

Antoine R. Dumont requested to merge generated-differential-D834-source into generated-differential-D834-target
  • git.updater: Rename to git.loader
  • loader.git.from_disk: Rename GitLoader to GitLoaderFromDisk
  • test_loader: Add missing copyright header
  • git.loader: Rename BulkUpdater to GitLoader
  • Update configuration files to be named appropriately
  • Update doc
  • Adaptations of the expected configuration files for the least surprise effect on the users.

This does not deal yet with other impacted modules (main documentation, swh-site, etc...).

Test Plan

tox

And/or from swh-docker-dev to start the relevant services:

docker-compose up

then, from remote test:

python3 -m swh.loader.git.loader --origin-url https://github.com/SoftwareHeritage/swh-storage.git

2018-12-17 14:01:19,948 12740 Creating git origin for https://github.com/SoftwareHeritage/swh-storage.git
2018-12-17 14:01:19,951 12740 Starting new HTTP connection (1): localhost:5002
2018-12-17 14:01:19,958 12740 http://localhost:5002 "POST /origin/add HTTP/1.1" 200 1
2018-12-17 14:01:19,959 12740 Done creating git origin for https://github.com/SoftwareHeritage/swh-storage.git
2018-12-17 14:01:19,959 12740 Creating origin_visit for origin 1 at time 2018-12-17 13:01:19.948234+00:00
2018-12-17 14:01:19,961 12740 Resetting dropped connection: localhost
2018-12-17 14:01:19,965 12740 http://localhost:5002 "POST /origin/visit/add HTTP/1.1" 200 16
2018-12-17 14:01:19,965 12740 Done Creating origin_visit for origin 1 at time 2018-12-17 13:01:19.948234+00:00
2018-12-17 14:01:19,967 12740 Resetting dropped connection: localhost
2018-12-17 14:01:19,971 12740 http://localhost:5002 "POST /fetch_history/start HTTP/1.1" 200 1
2018-12-17 14:01:19,977 12740 Resetting dropped connection: localhost
2018-12-17 14:01:19,982 12740 http://localhost:5002 "POST /snapshot/latest HTTP/1.1" 200 1
2018-12-17 14:01:19,982 12740 Starting new HTTPS connection (1): github.com:443
2018-12-17 14:01:20,352 12740 https://github.com:443 "GET /SoftwareHeritage/swh-storage.git/info/refs?service=git-upload-pack HTTP/1.1" 200 None
2018-12-17 14:01:20,460 12740 Resetting dropped connection: localhost
2018-12-17 14:01:20,466 12740 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 5776
2018-12-17 14:01:20,775 12740 https://github.com:443 "POST /SoftwareHeritage/swh-storage.git/git-upload-pack HTTP/1.1" 200 None
Enumerating objects: 241, done.
Counting objects: 100% (241/241), done.
Compressing objects: 100% (145/145), done.
Total 7480 (delta 130), reused 181 (delta 84), pack-reused 7239
2018-12-17 14:01:21,464 12740 Resetting dropped connection: localhost
2018-12-17 14:01:21,469 12740 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 5776
2018-12-17 14:01:21,470 12740 Listed 137 refs for repo https://github.com/SoftwareHeritage/swh-storage.git
2018-12-17 14:01:22,276 12740 Resetting dropped connection: localhost
2018-12-17 14:01:22,314 12740 http://localhost:5002 "POST /content/missing HTTP/1.1" 200 53375
2018-12-17 14:01:22,830 12740 Sending 2426 contents
2018-12-17 14:01:22,852 12740 Resetting dropped connection: localhost
2018-12-17 14:01:25,424 12740 http://localhost:5002 "POST /content/add HTTP/1.1" 200 1
2018-12-17 14:01:25,425 12740 Done sending 2426 contents
2018-12-17 14:01:25,430 12740 Resetting dropped connection: localhost
2018-12-17 14:01:25,456 12740 http://localhost:5002 "POST /directory/missing HTTP/1.1" 200 84637
2018-12-17 14:01:25,754 12740 Sending 3847 directories
2018-12-17 14:01:25,767 12740 Resetting dropped connection: localhost
2018-12-17 14:01:26,275 12740 http://localhost:5002 "POST /directory/add HTTP/1.1" 200 1
2018-12-17 14:01:26,276 12740 Done sending 3847 directories
2018-12-17 14:01:26,284 12740 Resetting dropped connection: localhost
2018-12-17 14:01:26,292 12740 http://localhost:5002 "POST /revision/missing HTTP/1.1" 200 23961
2018-12-17 14:01:26,523 12740 Sending 1089 revisions
2018-12-17 14:01:26,526 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,597 12740 http://localhost:5002 "POST /revision/add HTTP/1.1" 200 1
2018-12-17 14:01:27,597 12740 Done sending 1089 revisions
2018-12-17 14:01:27,601 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,605 12740 http://localhost:5002 "POST /release/missing HTTP/1.1" 200 2599
2018-12-17 14:01:27,827 12740 Sending 118 releases
2018-12-17 14:01:27,829 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,847 12740 http://localhost:5002 "POST /release/add HTTP/1.1" 200 1
2018-12-17 14:01:27,847 12740 Done sending 118 releases
2018-12-17 14:01:27,849 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,860 12740 http://localhost:5002 "POST /snapshot/add HTTP/1.1" 200 1
2018-12-17 14:01:27,862 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,865 12740 http://localhost:5002 "POST /fetch_history/end HTTP/1.1" 200 1
2018-12-17 14:01:27,866 12740 Updating origin_visit for origin 1 with status full
2018-12-17 14:01:27,868 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,870 12740 http://localhost:5002 "POST /origin/visit/update HTTP/1.1" 200 1
2018-12-17 14:01:27,871 12740 Done updating origin_visit for origin 1 with status full

from disk test:

python3 -m swh.loader.git.from_disk --origin-url swh-loader-git --git-directory $(pwd)
2018-12-17 14:02:03,006 13195 Creating git origin for swh-loader-git
2018-12-17 14:02:03,008 13195 Done creating git origin for swh-loader-git
2018-12-17 14:02:03,008 13195 Creating origin_visit for origin 6 at time 2018-12-17 13:02:03.003626+00:00
2018-12-17 14:02:03,009 13195 Done Creating origin_visit for origin 6 at time 2018-12-17 13:02:03.003626+00:00
2018-12-17 14:02:04,044 13195 Sending 1000 contents
2018-12-17 14:02:04,345 13195 Done sending 1000 contents
2018-12-17 14:02:04,435 13195 Sending 1000 contents
2018-12-17 14:02:04,690 13195 Done sending 1000 contents
2018-12-17 14:02:04,717 13195 Sending 292 contents
2018-12-17 14:02:04,800 13195 Done sending 292 contents
2018-12-17 14:02:04,975 13195 Sending 2500 directories
2018-12-17 14:02:05,222 13195 Done sending 2500 directories
2018-12-17 14:02:05,304 13195 Sending 1375 directories
2018-12-17 14:02:05,429 13195 Done sending 1375 directories
2018-12-17 14:02:05,532 13195 Sending 1000 revisions
2018-12-17 14:02:05,896 13195 Done sending 1000 revisions
2018-12-17 14:02:05,932 13195 Sending 465 revisions
2018-12-17 14:02:05,981 13195 Done sending 465 revisions
2018-12-17 14:02:05,990 13195 Sending 89 releases
2018-12-17 14:02:06,001 13195 Done sending 89 releases
2018-12-17 14:02:06,029 13195 Updating origin_visit for origin 6 with status full
2018-12-17 14:02:06,029 13195 Done updating origin_visit for origin 6 with status full

Migrated from D834 (view on Phabricator)

Merge request reports