Update value of User-Agent HTTP request header used by listers
Closed
requested to merge generated-differential-D8523-source into generated-differential-D8523-target
That HTTP header value will now contain the lister name but also a link to our contact form in order for sysadmins to easily reach us if needed.
The following template is used to generate it:
"Software Heritage <lister_name> lister v (+https://www.softwareheritage.org/contact)"
Depnds on !311 (closed)
Migrated from D8523 (view on Phabricator)
Merge request reports
Activity
Filter activity
Build is green
Patch application report for D8523 (id=30693)
Could not rebase; Attempt merge onto bd35d543...
Updating bd35d54..c2a2f7b Fast-forward docs/new_lister_template.py | 41 ++------------------- docs/tutorial.rst | 2 +- swh/lister/__init__.py | 14 ++++---- swh/lister/arch/lister.py | 31 ++-------------- swh/lister/aur/lister.py | 4 +-- swh/lister/bitbucket/lister.py | 33 +++-------------- swh/lister/bitbucket/tests/test_lister.py | 10 +++--- swh/lister/bower/lister.py | 35 +++--------------- swh/lister/cgit/lister.py | 15 ++------ swh/lister/cgit/tests/test_lister.py | 6 ++-- swh/lister/debian/lister.py | 17 ++++----- swh/lister/gitea/tests/test_lister.py | 2 +- swh/lister/github/lister.py | 4 +-- swh/lister/gitlab/lister.py | 13 +++---- swh/lister/gitlab/tests/test_lister.py | 11 +++--- swh/lister/gogs/lister.py | 52 ++++++++++----------------- swh/lister/gogs/tests/test_lister.py | 8 ++--- swh/lister/golang/lister.py | 28 ++------------- swh/lister/golang/tests/test_lister.py | 9 ++--- swh/lister/launchpad/lister.py | 4 +-- swh/lister/maven/lister.py | 56 +++++++++-------------------- swh/lister/maven/tests/test_lister.py | 5 +++ swh/lister/npm/lister.py | 30 +++------------- swh/lister/npm/tests/test_lister.py | 31 +++++++++++----- swh/lister/packagist/lister.py | 25 +++---------- swh/lister/pattern.py | 34 +++++++++++++++++- swh/lister/phabricator/lister.py | 28 +++------------ swh/lister/phabricator/tests/test_lister.py | 12 +++++-- swh/lister/pubdev/lister.py | 43 ++++------------------ swh/lister/pubdev/tests/test_lister.py | 8 +++-- swh/lister/pypi/lister.py | 6 ++-- swh/lister/sourceforge/lister.py | 43 +++++----------------- swh/lister/sourceforge/tests/test_lister.py | 16 +++++---- swh/lister/tests/test_utils.py | 32 +++++++++-------- swh/lister/tuleap/lister.py | 39 ++++---------------- swh/lister/tuleap/tests/test_lister.py | 7 +++- swh/lister/utils.py | 14 ++------ 37 files changed, 252 insertions(+), 516 deletions(-)Changes applied before test
commit c2a2f7ba588c5db2284f4ff5d41463824af291dd Author: Antoine Lambert <anlambert@softwareheritage.org> Date: Thu Sep 22 15:43:20 2022 +0200 Update value of User-Agent HTTP request header used by listers That HTTP header value will now contain the lister name but also a link to our contact form in order for sysadmins to easily reach us if needed. The following template is used to generate it: "Software Heritage <lister_name> lister v<swh-lister version> (+https://www.softwareheritage.org/contact)" commit d677988e22f0d71d6b4aa2dae24f2d41f2ce337a Author: Antoine Lambert <anlambert@softwareheritage.org> Date: Wed Sep 21 19:53:22 2022 +0200 Refactor and deduplicate HTTP requests code in listers Numerous listers were using the same page_request method or equivalent in their implementation so prefer to deduplicate that code by adding an http_request method in base lister class: swh.lister.pattern.Lister. That method simply wraps a call to requests.Session.request and logs some useful info for debugging and error reporting, also an HTTPError will be raised if a request ends up with an error. All listers using that new method now benefit of requests retry when an HTTP error occurs thanks to the use of the http_retry decorator. commit 6284d34b1725ce9c5e9f76359a2760da871edb7b Author: Antoine Lambert <anlambert@softwareheritage.org> Date: Wed Sep 21 16:56:34 2022 +0200 Use generic HTTP retry policy by default and rename dedicated decorator Instead of retrying HTTP requests only for 429 status code by default, prefer to use the generic retry policy enabling to also retry for status codes >= 500 but also on ConnectionError exceptions. Rename throttling_retry decorator to http_retry to reflect this change.See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/673/ for more details.
-
Build is green
Patch application report for D8523 (id=30741)
Could not rebase; Attempt merge onto 9b3e565c...
Updating 9b3e565..d5c30a3 Fast-forward docs/new_lister_template.py | 41 ++------------------- docs/tutorial.rst | 2 +- swh/lister/__init__.py | 14 ++++---- swh/lister/arch/lister.py | 31 ++-------------- swh/lister/aur/lister.py | 4 +-- swh/lister/bitbucket/lister.py | 33 +++-------------- swh/lister/bitbucket/tests/test_lister.py | 10 +++--- swh/lister/bower/lister.py | 35 +++--------------- swh/lister/cgit/lister.py | 15 ++------ swh/lister/cgit/tests/test_lister.py | 6 ++-- swh/lister/debian/lister.py | 17 ++++----- swh/lister/gitea/tests/test_lister.py | 2 +- swh/lister/github/lister.py | 4 +-- swh/lister/gitlab/lister.py | 13 +++---- swh/lister/gitlab/tests/test_lister.py | 11 +++--- swh/lister/gogs/lister.py | 52 ++++++++++----------------- swh/lister/gogs/tests/test_lister.py | 8 ++--- swh/lister/golang/lister.py | 28 ++------------- swh/lister/golang/tests/test_lister.py | 9 ++--- swh/lister/launchpad/lister.py | 4 +-- swh/lister/maven/lister.py | 56 +++++++++-------------------- swh/lister/maven/tests/test_lister.py | 5 +++ swh/lister/npm/lister.py | 30 +++------------- swh/lister/npm/tests/test_lister.py | 31 +++++++++++----- swh/lister/packagist/lister.py | 25 +++---------- swh/lister/pattern.py | 34 +++++++++++++++++- swh/lister/phabricator/lister.py | 28 +++------------ swh/lister/phabricator/tests/test_lister.py | 12 +++++-- swh/lister/pubdev/lister.py | 43 ++++------------------ swh/lister/pubdev/tests/test_lister.py | 8 +++-- swh/lister/pypi/lister.py | 6 ++-- swh/lister/sourceforge/lister.py | 43 +++++----------------- swh/lister/sourceforge/tests/test_lister.py | 16 +++++---- swh/lister/tests/test_utils.py | 32 +++++++++-------- swh/lister/tuleap/lister.py | 39 ++++---------------- swh/lister/tuleap/tests/test_lister.py | 7 +++- swh/lister/utils.py | 14 ++------ 37 files changed, 252 insertions(+), 516 deletions(-)Changes applied before test
commit d5c30a3ce371efa73cbabece72ce214bf18e3d3e Author: Antoine Lambert <anlambert@softwareheritage.org> Date: Thu Sep 22 15:43:20 2022 +0200 Update value of User-Agent HTTP request header used by listers That HTTP header value will now contain the lister name but also a link to our contact form in order for sysadmins to easily reach us if needed. The following template is used to generate it: "Software Heritage <lister_name> lister v<swh-lister version> (+https://www.softwareheritage.org/contact)" commit db6ce12e9e3bf527e81a3c8fb3b159d81f6a7d0f Author: Antoine Lambert <anlambert@softwareheritage.org> Date: Wed Sep 21 19:53:22 2022 +0200 Refactor and deduplicate HTTP requests code in listers Numerous listers were using the same page_request method or equivalent in their implementation so prefer to deduplicate that code by adding an http_request method in base lister class: swh.lister.pattern.Lister. That method simply wraps a call to requests.Session.request and logs some useful info for debugging and error reporting, also an HTTPError will be raised if a request ends up with an error. All listers using that new method now benefit of requests retry when an HTTP error occurs thanks to the use of the http_retry decorator. commit 9c55acd286091acb6f6094e9fe1c95aca1fdeeec Author: Antoine Lambert <anlambert@softwareheritage.org> Date: Wed Sep 21 16:56:34 2022 +0200 Use generic HTTP retry policy by default and rename dedicated decorator Instead of retrying HTTP requests only for 429 status code by default, prefer to use the generic retry policy enabling to also retry for status codes >= 500 but also on ConnectionError exceptions. Rename throttling_retry decorator to http_retry to reflect this change.See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/686/ for more details.
Please register or sign in to reply