Conda: List origins from anaconda.com, the Package, dependency and environment management for any language

Related #4547

---

Migrated from D8517 (view on Phabricator)

Build is green

Patch application report for D8517 (id=30668)

Rebasing onto bd35d543...

First, rewinding head to replay your work on top of it...
Applying: [WIP] Conda: List origins from anaconda.com, the Package, dependency and environment management for any language

Changes applied before test

commit 587b8853600d4e1d08c0930662fe4edaa21fe35c
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Sep 21 14:44:56 2022 +0200

    [WIP] Conda: List origins from anaconda.com, the Package, dependency and environment management for any language
    
    Related #4547

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/670/ for more details.

Overall, looks god to me but there is still some rooms for a couple of improvements (see inline comments).

Merge request was returned for changes

Lister improvment

Use json.bz2 endpoints instead of .json Remove msys related fixtures as those arch do not have source code to download Tests check artifacts

Build is green

Patch application report for D8517 (id=30697)

Rebasing onto 9b3e565c...

First, rewinding head to replay your work on top of it...
Applying: Conda: List origins from anaconda.com, the Package, dependency and environment management for any language

Changes applied before test

commit 384faae8b1edf07bf107caf555a2308ba56a28f5
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Sep 21 14:44:56 2022 +0200

    Conda: List origins from anaconda.com, the Package, dependency and environment management for any language
    
    Related #4547

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/674/ for more details.

Add documentation for lister usage

Build has FAILED

Patch application report for D8517 (id=30698)

Rebasing onto 9b3e565c...

First, rewinding head to replay your work on top of it...
Applying: Conda: List origins for Anaconda, the package manager that provides tooling for datascience

Changes applied before test

commit 011f60a8f0398dd6c60f8081200939f5bd62c2b4
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Sep 21 14:44:56 2022 +0200

    Conda: List origins for Anaconda, the package manager that provides tooling for datascience
    
    Related #4547

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/675/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/675/console

Add missing documentation link

Build is green

Patch application report for D8517 (id=30699)

Rebasing onto 9b3e565c...

First, rewinding head to replay your work on top of it...
Applying: Conda: List origins for Anaconda, the package manager that provides tooling for datascience

Changes applied before test

commit 0c3e50fe0fb41832250753011f7aaad5fff710c7
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Sep 21 14:44:56 2022 +0200

    Conda: List origins for Anaconda, the package manager that provides tooling for datascience
    
    Related #4547

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/676/ for more details.

Doc: Add missing quote to docker command example to add a conda task

Build is green

Patch application report for D8517 (id=30700)

Rebasing onto 9b3e565c...

First, rewinding head to replay your work on top of it...
Applying: Conda: List origins for Anaconda, the package manager that provides tooling for datascience

Changes applied before test

commit 3f53628d68f2c13ad86a32697abaa7945dc23f9c
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Sep 21 14:44:56 2022 +0200

    Conda: List origins for Anaconda, the package manager that provides tooling for datascience
    
    Related #4547

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/677/ for more details.

Some test report after running the lister (lister only, no loader yet) with following command:

swh scheduler task add list-conda channel="free" archs='["linux-64", "osx-64", "win-64"]' -p oneshot

swh-lister_1                         | [2022-09-23 07:39:37,418: INFO/ForkPoolWorker-1] Task swh.lister.conda.tasks.CondaListerTask[91139a37-dee3-4ebc-9ffd-2c8177e4b2d3] succeeded in 51.53754648496397s: {'pages': 3, 'origins': 2161}

Make use of http_retry instead of throttling_retry decorator after !310 (closed)

Build is green

Patch application report for D8517 (id=30752)

Rebasing onto d5c30a3c...

Current branch diff-target is up to date.

Changes applied before test

commit 928cd6baede495906ed1f07b3a997de851e332fd
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Sep 21 14:44:56 2022 +0200

    Conda: List origins for Anaconda, the package manager that provides tooling for datascience
    
    Related #4547

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/692/ for more details.

Still some changes to bring to that diff now HTTP requests and user-agent setting codes have been deduplicated.

Merge request was returned for changes

Make use of http_request after !311 (closed)

Build is green

Patch application report for D8517 (id=30789)

Rebasing onto fd1a4244...

Current branch diff-target is up to date.

Changes applied before test

commit 011157136966c7d93a7bddb2a49705103c93257a
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Sep 21 14:44:56 2022 +0200

    Conda: List origins for Anaconda, the package manager that provides tooling for datascience
    
    Related #4547

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/700/ for more details.

mentioned in issue #4547

Looks good to me, thanks !