@vlorentz As per tqdm, it's downloading 10k files in ~2.73 seconds. I got 30 files of nearly the same size (I'm calling them chunks) containing ~40M entries each. That's 1.2 billion files to process. So it will take 91 hours (((40_000_000 * 30) / 10_000) * 2.73 / 60 / 60) to just download all the files in batches of 100.

Do you have any suggestions on how to improve this?

I don't know how it compares, but the way I do it is just start a dozen threads which download in parallel.

91 hours isn't that bad considering you are working at the scale of the whole archive.

how many threads exactly? I'm using 20 at the moment.

Ohh, wait. I made a calculation mistake. It's downloading only 100 files in 2.73s. So that's 9100 hours (379 days). Pretty bad.

Do you have any example script that I can refer to optimize/rewrite my code?

https://gitlab.softwareheritage.org/swh/devel/swh-graph/-/blob/f0fd43b9f07a6180c0f8f9a4929413e87beca985/swh/graph/luigi/blobs_datasets.py#L587-800

Awesome. Thanks a lot!