> At the time of writing this post, the entire PyPI repository is somewhere in the neighbourhood of 1TB but, by using a selective download, I was able to get it down to 120GB or so.
That's the surprising part to me; that config doesn't look like it's filtering by popularity or such, just python versions? I guess packages aren't huge, but I'm still surprised that it's so small.
Maybe PyPI was that small when the post was written. It currently sits at 10.4TB; TensorFlow alone is more than 1TB. I suppose you can still keep the mirror relatively small if you exclude most of the giant ML wheels.
The average Python package distribution is pretty small: source distributions are (or should be) compressed source code, and binary distributions ("wheels") should be (mostly) compressed Python bytecode and the occasional platform-specific binary.
There are also relatively conservative per-project and per-releaase upload limits. PyPI's admins can increase those limits on a project or individual release, but the baseline limits have seemingly preserved the commons reasonably well.
That's the surprising part to me; that config doesn't look like it's filtering by popularity or such, just python versions? I guess packages aren't huge, but I'm still surprised that it's so small.