From my own (limited) experience, a 25-minute build time seems pretty unusual for an Alpine image pulling in Python dependencies. The lack of wheel support on Alpine definitely seems like a compelling argument against using it for Python apps, but that build time indicates to me that some confounding factor might be at play here. Maybe matplotlib and/or pandas are just way bigger than the libraries I've pulled into Alpine images before? Curious to hear what others think.
Pandas and numpy (which all things data science depend on) require compilation of C/C++ code which comes in addition to the download/install steps. Wheel files can contain precompiled .so files.