The advantage of tor over just E2EE is that it hides the sources of traffic and not only the traffic itself.

So it really depends on if your threat model can handle others knowing that you're using Matrix. It most likely can.