That solution simply does not work. If your browser will accept the certificate that the server presents, anyone can pretend to be anyone and your browser will accept it. Enforcing identity is just as important as encrypting the connection.
You can do that identity enforcement through some scheme (accept-first-time, or web-of-trust, or something else) but they all have significant downsides. It's not an easy problem.
Most browsers let you accept a particular certificate for a particular site already, just like SSH, so you can already have this if you want. Part of the reason it works so well for SSH, though, is "security through unpopularity": you don't have large numbers of naïve users doing online banking at Starbucks over SSH. If SSH became more popular, you'd need to verify the fingerprint, and then you're back to square one ("If I have a secure channel with which to verify the identity of the SSH key, why not just use this same channel to establish the secure session?").
You can do that identity enforcement through some scheme (accept-first-time, or web-of-trust, or something else) but they all have significant downsides. It's not an easy problem.