I don't want to reveal specific (and perhaps confidential) information about Heroku's architecture, but I can tell you a few things about working in AWS in general.
One: routing traffic in AWS typically means ELB. ELB doesn't necessarily give you fine-grained control over per-application routing.
Two: Proxies don't always send a single HTTP request per TCP connection.
Three: In streaming-data startups, I think it's pretty common that a customer does something inefficient, scales rapidly over the course of a week or two, and then realizes too late that their initial design is no longer suitable for their traffic load. Sometimes a single customer makes up a shockingly large proportion of your traffic load--and this load is hard to predict.
Four: The routing infrastructure for any given app still requires multiple load balancers in multiple availability zones. That means shards are more expensive--in terms of nodes and cost.
Five: Big systems come with lots of moving parts. It might be impractical to create an isolated, fault-tolerant copy of two hundred services for each reasonably-sized customer. There are... all sorts of issues with distributing state.
Thank you for the insight. 3, 4 and 5 are probably the reasons why they don't do it. They could, but it would be harder to operate, a bit more expensive and a lot less elastic. And maybe some customers at some points in time are, in fact, too large for that strategy.
As for 1, they do use ELB now, I don't know if they did during the "intelligent routing" era. It is only possible to create up to 10 different ELBs with a standard AWS account, but something as large as Heroku does not have the same limits.
I don't really understand point 2, could you explain what you mean? I can see how keep-alive can be a problem with some configurations but if you use the DNS to select the shard it shouldn't be a problem (you have one balancer per shard, with ELB it means different CNAME targets).
One: routing traffic in AWS typically means ELB. ELB doesn't necessarily give you fine-grained control over per-application routing.
Two: Proxies don't always send a single HTTP request per TCP connection.
Three: In streaming-data startups, I think it's pretty common that a customer does something inefficient, scales rapidly over the course of a week or two, and then realizes too late that their initial design is no longer suitable for their traffic load. Sometimes a single customer makes up a shockingly large proportion of your traffic load--and this load is hard to predict.
Four: The routing infrastructure for any given app still requires multiple load balancers in multiple availability zones. That means shards are more expensive--in terms of nodes and cost.
Five: Big systems come with lots of moving parts. It might be impractical to create an isolated, fault-tolerant copy of two hundred services for each reasonably-sized customer. There are... all sorts of issues with distributing state.