A lot of these AppEngine costs too much notices have been coming up recently, but upon further inspection, they all tend to boil down to operator error.
Unfortunately, AppEngine isn't forgiving of that and there is a real monetary value associated with questionable engineering design. Or, design that wasn't thought through enough in the context of a service like AppEngine.
This leads to a few people getting upset and making a lot of noise when the reality is that AppEngine is actually an amazing service.
So, to boil down the operator error from a quote in the thread:
"We're running a mapreduce to change the geobox sizes/precision for a large number of entities."
That is the real source of the problem. Instead of using geoboxes, they should be using geohashes, which allow arbitrary precision.
Finding if the location is in a box would be computing the hash from the lat/lng (there is free code out there to do that) and then doing an indexed 'in' query. The indexes would only need to be updated if the location of the entity changes, not when they want varying levels of precision.
First off, they mention that when the initial design decision was made, a similar operation cost ~$160, which is tenable for an operation that only happens once in a while. This is in fact a case of them getting bitten by the pricing structure changing after a reasonable design decision (at the time) was implemented.
Secondly, they mention that this is part of a larger issue:
"In our most common case we might have to add and delete
a couple items to the list property every once in a while. That would
still cost us well over $1,000 each time.
Most of the reasons for this type of data in our product is to
compensate for the fact that there isn't full text search yet. I know
they are beta testing full text, but I'm still worried that that also
might be too expensive per write."
This is a real problem that GAE needs to solve.
Finally, their problem doesn't seem to be that they need arbitrary precision, its that they seem to need fast location centric queries of a large database.
Geoboxes allow you to solve this problem correctly (and quickly), returning the results in the database that are closest to you. Matching on a geohash can end up serving the incorrect data unless you resort to hacks involving a number of queries.
1) You are making an assumption that the original design decision was made around cost. I bet the more likely fact is that they made the change and found out that it cost them $160. Remember, that was when AppEngine was a _beta_ product and it seems they got lucky the first time.
2) They seem to have an extreme use case. No one is going to argue that maybe AppEngine doesn't fit the bill for them. Or, one could argue that doing 6.5 billion writes times a large number of customers, across multiple datacenters is something that a lot of databases would choke on.
3) Running more queries, while admittedly hacky is less expensive than doing more writes.
We wrote some App Engine code to do geospatial lookups.
Before anyone gets the idea that the links in the parent are worth trying, they're not: the performance is absolutely atrocious – on our data set, 13 seconds per lookup.
We wrote our own approach on App Engine and now get stable performance on our datasets at ~300ms per lookup.
Like Foursquare, mobile clients send their current location, plus a search radius, and App Engine code returns a result set ordered by distance.
We were getting query times with realistic data set sizes in the ~13 second range from the code you linked to. With the code above, we're ~40x faster. YMMV.
You should write this up somewhere and post it to HN and the App Engine Reddit. Getting geospatial right is hard, and it sounds like you've done a good job.
Yes, blame the operators. The operators designed their apps based on the original pricing incentive published by Google but got the rug pulled under them when Google hiked the price to completely change the pricing incentive and invalidate their design.
I got burnt by AppEng, too. Picking AppEng as a platform is one of my worst technical decisions.
"Google App Engine is free to use during the preview release, but the amount of computing resources any app can use is limited. In the future, developers will be able to purchase additional computing resources as needed, but Google App Engine will always be free to get started."
You got 2+ years of use for 'free' and now they decided to turn it into a supported business model and are asking you to pay for what you use. Seems reasonable to me.
I don't disagree that they fubar'd their original pricing release announcement and should have had multithreading Python 2.7 for those folks.
But they did listen to the (loud) feedback, made adjustments and even apologized (were you there at the ThirstyBear meetup where they bought us all beers?).
Not having to hire an IT staff or be woken up in the middle of the night when AWS decides to reboot the host and your servers go down is worth its weight in gold.
Yes, stupid of me to trust the marketing speak in that blog post. I've already said it was one of my worst technical decisions. I've advocated to clients to use AppEng and I looked real stupid when the projects failed because of operation cost hike.
Software are architected and designed with constraints in mind. Features are feasible or unfeasible because of these constraints. Cost is one of the big constraints. With the new cost structure, the apps have to be re-architected and redesigned, lots of things aren't possible.
I didn't go to the ThirstyBear meetup because I have given up on AppEng and moved on. I simply do not have trust for them to be a platform vendor.
If you correctly architected your application around the constraints of AppEngine, then it wouldn't be costing you a lot of money. That is the whole point... you know the constraints, so you architect wisely and you spend money where you need to. You decide which indexes you need and which ones you don't need. You decide how many reads and writes you are going to make. You figure this stuff out in advance and if you make a mistake, you refactor your code to improve things.
I'm not saying it is easy, it isn't. It takes a skilled engineer to learn this stuff and make it work. But, when it does work, it really does work.
It sounds like you failed to create an application that took all of that into account and of course you are going to look 'real stupid' when your clients figure out that you took shortcuts. There is no way that could be the fault of AppEngine.
I'm curious where you went after AppEngine. Are you hosting on AWS now? Heroku? EngineYard? Do you have the same reliability and scalability as what AppEngine provides? Maybe that isn't something that is beneficial to you and in that case, I can see how AppEngine is not your cup of tea.
For me, AppEngine is amazing. I love the fact that I'll never have to hire a sysadmin. I love not carrying a pager. I love knowing that when my site gets an asston of traffic, I won't ever have to think about making it scale. I love not having to worry if my database is on big enough hardware, replicated across data centers, backed up. I'll never have to think about whether or not my OS needs an upgrade to plug a security hole or ssh'ing into a server at 2am to figure some esoteric problem out. To me, all of these things are worth the 'cost' of AppEngine. I'd rather spend my time adding features than doing sysadmin.
At certain point, I am not sure whether you are trolling or not. All I see is you kept repeating the AppEng marketing nonsense.
A skilled engineer would know not all problems are the same and one platform can't solve all the problems. You don't know what product requirements I had and you assert it's my failing since I couldn't beat my product into AppEng's square, despite the AppEng's square turned into a circle.
Just because your niche app happens to fit into AppEng's mold doesn't everyone else can do the same.
We don't know your application, so of course we can't tell if you made poor design decisions or not. But have you considered that the poor match went both ways?
Google said that they adjusted their prices because people were getting wrongly incentivized to do things that were expensive at their end. And the way free market is set up, pricing is one of the ways that signals are sent to tell customers that they should make different optimization decisions --- including, perhaps, switching to a different technology which is a better match for their requirements.
Yes, that can be painful --- but it's the free market. You might as well complain that some silly folks moved to exburbs hours away from their work, and bought gas-guzzling SUV's, and then got upset when the price of gasoline went up to 3-4 dollars/gallon. Whose fault was that? OPEC? Or the consumer for choosing to live far away from work and to buy a car that had horrible mileage?
Valid points except "people getting wrongly incentivized." People were rightly incentizivized to optimize their apps under Google's pricing guideline. It was Google who got it wrong to misalign its infrastructure cost and pricing, and unable to bring down their cost over the years.
I don't think your gas example are relevant but if you want to stretch it, it just means Google like OPEC is not a trusted platform vendor.
I'm definitely not trolling. I'm trying to have a conversation and understanding of your rationale for why GAE didn't work for you. So far, you've left me totally confused.
I heartily agree with you about GAE being a niche. Not all apps (or developers) belong on GAE.
I ask again, if GAE didn't work for you, where are you hosted now?
Now look who is trolling. First you respond with the now deleted statement, which I'll post back here:
"ww520: You were not trying to have a conversation. You were trying to have a bragging session. I don't find it constructive to continue the "conversation."
Then you come back and respond again? I'm so confused. Anyway, I'll consider this thread over, unless you actually want to have a real grown up conversation.
It's funny to see a hotshot troll feinted confusion. It's even funnier to see a rude brat pretending to have a grown up conversation. If you want a grown up conversation, act like one first.
> (were you there at the ThirstyBear meetup where they bought us all beers?)
How is "must live in the Bay Area in order to receive friendly customer service" a reasonable hidden addition to the T&C of a supposedly global service?
Unfortunately, AppEngine isn't forgiving of that and there is a real monetary value associated with questionable engineering design. Or, design that wasn't thought through enough in the context of a service like AppEngine.
This leads to a few people getting upset and making a lot of noise when the reality is that AppEngine is actually an amazing service.
So, to boil down the operator error from a quote in the thread:
"We're running a mapreduce to change the geobox sizes/precision for a large number of entities."
That is the real source of the problem. Instead of using geoboxes, they should be using geohashes, which allow arbitrary precision.
http://code.google.com/apis/maps/articles/geospatial.html http://en.wikipedia.org/wiki/Geohash
Instead of an indexed property that looks like this (what they currently have):
[u'37.3411|-121.8940|37.3395|-121.8926', u'37.3411|-121.8929|37.3395|-121.8916', ...]
They would have an indexed List<String> property that looks like this:
[8, 8f, 8f1, 8f12, 8f12a, 8f12ac, 8f12ac6, 8f12ac60, 8f12ac605, 8f12ac605f, 8f12ac605fb, 8f12ac605fb3, 8f12ac605fb34]
Finding if the location is in a box would be computing the hash from the lat/lng (there is free code out there to do that) and then doing an indexed 'in' query. The indexes would only need to be updated if the location of the entity changes, not when they want varying levels of precision.