I'm a big fan of RabbitMQ ( http://www.rabbitmq.com ). It's an AMQP message queue and has a bunch of very nice features like:

- Persistent queues

- Clustering - Topic based queues for an extremely flexible syndication model

- Transactional (acknowledged) queues

- Lots of adapters in different languages

- Fast, handles thousands of messages/sec

It's written in erlang so if you're already running CouchDB then you can maximise any language specific knowledge.

Replicated queues are tricky to set up with any queuing system, but if you're looking to a) not lose any messages and b) have high availability then the simplest thing is just to run 2 independent but identically configured queues in parallel which are configured to be persistent and configure the workers to pull jobs of each queue in turn. If one of them goes down you can restart it and recover the lost jobs whilst the other queue continues.

Alternatively you can write to both queues and deduplicate the messages when you pull them off. Beetle ( http://xing.github.com/beetle/ ) takes this approach and uses Redis to manage the deduplication.

Agreed! When using RabbitMQ / 0MQ or even Redis for queue replacement it's possible to build a great infrastructure around them, you're not sticking to the worker library launcher set. You simply have a queue, you may distribute it and get it working the way you want to.

You might want to check out Amazon SQS for this. It has all the advantages you're looking for (transactional, guaranteed not to lose messages, zero administration, crazy-simple API with solid client libraries in every conceivable language), with the only possible downside being that it's hosted externally.

It's priced so cheaply as to be essentially free, though you'll need to give it a credit card so that it can bill you eleven cents a month or whatever.

http://aws.amazon.com/sqs/

I've been using it for a few years now with good success.

+1 for SQS. It pretty much matches all your criteria to a tee. The replication guarantees messages won't be lost, and will occasionally result in duplicate messages.

I think another potential downside is that the messages get purged after 14 days.

You could have a look at http://celeryproject.org/

I've not used it so I'm not entirely sure it'll cover your needs, but I keep hearing good things about it.

We use rabbitmq and celery, both awesome projects. And both teams are always on irc if you need help.

Thanks. I'd seen it a long time ago, and forgotten about it.

Looking into it now. Seems it gives me choices for the backend including RabbitMQ and Redis.

I've worked mostly with Gearman (made by Danga who also created memcached, mogilefs and other awesomesauce).

It's stable, fast and has API libraries for most languages which can create clients and workers. You may have to do some configuration/customisation to get it to do exactly what you're after but it's a great place to start.

http://gearman.org/

+1 for gearman we use it to power 40 workers and it run without issues more imp learning curve and installation overhead is low ...

Maybe I'm missing something, but it looks like each Job Server is a separate queue? Not sure what happens if one of the job servers dies then.

It does handle load balancing in a nice way.

If you've been "backgrounding" jobs (for async tasks for example where the client wants to fire and forget) Gearman has the option to have a persistent queue. There're a few options for persistence...

1) You can use a local SQLite file. Fast and fine for jobs you don't mind losing once if your entire box goes down. (Cache busting comes to mind)

2) You can use MySQL. If a job server dies, you can restart it and point it at the same MySQL instance. If a job server dies and the entire MACHINE is down, you can spin up another gearmand instance on another machine and point it at the right place.

If you are submitting "foregrounded" tasks, meaning your client requires a response, Gearman's way of handling failure is pretty simple. When gearmand (the server) dies, the client will see you lost a socket connection. It is then up to the client to determine what to do in that failure scenario. It sounds like in your case, you just want to resubmit it. This should be pretty easy to do.

As an FYI, I'm currently the maintainer of the python-gearman 2.x series API. We (derwiki and I) have been using Gearman in production for the past few months now and it's worked out pretty well for us. Implementation's a snap and running the daemon's pretty trivial.

A decent set up is multiple job servers and having worker daemons for the difference queues on multiple machines. As long as you have at least one job server and worker daemon going, you're in business (and you should probably put a Nagios alert on that stuff anyway). It's naive balancing, but it will taking you surprisingly far (we've been using this in production for about 6 months). That said, it's got it's own set of quirks: job servers accepting jobs when they're unable to process them while they replay a transaction log for starters -- and there's a few other subtle "job complete" bugs we haven't been able to figure out yet.

I haven't tested a lot of queues but one solution that i played with recently and liked is zookeeper.

It is distributed, battle tested and has a small simple API. You can build some good distributed data structures(like queues for example) on top of it with minimal work.

For python examples - http://www.cloudera.com/blog/2009/05/building-a-distributed-...

and http://github.com/twitter/twitcher

I am currently using Resque(http://github.com/defunkt/resque) for a similar requirement. You might want to check it out.

edit: This is in Ruby though.

In addition, Delayed Job is also a battle-hardened/tested queue system too, with better retry / error rescuing (ironically enough). It is slower due to MySQL locking though -- Resque is built on top of Redis and is lightning fast.

We use DJ for jobs that MUST succeed (e.g. autopost) whereas we use Resque for more frequent but less essential jobs.

We use Delayed Job at AdGrok for both for async jobs kicked off from user activity as well as nightly maintenance jobs.

If you need support for "job prerequisites", (one job to not start before other jobs have finished successfully), we'll be adding that to our fork from collectiveidea soon. The DJ codebase is pretty nice to look at, too. Note also that there are capistrano dj deployment recipes in the git source.

Looks good too. Ruby is not a problem.

I've now got multiple choices that seem like they can do the job and just need to decide which one is the best fit.

Thanks

I second the vote for Resque: I have been amazed by how easy and stable it is.

MySQL has a storage engine Q4M, which is transactional and persistent. The performance depends on the server's performance, but I have not problem to achieve 10k/sec queue throughput on Core2 Duo 2GHz cpu. You can use the queue with standard mysql client.

Check out the http://q4m.31tools.com/, http://www.slideshare.net/kazuho/q4m-a-highperformance-messa...

I've used beanstalkd before. It has transaction, timeout and sleep features. What I don't like about it is that it adds complexity to server setup for such a simple thing. I've seen queue libraries built around redis which is a win since I'm already using that for other things (check out hotqueue http://richardhenry.github.com/hotqueue/).

I'm kind of a fan of using something like this ( http://lethain.com/entry/2010/sep/05/python-datastructures-b... ) and rolling your own.

If you are running Java or compatible, I recommend JPPF: http://www.jppf.org/

You can try condor http://www.cs.wisc.edu/condor/

We are using this: http://github.com/robey/kestrel

It is written in scala for use at Twitter. It is super simple to getting running and is so far robust. We are using it to pass JSON messages around our system for background processing tasks.