Recently at work, we were having some problems with our application. Most of the problems stemmed from the complicated nature of the application and some poor design that we had been trying to patch up for months. Finally, in November, we got clearance from my boss to rebuild the application as a series of mini-applications that would run behind the main UI.
Initially, we planned on using nanite to distribute the application, passing messages between each mini-application, however, there was a problem getting the nanite mapper to work, so we explored other options and within a few hours we had a working version of beanstalk—a simple message queuing system. Here’s a quick tutorial on setting up beanstalk and using it in your Ruby or Rails application.
Install libevent
First, if you’re on a Mac, you need to install libevent. I downloaded the latest version’s source code at http://www.monkey.org/~provos/libevent/ and compiled it on my mac. (You need the mac development libraries from your install disk to do this.)
If you’re on linux, libevent should be available in your package manager.
To install libevent on your mac, use these commands on the command line.
cd /path/to/libevent- ./configure make sudo make install
Install Beanstalkd
Once you have libevent installed, you can download, compile, and install beanstalkd.
cd /path/to/beanstalkd-.tar.gz tar xvfz beanstalkd-.tar.gz cd beanstalkd- ./configure make cp beanstalkd /usr/bin/beanstalkd
Start Beanstalkd daemon
Now that you have beanstalkd ready to go, start a beanstalk server:
beanstalkd -d -l 127.0.0.1 -p 11300
Install the Beanstalk Client gem
This will start beanstalk on your local machine as a daemon on port 11300.
With beanstalk running, we can now install the ‘beanstalk-client’ gem to give us programmatic access to the beanstalk server.
sudo gem install beanstalk-client
Talking to Beanstalk
Since beanstalk is a message queue, the simplest use is putting things on the queue and taking them back off the queue.
To put messages on the queue, you use the ‘put’ command:
beanstalk = Beanstalk::Pool.new(['127.0.0.1:11300'])
beanstalk.put("message")
When sending an object, hash, or array, you can have your object serialized to YAML before being placed on the queue by using the ‘yput’ method:
beanstalk = Beanstalk::Pool.new(['127.0.0.1:11300'])
beanstalk.yput({:email => "chuck@charlesmaxwood.com"})
If you want to send certain messages to certain queues—beanstalk calls them tubes—call the ‘use’ method:
beanstalk = Beanstalk::Pool.new(['127.0.0.1:11300'])
beanstalk.use("emails")
beanstalk.yput({:email => "chuck@charlesmaxwood.com"})
Listening to Beanstalk
If you want to get a message from the beanstalk queue, you reserve it. When you’re done with it, you delete it.
beanstalk = Beanstalk::Pool.new(['127.0.0.1:11300']) job = beanstalk.reserve puts job.body job.delete
If you want to put the job back in the queue for something else to process after reserving it, you can release it.
beanstalk = Beanstalk::Pool.new(['127.0.0.1:11300']) job = beanstalk.reserve puts job.body job.release
Finally, to consume a queue other than the default queue, you need to watch the other queue and ignore the default queue.
beanstalk = Beanstalk::Pool.new(['127.0.0.1:11300'])
beanstalk.watch('my-queue')
beanstalk.ignore('default')
job = beanstalk.reserve
puts job.body
job.delete
A few things to note are:
- Queue names don’t support underscores. I’ve just been using dashes.
- Max message size is around 65000 characters. To change it, use the -z flag when starting the beanstalk daemon.
Give it a try and let me know how you’re using it!
{ 1 trackback }