workling 0.4.9.7

Sign up to get free protection for your applications and to get access to all the features.
Files changed (43) hide show
  1. data/CHANGES.markdown +82 -0
  2. data/README.markdown +543 -0
  3. data/TODO.markdown +27 -0
  4. data/VERSION.yml +4 -0
  5. data/bin/workling_client +29 -0
  6. data/contrib/bj_invoker.rb +11 -0
  7. data/contrib/starling_status.rb +37 -0
  8. data/lib/extensions/cattr_accessor.rb +51 -0
  9. data/lib/extensions/mattr_accessor.rb +55 -0
  10. data/lib/workling.rb +213 -0
  11. data/lib/workling/base.rb +110 -0
  12. data/lib/workling/clients/amqp_client.rb +51 -0
  13. data/lib/workling/clients/amqp_exchange_client.rb +58 -0
  14. data/lib/workling/clients/backgroundjob_client.rb +25 -0
  15. data/lib/workling/clients/base.rb +89 -0
  16. data/lib/workling/clients/broker_base.rb +63 -0
  17. data/lib/workling/clients/memcache_queue_client.rb +104 -0
  18. data/lib/workling/clients/memory_queue_client.rb +34 -0
  19. data/lib/workling/clients/not_client.rb +14 -0
  20. data/lib/workling/clients/not_remote_client.rb +17 -0
  21. data/lib/workling/clients/rude_q_client.rb +47 -0
  22. data/lib/workling/clients/spawn_client.rb +46 -0
  23. data/lib/workling/clients/sqs_client.rb +163 -0
  24. data/lib/workling/clients/thread_client.rb +18 -0
  25. data/lib/workling/clients/xmpp_client.rb +110 -0
  26. data/lib/workling/discovery.rb +16 -0
  27. data/lib/workling/invokers/amqp_single_subscriber.rb +42 -0
  28. data/lib/workling/invokers/base.rb +124 -0
  29. data/lib/workling/invokers/basic_poller.rb +38 -0
  30. data/lib/workling/invokers/eventmachine_subscriber.rb +38 -0
  31. data/lib/workling/invokers/looped_subscriber.rb +34 -0
  32. data/lib/workling/invokers/thread_pool_poller.rb +165 -0
  33. data/lib/workling/invokers/threaded_poller.rb +149 -0
  34. data/lib/workling/remote.rb +38 -0
  35. data/lib/workling/return/store/base.rb +42 -0
  36. data/lib/workling/return/store/iterator.rb +24 -0
  37. data/lib/workling/return/store/memory_return_store.rb +24 -0
  38. data/lib/workling/return/store/starling_return_store.rb +30 -0
  39. data/lib/workling/routing/base.rb +13 -0
  40. data/lib/workling/routing/class_and_method_routing.rb +55 -0
  41. data/lib/workling/routing/static_routing.rb +43 -0
  42. data/lib/workling_daemon.rb +111 -0
  43. metadata +96 -0
data/CHANGES.markdown ADDED
@@ -0,0 +1,82 @@
1
+ Version 0.4.8, 07.05.2009
2
+ - implemented XMPP listener (experimental)
3
+ - implemented method for explicitly naming queues, this is useful in conjunction with the XMPP listener
4
+ - rewrote the workling_client script, now allows specifying configuration options on the command line. Makes it easier to manage workling instances in production, and allows running worklings for multiple apps on one machine
5
+ - integrated support for custom exception notifications
6
+
7
+ Version 0.4.2.3, 31.01.2009
8
+ - introduced Workling.raises_exceptions. by default, this is true in test and development to help with bug tracking.
9
+ - added :threaded as the default spawn runner for test and development. helps problem tracing.
10
+
11
+ Version 0.4.2.2, 29.11.08
12
+ - turned Workling.load_path into an Array.
13
+
14
+ Version 0.4.2.1, 27.11.08
15
+ - fixed raise exceptions if non existing worker methods are called
16
+
17
+ Version 0.4.2, 10.11.08
18
+ - added information about invokers and clients to the readme
19
+ - fixed dependence on amqp library
20
+ - nicer error messages with amqp / rabbitmq
21
+
22
+ Version 0.4.1 08.11.08
23
+ - added a generic client runner. deprecated starling_runner since it is now redundant
24
+ - moved connection exception handling code into MemcacheQueueClient and out of pollers
25
+
26
+ Version 0.4.0, 04.11.08
27
+ - more refactored clients and invokers. introduced clear base classes
28
+ - support for 3 invoker strategies: basic poller, threaded poller, eventmachine subscriber
29
+ - amqp support
30
+
31
+ Version 0.3.8, 03.11.08
32
+ - full support for rudeq
33
+ - refactored pollers. now now longer mainly about starling
34
+ - refactored starling client, converted to generalized memcachequeue client.
35
+ - changed runner script to be more generic
36
+
37
+ Version 0.3.1, 15.10.08
38
+ - fixed to autodiscovery code bugs.
39
+ - introduced Workling::VERSION
40
+ - fixed test suite for the case that no memcache client is installed at all
41
+ - fixed AR reconnecting code for Multicore systems (Thanks Brent)
42
+
43
+ Version 0.3, 25.09.08
44
+ - added backgroundjob runner
45
+ - added automatic detection of starling, spawn and backgroundjob to set default runner
46
+ - made logging of exceptions more consistent across runners.
47
+ - added friendlier error message if starling was started on the wrong port.
48
+ - play nice without fiveruns-memcache-client.
49
+ - added better documentation in README and RDOC
50
+
51
+ Version 0.2.5, 02.09.08
52
+ - added automatic setting of spawn runner if the spawn plugin is installed.
53
+
54
+ Version 0.2.4, 08.06.08
55
+ - accept both async_ and asynch_ as prefixes for workling method invocation. thank you francois beausoleil!
56
+ - added memcached configuration options to workling.yml. see example yml for details. thank you larry diehl!
57
+ - re-raise exceptions if there is a problem adding an item to the starling queue. thank you digitalronin!
58
+ - added status script for starling client. thank you andrew carter!
59
+ - applied patches from dave dupre: http://davedupre.com/2008/03/29/ruby-background-tasks-with-starling-part-2/
60
+ - added threading to starling poller. One polling thread can now be set to run per queue.
61
+ - default routing no longer producing queues like a:b:c, this was conflicting with MemCacheClient#stat
62
+ - added handling for memcache exceptions
63
+ - keep the database connection alive
64
+
65
+ Version 0.2.2, 15.02.08, rev 31
66
+ - added blaine cook's suggestion: worklings can now (also) be invoked like this: YourWorkling.asynch_your_method(options)
67
+ - added similar for remote store, which can now be called like this: Workling::Return::Store.set(:key, "value")
68
+
69
+ Version 0.2.1, 14.02.08 rev. 24
70
+ - added WorklingError classes.
71
+ - all runners now suppresses workling exceptions. This brings the local behaviour in line with the remote runners.
72
+
73
+ Version 0.2, 13.02.08 rev. 21
74
+ - progress bars or returning results now possible with return stores. use these to communicate back from your workling.
75
+ - memory store for testing and starling store added.
76
+ - now generates uids for workling jobs. these are returned by the runner.
77
+ - extracted Workling::Clients::Starling
78
+ - clearer file structure for workling
79
+
80
+ Version 0.1, 06.02.08
81
+ - initial release
82
+ - see http://playtype.net/past/2008/2/6/starling_and_asynchrous_tasks_in_ruby_on_rails/ for details.
data/README.markdown ADDED
@@ -0,0 +1,543 @@
1
+ # *digitalhobbit/workling Fork Notes*
2
+
3
+ *This particular Workling fork provides an SQS Client. See instructions below.*
4
+
5
+ # Workling
6
+
7
+ Workling gives your Rails App a simple API that you can use to make code run in the background, outside of the your request.
8
+
9
+ You can configure how the background code will be run. Currently, workling supports Starling, BackgroundJob and Spawn Runners. Workling is a bit like Actve* for background work: you can write your code once, then swap in any of the supported background Runners later. This keeps things flexible.
10
+
11
+ ## Installing Workling
12
+
13
+ The easiest way of getting started with workling is like this:
14
+
15
+ script/plugin install git://github.com/purzelrakete/workling.git
16
+ script/plugin install git://github.com/tra/spawn.git
17
+
18
+ If you're on an older Rails version, there's also a subversion mirror wor workling (I'll do my best to keep it synched) at:
19
+
20
+ script/plugin install http://svn.playtype.net/plugins/workling/
21
+
22
+ ## Writing and calling Workers
23
+
24
+ This is pretty easy. Just put `cow_worker.rb` into into `app/workers`, and subclass `Workling::Base`:
25
+
26
+ # handle asynchronous mooing.
27
+ class CowWorker < Workling::Base
28
+ def moo(options)
29
+ cow = Cow.find(options[:id])
30
+ logger.info("about to moo.")
31
+ cow.moo
32
+ end
33
+ end
34
+
35
+ Make sure you have exactly one hash parameter in your methods, workling passes the job :uid into here. Btw, in case you want to follow along with the Mooing, grab 'cows-not-kittens' off github, it's an example workling project. Look at the branches, there's one for each Runner.
36
+
37
+ Next, you'll want to call your workling in a controller. Your controller might looks like this:
38
+
39
+ class CowsController < ApplicationController
40
+
41
+ # milking has the side effect of causing
42
+ # the cow to moo. we don't want to
43
+ # wait for this while milking, though,
44
+ # it would be a terrible waste ouf our time.
45
+ def milk
46
+ @cow = Cow.find(params[:id])
47
+ CowWorker.asynch_moo(:id => @cow.id)
48
+ end
49
+ end
50
+
51
+ Notice the `asynch_moo` call to `CowWorker`. This will call the `moo` method on the `CowWorker` in the background, passing any parameters you like on. In fact, workling will call whatever comes after asynch_ as a method on the worker instance.
52
+
53
+ ## Worker Lifecycle
54
+
55
+ All worker classes must inherit from this class, and be saved in `app/workers`. The Worker is loaded once, at which point the instance method `create` is called.
56
+
57
+ Calling `async_my_method` on the worker class will trigger background work. This means that the loaded Worker instance will receive a call to the method `my_method(:uid => "thisjobsuid2348732947923")`.
58
+
59
+ ## Exception handling in Workers
60
+
61
+ If an exception is raised in your Worker, it will not be propagated to the calling code by workling. This is because the code is called asynchronously, meaning that exceptions may be raised after the calling code has already returned. If you need your calling code to handle exceptional situations, you have to pass the error into the return store.
62
+
63
+ Workling does log all exceptions that propagate out of the worker methods.
64
+
65
+ Furthermore you can provide custom exception handling by defining the
66
+ notify_exception(exception, method, options)
67
+ in your worker class. If it is present it will be called for every exception
68
+
69
+ ## Logging with Workling
70
+
71
+ `RAILS_DEFAULT_LOGGER` is available in all workers. Workers also have a logger method which returns the default logger, so you can log like this:
72
+
73
+ logger.info("about to moo.")
74
+
75
+ ## Running Workling (in production)
76
+
77
+ The workling daemon can be invoked using the command
78
+
79
+ script/workling_client run
80
+
81
+ This will make it run in the development environment, with the default settings unless overridden in your development.rb (see below)
82
+
83
+ For production use the script takes a couple of options
84
+
85
+ script/workling_client <daemon_options> -- <app_options>
86
+
87
+ The daemon_options configure the runtime environment of the daemon and can be one of the following options:
88
+
89
+ --app-name APP_NAME
90
+ --dir DIR
91
+ --monitor
92
+ --ontop
93
+
94
+ The app-name option is useful if you want to run worklings for multiple Rails apps on the same machine. Each app will need to have its unique name.
95
+ The monitor option should not be specified if you are using Monit or god.
96
+ The dir options specifies where the logs and the pid files are saved.
97
+
98
+ The app_options allow you to specify the configuration of the workling interfaces via the command line:
99
+
100
+ --client CLIENT
101
+ --invoker INVOKER
102
+ --routing ROUTING
103
+ --load-path LOADPATH
104
+ --environment ENVIRONMENT
105
+
106
+ The client, invoker and routing params take the full class names of the relevant plugin classes. load-path allows overriding where workling looks for workers and environment should be self explanatory.
107
+
108
+ The following is a sample of how workling_client can be used with god and AMQP:
109
+
110
+ God.watch do |w|
111
+ script = "cd #{RAILS_ROOT} && workling_client"
112
+ w.name = "myapp-workling"
113
+ w.start = "#{script} start -a myapp-workling -- -e production -i Workling::Remote::Invokers::EventmachineSubscriber -c Workling::Clients::AmqpClient"
114
+ w.restart = "#{script} restart -a myapp-workling -- -e production -i Workling::Remote::Invokers::EventmachineSubscriber -c Workling::Clients::AmqpClient"
115
+ w.stop = "#{script} stop -a myapp-workling"
116
+
117
+ w.pid_file = "#{RAILS_ROOT}/log/myapp-workling.pid"
118
+ end
119
+
120
+
121
+ ## What should I know about the Spawn Runner?
122
+
123
+ Workling automatically detects and uses Spawn, if installed. Spawn basically forks Rails every time you invoke a workling. To see what sort of characteristics this has, go into script/console, and run this:
124
+
125
+ >> fork { sleep 100 }
126
+ => 1060 (the pid is returned)
127
+
128
+ You'll see that this executes pretty much instantly. Run 'top' in another terminal window, and look for the new ruby process. This might be around 30 MB. This tells you that using spawn as a runner will result low latency, but will take at least 30MB for each request you make.
129
+
130
+ You cannot run your workers on a remote machine or cluster them with spawn. You also have no persistence: if you've fired of a lot of work and everything dies, there's no way of picking up where you left off.
131
+
132
+
133
+ ## Gohanlon addition
134
+
135
+ To use Workling with Spawn, you can safely delete the config/workling.yml file generated by 'script/plugin install'.
136
+
137
+ Also in your environment.rb file the following are useful
138
+
139
+ # Run all jobs to be executed in the foreground
140
+ # Workling::Remote.dispatcher = Workling::Remote::Runners::NotRemoteRunner.new
141
+
142
+ # Execute jobs in a forked process using Spawn
143
+ Workling::Remote::Runners::SpawnRunner.options = { :method => :spawn }
144
+ Workling::Remote.dispatcher = Workling::Remote::Runners::SpawnRunner.new
145
+
146
+
147
+ ## Installing Starling
148
+
149
+ As of 27. September 2008, the recommended Starling setup is as follows:
150
+
151
+ gem sources -a http://gems.github.com/
152
+ sudo gem install starling-starling
153
+ mkdir /var/spool/starling
154
+
155
+ The robot Co-Op Memcached Gem version 1.5.0 has several bugs, which have been fixed in the fiveruns-memcache-client gem. The starling-starling gem will install this as a dependency. Refer to the fiveruns README to see what the exact fixes are.
156
+
157
+ The Rubyforge Starling gem is also out of date. Currently, the most authorative Project is starling-starling on github (27. September 2008).
158
+
159
+ Workling will now automatically detect and use Starling, unless you have also installed Spawn. If you have Spawn installed, you need to tell Workling to use Starling by putting this in your environment.rb:
160
+
161
+ Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
162
+
163
+ ## Starting up the required processes
164
+
165
+ Here's what you need to get up and started in development mode. Look in config/workling.yml to see what the default ports are for other environments.
166
+
167
+ sudo starling -d -p 22122
168
+ script/workling_client start
169
+
170
+ ## Configuring workling.yml
171
+
172
+ Workling copies a file called workling.yml into your applications config directory. The config file tells Workling on which port Starling is listening.
173
+
174
+ Notice that the default production port is 15151. This means you'll need to start Starling with -p 15151 on production.
175
+
176
+ You can also use this config file to pass configuration options to the memcache client which workling uses to connect to starling. use the key 'memcache_options' for this.
177
+
178
+ In addition, you can use this config to set the type of marshaling that the amqp client uses. Use the 'ymj' key for this. Current options are 'yaml' to use YAML.dump and YAML.load and 'marshal' to use Marshal.dump and Marshal.load. If no value is set, it defaults to 'marshal'.
179
+
180
+ You can also set sleep time for each Worker. See the key 'listeners' for this. Put in the modularized Class name as a key.
181
+
182
+ development:
183
+ listens_on: localhost:22122
184
+ sleep_time: 2
185
+ reset_time: 30
186
+ listeners:
187
+ Util:
188
+ sleep_time: 20
189
+ memcache_options:
190
+ namespace: myapp_development
191
+ ymj: yaml
192
+
193
+ production:
194
+ listens_on: localhost:22122, localhost:221223, localhost:221224
195
+ sleep_time: 2
196
+ reset_time: 30
197
+ ymj: marshal
198
+
199
+ Note that you can cluster Starling instances by passing a comma separated list of values to
200
+
201
+ Sleep time determines the wait time between polls against polls. A single poll will do one .get on every queue (there is a corresponding queue for each worker method).
202
+
203
+ If there is a memcache error, the Poller will hang for a bit to give it a chance to fire up again and reset the connection. The wait time can be set with the key reset_time.
204
+
205
+ ## Seeing what Starling is doing
206
+
207
+ Starling comes with it's own script, starling_top. If you want statistics specific to workling, run:
208
+
209
+ script/starling_status.rb
210
+
211
+ ## A Quick Starling Primer
212
+
213
+ You might wonder what exactly starling does. Here's a little snippet you can play with to illustrate how it works:
214
+
215
+ 4 # Put messages onto a queue:
216
+ 5 require 'memcache'
217
+ 6 starling = MemCache.new('localhost:22122')
218
+ 7 starling.set('my_queue', 1)
219
+ 8
220
+ 9 # Get messages from the queue:
221
+ 10 require 'memcache'
222
+ 11 starling = MemCache.new('localhost:22122')
223
+ 12 loop { puts starling.get('my_queue') }
224
+ 13
225
+
226
+ # Using RabbitMQ or any Queue Server that supports AMQP
227
+
228
+ RabbitMQ is a reliable, high performance queue server written in erlang. If you're doing high volume messaging and need a high degree of reliability, you should definitely consider using RabbitMQ over Starling.
229
+
230
+ A lot of Ruby people have been talking about using RabbitMQ as their Queue of choice. Soundcloud.com are using it, as is new bamboo founder Johnathan Conway, who is using it at his video startup http://www.vzaar.com/. He says:
231
+
232
+ > RabbitMQ – Now this is the matrons knockers when it comes to kick ass, ultra fast and scalable messaging. It simply rocks, with performance off the hook. It’s written in Erlang and supports the AMPQ protocol.
233
+
234
+ If you're on OSX, you can get started with RabbitMQ by following the installation instructions [here](http://playtype.net/past/2008/10/9/installing_rabbitmq_on_osx/). To get an idea of how to directly connect to RabbitMQ using ruby, have a look at [this article](http://playtype.net/past/2008/10/10/kickass_queuing_over_ruby_using_amqp).
235
+
236
+ Once you've installed RabbitMQ, install the ruby amqp library:
237
+
238
+ gem sources -a http://gems.github.com/ (if necessary)
239
+ sudo gem install tmm1-amqp
240
+
241
+ then configure configure your application to use Amqp by adding this:
242
+
243
+ Workling::Remote.invoker = Workling::Remote::Invokers::EventmachineSubscriber
244
+ Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
245
+ Workling::Remote.dispatcher.client = Workling::Clients::AmqpClient.new
246
+
247
+ Then start the workling Client:
248
+
249
+ 1 ./script/workling_client start
250
+
251
+ You're good.
252
+
253
+ # Using RudeQueue
254
+
255
+ RudeQueue is a Starling-like Queue that runs on top of your database and requires no extra processes. Use this if you don't need very fast job processing and want to avoid managing the extra process starling requires.
256
+
257
+ Install the RudeQ plugin like this:
258
+
259
+ 1 ./script/plugin install git://github.com/matthewrudy/rudeq.git
260
+ 2 rake queue:setup
261
+ 3 rake db:migrate
262
+
263
+ Configure Workling to use RudeQ. Add this to your environment:
264
+
265
+ Workling::Clients::MemcacheQueueClient.memcache_client_class = RudeQ::Client
266
+ Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
267
+
268
+ Now start the Workling Client:
269
+
270
+ 1 ./script/workling_client start
271
+
272
+ You're good.
273
+
274
+ # Using BackgroundJob
275
+
276
+ If you don't want to bother with seperate processes, are not worried about latence or memory footprint, then you might want to use Bj to power workling.
277
+
278
+ Install the Bj plugin like this:
279
+
280
+ 1 ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
281
+ 2 ./script/bj setup
282
+
283
+ Workling will now automatically detect and use Bj, unless you have also installed Starling. If you have Starling installed, you need to tell Workling to use Bj by putting this in your environment.rb:
284
+
285
+ Workling::Remote.dispatcher = Workling::Remote::Runners::BackgroundjobRunner.new
286
+
287
+
288
+ # Using XMPP listener
289
+
290
+ NOTE: this code is highly experimental. It was implemented in order to facilitate the async communication between multiple Rails app running in the same domain/datacentre. However it is no longer in production use, since the setup proved to be both too complex to maintain and quite unreliable. The AMQP support turned out to be much more appropriate.
291
+ I'm putting this here as a starting base for someone interested in using XMPP in such a way. Contact me at derfred on github if you want pointers.
292
+
293
+ this client requires the xmpp4r gem
294
+
295
+ in the config/environments/development.rb file (or production.rb etc)
296
+
297
+ Workling::Remote::Runners::ClientRunner.client = Workling::Clients::XmppClient.new
298
+ Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new # dont use the standard runner
299
+ Workling::Remote.invoker = Workling::Remote::Invokers::LoopedSubscriber # does not work with the EventmachineSubscriber Invoker
300
+
301
+ furthermore in the workling.yml file you need to set up the server details for your XMPP server
302
+
303
+ development:
304
+ listens_on: "localhost:22122"
305
+ jabber_id: "sub@localhost/laptop"
306
+ jabber_server: "localhost"
307
+ jabber_password: "sub"
308
+ jabber_service: "pubsub.derfredtop.local"
309
+
310
+ for details on how to configure your XMPP server (ejabberd) check out the following howto:
311
+
312
+ http://keoko.wordpress.com/2008/12/17/xmpp-pubsub-with-ejabberd-and-xmpp4r/
313
+
314
+
315
+ finally you need to expose your worker methods to XMPP nodes like so:
316
+
317
+ class NotificationWorker < Workling::Base
318
+
319
+ expose :receive_notification, :as => "/home/localhost/pub/sub"
320
+
321
+ def receive_notification(input)
322
+ # something here
323
+ end
324
+
325
+ end
326
+
327
+ # Using SQS
328
+
329
+ If you're running on Amazon EC2, you may want to leverage SQS (Simple Queue Service) to benefit from this highly scalable queue implementation without having to install any software.
330
+
331
+ The SQS Client namespaces queues with an optional prefix as well as with the Rails environment, allowing us to distinguish between production and staging queues, for example. Queues are automatically created the first time they are accessed.
332
+
333
+ Configuring Workling to use SQS is very straightforward and requires no additional software, with the exception of the RightAws gem.
334
+
335
+ ## Installing the SQS Client
336
+
337
+ Install the RightAws gem:
338
+
339
+ 1. sudo gem install right_aws
340
+
341
+ Configure Workling to use the SqsClient. Add this to your environment:
342
+
343
+ Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
344
+ Workling::Remote.dispatcher.client = Workling::Clients::SqsClient.new
345
+
346
+ Add your AWS key id and secret key to workling.yml:
347
+
348
+ production:
349
+ sqs_options:
350
+ aws_access_key_id: <your AWS access key id>
351
+ aws_secret_access_key: <your AWS secret access key>
352
+
353
+ You can optionally override the following settings, although the defaults
354
+ will likely be sufficient:
355
+
356
+ # Queue names consist of an optional prefix, followed by the environment
357
+ # and the name of the key.
358
+ prefix: foo_
359
+
360
+ # The number of SQS messages to retrieve at once. The maximum and default
361
+ # value is 10.
362
+ messages_per_req: 10
363
+
364
+ # The SQS visibility timeout for retrieved messages. Defaults to 30 seconds.
365
+ visibility_timeout: 30
366
+
367
+ Now start the Workling Client:
368
+
369
+ 1 ./script/workling_client start
370
+
371
+ You're good.
372
+
373
+ ## Limitations
374
+
375
+ SQS messages need to be explicitly deleted from the queue. Otherwise, they will reappear after the visibility timeout. The SQS client currently deletes a message immediately before handing it to a worker, assuming that it will be processed successfully. A more robust implementation (which would require additional hooks in the Workling framework) would be to defer the deletion until after the message was successfully processed, allowing us to retry the message processing in case of an error.
376
+
377
+ # Progress indicators and return stores
378
+
379
+ Your worklings can write back to a return store. This allows you to write progress indicators, or access results from your workling. As above, this is fairly slim. Again, you can swap in any return store implementation you like without changing your code. They all behave like memcached. For tests, there is a memory return store, for production use there is currently a starling return store. You can easily add a new return store (over the database for instance) by subclassing `Workling::Return::Store::Base`. Configure it like this in your test environment:
380
+
381
+ Workling::Return::Store.instance = Workling::Return::Store::MemoryReturnStore.new
382
+
383
+ Setting and getting values works as follows. Read the next paragraph to see where the job-id comes from.
384
+
385
+ Workling.return.set("job-id-1", "moo")
386
+ Workling.return.get("job-id-1") => "moo"
387
+
388
+ Here is an example worker that crawls an addressbook and puts results into a return store. Workling makes sure you have a :uid in your argument hash - set the value into the return store using this uid as a key:
389
+
390
+ require 'blackbook'
391
+ class NetworkWorker < Workling::Base
392
+ def search(options)
393
+ results = Blackbook.get(options[:key], options[:username], options[:password])
394
+ Workling.return.set(options[:uid], results)
395
+ end
396
+ end
397
+
398
+ call your workling as above:
399
+
400
+ @uid = NetworkWorker.asynch_search(:key => :gmail, :username => "foo@gmail.com", :password => "bar")
401
+
402
+ you can now use the @uid to query the return store:
403
+
404
+ results = Workling.return.get(@uid)
405
+
406
+ of course, you can use this for progress indicators. just put the progress into the return store.
407
+
408
+ enjoy!
409
+
410
+ ## Adding new work brokers to Workling
411
+
412
+ There are two new base classes you can extend to add new brokers. I'll describe how this is done usin amqp as an example. The code i show is already a part of workling.
413
+
414
+ ### Clients
415
+
416
+ Clients help workling to connect to job brokers. To add an AmqpClient, we need to extend from `Workling::Client::Base` and implement a couple of methods.
417
+
418
+ require 'workling/clients/base'
419
+ require 'mq'
420
+
421
+ #
422
+ # An Ampq client
423
+ #
424
+ module Workling
425
+ module Clients
426
+ class AmqpClient < Workling::Clients::Base
427
+
428
+ # starts the client.
429
+ def connect
430
+ @amq = MQ.new
431
+ end
432
+
433
+ # stops the client.
434
+ def close
435
+ @amq.close
436
+ end
437
+
438
+ # request work
439
+ def request(queue, value)
440
+ @amq.queue(queue).publish(value)
441
+ end
442
+
443
+ # retrieve work
444
+ def retrieve(queue)
445
+ @amq.queue(queue)
446
+ end
447
+
448
+ # subscribe to a queue
449
+ def subscribe(queue)
450
+ @amq.queue(queue).subscribe do |value|
451
+ yield value
452
+ end
453
+ end
454
+
455
+ end
456
+ end
457
+ end
458
+
459
+ Were's using the eventmachine amqp client for this, you can find it [up on github](http://github.com/tmm1/amqp/tree/master). `connect` and `close` do exactly what it says on the tin: connecting to rabbitmq and closing the connection.
460
+
461
+ `request` and `retrieve` are responsible for placing work on rabbitmq. The methods are passed the correct queue, and a value that contains the worker method arguments. If you need control over the queue names, look at the RDoc for Workling::Routing::Base. In our case, there's no special requirement here.
462
+
463
+ Finally, we implement a `subscribe` method. Use this if your broker supports callbacks, as is the case with amqp. This method expects to a block, which we pass into the amqp subscribe method here. The block will be called when a message is available on the queue, and the result is yielded into the block.
464
+
465
+ Having subscription callbacks is very nice, because this way, we don't need to keep calling `get` on the queue to see if something new is waiting.
466
+
467
+ So now we're done! That's all you need to add RabbitMQ to workling. Configure it in your application as descibed below.
468
+
469
+ ### Invokers
470
+
471
+ There's still potential to improve things though. Workling 0.4.0 introduces the idea of invokers. Invokers grab work off a job broker, using a client (see above). They subclass Workling::Remote::Invokers::Base. Read the RDoc for a description of the methods.
472
+
473
+ Workling comes with a couple of standard invokers, like the BasicPoller. This invoker simply keeps hitting the broker every n seconds, checking for new work and executing it immediately. The ThreadedInvoker does the same, but spawns a Thread for every Worker class the project defines.
474
+
475
+ So Amqp: it would be nice if we had an invoker that makes use of the subscription callbacks. Easily done, lets have a look:
476
+
477
+ require 'eventmachine'
478
+ require 'workling/remote/invokers/base'
479
+
480
+ #
481
+ # Subscribes the workers to the correct queues.
482
+ #
483
+ module Workling
484
+ module Remote
485
+ module Invokers
486
+ class EventmachineSubscriber < Workling::Remote::Invokers::Base
487
+
488
+ def initialize(routing, client_class)
489
+ super
490
+ end
491
+
492
+ #
493
+ # Starts EM loop and sets up subscription callbacks for workers.
494
+ #
495
+ def listen
496
+ EM.run do
497
+ connect do
498
+ routes.each do |queue|
499
+ @client.subscribe(queue) do |args|
500
+ run(queue, args)
501
+ end
502
+ end
503
+ end
504
+ end
505
+ end
506
+
507
+ def stop
508
+ EM.stop if EM.reactor_running?
509
+ end
510
+ end
511
+ end
512
+ end
513
+ end
514
+
515
+ Invokers have to implement two methods, `listen` and `stop`. Listen starts the main listener loop, which is responsible for starting work when it becomes available.
516
+
517
+ In our case, we need to start an EM loop around `listen`. This is because the Ruby AMQP library needs to run inside of an eventmachine reactor loop.
518
+
519
+ Next, inside of `listen`, we need to iterate through all defined routes. There is a route for each worker method you defined in your application. The routes double as queue names. For this, you can use the helper method `routes`. Now we attach a callback to each queue. We can use the helper method `run`, which executes the worker method associated with the queue, passing along any supplied arguments.
520
+
521
+ That's it! We now have a more effective Invoker.
522
+
523
+
524
+
525
+ # Contributors
526
+
527
+ The following people contributed code to workling so far. Many thanks :) If I forgot anybody, I aplogise. Just drop me a note and I'll add you to the project so that you can amend this!
528
+
529
+ Anybody who contributes fixes (with tests), or new functionality (with tests) which is pulled into the main project, will also be be added to the project.
530
+
531
+ * Andrew Carter (ascarter)
532
+ * Chris Gaffney (gaffneyc)
533
+ * Matthew Rudy (matthewrudy)
534
+ * Larry Diehl (reeze)
535
+ * grantr (francios)
536
+ * David (digitalronin)
537
+ * Dave Dupré
538
+ * Douglas Shearer (dougal)
539
+ * Nick Plante (zapnap)
540
+ * Brent
541
+ * Evan Light (elight)
542
+
543
+ Copyright (c) 2008 play/type GmbH, released under the MIT license