reqless 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (81) hide show
  1. checksums.yaml +7 -0
  2. data/Gemfile +8 -0
  3. data/README.md +648 -0
  4. data/Rakefile +117 -0
  5. data/bin/docker-build-and-test +22 -0
  6. data/exe/reqless-web +11 -0
  7. data/lib/reqless/config.rb +31 -0
  8. data/lib/reqless/failure_formatter.rb +43 -0
  9. data/lib/reqless/job.rb +496 -0
  10. data/lib/reqless/job_reservers/ordered.rb +29 -0
  11. data/lib/reqless/job_reservers/round_robin.rb +46 -0
  12. data/lib/reqless/job_reservers/shuffled_round_robin.rb +21 -0
  13. data/lib/reqless/lua/reqless-lib.lua +2965 -0
  14. data/lib/reqless/lua/reqless.lua +2545 -0
  15. data/lib/reqless/lua_script.rb +90 -0
  16. data/lib/reqless/middleware/requeue_exceptions.rb +94 -0
  17. data/lib/reqless/middleware/retry_exceptions.rb +72 -0
  18. data/lib/reqless/middleware/sentry.rb +66 -0
  19. data/lib/reqless/middleware/timeout.rb +63 -0
  20. data/lib/reqless/queue.rb +189 -0
  21. data/lib/reqless/queue_priority_pattern.rb +16 -0
  22. data/lib/reqless/server/static/css/bootstrap-responsive.css +686 -0
  23. data/lib/reqless/server/static/css/bootstrap-responsive.min.css +12 -0
  24. data/lib/reqless/server/static/css/bootstrap.css +3991 -0
  25. data/lib/reqless/server/static/css/bootstrap.min.css +689 -0
  26. data/lib/reqless/server/static/css/codemirror.css +112 -0
  27. data/lib/reqless/server/static/css/docs.css +839 -0
  28. data/lib/reqless/server/static/css/jquery.noty.css +105 -0
  29. data/lib/reqless/server/static/css/noty_theme_twitter.css +137 -0
  30. data/lib/reqless/server/static/css/style.css +200 -0
  31. data/lib/reqless/server/static/favicon.ico +0 -0
  32. data/lib/reqless/server/static/img/glyphicons-halflings-white.png +0 -0
  33. data/lib/reqless/server/static/img/glyphicons-halflings.png +0 -0
  34. data/lib/reqless/server/static/js/bootstrap-alert.js +94 -0
  35. data/lib/reqless/server/static/js/bootstrap-scrollspy.js +125 -0
  36. data/lib/reqless/server/static/js/bootstrap-tab.js +130 -0
  37. data/lib/reqless/server/static/js/bootstrap-tooltip.js +270 -0
  38. data/lib/reqless/server/static/js/bootstrap-typeahead.js +285 -0
  39. data/lib/reqless/server/static/js/bootstrap.js +1726 -0
  40. data/lib/reqless/server/static/js/bootstrap.min.js +6 -0
  41. data/lib/reqless/server/static/js/codemirror.js +2972 -0
  42. data/lib/reqless/server/static/js/jquery.noty.js +220 -0
  43. data/lib/reqless/server/static/js/mode/javascript.js +360 -0
  44. data/lib/reqless/server/static/js/theme/cobalt.css +18 -0
  45. data/lib/reqless/server/static/js/theme/eclipse.css +25 -0
  46. data/lib/reqless/server/static/js/theme/elegant.css +10 -0
  47. data/lib/reqless/server/static/js/theme/lesser-dark.css +45 -0
  48. data/lib/reqless/server/static/js/theme/monokai.css +28 -0
  49. data/lib/reqless/server/static/js/theme/neat.css +9 -0
  50. data/lib/reqless/server/static/js/theme/night.css +21 -0
  51. data/lib/reqless/server/static/js/theme/rubyblue.css +21 -0
  52. data/lib/reqless/server/static/js/theme/xq-dark.css +46 -0
  53. data/lib/reqless/server/views/_job.erb +259 -0
  54. data/lib/reqless/server/views/_job_list.erb +8 -0
  55. data/lib/reqless/server/views/_pagination.erb +7 -0
  56. data/lib/reqless/server/views/about.erb +130 -0
  57. data/lib/reqless/server/views/completed.erb +11 -0
  58. data/lib/reqless/server/views/config.erb +14 -0
  59. data/lib/reqless/server/views/failed.erb +48 -0
  60. data/lib/reqless/server/views/failed_type.erb +18 -0
  61. data/lib/reqless/server/views/job.erb +17 -0
  62. data/lib/reqless/server/views/layout.erb +451 -0
  63. data/lib/reqless/server/views/overview.erb +137 -0
  64. data/lib/reqless/server/views/queue.erb +125 -0
  65. data/lib/reqless/server/views/queues.erb +45 -0
  66. data/lib/reqless/server/views/tag.erb +6 -0
  67. data/lib/reqless/server/views/throttles.erb +38 -0
  68. data/lib/reqless/server/views/track.erb +75 -0
  69. data/lib/reqless/server/views/worker.erb +34 -0
  70. data/lib/reqless/server/views/workers.erb +14 -0
  71. data/lib/reqless/server.rb +549 -0
  72. data/lib/reqless/subscriber.rb +74 -0
  73. data/lib/reqless/test_helpers/worker_helpers.rb +55 -0
  74. data/lib/reqless/throttle.rb +57 -0
  75. data/lib/reqless/version.rb +5 -0
  76. data/lib/reqless/worker/base.rb +237 -0
  77. data/lib/reqless/worker/forking.rb +215 -0
  78. data/lib/reqless/worker/serial.rb +41 -0
  79. data/lib/reqless/worker.rb +5 -0
  80. data/lib/reqless.rb +309 -0
  81. metadata +399 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 7ee6251511542e00f09fbb78040abb3da7432ef449169bae1ee15feae9404822
4
+ data.tar.gz: ed756c0fb6a7e2a1e16b0072feb1518c533c4dbcc184426117099933e3db0b04
5
+ SHA512:
6
+ metadata.gz: f9079197e0f9d77fb846ea8a37c881e95578f1208e0a58ea5dcec7ebdf70b15cfd63f119c5b0c3fa8a0af820b7f7510d6c44bdd81e17e2ea0a4d26430d3ee215
7
+ data.tar.gz: c7004e03c3facd74f28789a44284cc8c38fab213ab99cec91f6086d83a2f95724216d1fdbc73bbc136ca65a1a2279029b92901271dfeef072db57db981d3087b
data/Gemfile ADDED
@@ -0,0 +1,8 @@
1
+ source 'http://rubygems.org'
2
+
3
+ gemspec
4
+
5
+ group :development do
6
+ gem 'pry', '~> 0.14.2'
7
+ gem 'pry-stack_explorer', '~> 0.6.1'
8
+ end
data/README.md ADDED
@@ -0,0 +1,648 @@
1
+ reqless
2
+ =====
3
+
4
+ Reqless is a powerful `Redis`-based job queueing system inspired by
5
+ [resque](https://github.com/defunkt/resque#readme),
6
+ but built on a collection of Lua scripts, maintained in the
7
+ [reqless-core](https://github.com/tdg5/reqless-core) repo.
8
+
9
+ Philosophy and Nomenclature
10
+ ===========================
11
+ A `job` is a unit of work identified by a job id or `jid`. A `queue` can contain
12
+ several jobs that are scheduled to be run at a certain time, several jobs that are
13
+ waiting to run, and jobs that are currently running. A `worker` is a process on a
14
+ host, identified uniquely, that asks for jobs from the queue, performs some process
15
+ associated with that job, and then marks it as complete. When it's completed, it
16
+ can be put into another queue.
17
+
18
+ Jobs can only be in one queue at a time. That queue is whatever queue they were last
19
+ put in. So if a worker is working on a job, and you move it, the worker's request to
20
+ complete the job will be ignored.
21
+
22
+ A job can be `canceled`, which means it disappears into the ether, and we'll never
23
+ pay it any mind ever again. A job can be `dropped`, which is when a worker fails
24
+ to heartbeat or complete the job in a timely fashion, or a job can be `failed`,
25
+ which is when a host recognizes some systematically problematic state about the
26
+ job. A worker should only fail a job if the error is likely not a transient one;
27
+ otherwise, that worker should just drop it and let the system reclaim it.
28
+
29
+ Features
30
+ ========
31
+
32
+ 1. __Jobs don't get dropped on the floor__ -- Sometimes workers drop jobs. Reqless
33
+ automatically picks them back up and gives them to another worker
34
+ 1. __Tagging / Tracking__ -- Some jobs are more interesting than others. Track those
35
+ jobs to get updates on their progress. Tag jobs with meaningful identifiers to
36
+ find them quickly in the UI.
37
+ 1. __Job Dependencies__ -- One job might need to wait for another job to complete
38
+ 1. __Stats__ -- `reqless` automatically keeps statistics about how long jobs wait
39
+ to be processed and how long they take to be processed. Currently, we keep
40
+ track of the count, mean, standard deviation, and a histogram of these times.
41
+ 1. __Job data is stored temporarily__ -- Job info sticks around for a configurable
42
+ amount of time so you can still look back on a job's history, data, etc.
43
+ 1. __Priority__ -- Jobs with the same priority get popped in the order they were
44
+ inserted; a higher priority means that it gets popped faster
45
+ 1. __Retry logic__ -- Every job has a number of retries associated with it, which are
46
+ renewed when it is put into a new queue or completed. If a job is repeatedly
47
+ dropped, then it is presumed to be problematic, and is automatically failed.
48
+ 1. __Web App__ -- With the advent of a Ruby client, there is a Sinatra-based web
49
+ app that gives you control over certain operational issues
50
+ 1. __Scheduled Work__ -- Until a job waits for a specified delay (defaults to 0),
51
+ jobs cannot be popped by workers
52
+ 1. __Recurring Jobs__ -- Scheduling's all well and good, but we also support
53
+ jobs that need to recur periodically.
54
+ 1. __Notifications__ -- Tracked jobs emit events on pubsub channels as they get
55
+ completed, failed, put, popped, etc. Use these events to get notified of
56
+ progress on jobs you're interested in.
57
+
58
+ Enqueing Jobs
59
+ =============
60
+ First things first, require `reqless` and create a client. The client accepts all the
61
+ same arguments that you'd use when constructing a redis client.
62
+
63
+ ``` ruby
64
+ require 'reqless'
65
+
66
+ # Connect to localhost
67
+ client = Reqless::Client.new
68
+ # Connect to somewhere else
69
+ client = Reqless::Client.new(:host => 'foo.bar.com', :port => 1234)
70
+ ```
71
+
72
+ Jobs should be classes or modules that define a `perform` method, which
73
+ must accept a single `job` argument:
74
+
75
+ ``` ruby
76
+ class MyJobClass
77
+ def self.perform(job)
78
+ # job is an instance of `Reqless::Job` and provides access to
79
+ # job.data, a means to cancel the job (job.cancel), and more.
80
+ end
81
+ end
82
+ ```
83
+
84
+ Now you can access a queue, and add a job to that queue.
85
+
86
+ ``` ruby
87
+ # This references a new or existing queue 'testing'
88
+ queue = client.queues['testing']
89
+ # Let's add a job, with some data. Returns Job ID
90
+ queue.put(MyJobClass, :hello => 'howdy')
91
+ # => "0c53b0404c56012f69fa482a1427ab7d"
92
+ # Now we can ask for a job
93
+ job = queue.pop
94
+ # => <Reqless::Job 0c53b0404c56012f69fa482a1427ab7d (MyJobClass / testing)>
95
+ # And we can do the work associated with it!
96
+ job.perform
97
+ ```
98
+
99
+ The job data must be serializable to JSON, and it is recommended
100
+ that you use a hash for it. See below for a list of the supported job options.
101
+
102
+ The argument returned by `queue.put` is the job ID, or jid. Every Reqless
103
+ job has a unique jid, and it provides a means to interact with an
104
+ existing job:
105
+
106
+ ``` ruby
107
+ # find an existing job by it's jid
108
+ job = client.jobs[jid]
109
+
110
+ # Query it to find out details about it:
111
+ job.klass # => the class of the job
112
+ job.queue # => the queue the job is in
113
+ job.data # => the data for the job
114
+ job.history # => the history of what has happened to the job sofar
115
+ job.dependencies # => the jids of other jobs that must complete before this one
116
+ job.dependents # => the jids of other jobs that depend on this one
117
+ job.priority # => the priority of this job
118
+ job.tags # => array of tags for this job
119
+ job.original_retries # => the number of times the job is allowed to be retried
120
+ job.retries_left # => the number of retries left
121
+
122
+ # You can also change the job in various ways:
123
+ job.requeue("some_other_queue") # move it to a new queue
124
+ job.cancel # cancel the job
125
+ job.tag("foo") # add a tag
126
+ job.untag("foo") # remove a tag
127
+ ```
128
+
129
+ Running A Worker
130
+ ================
131
+
132
+ The Reqless ruby worker was heavily inspired by Resque's worker,
133
+ but thanks to the power of the reqless-core lua scripts, it is
134
+ *much* simpler and you are welcome to write your own (e.g. if
135
+ you'd rather save memory by not forking the worker for each job).
136
+
137
+ As with resque...
138
+
139
+ * The worker forks a child process for each job in order to provide
140
+ resilience against memory leaks. Pass the `RUN_AS_SINGLE_PROCESS`
141
+ environment variable to force Reqless to not fork the child process.
142
+ Single process mode should only be used in some test/dev
143
+ environments.
144
+ * The worker updates its procline with its status so you can see
145
+ what workers are doing using `ps`.
146
+ * The worker registers signal handlers so that you can control it
147
+ by sending it signals.
148
+ * The worker is given a list of queues to pop jobs off of.
149
+ * The worker logs out put based on `VERBOSE` or `VVERBOSE` (very
150
+ verbose) environment variables.
151
+ * Reqless ships with a rake task (`reqless:work`) for running workers.
152
+ It runs `reqless:setup` before starting the main work loop so that
153
+ users can load their environment in that task.
154
+ * The sleep interval (for when there is no jobs available) can be
155
+ configured with the `INTERVAL` environment variable.
156
+
157
+ Resque uses queues for its notion of priority. In contrast, reqless
158
+ has priority support built-in. Thus, the worker supports two strategies
159
+ for what order to pop jobs off the queues: ordered and round-robin.
160
+ The ordered reserver will keep popping jobs off the first queue until
161
+ it is empty, before trying to pop job off the second queue. The
162
+ round-robin reserver will pop a job off the first queue, then the second
163
+ queue, and so on. You could also easily implement your own.
164
+
165
+ To start a worker, write a bit of Ruby code that instantiates a
166
+ worker and runs it. You could write a rake task to do this, for
167
+ example:
168
+
169
+ ``` ruby
170
+ namespace :reqless do
171
+ desc "Run a Reqless worker"
172
+ task :work do
173
+ # Load your application code. All job classes must be loaded.
174
+ require 'my_app/environment'
175
+
176
+ # Require the parts of reqless you need
177
+ require 'reqless'
178
+ require 'reqless/job_reservers/ordered'
179
+ require 'reqless/worker'
180
+
181
+ # Create a client
182
+ client = Reqless::Client.new(:host => 'foo.bar.com', :port => 1234)
183
+
184
+ # Get the queues you use
185
+ queues = %w[ queue_1 queue_2 ].map do |name|
186
+ client.queues[name]
187
+ end
188
+
189
+ # Create a job reserver; different reservers use different
190
+ # strategies for which order jobs are popped off of queues
191
+ reserver = Reqless::JobReservers::Ordered.new(queues)
192
+
193
+ # Create a forking worker that uses the given reserver to pop jobs.
194
+ worker = Reqless::Workers::ForkingWorker.new(reserver)
195
+
196
+ # Start the worker!
197
+ worker.run
198
+ end
199
+ end
200
+ ```
201
+
202
+ The following signals are supported in the parent process:
203
+
204
+ * TERM: Shutdown immediately, stop processing jobs.
205
+ * INT: Shutdown immediately, stop processing jobs.
206
+ * QUIT: Shutdown after the current job has finished processing.
207
+ * USR1: Kill the forked child immediately, continue processing jobs.
208
+ * USR2: Don't process any new jobs, and dump the current backtrace.
209
+ * CONT: Start processing jobs again after a USR2
210
+
211
+ You should send these to the master process, not the child.
212
+
213
+ The child process supports the `USR2` signal, whch causes it to
214
+ dump its current backtrace.
215
+
216
+ Workers also support middleware modules that can be used to inject
217
+ logic before, after or around the processing of a single job in
218
+ the child process. This can be useful, for example, when you need to
219
+ re-establish a connection to your database in each job.
220
+
221
+ Define a module with an `around_perform` method that calls `super` where you
222
+ want the job to be processed:
223
+
224
+ ``` ruby
225
+ module ReEstablishDBConnection
226
+ def around_perform(job)
227
+ MyORM.establish_connection
228
+ super
229
+ end
230
+ end
231
+ ```
232
+
233
+ Then, mix-it into the worker class. You can mix-in as many
234
+ middleware modules as you like:
235
+
236
+ ``` ruby
237
+ require 'reqless/worker'
238
+ Reqless::Worker.class_eval do
239
+ include ReEstablishDBConnection
240
+ include SomeOtherAwesomeMiddleware
241
+ end
242
+ ```
243
+
244
+ Per-Job Middlewares
245
+ ===================
246
+
247
+ Reqless also supports middleware on a per-job basis, when you have some
248
+ orthogonal logic to run in the context of some (but not all) jobs.
249
+
250
+ Per-job middlewares are defined the same as worker middlewares:
251
+
252
+ ``` ruby
253
+ module ReEstablishDBConnection
254
+ def around_perform(job)
255
+ MyORM.establish_connection
256
+ super
257
+ end
258
+ end
259
+ ```
260
+
261
+ To add them to a job class, you first have to make your job class
262
+ middleware-enabled by extending it with
263
+ `Reqless::Job::SupportsMiddleware`, then extend your middleware
264
+ modules:
265
+
266
+ ``` ruby
267
+ class MyJobClass
268
+ extend Reqless::Job::SupportsMiddleware
269
+ extend ReEstablishDBConnection
270
+ extend MyOtherAwesomeMiddleware
271
+
272
+ def self.perform(job)
273
+ end
274
+ end
275
+ ```
276
+
277
+ Note that `Reqless::Job::SupportsMiddleware` must be extended onto your
278
+ job class _before_ any other middleware modules.
279
+
280
+ Web Interface
281
+ =============
282
+
283
+ Reqless ships with a resque-inspired web app that lets you easily
284
+ deal with failures and see what it is processing. If you're project
285
+ has a rack-based ruby web app, we recommend you mount Reqless's web app
286
+ in it. Here's how you can do that with `Rack::Builder` in your `config.ru`:
287
+
288
+ ``` ruby
289
+ client = Reqless::Client.new(:host => "some-host", :port => 7000)
290
+
291
+ Rack::Builder.new do
292
+ use SomeMiddleware
293
+
294
+ map('/some-other-app') { run Apps::Something.new }
295
+ map('/reqless') { run Reqless::Server.new(client) }
296
+ end
297
+ ```
298
+
299
+ For an app using Rails 3+, check the router documentation for how to mount
300
+ rack apps.
301
+
302
+ Job Dependencies
303
+ ================
304
+ Let's say you have one job that depends on another, but the task definitions are
305
+ fundamentally different. You need to bake a turkey, and you need to make stuffing,
306
+ but you can't make the turkey until the stuffing is made:
307
+
308
+ ``` ruby
309
+ queue = client.queues['cook']
310
+ stuffing_jid = queue.put(MakeStuffing, {:lots => 'of butter'})
311
+ turkey_jid = queue.put(MakeTurkey , {:with => 'stuffing'}, :depends=>[stuffing_jid])
312
+ ```
313
+
314
+ When the stuffing job completes, the turkey job is unlocked and free to be processed.
315
+
316
+ Priority
317
+ ========
318
+ Some jobs need to get popped sooner than others. Whether it's a trouble ticket, or
319
+ debugging, you can do this pretty easily when you put a job in a queue:
320
+
321
+ ``` ruby
322
+ queue.put(MyJobClass, {:foo => 'bar'}, :priority => 10)
323
+ ```
324
+
325
+ What happens when you want to adjust a job's priority while it's still waiting in
326
+ a queue?
327
+
328
+ ``` ruby
329
+ job = client.jobs['0c53b0404c56012f69fa482a1427ab7d']
330
+ job.priority = 10
331
+ # Now this will get popped before any job of lower priority
332
+ ```
333
+
334
+ Scheduled Jobs
335
+ ==============
336
+ If you don't want a job to be run right away but some time in the future, you can
337
+ specify a delay:
338
+
339
+ ``` ruby
340
+ # Run at least 10 minutes from now
341
+ queue.put(MyJobClass, {:foo => 'bar'}, :delay => 600)
342
+ ```
343
+
344
+ This doesn't guarantee that job will be run exactly at 10 minutes. You can accomplish
345
+ this by changing the job's priority so that once 10 minutes has elapsed, it's put before
346
+ lesser-priority jobs:
347
+
348
+ ``` ruby
349
+ # Run in 10 minutes
350
+ queue.put(MyJobClass, {:foo => 'bar'}, :delay => 600, :priority => 100)
351
+ ```
352
+
353
+ Recurring Jobs
354
+ ==============
355
+ Sometimes it's not enough simply to schedule one job, but you want to run jobs regularly.
356
+ In particular, maybe you have some batch operation that needs to get run once an hour and
357
+ you don't care what worker runs it. Recurring jobs are specified much like other jobs:
358
+
359
+ ``` ruby
360
+ # Run every hour
361
+ queue.recur(MyJobClass, {:widget => 'warble'}, 3600)
362
+ # => 22ac75008a8011e182b24cf9ab3a8f3b
363
+ ```
364
+
365
+ You can even access them in much the same way as you would normal jobs:
366
+
367
+ ``` ruby
368
+ job = client.jobs['22ac75008a8011e182b24cf9ab3a8f3b']
369
+ # => < Reqless::RecurringJob 22ac75008a8011e182b24cf9ab3a8f3b >
370
+ ```
371
+
372
+ Changing the interval at which it runs after the fact is trivial:
373
+
374
+ ``` ruby
375
+ # I think I only need it to run once every two hours
376
+ job.interval = 7200
377
+ ```
378
+
379
+ If you want it to run every hour on the hour, but it's 2:37 right now, you can specify
380
+ an offset which is how long it should wait before popping the first job:
381
+
382
+ ``` ruby
383
+ # 23 minutes of waiting until it should go
384
+ queue.recur(MyJobClass, {:howdy => 'hello'}, 3600, :offset => 23 * 60)
385
+ ```
386
+
387
+ Recurring jobs also have priority, a configurable number of retries, and tags. These
388
+ settings don't apply to the recurring jobs, but rather the jobs that they create. In the
389
+ case where more than one interval passes before a worker tries to pop the job, __more than
390
+ one job is created__. The thinking is that while it's completely client-managed, the state
391
+ should not be dependent on how often workers are trying to pop jobs.
392
+
393
+ ``` ruby
394
+ # Recur every minute
395
+ queue.recur(MyJobClass, {:lots => 'of jobs'}, 60)
396
+ # Wait 5 minutes
397
+ queue.pop(10).length
398
+ # => 5 jobs got popped
399
+ ```
400
+
401
+ Configuration Options
402
+ =====================
403
+ You can get and set global (read: in the context of the same Redis instance) configuration
404
+ to change the behavior for heartbeating, and so forth. There aren't a tremendous number
405
+ of configuration options, but an important one is how long job data is kept around. Job
406
+ data is expired after it has been completed for `jobs-history` seconds, but is limited to
407
+ the last `jobs-history-count` completed jobs. These default to 50k jobs, and 30 days, but
408
+ depending on volume, your needs may change. To only keep the last 500 jobs for up to 7 days:
409
+
410
+ ``` ruby
411
+ client.config['jobs-history'] = 7 * 86400
412
+ client.config['jobs-history-count'] = 500
413
+ ```
414
+
415
+ Tagging / Tracking
416
+ ==================
417
+ In reqless, 'tracking' means flagging a job as important. Tracked jobs have a tab reserved
418
+ for them in the web interface, and they also emit subscribable events as they make progress
419
+ (more on that below). You can flag a job from the web interface, or the corresponding code:
420
+
421
+ ``` ruby
422
+ client.jobs['b1882e009a3d11e192d0b174d751779d'].track
423
+ ```
424
+
425
+ Jobs can be tagged with strings which are indexed for quick searches. For example, jobs
426
+ might be associated with customer accounts, or some other key that makes sense for your
427
+ project.
428
+
429
+ ``` ruby
430
+ queue.put(MyJobClass, {:tags => 'aplenty'}, :tags => ['12345', 'foo', 'bar'])
431
+ ```
432
+
433
+ This makes them searchable in the web interface, or from code:
434
+
435
+ ``` ruby
436
+ jids = client.jobs.tagged('foo')
437
+ ```
438
+
439
+ You can add or remove tags at will, too:
440
+
441
+ ``` ruby
442
+ job = client.jobs['b1882e009a3d11e192d0b174d751779d']
443
+ job.tag('howdy', 'hello')
444
+ job.untag('foo', 'bar')
445
+ ```
446
+
447
+ Notifications
448
+ =============
449
+ Tracked jobs emit events on specific pubsub channels as things happen to them. Whether
450
+ it's getting popped off of a queue, completed by a worker, etc. A good example of how
451
+ to make use of this is in the `reqless-campfire` or `reqless-growl`. The jist of it goes like
452
+ this, though:
453
+
454
+ ``` ruby
455
+ client.events do |on|
456
+ on.canceled { |jid| puts "#{jid} canceled" }
457
+ on.stalled { |jid| puts "#{jid} stalled" }
458
+ on.track { |jid| puts "tracking #{jid}" }
459
+ on.untrack { |jid| puts "untracking #{jid}" }
460
+ on.completed { |jid| puts "#{jid} completed" }
461
+ on.failed { |jid| puts "#{jid} failed" }
462
+ on.popped { |jid| puts "#{jid} popped" }
463
+ on.put { |jid| puts "#{jid} put" }
464
+ end
465
+ ```
466
+
467
+ Those familiar with redis pubsub will note that a redis connection can only be used
468
+ for pubsub-y commands once listening. For this reason, invoking `client.events` actually
469
+ creates a second connection so that `client` can still be used as it normally would be:
470
+
471
+ ``` ruby
472
+ client.events do |on|
473
+ on.failed do |jid|
474
+ puts "#{jid} failed in #{client.jobs[jid].queue_name}"
475
+ end
476
+ end
477
+ ```
478
+
479
+ Heartbeating
480
+ ============
481
+ When a worker is given a job, it is given an exclusive lock to that job. That means
482
+ that job won't be given to any other worker, so long as the worker checks in with
483
+ progress on the job. By default, jobs have to either report back progress every 60
484
+ seconds, or complete it, but that's a configurable option. For longer jobs, this
485
+ may not make sense.
486
+
487
+ ``` ruby
488
+ # Hooray! We've got a piece of work!
489
+ job = queue.pop
490
+ # How long until I have to check in?
491
+ job.ttl
492
+ # => 59
493
+ # Hey! I'm still working on it!
494
+ job.heartbeat
495
+ # => 1331326141.0
496
+ # Ok, I've got some more time. Oh! Now I'm done!
497
+ job.complete
498
+ ```
499
+
500
+ If you want to increase the heartbeat in all queues,
501
+
502
+ ``` ruby
503
+ # Now jobs get 10 minutes to check in
504
+ client.config['heartbeat'] = 600
505
+ # But the testing queue doesn't get as long.
506
+ client.queues['testing'].heartbeat = 300
507
+ ```
508
+
509
+ When choosing a heartbeat interval, realize that this is the amount of time that
510
+ can pass before reqless realizes if a job has been dropped. At the same time, you don't
511
+ want to burden reqless with heartbeating every 10 seconds if your job is expected to
512
+ take several hours.
513
+
514
+ An idiom you're encouraged to use for long-running jobs that want to check in their
515
+ progress periodically:
516
+
517
+ ``` ruby
518
+ # Wait until we have 5 minutes left on the heartbeat, and if we find that
519
+ # we've lost our lock on a job, then honorably fall on our sword
520
+ if (job.ttl < 300) && !job.heartbeat
521
+ return / die / exit
522
+ end
523
+ ```
524
+
525
+ Stats
526
+ =====
527
+ One nice feature of `reqless` is that you can get statistics about usage. Stats are
528
+ aggregated by day, so when you want stats about a queue, you need to say what queue
529
+ and what day you're talking about. By default, you just get the stats for today.
530
+ These stats include information about the mean job wait time, standard deviation,
531
+ and histogram. This same data is also provided for job completion:
532
+
533
+ ``` ruby
534
+ # So, how're we doing today?
535
+ stats = client.stats.get('testing')
536
+ # => { 'run' => {'mean' => ..., }, 'wait' => {'mean' => ..., }}
537
+ ```
538
+
539
+ Time
540
+ ====
541
+ It's important to note that Redis doesn't allow access to the system time if you're
542
+ going to be making any manipulations to data (which our scripts do). And yet, we
543
+ have heartbeating. This means that the clients actually send the current time when
544
+ making most requests, and for consistency's sake, means that your workers must be
545
+ relatively synchronized. This doesn't mean down to the tens of milliseconds, but if
546
+ you're experiencing appreciable clock drift, you should investigate NTP. For what it's
547
+ worth, this hasn't been a problem for us, but most of our jobs have heartbeat intervals
548
+ of 30 minutes or more.
549
+
550
+ Ensuring Job Uniqueness
551
+ =======================
552
+
553
+ As mentioned above, Jobs are uniquely identied by an id--their jid.
554
+ Reqless will generate a UUID for each enqueued job or you can specify
555
+ one manually:
556
+
557
+ ``` ruby
558
+ queue.put(MyJobClass, { :hello => 'howdy' }, :jid => 'my-job-jid')
559
+ ```
560
+
561
+ This can be useful when you want to ensure a job's uniqueness: simply
562
+ create a jid that is a function of the Job's class and data, it'll
563
+ guaranteed that Reqless won't have multiple jobs with the same class
564
+ and data.
565
+
566
+ Setting Default Job Options
567
+ ===========================
568
+
569
+ `Reqless::Queue#put` accepts a number of job options (see above for their
570
+ semantics):
571
+
572
+ * jid
573
+ * delay
574
+ * priority
575
+ * tags
576
+ * retries
577
+ * depends
578
+
579
+ When enqueueing the same kind of job with the same args in multiple
580
+ places it's a pain to have to declare the job options every time.
581
+ Instead, you can define default job options directly on the job class:
582
+
583
+ ``` ruby
584
+ class MyJobClass
585
+ def self.default_job_options(data)
586
+ { :priority => 10, :delay => 100 }
587
+ end
588
+ end
589
+
590
+ queue.put(MyJobClass, { :some => "data" }, :delay => 10)
591
+ ```
592
+
593
+ Individual jobs can still specify options, so in this example,
594
+ the job would be enqueued with a priority of 10 and a delay of 10.
595
+
596
+ Testing Jobs
597
+ ============
598
+ When unit testing your jobs, you will probably want to avoid the
599
+ overhead of round-tripping them through redis. You can of course
600
+ use a mock job object and pass it to your job class's `perform`
601
+ method. Alternately, if you want a real full-fledged `Reqless::Job`
602
+ instance without round-tripping it through Redis, use `Reqless::Job.build`:
603
+
604
+ ``` ruby
605
+ describe MyJobClass do
606
+ let(:client) { Reqless::Client.new }
607
+ let(:job) { Reqless::Job.build(client, MyJobClass, :data => { "some" => "data" }) }
608
+
609
+ it 'does something' do
610
+ MyJobClass.perform(job)
611
+ # make an assertion about what happened
612
+ end
613
+ end
614
+ ```
615
+
616
+ The options hash passed to `Reqless::Job.build` supports all the same
617
+ options a normal job supports. See
618
+ [the source](https://github.com/tdg5/reqless-rb/blob/main/lib/reqless/job.rb)
619
+ for a full list.
620
+
621
+ Contributing
622
+ ============
623
+
624
+ To bootstrap an environment, first setup a redis instance.
625
+
626
+ Have `rvm` or `rbenv`. Then to install the dependencies:
627
+
628
+ ```bash
629
+ rbenv install # rbenv only. Install bundler if you need it.
630
+ bundle install
631
+ ./exe/install_phantomjs # Bring in phantomjs 1.7.0 for tests.
632
+ rbenv rehash # rbenv only
633
+ git submodule init
634
+ git submodule update
635
+ bundle exec rake core:build
636
+ ```
637
+
638
+ To run the tests:
639
+
640
+ ```
641
+ bundle exec rake spec
642
+ ```
643
+
644
+ **The locally installed redis will be flushed before and after each test run.**
645
+
646
+ To change the redis instance used in tests, put the connection information into [`./spec/redis.config.yml`](https://github.com/tdg5/reqless-rb/blob/92904532aee82aaf1078957ccadfa6fcd27ae408/spec/spec_helper.rb#L26).
647
+
648
+ To contribute, fork the repo, use feature branches, run the tests and open PRs.