backburner-allq 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (63) hide show
  1. checksums.yaml +7 -0
  2. data/.gitignore +17 -0
  3. data/.travis.yml +29 -0
  4. data/CHANGELOG.md +133 -0
  5. data/CONTRIBUTING.md +37 -0
  6. data/Gemfile +4 -0
  7. data/HOOKS.md +99 -0
  8. data/LICENSE +22 -0
  9. data/README.md +658 -0
  10. data/Rakefile +17 -0
  11. data/TODO +4 -0
  12. data/backburner-allq.gemspec +26 -0
  13. data/bin/backburner +7 -0
  14. data/circle.yml +3 -0
  15. data/deploy.sh +3 -0
  16. data/examples/custom.rb +25 -0
  17. data/examples/demo.rb +60 -0
  18. data/examples/god.rb +46 -0
  19. data/examples/hooked.rb +87 -0
  20. data/examples/retried.rb +31 -0
  21. data/examples/simple.rb +43 -0
  22. data/examples/stress.rb +31 -0
  23. data/lib/backburner.rb +75 -0
  24. data/lib/backburner/allq_wrapper.rb +317 -0
  25. data/lib/backburner/async_proxy.rb +25 -0
  26. data/lib/backburner/cli.rb +53 -0
  27. data/lib/backburner/configuration.rb +48 -0
  28. data/lib/backburner/connection.rb +157 -0
  29. data/lib/backburner/helpers.rb +193 -0
  30. data/lib/backburner/hooks.rb +53 -0
  31. data/lib/backburner/job.rb +118 -0
  32. data/lib/backburner/logger.rb +53 -0
  33. data/lib/backburner/performable.rb +95 -0
  34. data/lib/backburner/queue.rb +145 -0
  35. data/lib/backburner/tasks.rb +54 -0
  36. data/lib/backburner/version.rb +3 -0
  37. data/lib/backburner/worker.rb +221 -0
  38. data/lib/backburner/workers/forking.rb +52 -0
  39. data/lib/backburner/workers/simple.rb +29 -0
  40. data/lib/backburner/workers/threading.rb +163 -0
  41. data/lib/backburner/workers/threads_on_fork.rb +263 -0
  42. data/test/async_proxy_test.rb +36 -0
  43. data/test/back_burner_test.rb +88 -0
  44. data/test/connection_test.rb +179 -0
  45. data/test/fixtures/hooked.rb +122 -0
  46. data/test/fixtures/test_fork_jobs.rb +72 -0
  47. data/test/fixtures/test_forking_jobs.rb +56 -0
  48. data/test/fixtures/test_jobs.rb +87 -0
  49. data/test/fixtures/test_queue_settings.rb +14 -0
  50. data/test/helpers/templogger.rb +22 -0
  51. data/test/helpers_test.rb +278 -0
  52. data/test/hooks_test.rb +112 -0
  53. data/test/job_test.rb +185 -0
  54. data/test/logger_test.rb +44 -0
  55. data/test/performable_test.rb +88 -0
  56. data/test/queue_test.rb +69 -0
  57. data/test/test_helper.rb +128 -0
  58. data/test/worker_test.rb +157 -0
  59. data/test/workers/forking_worker_test.rb +181 -0
  60. data/test/workers/simple_worker_test.rb +350 -0
  61. data/test/workers/threading_worker_test.rb +104 -0
  62. data/test/workers/threads_on_fork_worker_test.rb +484 -0
  63. metadata +217 -0
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: b341ad669c86cbe42e3cb8b8c521fdf51eb62fedaf02c7f5fc35c2701b275995
4
+ data.tar.gz: d5cd03e528baa91099f161e3ca98f9bcdc482058bbfa7183fdfb2dc779955b71
5
+ SHA512:
6
+ metadata.gz: 37f3bb1322f1b8366c5e033dc2818bc2adecb2f2f8ae414d55c2c352826a615df31e371422c478aa41a8e690546caf1f71b0797aed3df7fa6d8691204470a51d
7
+ data.tar.gz: 2e7029d2e04e6f37a6b634e7be2a68c59500dadfef25b4332a350bbd7d87000c593d9e88050935b7943196ce4fa91a51291f532136b120c5212fee2470bb37d4
@@ -0,0 +1,17 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ Gemfile.lock
7
+ InstalledFiles
8
+ _yardoc
9
+ coverage
10
+ doc/
11
+ lib/bundler/man
12
+ pkg
13
+ rdoc
14
+ spec/reports
15
+ test/tmp
16
+ test/version_tmp
17
+ tmp
@@ -0,0 +1,29 @@
1
+ # http://about.travis-ci.org/docs/user/build-configuration/
2
+ rvm:
3
+ - 1.9.3
4
+ - 2.0.0
5
+ - 2.1
6
+ - 2.2
7
+ - 2.3
8
+ - 2.4
9
+ - 2.5
10
+ - rbx-2
11
+ before_install:
12
+ - curl -L https://github.com/kr/beanstalkd/archive/v1.9.tar.gz | tar xz -C /tmp
13
+ - cd /tmp/beanstalkd-1.9/
14
+ - make
15
+ - ./beanstalkd &
16
+ - cd $TRAVIS_BUILD_DIR
17
+ - gem update --system
18
+ - gem update bundler
19
+ matrix:
20
+ allow_failures:
21
+ - rvm: rbx-2
22
+ - rvm: 2.0.0
23
+ script:
24
+ - bundle exec rake test
25
+ gemfile: Gemfile
26
+ notifications:
27
+ recipients:
28
+ - nesquena@gmail.com
29
+ - therealdave.myron@gmail.com
@@ -0,0 +1,133 @@
1
+ # CHANGELOG
2
+
3
+ ## Version 1.5.0 (September 10 2018)
4
+
5
+ * TBD
6
+
7
+ ## Version 1.4.1 (June 10 2017)
8
+
9
+ * Fix warning for constant ::Fixnum is deprecated (@amatsuda)
10
+
11
+ ## Version 1.4.0 (May 13 2017)
12
+
13
+ * Fix unit tests to be more consistent (@eltone)
14
+ * Ensure job supports body hash with symbol keys (@eltone)
15
+ * Add support for custom serialization formats (@eltone)
16
+ * Log the params when a job timeout occurs (@nathantsoi)
17
+
18
+ ## Version 1.3.1 (April 21 2016)
19
+
20
+ * Addition of thread-pool-based concurrency (@contentfree)
21
+
22
+ ## Version 1.3.0 (February 05 2016)
23
+
24
+ * Enqueue command now responds with beanstalk response details
25
+
26
+ ## Version 1.2.0 (November 01 2015)
27
+
28
+ * FIX Made connections to beanstalkd more resilient (@contentfree)
29
+
30
+ ## Version 1.2.0.pre (October 24 2015)
31
+
32
+ * FIX Replace static Beaneater connection with individual connections per worker instance/thread (@contentfree)
33
+ * FIX Beaneater connections try really hard to repair themselves if disconnected accidentally (@contentfree)
34
+ * NEW Event hook for workers: on_reconnect (@contentfree)
35
+
36
+ ## Version 1.1.0 (September 14 2015)
37
+
38
+ * NEW Ability to configure namespace separator (@bfolkens)
39
+ * NEW Avoid timeouts altogether by setting queue_respond_timeout to 0 (@zacviandier)
40
+ * NEW Event hooks for on_retry and on_bury (@contentfree)
41
+ * NEW Support lambdas for queue names (@contentfree)
42
+ * NEW Allow for control of delay calculation (@contentfree)
43
+ * NEW Ability to specify environment when running the CLI (@contentfree)
44
+ * NEW Control default async behavior of methods (@contentfree)
45
+
46
+ ## Version 1.0.0 (April 26 2015)
47
+
48
+ * NEW Updating to Beaneater 1.0 (@alup)
49
+
50
+ ## Version 0.4.6 (October 26 2014)
51
+
52
+ * NEW Add job to on_error handler if the handler has a 4th argument (@Nitrodist)
53
+ * NEW Use a timeout when looking for a job to reserve (@EasyPost)
54
+ * NEW Support configuring settings on threads on fork class (@silentshade)
55
+ * FIX queue override by existing queues (@silentshade)
56
+ * FIX Use thread to log exit message (@silentshade)
57
+
58
+ ## Version 0.4.5 (December 16 2013)
59
+
60
+ * FIX #47 Create a backburner connection per thread (Thanks @thcrock)
61
+
62
+ ## Version 0.4.4 (October 27 2013)
63
+
64
+ * NEW #51 Added ability to set per-queue default ttr's (Thanks @ryanjohns)
65
+
66
+ ## Version 0.4.3 (July 19 2013)
67
+
68
+ * FIX #44 Additional fix to issue introduced in 0.4.2
69
+ * FIX #45 More graceful shutdown using Kernel.exit and rescuing SystemExit. (Thanks @ryanjohns)
70
+
71
+ ## Version 0.4.2 (July 3 2013)
72
+
73
+ * FIX #44 Properly retry to connect to beanstalkd when connection fails.
74
+
75
+ ## Version 0.4.1 (June 28 2013)
76
+
77
+ * FIX #43 Properly support CLI options and smart load the app environment.
78
+
79
+ ## Version 0.4.0 (June 28 2013)
80
+
81
+ NOTE: This is the start of working with @bradgessler to improve backburner and merge with quebert
82
+
83
+ * NEW #26 #27 Remove need for Queue mixin, allow plain ruby objects
84
+ * NEW Default all jobs to a single general queue rather than separate queues
85
+ * NEW Add support for named priorities, allowing shorthand names for priority values
86
+
87
+ ## Version 0.3.4 (April 23 2013)
88
+
89
+ * FIX #22 Adds signal handlers for worker to manage proper shutdown (Thanks @tkiley)
90
+
91
+ ## Version 0.3.3 (April 19 2013)
92
+
93
+ * Fix naming conflict rename 'config' to 'queue_config'
94
+
95
+ ## Version 0.3.2 (Jan 23 2013)
96
+
97
+ * Bump version of beaneater to 0.3.0 (better socket handling)
98
+
99
+ ## Version 0.3.1 (Dec 28 2012)
100
+
101
+ * Adds basic forking processing strategy and rake tasks (Thanks @danielfarrell)
102
+
103
+ ## Version 0.3.0 (Nov 14 2012)
104
+
105
+ * Major update with support for a 'threads_on_fork' processing strategy (Thanks @ShadowBelmolve)
106
+ * Different workers have different rake tasks (Thanks @ShadowBelmolve)
107
+ * Added processing strategy specific examples i.e stress.rb and adds new unit tests. (Thanks @ShadowBelmolve)
108
+
109
+ ## Version 0.2.6 (Nov 12 2012)
110
+
111
+ * Upgrade to beaneater 0.2.0
112
+
113
+ ## Version 0.2.5 (Nov 9 2012)
114
+
115
+ * Add support for multiple worker processing strategies through subclassing.
116
+
117
+ ## Version 0.2.0 (Nov 7 2012)
118
+
119
+ * Add new plugin hooks feature (see HOOKS.md)
120
+
121
+ ## Version 0.1.2 (Nov 7 2012)
122
+
123
+ * Adds ability to specify a custom logger.
124
+ * Adds job retry configuration and worker support.
125
+
126
+ ## Version 0.1.1 (Nov 6 2012)
127
+
128
+ * Fix issue with timed out reserves
129
+
130
+ ## Version 0.1.0 (Nov 4 2012)
131
+
132
+ * Switch to beaneater as new ruby beanstalkd client
133
+ * Add support for array of connections in `beanstalk_url`
@@ -0,0 +1,37 @@
1
+ We love pull requests. Here's a quick guide:
2
+
3
+ 1. Fork the repo.
4
+
5
+ 2. Run the tests. We only take pull requests with passing tests, and it's great
6
+ to know that you have a clean slate: `bundle && rake test`
7
+
8
+ 3. Add a test for your change. Only refactoring and documentation changes
9
+ require no new tests. If you are adding functionality or fixing a bug, we need
10
+ a test!
11
+
12
+ 4. Make the test pass.
13
+
14
+ 5. Push to your fork and submit a pull request.
15
+
16
+ At this point you're waiting on us. We like to at least comment on, if not
17
+ accept, pull requests within three business days (and, typically, one business
18
+ day). We may suggest some changes or improvements or alternatives.
19
+
20
+ Some things that will increase the chance that your pull request is accepted:
21
+
22
+ * Use Rails idioms and helpers
23
+ * Include tests that fail without your code, and pass with it
24
+ * Update the documentation and README for anything affected by your contribution
25
+
26
+ Syntax:
27
+
28
+ * Two spaces, no tabs.
29
+ * No trailing whitespace. Blank lines should not have any space.
30
+ * Prefer &&/|| over and/or.
31
+ * MyClass.my_method(my_arg) not my_method( my_arg ) or my_method my_arg.
32
+ * a = b and not a=b.
33
+ * Follow the conventions you see used in the source already.
34
+
35
+ And in case we didn't emphasize it enough: we love tests!
36
+
37
+ NOTE: Adapted from https://raw.github.com/thoughtbot/factory_girl_rails/master/CONTRIBUTING.md
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in backburner.gemspec
4
+ gemspec
@@ -0,0 +1,99 @@
1
+ # Backburner Hooks
2
+
3
+ You can customize Backburner or write plugins using its hook API.
4
+ In many cases you can use a hook rather than mess around with Backburner's internals.
5
+
6
+ ## Job Hooks
7
+
8
+ Hooks are transparently adapted from [Resque](https://github.com/resque/resque/blob/master/docs/HOOKS.md), so
9
+ if you are familiar with their hook API, now you can use nearly the same ones with beanstalkd and backburner!
10
+
11
+ There are a variety of hooks available that are triggered during the lifecycle of a job:
12
+
13
+ * `before_enqueue`: Called with the job args before a job is placed on the queue.
14
+ If the hook returns `false`, the job will not be placed on the queue.
15
+
16
+ * `after_enqueue`: Called with the job args after a job is placed on the queue.
17
+ Any exception raised propagates up to the code which queued the job.
18
+
19
+ * `before_perform`: Called with the job args before perform. If a hook returns false,
20
+ the job is aborted. Other exceptions are treated like regular job exceptions.
21
+
22
+ * `after_perform`: Called with the job args after it performs. Uncaught
23
+ exceptions will be treated like regular job exceptions.
24
+
25
+ * `around_perform`: Called with the job args. It is expected to yield in order
26
+ to perform the job (but is not required to do so). It may handle exceptions
27
+ thrown by perform, but uncaught exceptions will be treated like regular job exceptions.
28
+
29
+ * `on_retry`: Called with the retry count, the delay and the job args whenever a job is retried.
30
+
31
+ * `on_bury`: Called with the job args when the job is buried.
32
+
33
+ * `on_failure`: Called with the exception and job args if any exception occurs
34
+ while performing the job (or hooks).
35
+
36
+ Hooks are just methods prefixed with the hook type. For example:
37
+
38
+ ```ruby
39
+ class SomeJob
40
+ def self.before_perform_log_job(*args)
41
+ logger.info "About to perform #{self} with #{args.inspect}"
42
+ end
43
+
44
+ def self.on_failure_bury(e, *args)
45
+ logger.info "Performing #{self} caused an exception (#{e})"
46
+ self.bury
47
+ end
48
+
49
+ def self.perform(*args)
50
+ # ...
51
+ end
52
+
53
+ def self.logger
54
+ @_logger ||= Logger.new(STDOUT)
55
+ end
56
+ end
57
+ ```
58
+
59
+ You can also setup modules to create compose-able and reusable hooks for your jobs. For example:
60
+
61
+ ```ruby
62
+ module LoggedJob
63
+ def before_perform_log_job(*args)
64
+ Logger.info "About to perform #{self} with #{args.inspect}"
65
+ end
66
+ end
67
+
68
+ module BuriedJob
69
+ def on_failure_bury(e, *args)
70
+ Logger.info "Performing #{self} caused an exception (#{e}). Retrying..."
71
+ self.bury
72
+ end
73
+ end
74
+
75
+ class MyJob
76
+ extend LoggedJob
77
+ extend BuriedJob
78
+
79
+ def self.perform(*args)
80
+ # ...
81
+ end
82
+ end
83
+ ```
84
+
85
+ ## Worker Hooks
86
+
87
+ Currently, there is just one hook:
88
+
89
+ * `on_reconnect`: Called on the worker whose connection has been reset. The connection
90
+ is given as the argument
91
+
92
+ An example:
93
+
94
+ ```ruby
95
+ class MyWorker < Backburner::Worker
96
+ def on_reconnect(conn)
97
+ prepare
98
+ end
99
+ end
data/LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2012 Nathan Esquenazi
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,658 @@
1
+ # Backburner [![Build Status](https://travis-ci.org/nesquena/backburner.svg?branch=master)](https://travis-ci.org/nesquena/backburner)
2
+
3
+ Backburner is a [beanstalkd](http://kr.github.com/beanstalkd/)-powered job queue that can handle a very high volume of jobs.
4
+ You create background jobs and place them on multiple work queues to be processed later.
5
+
6
+ Processing background jobs reliably has never been easier than with beanstalkd and Backburner. This gem works with any ruby-based
7
+ web framework, but is especially suited for use with [Sinatra](http://sinatrarb.com), [Padrino](http://padrinorb.com) and Rails.
8
+
9
+ If you want to use beanstalk for your job processing, consider using Backburner.
10
+ Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads.
11
+ Persistent queues are supported when beanstalkd persistence mode is enabled.
12
+
13
+ Backburner supports multiple queues, job priorities, delays, and timeouts. In addition,
14
+ Backburner has robust support for retrying failed jobs, handling error cases,
15
+ custom logging, and extensible plugin hooks.
16
+
17
+ ## Why Backburner?
18
+
19
+ Backburner is well tested and has a familiar, no-nonsense approach to job processing, but that is of secondary importance.
20
+ Let's face it, there are a lot of options for background job processing. [DelayedJob](https://github.com/collectiveidea/delayed_job),
21
+ and [Resque](https://github.com/defunkt/resque) are the first that come to mind immediately. So, how do we make sense
22
+ of which one to use? And why use Backburner over other alternatives?
23
+
24
+ The key to understanding the differences lies in understanding the different projects and protocols that power these popular queue
25
+ libraries under the hood. Every job queue requires a queue store that jobs are put into and pulled out of.
26
+ In the case of Resque, jobs are processed through **Redis**, a persistent key-value store. In the case of DelayedJob, jobs are processed through
27
+ **ActiveRecord** and a database such as PostgreSQL.
28
+
29
+ The work queue underlying these gems tells you infinitely more about the differences than anything else.
30
+ Beanstalk is probably the best solution for job queues available today for many reasons.
31
+ The real question then is... "Why Beanstalk?".
32
+
33
+ ## Why Beanstalk?
34
+
35
+ Illya has an excellent blog post
36
+ [Scalable Work Queues with Beanstalk](http://www.igvita.com/2010/05/20/scalable-work-queues-with-beanstalk/) and
37
+ Adam Wiggins posted [an excellent comparison](http://adam.herokuapp.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/).
38
+
39
+ You will quickly see that **beanstalkd** is an underrated but incredible project that is extremely well-suited as a job queue.
40
+ Significantly better suited for this task than Redis or a database. Beanstalk is a simple,
41
+ and a very fast work queue service rolled into a single binary - it is the memcached of work queues.
42
+ Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project.
43
+ [PostRank](http://www.postrank.com) uses beanstalk to reliably process millions of jobs a day.
44
+
45
+ A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size)
46
+ because it is an in-memory, event-driven system. Powered by libevent under the hood,
47
+ it requires zero setup (launch and forget, à la memcached), optional log based persistence, an easily parsed ASCII protocol,
48
+ and a rich set of tools for job management that go well beyond a simple FIFO work queue.
49
+
50
+ Beanstalkd supports the following features out of the box:
51
+
52
+ | Feature | Description |
53
+ | ------- | ------------------------------- |
54
+ | **Parallelized** | Supports multiple work queues created on demand. |
55
+ | **Reliable** | Beanstalk’s reserve, work, delete cycle ensures reliable processing. |
56
+ | **Scheduling** | Delay enqueuing jobs by a specified interval to schedule processing later |
57
+ | **Fast** | Processes thousands of jobs per second without breaking a sweat. |
58
+ | **Priorities** | Specify priority so important jobs can be processed quickly. |
59
+ | **Persistence** | Jobs are stored in memory for speed, but logged to disk for safe keeping. |
60
+ | **Federation** | Horizontal scalability provided through federation by the client. |
61
+ | **Error Handling** | Bury any job which causes an error for later debugging and inspection.|
62
+
63
+ Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support.
64
+ In the end, **beanstalk is the ideal job queue** while also being ridiculously easy to install and setup.
65
+
66
+ ## Installation
67
+
68
+ First, you probably want to [install beanstalkd](http://kr.github.com/beanstalkd/download.html), which powers the job queues.
69
+ Depending on your platform, this should be as simple as (for Ubuntu):
70
+
71
+ $ sudo apt-get install beanstalkd
72
+
73
+ Add this line to your application's Gemfile:
74
+
75
+ gem 'backburner'
76
+
77
+ And then execute:
78
+
79
+ $ bundle
80
+
81
+ Or install it yourself as:
82
+
83
+ $ gem install backburner
84
+
85
+ ## Configuration ##
86
+
87
+ Backburner is extremely simple to setup. Just configure basic settings for backburner:
88
+
89
+ ```ruby
90
+ Backburner.configure do |config|
91
+ config.beanstalk_url = "beanstalk://127.0.0.1"
92
+ config.tube_namespace = "some.app.production"
93
+ config.namespace_separator = "."
94
+ config.on_error = lambda { |e| puts e }
95
+ config.max_job_retries = 3 # default 0 retries
96
+ config.retry_delay = 2 # default 5 seconds
97
+ config.retry_delay_proc = lambda { |min_retry_delay, num_retries| min_retry_delay + (num_retries ** 3) }
98
+ config.default_priority = 65536
99
+ config.respond_timeout = 120
100
+ config.default_worker = Backburner::Workers::Simple
101
+ config.logger = Logger.new(STDOUT)
102
+ config.primary_queue = "backburner-jobs"
103
+ config.priority_labels = { :custom => 50, :useless => 1000 }
104
+ config.reserve_timeout = nil
105
+ config.job_serializer_proc = lambda { |body| JSON.dump(body) }
106
+ config.job_parser_proc = lambda { |body| JSON.parse(body) }
107
+
108
+ end
109
+ ```
110
+
111
+ The key options available are:
112
+
113
+ | Option | Description |
114
+ | ----------------- | ------------------------------- |
115
+ | `beanstalk_url` | Address for beanstalkd connection i.e 'beanstalk://127.0.0.1' |
116
+ | `tube_namespace` | Prefix used for all tubes related to this backburner queue. |
117
+ | `namespace_separator` | Separator used for namespace and queue name |
118
+ | `on_error` | Lambda invoked with the error whenever any job in the system fails. |
119
+ | `max_job_retries` | Integer defines how many times to retry a job before burying. |
120
+ | `retry_delay` | Integer defines the base time to wait (in secs) between job retries. |
121
+ | `retry_delay_proc` | Lambda calculates the delay used, allowing for exponential back-off. |
122
+ | `default_priority` | Integer The default priority of jobs |
123
+ | `respond_timeout` | Integer defines how long a job has to complete its task |
124
+ | `default_worker` | Worker class that will be used if no other worker is specified. |
125
+ | `logger` | Logger recorded to when backburner wants to report info or errors. |
126
+ | `primary_queue` | Primary queue used for a job when an alternate queue is not given. |
127
+ | `priority_labels` | Hash of named priority definitions for your app. |
128
+ | `reserve_timeout` | Duration to wait for work from a single server, or nil for forever. |
129
+ | `job_serializer_proc` | Lambda serializes a job body to a string to write to the task |
130
+ | `job_parser_proc` | Lambda parses a task body string to a hash |
131
+
132
+ ## Breaking Changes
133
+
134
+ Before **v0.4.0**: Jobs were placed into default queues based on the name of the class creating the queue. i.e NewsletterJob would
135
+ be put into a 'newsletter-job' queue. As of 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs"
136
+ unless otherwise specified.
137
+
138
+ ## Usage
139
+
140
+ Backburner allows you to create jobs and place them onto any number of beanstalk tubes, and later pull those jobs off the tubes and
141
+ process them asynchronously with a worker.
142
+
143
+ ### Enqueuing Jobs ###
144
+
145
+ At the core, Backburner is about jobs that can be processed asynchronously. Jobs are simple ruby objects which respond to `perform`.
146
+
147
+ Job objects are queued as JSON onto a tube to be later processed by a worker. Here's an example:
148
+
149
+ ```ruby
150
+ class NewsletterJob
151
+ # required
152
+ def self.perform(email, body)
153
+ NewsletterMailer.deliver_text_to_email(email, body)
154
+ end
155
+
156
+ # optional, defaults to 'backburner-jobs' tube
157
+ def self.queue
158
+ "newsletter-sender"
159
+ end
160
+
161
+ # optional, defaults to default_priority
162
+ def self.queue_priority
163
+ 1000 # most urgent priority is 0
164
+ end
165
+
166
+ # optional, defaults to respond_timeout in config
167
+ def self.queue_respond_timeout
168
+ 300 # number of seconds before job times out, 0 to avoid timeout. NB: A timeout of 1 second will likely lead to race conditions between Backburner and beanstalkd and should be avoided
169
+ end
170
+
171
+ # optional, defaults to retry_delay_proc in config
172
+ def self.queue_retry_delay_proc
173
+ lambda { |min_retry_delay, num_retries| min_retry_delay + (num_retries ** 5) }
174
+ end
175
+
176
+ # optional, defaults to retry_delay in config
177
+ def self.queue_retry_delay
178
+ 5
179
+ end
180
+
181
+ # optional, defaults to max_job_retries in config
182
+ def self.queue_max_job_retries
183
+ 5
184
+ end
185
+ end
186
+ ```
187
+
188
+ You can include the optional `Backburner::Queue` module so you can easily specify queue settings for this job:
189
+
190
+ ```ruby
191
+ class NewsletterJob
192
+ include Backburner::Queue
193
+ queue "newsletter-sender" # defaults to 'backburner-jobs' tube
194
+ queue_priority 1000 # most urgent priority is 0
195
+ queue_respond_timeout 300 # number of seconds before job times out, 0 to avoid timeout
196
+
197
+ def self.perform(email, body)
198
+ NewsletterMailer.deliver_text_to_email(email, body)
199
+ end
200
+ end
201
+ ```
202
+
203
+ Jobs can be enqueued with:
204
+
205
+ ```ruby
206
+ Backburner.enqueue NewsletterJob, 'foo@admin.com', 'lorem ipsum...'
207
+ ```
208
+
209
+ `Backburner.enqueue` accepts first a ruby object that supports `perform` and then a series of parameters
210
+ to that object's `perform` method. The queue name used by default is `{namespace}.backburner-jobs`
211
+ unless otherwise specified.
212
+
213
+ You may also pass a lambda as the queue name and it will be evaluated when enqueuing a
214
+ job (and passed the Job's class as an argument). This is especially useful when combined
215
+ with "Simple Async Jobs" (see below).
216
+
217
+ ### Simple Async Jobs ###
218
+
219
+ In addition to defining custom jobs, a job can also be enqueued by invoking the `async` method on any object which
220
+ includes `Backburner::Performable`. Async enqueuing works for both instance and class methods on any _performable_ object.
221
+
222
+ ```ruby
223
+ class User
224
+ include Backburner::Performable
225
+ queue "user-jobs" # defaults to 'user'
226
+ queue_priority 500 # most urgent priority is 0
227
+ queue_respond_timeout 300 # number of seconds before job times out, 0 to avoid timeout
228
+
229
+ def activate(device_id)
230
+ @device = Device.find(device_id)
231
+ # ...
232
+ end
233
+
234
+ def self.reset_password(user_id)
235
+ # ...
236
+ end
237
+ end
238
+
239
+ # Async works for instance methods on a persisted object with an `id`
240
+ @user = User.first
241
+ @user.async(:ttr => 100, :queue => "activate").activate(@device.id)
242
+ # ..and for class methods
243
+ User.async(:pri => 100, :delay => 10.seconds).reset_password(@user.id)
244
+ ```
245
+
246
+ This automatically enqueues a job for that user record that will run `activate` with the specified argument.
247
+ Note that you can set the queue name and queue priority at the class level and
248
+ you are also able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`.
249
+
250
+ The queue name used by default is `{namespace}.backburner-jobs` if not otherwise
251
+ specified.
252
+
253
+ If a lambda is given for `queue`, then it will be called and given the
254
+ _performable_ object's class as an argument:
255
+
256
+ ```ruby
257
+ # Given the User class above
258
+ User.async(:queue => lambda { |user_klass| ["queue1","queue2"].sample(1).first }).do_hard_work # would add the job to either queue1 or queue2 randomly
259
+ ```
260
+
261
+ ### Using Async Asynchronously ###
262
+
263
+ It's often useful to be able to configure your app in production such that every invocation of a method is asynchronous by default as seen in [delayed_job](https://github.com/collectiveidea/delayed_job#queuing-jobs). To accomplish this, the `Backburner::Performable` module exposes two `handle_asynchronously` convenience methods
264
+ which accept the same options as the `async` method:
265
+
266
+ ```ruby
267
+ class User
268
+ include Backburner::Performable
269
+
270
+ def send_welcome_email
271
+ # ...
272
+ end
273
+
274
+ # ---> For instance methods
275
+ handle_asynchronously :send_welcome_email, queue: 'send-mail', pri: 5000, ttr: 60
276
+
277
+ def self.update_recent_visitors
278
+ # ...
279
+ end
280
+
281
+ # ---> For class methods
282
+ handle_static_asynchronously :update_recent_visitors, queue: 'long-tasks', ttr: 300
283
+ end
284
+ ```
285
+
286
+ Now, all calls to `User.update_recent_visitors` or `User#send_welcome_email` will automatically be handled asynchronously when invoked. Similarly, you can call these methods directly on the `Backburner::Performable` module to apply async behavior outside the class:
287
+
288
+ ```ruby
289
+ # Given the User class above
290
+ Backburner::Performable.handle_asynchronously(User, :activate, ttr: 100, queue: 'activate')
291
+ ```
292
+
293
+ Now all calls to the `activate` method on a `User` instance will be async with the provided options.
294
+
295
+ #### A Note About Auto-Async
296
+
297
+ Because an async proxy is injected and used in place of the original method, you must not rely on the return value of the method. Using the example `User` class above, if my `send_welcome_email` returned the status of an email submission and I relied on that to take some further action, I will be surprised after rewiring things with `handle_asynchronously` because the async proxy actually returns the (boolean) result of `Backburner::Worker.enqueue`.
298
+
299
+ ### Working Jobs
300
+
301
+ Backburner workers are processes that run forever handling jobs that are reserved from the queue. Starting a worker in ruby code is simple:
302
+
303
+ ```ruby
304
+ Backburner.work
305
+ ```
306
+
307
+ This will process jobs in all queues but you can also restrict processing to specific queues:
308
+
309
+ ```ruby
310
+ Backburner.work('newsletter-sender', 'push-notifier')
311
+ ```
312
+
313
+ The Backburner worker also exists as a rake task:
314
+
315
+ ```ruby
316
+ require 'backburner/tasks'
317
+ ```
318
+
319
+ so you can run:
320
+
321
+ ```
322
+ $ QUEUE=newsletter-sender,push-notifier rake backburner:work
323
+ ```
324
+
325
+ You can also run the backburner binary for a convenient worker:
326
+
327
+ ```
328
+ bundle exec backburner -q newsletter-sender,push-notifier -d -P /var/run/backburner.pid -l /var/log/backburner.log
329
+ ```
330
+
331
+ This will daemonize the worker and store the pid and logs automatically. For Rails and Padrino, the environment should
332
+ load automatically. For other cases, use the `-r` flag to specify a file to require.
333
+
334
+ ### Delaying Jobs
335
+
336
+ In Backburner, jobs can be delayed by specifying the `delay` option whenever you enqueue a job. If you want to schedule a job for an hour from now, simply add that option while enqueuing the standard job:
337
+
338
+ ```ruby
339
+ Backburner::Worker.enqueue(NewsletterJob, ['foo@admin.com', 'lorem ipsum...'], :delay => 1.hour)
340
+ ```
341
+
342
+ or while you schedule an async method call:
343
+
344
+ ```ruby
345
+ User.async(:delay => 1.hour).reset_password(@user.id)
346
+ ```
347
+
348
+ Backburner will take care of the rest!
349
+
350
+ ### Persistence
351
+
352
+ Jobs are persisted to queues as JSON objects. Let's take our `User`
353
+ example from above. We'll run the following code to create a job:
354
+
355
+ ``` ruby
356
+ User.async.reset_password(@user.id)
357
+ ```
358
+
359
+ The following JSON will be put on the `{namespace}.backburner-jobs` queue:
360
+
361
+ ``` javascript
362
+ {
363
+ 'class': 'User',
364
+ 'args': [nil, 'reset_password', 123]
365
+ }
366
+ ```
367
+
368
+ The first argument is the 'id' of the object in the case of an instance method being async'ed. For example:
369
+
370
+ ```ruby
371
+ @device = Device.find(987)
372
+ @user = User.find(246)
373
+ @user.async.activate(@device.id)
374
+ ```
375
+
376
+ would be stored as:
377
+
378
+ ``` javascript
379
+ {
380
+ 'class': 'User',
381
+ 'args': [246, 'activate', 987]
382
+ }
383
+ ```
384
+
385
+ Since all jobs are persisted in JSON, your jobs must only accept arguments that can be encoded into that format.
386
+ This is why our examples use object IDs instead of passing around objects.
387
+
388
+ ### Named Priorities
389
+
390
+ As of v0.4.0, Backburner has support for named priorities. beanstalkd priorities are numerical but
391
+ backburner supports a mapping between a word and a numerical value. The following priorities are
392
+ available by default: `high` is 0, `medium` is 100, and `low` is 200.
393
+
394
+ Priorities can be customized with:
395
+
396
+ ```ruby
397
+ Backburner.configure do |config|
398
+ config.priority_labels = { :custom => 50, :useful => 5 }
399
+ # or append to default priorities with
400
+ # config.priority_labels = Backburner::Configuration::PRIORITY_LABELS.merge(:foo => 5)
401
+ end
402
+ ```
403
+
404
+ and then these aliases can be used anywhere that a numerical value can:
405
+
406
+ ```ruby
407
+ Backburner::Worker.enqueue NewsletterJob, ["foo", "bar"], :pri => :custom
408
+ User.async(:pri => :useful, :delay => 10.seconds).reset_password(@user.id)
409
+ ```
410
+
411
+ Using named priorities can greatly simplify priority management.
412
+
413
+ ### Processing Strategies
414
+
415
+ In Backburner, there are several different strategies for processing jobs
416
+ which are reflected by multiple worker subclasses.
417
+ Custom workers can be [defined fairly easily](https://github.com/nesquena/backburner/wiki/Defining-Workers).
418
+ By default, Backburner comes with the following workers built-in:
419
+
420
+ | Worker | Description |
421
+ | ------- | ------------------------------- |
422
+ | `Backburner::Workers::Simple` | Single threaded, no forking worker. Simplest option. |
423
+ | `Backburner::Workers::Forking` | Basic forking worker that manages crashes and memory bloat. |
424
+ | `Backburner::Workers::ThreadsOnFork` | Forking worker that utilizes threads for concurrent processing. |
425
+ | `Backburner::Workers::Threading` | Utilizes thread pools for concurrent processing. |
426
+
427
+ You can select the default worker for processing with:
428
+
429
+ ```ruby
430
+ Backburner.configure do |config|
431
+ config.default_worker = Backburner::Workers::Forking
432
+ end
433
+ ```
434
+
435
+ or determine the worker on the fly when invoking `work`:
436
+
437
+ ```ruby
438
+ Backburner.work('newsletter-sender', :worker => Backburner::Workers::ThreadsOnFork)
439
+ ```
440
+
441
+ or through associated rake tasks with:
442
+
443
+ ```
444
+ $ QUEUE=newsletter-sender,push-message THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work
445
+ ```
446
+
447
+ When running on MRI or another Ruby implementation with a Global Interpreter Lock (GIL), do not be surprised if you're unable to saturate multiple cores, even with the threads_on_fork worker. To utilize multiple cores, you must run multiple worker processes.
448
+
449
+ Additional concurrency strategies will hopefully be contributed in the future.
450
+ If you are interested in helping out, please let us know.
451
+
452
+ #### More info: Threads on Fork Worker
453
+
454
+ For more information on the threads_on_fork worker, check out the
455
+ [ThreadsOnFork Worker](https://github.com/nesquena/backburner/wiki/ThreadsOnFork-worker) documentation. Please note that the `ThreadsOnFork` worker does not work on Windows due to its lack of `fork`.
456
+
457
+ #### More info: Threading Worker (thread-pool-based)
458
+
459
+ Configuration options for the Threading worker are similar to the threads_on_fork worker, sans the garbage option. When running via the `backburner` CLI, it's simplest to provide the queue names and maximum number of threads in the format "{queue name}:{max threads in pool}[,{name}:{threads}]":
460
+
461
+ ```
462
+ $ bundle exec backburner -q queue1:4,queue2:4 # and then other options, like environment, pidfile, app root, etc. See docs for the CLI
463
+ ```
464
+
465
+ ### Default Queues
466
+
467
+ Workers can be easily restricted to processing only a specific set of queues as shown above. However, if you want a worker to
468
+ process **all** queues instead, then you can leave the queue list blank.
469
+
470
+ When you execute a worker without any queues specified, queues for known job queue class with `include Backburner::Queue` will be processed.
471
+ To access the list of known queue classes, you can use:
472
+
473
+ ```ruby
474
+ Backburner::Worker.known_queue_classes
475
+ # => [NewsletterJob, SomeOtherJob]
476
+ ```
477
+
478
+ Dynamic queues created by passing queue options **will not be processed** by a default worker. For this reason, you may want to take control over the default list of
479
+ queues processed when none are specified. To do this, you can use the `default_queues` class method:
480
+
481
+ ```ruby
482
+ Backburner.default_queues.concat(["foo", "bar"])
483
+ ```
484
+
485
+ This will ensure that the _foo_ and _bar_ queues are processed by any default workers. You can also add job queue names with:
486
+
487
+ ```ruby
488
+ Backburner.default_queues << NewsletterJob.queue
489
+ ```
490
+
491
+ The `default_queues` stores the specific list of queues that should be processed by default by a worker.
492
+
493
+ ### Failures
494
+
495
+ When a job fails in backburner (usually because an exception was raised), the job will be released
496
+ and retried again until the `max_job_retries` configuration is reached.
497
+
498
+ ```ruby
499
+ Backburner.configure do |config|
500
+ config.max_job_retries = 3 # retry jobs 3 times
501
+ config.retry_delay = 2 # wait 2 seconds in between retries
502
+ end
503
+ ```
504
+
505
+ Note the default `max_job_retries` is 0, meaning that by default **jobs are not retried**.
506
+
507
+ As jobs are retried, a progressively-increasing delay is added to give time for transient
508
+ problems to resolve themselves. This may be configured using `retry_delay_proc`. It expects
509
+ an object that responds to `#call` and receives the value of `retry_delay` and the number
510
+ of times the job has been retried already. The default is a cubic back-off, eg:
511
+
512
+ ```ruby
513
+ Backburner.configure do |config|
514
+ config.retry_delay = 2 # The minimum number of seconds a retry will be delayed
515
+ config.retry_delay_proc = lambda { |min_retry_delay, num_retries| min_retry_delay + (num_retries ** 3) }
516
+ end
517
+ ```
518
+
519
+ If continued retry attempts fail, the job will be buried and can be 'kicked' later for inspection.
520
+
521
+ You can also setup a custom error handler for jobs using configure:
522
+
523
+ ```ruby
524
+ Backburner.configure do |config|
525
+ config.on_error = lambda { |ex| Airbrake.notify(ex) }
526
+ end
527
+ ```
528
+
529
+ Now all backburner queue errors will appear on airbrake for deeper inspection.
530
+
531
+ If you wish to retry a job without logging an error (for example when handling transient issues in a cloud or service oriented environment),
532
+ simply raise a `Backburner::Job::RetryJob` error.
533
+
534
+ ### Logging
535
+
536
+ Logging in backburner is rather simple. When a job is run, the log records that. When a job
537
+ fails, the log records that. When any exceptions occur during processing, the log records that.
538
+
539
+ By default, the log will print to standard out. You can customize the log to output to any
540
+ standard logger by controlling the configuration option:
541
+
542
+ ```ruby
543
+ Backburner.configure do |config|
544
+ config.logger = Logger.new(STDOUT)
545
+ end
546
+ ```
547
+
548
+ Be sure to check logs whenever things do not seem to be processing.
549
+
550
+ ### Hooks
551
+
552
+ Backburner is highly extensible and can be tailored to your needs by using various hooks that
553
+ can be triggered across the job processing lifecycle.
554
+ Often using hooks is much easier then trying to monkey patch the externals.
555
+
556
+ Check out [HOOKS.md](https://github.com/nesquena/backburner/blob/master/HOOKS.md) for a detailed overview on using hooks.
557
+
558
+ ### Workers in Production
559
+
560
+ Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](http://kr.github.com/beanstalkd/download.html)
561
+ is installed, your best bet is to use the built-in rake task that comes with Backburner. Simply add the task to your Rakefile:
562
+
563
+ ```ruby
564
+ # Rakefile
565
+ require 'backburner/tasks'
566
+ ```
567
+
568
+ and then you can start the rake task with:
569
+
570
+ ```bash
571
+ $ rake backburner:work
572
+ $ QUEUE=newsletter-sender,push-notifier rake backburner:work
573
+ ```
574
+
575
+ The best way to deploy these rake tasks is using a monitoring library. We suggest [God](https://github.com/mojombo/god/)
576
+ which watches processes and ensures their stability. A simple God recipe for Backburner can be found in
577
+ [examples/god](https://github.com/nesquena/backburner/blob/master/examples/god.rb).
578
+
579
+ #### Command-Line Interface
580
+
581
+ Instead of using the Rake tasks, you can use Backburner's command-line interface (CLI) – powered by the [Dante gem](https://github.com/nesquena/dante) – to launch daemonized workers. Several flags are available to control the process. Many of these are provided by Dante itself, such as flags for logging (`-l`), the process' PID (`-P`), whether to daemonize (`-d`) or kill a running process (`-k`). Backburner provides a few more:
582
+
583
+
584
+ ##### Queues (`-q`)
585
+
586
+ Control which queues the worker will watch with the `-q` flag. Comma-separate multiple queue names and, if you're using the `ThreadsOnFork` worker, colon-separate the settings for thread limit, garbage limit and retries limit (eg. `send_mail:4:10:3`). See its [wiki page](https://github.com/nesquena/backburner/wiki/ThreadsOnFork-worker) for some more details.
587
+
588
+ ```ruby
589
+ backburner -q send_mail,create_thumbnail # You may need to use `bundle exec`
590
+ ```
591
+
592
+ ##### Boot an app (`-r`)
593
+
594
+ Load an app with the `-r` flag. Backburner supports automatic loading for both Rails and Padrino apps when started from the their root folder. However, you may point to a specific app's root using this flag, which is very useful when running workers from a service script.
595
+
596
+ ```ruby
597
+ path="/var/www/my-app/current"
598
+ backburner -r "$path"
599
+ ```
600
+
601
+ ##### Load an environment (`-e`)
602
+
603
+ Use the `-e` flag to control which environment your app should use:
604
+
605
+ ```ruby
606
+ environment="production"
607
+ backburner -e $environment
608
+ ```
609
+
610
+ #### Reconnecting
611
+
612
+ In Backburner, if the beanstalkd connection is temporarily severed, several retries to establish the connection will be attempted.
613
+ After several retries, if the connection is still not able to be made, a `Beaneater::NotConnected` exception will be raised.
614
+ You can manually catch this exception, and attempt another manual retry using `Backburner::Worker.retry_connection!`.
615
+
616
+ ### Web Front-end
617
+
618
+ Be sure to check out the Sinatra-powered project [beanstalkd_view](https://github.com/denniskuczynski/beanstalkd_view)
619
+ by [denniskuczynski](http://github.com/denniskuczynski) which provides an excellent overview of the tubes and
620
+ jobs processed by your beanstalk workers. An excellent addition to your Backburner setup.
621
+
622
+ ## Acknowledgements
623
+
624
+ * [Nathan Esquenazi](https://github.com/nesquena) - Project maintainer
625
+ * [Dave Myron](https://github.com/contentfree) - Multiple features and doc improvements
626
+ * Kristen Tucker - Coming up with the gem name
627
+ * [Tim Lee](https://github.com/timothy1ee), [Josh Hull](https://github.com/joshbuddy), [Nico Taing](https://github.com/Nico-Taing) - Helping me work through the idea
628
+ * [Miso](http://gomiso.com) - Open-source friendly place to work
629
+ * [Evgeniy Denisov](https://github.com/silentshade) - Multiple fixes and cleanups
630
+ * [Andy Bakun](https://github.com/thwarted) - Fixes to how multiple beanstalkd instances are processed
631
+ * [Renan T. Fernandes](https://github.com/ShadowBelmolve) - Added threads_on_fork worker
632
+ * [Daniel Farrell](https://github.com/danielfarrell) - Added forking worker
633
+
634
+ ## Contributing
635
+
636
+ 1. Fork it
637
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
638
+ 3. Commit your changes (`git commit -am 'Added some feature'`)
639
+ 4. Push to the branch (`git push origin my-new-feature`)
640
+ 5. Create new Pull Request
641
+
642
+ ## References
643
+
644
+ The code in this project has been made in light of a few excellent projects:
645
+
646
+ * [DelayedJob](https://github.com/collectiveidea/delayed_job)
647
+ * [Resque](https://github.com/defunkt/resque)
648
+ * [Stalker](https://github.com/han/stalker)
649
+
650
+ Thanks to these projects for inspiration and certain design and implementation decisions.
651
+
652
+ ## Links
653
+
654
+ * Code: `git clone git://github.com/nesquena/backburner.git`
655
+ * Home: <http://github.com/nesquena/backburner>
656
+ * Docs: <http://rdoc.info/github/nesquena/backburner/master/frames>
657
+ * Bugs: <http://github.com/nesquena/backburner/issues>
658
+ * Gems: <http://gemcutter.org/gems/backburner>