chore-core 1.10.0 → 4.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -13
- data/LICENSE.txt +1 -1
- data/README.md +172 -153
- data/chore-core.gemspec +3 -3
- data/lib/chore.rb +29 -5
- data/lib/chore/cli.rb +22 -4
- data/lib/chore/configuration.rb +1 -1
- data/lib/chore/consumer.rb +54 -12
- data/lib/chore/fetcher.rb +12 -7
- data/lib/chore/hooks.rb +2 -1
- data/lib/chore/job.rb +19 -0
- data/lib/chore/manager.rb +17 -2
- data/lib/chore/publisher.rb +18 -2
- data/lib/chore/queues/filesystem/consumer.rb +126 -64
- data/lib/chore/queues/filesystem/filesystem_queue.rb +19 -0
- data/lib/chore/queues/filesystem/publisher.rb +10 -16
- data/lib/chore/queues/sqs.rb +22 -13
- data/lib/chore/queues/sqs/consumer.rb +64 -51
- data/lib/chore/queues/sqs/publisher.rb +26 -17
- data/lib/chore/strategies/consumer/batcher.rb +6 -6
- data/lib/chore/strategies/consumer/single_consumer_strategy.rb +5 -5
- data/lib/chore/strategies/consumer/threaded_consumer_strategy.rb +7 -6
- data/lib/chore/strategies/consumer/throttled_consumer_strategy.rb +120 -0
- data/lib/chore/strategies/worker/forked_worker_strategy.rb +5 -6
- data/lib/chore/strategies/worker/helpers/ipc.rb +87 -0
- data/lib/chore/strategies/worker/helpers/preforked_worker.rb +163 -0
- data/lib/chore/strategies/worker/helpers/work_distributor.rb +65 -0
- data/lib/chore/strategies/worker/helpers/worker_info.rb +13 -0
- data/lib/chore/strategies/worker/helpers/worker_killer.rb +40 -0
- data/lib/chore/strategies/worker/helpers/worker_manager.rb +183 -0
- data/lib/chore/strategies/worker/preforked_worker_strategy.rb +150 -0
- data/lib/chore/unit_of_work.rb +2 -1
- data/lib/chore/util.rb +5 -1
- data/lib/chore/version.rb +2 -2
- data/lib/chore/worker.rb +30 -3
- data/spec/chore/cli_spec.rb +2 -2
- data/spec/chore/consumer_spec.rb +1 -5
- data/spec/chore/duplicate_detector_spec.rb +17 -5
- data/spec/chore/fetcher_spec.rb +0 -11
- data/spec/chore/manager_spec.rb +7 -0
- data/spec/chore/queues/filesystem/filesystem_consumer_spec.rb +74 -16
- data/spec/chore/queues/sqs/consumer_spec.rb +117 -78
- data/spec/chore/queues/sqs/publisher_spec.rb +49 -60
- data/spec/chore/queues/sqs_spec.rb +32 -41
- data/spec/chore/strategies/consumer/single_consumer_strategy_spec.rb +3 -3
- data/spec/chore/strategies/consumer/threaded_consumer_strategy_spec.rb +6 -6
- data/spec/chore/strategies/consumer/throttled_consumer_strategy_spec.rb +165 -0
- data/spec/chore/strategies/worker/forked_worker_strategy_spec.rb +6 -1
- data/spec/chore/strategies/worker/helpers/ipc_spec.rb +127 -0
- data/spec/chore/strategies/worker/helpers/preforked_worker_spec.rb +236 -0
- data/spec/chore/strategies/worker/helpers/work_distributor_spec.rb +131 -0
- data/spec/chore/strategies/worker/helpers/worker_info_spec.rb +14 -0
- data/spec/chore/strategies/worker/helpers/worker_killer_spec.rb +97 -0
- data/spec/chore/strategies/worker/helpers/worker_manager_spec.rb +304 -0
- data/spec/chore/strategies/worker/preforked_worker_strategy_spec.rb +183 -0
- data/spec/chore/strategies/worker/single_worker_strategy_spec.rb +1 -1
- data/spec/chore/worker_spec.rb +70 -15
- data/spec/spec_helper.rb +1 -1
- data/spec/support/queues/sqs/fake_objects.rb +18 -0
- metadata +53 -29
checksums.yaml
CHANGED
@@ -1,15 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
|
5
|
-
data.tar.gz: !binary |-
|
6
|
-
ZTliZWFkNzRhZDZlZDBiZWZlNjg4MDcxNDEzMTlhNDkwNmJkOWU4NA==
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 9c18dd2782818f483d141ed7fff2daac3b9dacab
|
4
|
+
data.tar.gz: 78b658a9ad9b2e79126f194a56090e8e9da16d8f
|
7
5
|
SHA512:
|
8
|
-
metadata.gz:
|
9
|
-
|
10
|
-
YTUzNjBlMDIxYjM1MTcxNjE4NzU1YjdkNGVmOWY1Y2IzODg1MTQ5YjIxZGUx
|
11
|
-
ZWI5N2U4ZGQwZGRhYTVkZTRiZmMxYzY0NGM1MjQ1YzM5NmJiZTM=
|
12
|
-
data.tar.gz: !binary |-
|
13
|
-
Mjc2ODk3ZmFjZjc2ZDdiODlkMTA4ODQzM2I1N2JiZDI5M2QwZjU2ODNkOWRl
|
14
|
-
NGY4ZDg3NDRiMDAzMTUxNzM0MGQxNTNiNDM4ZmMwMmVhOTQzZTgwOTc4ZTc2
|
15
|
-
MTg1YjI4Y2M1ZjE2YTM5ZDJiNTMwY2NkMjNjNTlmMTAyZmM1YTQ=
|
6
|
+
metadata.gz: 24ef03d8f8a42f2632073c2bfa4df9e466aa9fea90350f8df496dc117cae946aff83fcc1f4d458c00f1b8149067c85d3f990ebddac7b85d8d443901cf94062e9
|
7
|
+
data.tar.gz: 0925bbf1e586472db8a894116b2b9b6f4fe13c54941e250ae1006f400baf6fe35261314c63934c40d2f1b683dea2dd33c684d134f2edb07fecf7c1ae88001c48
|
data/LICENSE.txt
CHANGED
data/README.md
CHANGED
@@ -1,90 +1,90 @@
|
|
1
|
-
# Chore: Job
|
1
|
+
# Chore: Async Job Processing Framework For Ruby
|
2
|
+
|
3
|
+
[](https://travis-ci.org/Tapjoy/chore)
|
2
4
|
|
3
5
|
## About
|
4
6
|
|
5
|
-
Chore is a pluggable, multi-backend job
|
6
|
-
|
7
|
+
Chore is a pluggable, multi-backend job processing framework. It was built from the ground up to be extremely flexible.
|
8
|
+
We hope that you find integrating and using Chore to be as pleasant as we do.
|
7
9
|
|
8
|
-
The full docs for Chore can always be found at
|
10
|
+
The full docs for Chore can always be found at https://tapjoy.github.io/chore.
|
9
11
|
|
10
12
|
## Configuration
|
11
13
|
|
12
14
|
Chore can be integrated with any Ruby-based project by following these instructions:
|
13
15
|
|
14
|
-
|
16
|
+
1. Add `chore-core` to the Gemfile
|
17
|
+
|
18
|
+
```ruby
|
19
|
+
gem 'chore-core', '~> 4.0.0'
|
20
|
+
```
|
15
21
|
|
16
|
-
|
22
|
+
When using SQS, also add `dalli` to use for `memcached`-based deduplication:
|
17
23
|
|
24
|
+
```ruby
|
18
25
|
gem 'dalli'
|
26
|
+
```
|
19
27
|
|
20
|
-
Create a `Chorefile` file in the root of
|
28
|
+
1. Create a `Chorefile` file in the root of the project directory. While Chore itself can be configured from this file,
|
29
|
+
it's primarily used to direct the Chore binstub toward the root of the application so that it can locate all of the
|
30
|
+
dependencies and required code.
|
21
31
|
|
32
|
+
```
|
22
33
|
--require=./<FILE_TO_LOAD>
|
34
|
+
```
|
23
35
|
|
24
|
-
Make sure that `--require` points to the main entry point for
|
25
|
-
|
26
|
-
Other options include:
|
27
|
-
|
28
|
-
--concurrency 16 # number of concurrent worker processes, if using forked worker strategy
|
29
|
-
--worker-strategy Chore::Strategy::ForkedWorkerStrategy # which worker strategy class to use
|
30
|
-
--consumer Chore::Queues::SQS::Consumer # which consumer class to use Options are SQS::Consumer and Filesystem::Consumer. Filesystem is recommended for local and testing purposes only.
|
31
|
-
--consumer-strategy Chore::Queues::Strategies::Consumer::ThreadedConsumerStrategy # which consuming strategy to use. Options are SingleConsumerStrategy and ThreadedConsumerStrategy. Threaded is recommended for better tuning your consuming profile
|
32
|
-
--consumer-sleep-interval 1.0 # The amount of time in seconds to sleep when a consumer doesn't receive any messages. Sub-second values are accepted. The default varies by consumer implementation. This is a weak form of backoff for when there is no work to do.
|
33
|
-
--threads-per-queue 4 # number of threads per queue for consuming from a given queue.
|
34
|
-
--dedupe-servers # if using SQS or similiar queue with at-least once delivery and your memcache is running on something other than localhost
|
35
|
-
--batch-size 50 # how many messages are batched together before handing them to a worker
|
36
|
-
--batch-timeout 20 # maximum number of seconds to wait until handing a message over to a worker
|
37
|
-
--queue_prefix prefixy # A prefix to prepend to queue names, mainly for development and qa testing purposes
|
38
|
-
--max-attempts 100 # The maximum number of times a job can be attempted
|
39
|
-
--dupe-on-cache-failure # Determines the deduping behavior when a cache connection error occurs. When set to `false`, the message is assumed not to be a duplicate. Defaults to `false`.
|
40
|
-
--queue-polling-size 10 # If your particular queueing system supports responding with messages in batches of a certain size, you can control that with this flag. SQS has a built in upper-limit of 10, but other systems will vary.
|
41
|
-
|
42
|
-
If you're using SQS, you'll want to add AWS keys so that Chore can authenticate with AWS.
|
36
|
+
Make sure that `--require` points to the main entry point for the application. If integrating with a Rails app,
|
37
|
+
point it to the application directory and Chore will handle loading the correct files on its own.
|
43
38
|
|
44
|
-
|
45
|
-
|
39
|
+
1. When using SQS, ensure that AWS credentials exist in the environment (e.g. but not limited to `AWS_ACCESS_KEY_ID` &
|
40
|
+
`AWS_SECRET_ACCESS_KEY` environment variables) and an AWS region is set (e.g. `AWS_REGION` environment variable) so that
|
41
|
+
Chore can authenticate with AWS.
|
46
42
|
|
47
|
-
By default, Chore will run over all queues it detects among the required files. If
|
43
|
+
By default, Chore will run over all queues it detects among the required files. If different behavior is desired,
|
44
|
+
use one of the following flags:
|
48
45
|
|
46
|
+
```
|
47
|
+
# Note that only one of these options may be used, not both. Chore will quit
|
48
|
+
# if both options are specified.
|
49
49
|
--queues QUEUE1,QUEUE2... # a list of queues to process
|
50
50
|
--except-queues QUEUE1,QUEUE2... # a list of queues _not_ to process
|
51
|
+
```
|
51
52
|
|
52
|
-
|
53
|
+
1. Chore has many more options, which can be viewed by executing `bundle exec chore --help`
|
53
54
|
|
54
|
-
### Tips
|
55
|
+
### Tips For Configuring Chore
|
55
56
|
|
56
|
-
For Rails, it can be necessary to add the
|
57
|
-
|
58
|
-
|
57
|
+
For Rails, it can be necessary to add the jobs directory to the eager loading path, found in `application.rb`. A similar
|
58
|
+
approach for most apps using jobs is likely needed, unless the jobs are placed into a directory that is already eager
|
59
|
+
loaded by the application. One example of this might be:
|
59
60
|
|
60
61
|
```ruby
|
61
62
|
config.eager_load_paths += File.join(config.root, "app", "jobs")
|
62
63
|
```
|
63
64
|
|
64
|
-
However, due to the way eager_load_paths works in Rails, this may only solve the issue
|
65
|
-
|
66
|
-
|
67
|
-
can choose to load the job files in any way you like:
|
65
|
+
However, due to the way `eager_load_paths` works in Rails, this may only solve the issue in the production environment.
|
66
|
+
It can also be useful useful for other environments to have something like this in an `config/initializers/chore.rb`
|
67
|
+
file, although the job files can be loaded in just about any way.
|
68
68
|
|
69
69
|
```ruby
|
70
|
-
|
71
|
-
|
72
|
-
|
70
|
+
if !Rails.env.production?
|
71
|
+
Dir["#{Rails.root}/app/jobs/**/*"].each do |file|
|
72
|
+
require file unless File.directory?(file)
|
73
|
+
end
|
74
|
+
end
|
73
75
|
```
|
74
76
|
|
75
|
-
### Producing
|
76
|
-
|
77
|
-
When it comes to configuring Chore, you have 2 main use cases - as a producer of messages, or as a consumer of messages (the consumer is also able to produce messages if need be, but is running as it's own isolated instance of your application).
|
77
|
+
### Producing & Consuming Jobs
|
78
78
|
|
79
|
-
|
79
|
+
When it comes to configuring Chore, there are 2 main use configurations - as a producer of messages, or as a consumer of
|
80
|
+
messages. The consuming context may also messages if necessary, as it is running as its own isolated instance of the
|
81
|
+
application.
|
80
82
|
|
81
|
-
For
|
83
|
+
For producers, all of the Chore configuration must be in an initializer.
|
82
84
|
|
83
|
-
|
85
|
+
For consumers, a Chorefile must be used. A Chorefile _plus_ an initializer is also a good pattern.
|
84
86
|
|
85
|
-
|
86
|
-
|
87
|
-
An example of how to configure chore via an initializer:
|
87
|
+
Here's example of how to configure chore via an initializer:
|
88
88
|
|
89
89
|
```ruby
|
90
90
|
Chore.configure do |c|
|
@@ -97,27 +97,42 @@ Chore.configure do |c|
|
|
97
97
|
end
|
98
98
|
```
|
99
99
|
|
100
|
+
Because it is like that the same application serves as the basis for both producing and consuming messages, and there
|
101
|
+
will already be a considerable amount of configuration in the Producer, it makes sense to use Chorefile to simply
|
102
|
+
provide the `require` option and stick to the initializer for the rest of the configuration to keep things DRY.
|
103
|
+
|
104
|
+
However, like many aspects of Chore, it is ultimately up to the developer to decide which use case fits their needs
|
105
|
+
best. Chore is happy to be configured in almost any way a developer desires.
|
106
|
+
|
100
107
|
## Integration
|
101
108
|
|
102
|
-
|
109
|
+
This section assumes `foreman` is being used to execute (or export the run commands of) the application, but it is not
|
110
|
+
strictly necessary.
|
111
|
+
|
112
|
+
1. Add an appropriate line to the `Procfile`:
|
103
113
|
|
114
|
+
```
|
104
115
|
jobs: bundle exec chore -c config/chore.config
|
116
|
+
```
|
105
117
|
|
106
|
-
If
|
118
|
+
1. If the queues do not exist, they must be created before the application can produce/consume Chore jobs:
|
107
119
|
|
108
|
-
```ruby
|
109
|
-
require 'aws-sdk'
|
110
|
-
sqs =
|
111
|
-
sqs.
|
112
|
-
```
|
120
|
+
```ruby
|
121
|
+
require 'aws-sdk-sqs'
|
122
|
+
sqs = Aws::SQS::Client.new
|
123
|
+
sqs.create_queue(queue_name: "test_queue")
|
124
|
+
```
|
113
125
|
|
114
|
-
Finally, start
|
126
|
+
1. Finally, start the application as usual
|
115
127
|
|
128
|
+
```
|
116
129
|
bundle exec foreman start
|
130
|
+
```
|
117
131
|
|
118
|
-
## Chore::Job
|
132
|
+
## `Chore::Job`
|
119
133
|
|
120
|
-
A Chore::Job is any class
|
134
|
+
A `Chore::Job` is any class with `include Chore::Job` and implements a `perform(*args)` instance method. Here is an
|
135
|
+
example job class:
|
121
136
|
|
122
137
|
```ruby
|
123
138
|
class TestJob
|
@@ -127,30 +142,36 @@ class TestJob
|
|
127
142
|
def perform(args={})
|
128
143
|
Chore.logger.debug "My first async job"
|
129
144
|
end
|
130
|
-
|
131
145
|
end
|
132
146
|
```
|
133
147
|
|
134
|
-
This job declares that the name of the queue it uses is `test_queue`, set in the queue_options method.
|
148
|
+
This job declares that the name of the queue it uses is `test_queue`, set in the `queue_options` method.
|
135
149
|
|
136
|
-
### Chore::Job
|
150
|
+
### `Chore::Job` & `perform` Signatures
|
137
151
|
|
138
|
-
The perform method signature can have explicit argument names, but in practice this makes changing the signature more
|
152
|
+
The perform method signature can have explicit argument names, but in practice this makes changing the signature more
|
153
|
+
difficult later on. Once a `Chore::Job` is in production and being used at a constant rate, it becomes problematic to
|
154
|
+
begin mixing versions of the job with non-matching signatures.
|
139
155
|
|
140
|
-
While this is able to be overcome with a number of techniques, such as versioning
|
156
|
+
While this is able to be overcome with a number of techniques, such as versioning jobs/queues, it increases the
|
157
|
+
complexity of making changes.
|
141
158
|
|
142
|
-
The simplest way to structure job signatures is to treat the arguments as a hash. This
|
159
|
+
The simplest way to structure job signatures is to treat the arguments as a hash. This enables maintaining forwards and
|
160
|
+
backwards compatibility between signature changes with the same job class.
|
143
161
|
|
144
|
-
However, Chore is ultimately agnostic
|
162
|
+
However, Chore is ultimately agnostic in this regard and will allow explicit arguments in signatures as easily as using
|
163
|
+
a simple hash; the choice is left to the developer.
|
145
164
|
|
146
|
-
### Chore::Job
|
165
|
+
### `Chore::Job` & Publishing Jobs
|
166
|
+
|
167
|
+
Now that there's a test job, publishing an instance of the job is as simple as:
|
147
168
|
|
148
|
-
Now that you've got a test job, if you wanted to publish to that job it's as simple as:
|
149
169
|
```ruby
|
150
170
|
TestJob.perform_async({"message"=>"YES, DO THAT THING."})
|
151
171
|
```
|
152
172
|
|
153
|
-
It's advisable to specify the Publisher
|
173
|
+
It's advisable to specify the Publisher Chore uses to send messages globally, so that it can easily be modified based on
|
174
|
+
the environment. To do this, add a configuration block to an initializer:
|
154
175
|
|
155
176
|
```ruby
|
156
177
|
Chore.configure do |c|
|
@@ -158,14 +179,15 @@ Chore.configure do |c|
|
|
158
179
|
end
|
159
180
|
```
|
160
181
|
|
161
|
-
It is worth noting that any option that can be set via config file or command-line args can also be set in a configure
|
182
|
+
It is worth noting that any option that can be set via config file or command-line args can also be set in a configure
|
183
|
+
block.
|
162
184
|
|
163
185
|
If a global publisher is set, it can be overridden on a per-job basis by specifying the publisher in `queue_options`.
|
164
186
|
|
165
187
|
## Retry Backoff Strategy
|
166
188
|
|
167
|
-
Chore has basic support for delaying retries
|
168
|
-
supports this functionality is SQS
|
189
|
+
Chore has basic support for delaying retries of a failed job using a step function. Currently the only queue that
|
190
|
+
supports this functionality is SQS; all others will simply ignore the delay setting.
|
169
191
|
|
170
192
|
### Setup
|
171
193
|
|
@@ -177,51 +199,52 @@ queue_options :name => 'nameOfQueue',
|
|
177
199
|
:backoff => lambda { |work| work.current_attempt ** 2 } # Exponential backoff
|
178
200
|
```
|
179
201
|
|
180
|
-
### Using
|
202
|
+
### Using The Backoff
|
181
203
|
|
182
204
|
If there is a `:backoff` option supplied, any failures will delay the next attempt by the result of that lambda.
|
183
205
|
|
184
|
-
### Notes
|
206
|
+
### Notes On SQS & Delays
|
185
207
|
|
186
|
-
Read more details about SQS and Delays [here](docs/Delayed
|
208
|
+
Read more details about SQS and Delays [here](docs/Delayed%20Jobs.md)
|
187
209
|
|
188
210
|
## Hooks
|
189
211
|
|
190
|
-
A number of hooks, both global and per-job, exist in Chore for
|
191
|
-
|
192
|
-
Global Hooks:
|
212
|
+
A number of hooks, both global and per-job, exist in Chore for flexibility and convencience. Hooks should be named
|
213
|
+
`hook_name_identifier` where `identifier` is a descriptive string of chosen by the developer.
|
193
214
|
|
194
|
-
|
195
|
-
* before_first_fork
|
196
|
-
* before_fork
|
197
|
-
* after_fork
|
198
|
-
* around_fork
|
199
|
-
* within_fork
|
200
|
-
* before_shutdown
|
215
|
+
### Global Hooks
|
201
216
|
|
202
|
-
|
217
|
+
* `before_start`
|
218
|
+
* `before_first_fork`
|
219
|
+
* `before_fork`
|
220
|
+
* `after_fork`
|
221
|
+
* `around_fork`
|
222
|
+
* `within_fork`
|
223
|
+
* behaves similarly to `around_fork`, except that it is called _after_ the worker process has been forked.
|
224
|
+
In contrast, `around_fork` is called by the parent process ( chore-master`)
|
225
|
+
* `before_shutdown`
|
203
226
|
|
204
|
-
Filesystem Consumer/Publisher
|
227
|
+
## Filesystem Consumer/Publisher Hooks
|
205
228
|
|
206
|
-
* on_fetch(job_file, job_json)
|
229
|
+
* `on_fetch(job_file, job_json)`
|
207
230
|
|
208
|
-
SQS Consumer
|
231
|
+
#### SQS Consumer Hooks
|
209
232
|
|
210
|
-
* on_fetch(handle, body)
|
233
|
+
* `on_fetch(handle, body)`
|
211
234
|
|
212
|
-
Per Job
|
235
|
+
### Per Job
|
213
236
|
|
214
|
-
* before_publish
|
215
|
-
* after_publish
|
216
|
-
* before_perform(message)
|
217
|
-
* after_perform(message)
|
218
|
-
* on_rejected(message)
|
219
|
-
* on_failure(message, error)
|
220
|
-
* on_permanent_failure(queue_name, message, error)
|
237
|
+
* `before_publish`
|
238
|
+
* `after_publish`
|
239
|
+
* `before_perform(message)`
|
240
|
+
* `after_perform(message)`
|
241
|
+
* `on_rejected(message)`
|
242
|
+
* `on_failure(message, error)`
|
243
|
+
* `on_permanent_failure(queue_name, message, error)`
|
221
244
|
|
222
245
|
All per-job hooks can also be global hooks.
|
223
246
|
|
224
|
-
Hooks can be added to a job class
|
247
|
+
Hooks can be added to a job class like so:
|
225
248
|
|
226
249
|
```ruby
|
227
250
|
class TestJob
|
@@ -237,56 +260,52 @@ class TestJob
|
|
237
260
|
end
|
238
261
|
end
|
239
262
|
```
|
263
|
+
|
240
264
|
Global hooks can also be registered like so:
|
241
265
|
|
242
266
|
```ruby
|
243
267
|
Chore.add_hook :after_publish do
|
244
|
-
#
|
268
|
+
# Add handler code here
|
245
269
|
end
|
246
270
|
```
|
247
271
|
|
248
272
|
## Signals
|
249
273
|
|
250
|
-
Signal handling can get complicated when
|
251
|
-
|
274
|
+
Signal handling can get complicated when there are multiple threads, process forks, and both signal handlers and
|
275
|
+
application code making use of mutexes.
|
252
276
|
|
253
|
-
To simplify the complexities around this, Chore introduces some additional
|
254
|
-
|
255
|
-
functionality is primarily inspired by sidekiq's signal handling @
|
277
|
+
To simplify the complexities around this, Chore introduces some additional behaviors on top of Ruby's default
|
278
|
+
`Signal.trap` implementation. This functionality is primarily inspired by `sidekiq`'s signal handling @
|
256
279
|
https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb.
|
257
280
|
|
258
|
-
In particular
|
259
|
-
|
260
|
-
on the differences between Ruby's `Signal.trap` and Chore's `Chore::Signal.trap`.
|
281
|
+
In particular Chore handles signals in a separate thread, and does so sequentially instead of being interrupt-driven.
|
282
|
+
See `Chore::Signal` for more details on the differences between Ruby's `Signal.trap` and Chore's `Chore::Signal.trap`.
|
261
283
|
|
262
|
-
Chore will respond to the following
|
284
|
+
Chore will respond to the following signals:
|
263
285
|
|
264
|
-
* INT , TERM
|
265
|
-
|
286
|
+
* `INT` , `TERM`, `QUIT` - Chore will begin shutting down, taking steps to safely terminate workers and not interrupt
|
287
|
+
jobs in progress unless it believes they may be hung
|
288
|
+
* `USR1` - Re-opens logfiles, useful for handling log rotations
|
266
289
|
|
267
290
|
## Timeouts
|
268
291
|
|
269
|
-
When using the forked worker strategy for processing jobs, inevitably there are
|
270
|
-
|
271
|
-
hung network calls, tight loops, etc. When these jobs hang, they consume
|
292
|
+
When using the forked worker strategy for processing jobs, inevitably there are cases in which child processes become
|
293
|
+
stuck. This could result from deadlocks, hung network calls, tight loops, etc. When these jobs hang, they consume
|
272
294
|
resources and can affect throughput.
|
273
295
|
|
274
|
-
To mitigate this, Chore has built-in monitoring of forked child processes.
|
275
|
-
|
276
|
-
|
277
|
-
a KILL signal.
|
296
|
+
To mitigate this, Chore has built-in monitoring of forked child processes. When a fork is created to process a batch of
|
297
|
+
work, that fork is assigned an expiration time -- if it doesn't complete by that time, the process is sent a `KILL`
|
298
|
+
signal.
|
278
299
|
|
279
300
|
Fork expiration times are determined from one of two places:
|
280
|
-
1. The timeout associated with the queue. For SQS, this is the visibility
|
281
|
-
timeout.
|
282
|
-
2. The default queue timeout configured for Chore. For Filesystem queues,
|
283
|
-
this is the value used.
|
284
301
|
|
285
|
-
|
286
|
-
|
302
|
+
1. The timeout associated with the queue. For SQS queues, this is the visibility timeout.
|
303
|
+
1. The default queue timeout configured for Chore. For filesystem queues, this is the value used.
|
287
304
|
|
288
|
-
|
289
|
-
the
|
305
|
+
For example, if a worker is processing a batch of 5 jobs and each job's queue has a timeout of 60s, then the expiration
|
306
|
+
time will be 5 minutes for the worker.
|
307
|
+
|
308
|
+
To change the default queue timeout (when one can't be inferred), do the following:
|
290
309
|
|
291
310
|
```ruby
|
292
311
|
Chore.configure do |c|
|
@@ -294,62 +313,62 @@ Chore.configure do |c|
|
|
294
313
|
end
|
295
314
|
```
|
296
315
|
|
297
|
-
A reasonable timeout would be based on the maximum amount of time
|
298
|
-
|
299
|
-
get killed if the job is running for too long.
|
316
|
+
A reasonable timeout would be based on the maximum amount of time any job in the system is expected to run. Keep in
|
317
|
+
mind that the process running the job may get killed if the job is running for too long.
|
300
318
|
|
301
319
|
## Plugins
|
302
320
|
|
303
|
-
Chore has several plugin gems available, which extend
|
321
|
+
Chore has several plugin gems available, which extend its core functionality
|
304
322
|
|
305
323
|
[New Relic](https://github.com/Tapjoy/chore-new_relic) - Integrating Chore with New Relic
|
306
324
|
|
307
325
|
[Airbrake](https://github.com/Tapjoy/chore-airbrake) - Integrating Chore with Airbrake
|
308
326
|
|
309
|
-
## Managing Chore
|
327
|
+
## Managing Chore Processes
|
310
328
|
|
311
329
|
### Sample Upstart
|
312
330
|
|
313
|
-
There are lots of ways to create upstart scripts, so it's difficult to give a prescriptive
|
314
|
-
|
315
|
-
at Tapjoy:
|
331
|
+
There are lots of ways to create upstart scripts, so it's difficult to give a prescriptive example of the "right" way to
|
332
|
+
do it. However, here are some ideas from how we run it in production at Tapjoy:
|
316
333
|
|
317
|
-
|
318
|
-
|
334
|
+
For security reasons, a specific user should be specified that the process runs as. Switch to this user at the beginning
|
335
|
+
of the exec line
|
319
336
|
|
320
337
|
```bash
|
321
338
|
su - $USER --command '...'
|
322
339
|
```
|
323
340
|
|
324
|
-
For the command to run Chore itself keeping all of the necessary environment variables in an env
|
325
|
-
|
326
|
-
|
341
|
+
For the command to run Chore itself keeping all of the necessary environment variables in an env file that Upstart can
|
342
|
+
source on it's exec line, to prevent having to mix changing environment variables with having to change the upstart
|
343
|
+
script itself
|
327
344
|
|
328
345
|
```bash
|
329
346
|
source $PATHTOENVVARS ;
|
330
347
|
```
|
331
348
|
|
332
|
-
After that,
|
333
|
-
|
334
|
-
|
349
|
+
After that, ensure Chore is running under the right ruby version. Additionally, `STDOUT` and `STDERR` can be redirected
|
350
|
+
to `logger` with an app name. This makes it easy to find information in syslog later on. Putting that all together looks
|
351
|
+
like:
|
335
352
|
|
336
353
|
```bash
|
337
354
|
rvm use $RUBYVERSION do bundle exec chore -c Chorefile 2>&1 | logger -t $APPNAME
|
338
355
|
```
|
339
356
|
|
340
|
-
There are many other ways
|
341
|
-
|
357
|
+
There are many other ways to manage the Upstart file, but these are a few of the ways we prefer to do it. Putting it all
|
358
|
+
together, it looks something like:
|
342
359
|
|
343
360
|
```bash
|
344
|
-
exec su - special_user --command '
|
361
|
+
exec su - special_user --command '\
|
362
|
+
source /the/path/to/env ;\
|
363
|
+
rvm use 2.4.1 do bundle exec chore -c Chorefile 2>&1 | logger chore-app ;'
|
345
364
|
```
|
346
365
|
|
347
366
|
### Locating Processes
|
348
367
|
|
349
|
-
As Chore does not keep a
|
350
|
-
|
368
|
+
As Chore does not keep a PID file, and has both a master and a potential number of workers, it may be difficult to
|
369
|
+
isolate the exact PID for the master process.
|
351
370
|
|
352
|
-
To find Chore master processes via
|
371
|
+
To find Chore master processes via `ps`, run the following:
|
353
372
|
|
354
373
|
```bash
|
355
374
|
ps aux | grep bin/chore
|
@@ -364,15 +383,15 @@ pgrep -f bin/chore
|
|
364
383
|
To find a list of only Chore worker processes:
|
365
384
|
|
366
385
|
```bash
|
367
|
-
ps aux | grep chore-
|
386
|
+
ps aux | grep chore-worker
|
368
387
|
```
|
369
388
|
|
370
389
|
or
|
371
390
|
|
372
391
|
```bash
|
373
|
-
pgrep -f chore-
|
392
|
+
pgrep -f chore-worker
|
374
393
|
```
|
394
|
+
|
375
395
|
## Copyright
|
376
396
|
|
377
|
-
Copyright (c) 2013 -
|
378
|
-
further details.
|
397
|
+
Copyright (c) 2013 - 2020 Tapjoy. See [LICENSE.txt](LICENSE.txt) for further details.
|