lowkiq 1.0.4 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/Gemfile.lock +14 -4
- data/README.md +133 -66
- data/README.ru.md +2 -2
- data/assets/app.js +1 -1
- data/lib/lowkiq/queue/fetch.rb +4 -4
- data/lib/lowkiq/queue/keys.rb +16 -4
- data/lib/lowkiq/queue/queue.rb +99 -48
- data/lib/lowkiq/queue/queue_metrics.rb +3 -5
- data/lib/lowkiq/queue/shard_metrics.rb +3 -4
- data/lib/lowkiq/schedulers/lag.rb +1 -1
- data/lib/lowkiq/script.rb +42 -0
- data/lib/lowkiq/server.rb +4 -0
- data/lib/lowkiq/shard_handler.rb +4 -4
- data/lib/lowkiq/utils.rb +1 -1
- data/lib/lowkiq/version.rb +1 -1
- data/lib/lowkiq/web/api.rb +1 -1
- data/lib/lowkiq/web.rb +2 -0
- data/lib/lowkiq/worker.rb +0 -2
- data/lib/lowkiq.rb +12 -7
- data/lowkiq.gemspec +1 -0
- metadata +19 -5
- data/lib/lowkiq/extend_tracker.rb +0 -13
- data/lib/lowkiq/queue/marshal.rb +0 -23
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 31b5b7d70917347fd443b4486b997b5d282fe55906e0c5e9c9e2e0619fa8037e
|
4
|
+
data.tar.gz: f609df71cb9709bea7141a7439f9e77a8fc6c1ef76b033a91758f36fc629f6a5
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7489ef14cfdcb01ececd8642f0715797e3f4284b48da8ca5f6189a6ce962746f117d181f9020f052ac5c38d27195fddcea68adbfdc678966b2d797c0d49f4010
|
7
|
+
data.tar.gz: cd6f3cc00871540775a457135c55d7e6a652f24d3768d7b794d0290a2336da32420384ef27b13ba57f4a762b9a8ea2b2f009b6d34c39de832dc5f2b14c0999fb
|
data/CHANGELOG.md
ADDED
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
lowkiq (1.
|
4
|
+
lowkiq (1.1.0)
|
5
5
|
connection_pool (~> 2.2, >= 2.2.2)
|
6
6
|
rack (>= 1.5.0)
|
7
7
|
redis (>= 4.0.1, < 5)
|
@@ -9,13 +9,22 @@ PATH
|
|
9
9
|
GEM
|
10
10
|
remote: https://rubygems.org/
|
11
11
|
specs:
|
12
|
-
|
12
|
+
byebug (11.1.3)
|
13
|
+
coderay (1.1.3)
|
14
|
+
connection_pool (2.2.3)
|
13
15
|
diff-lcs (1.3)
|
16
|
+
method_source (1.0.0)
|
17
|
+
pry (0.13.1)
|
18
|
+
coderay (~> 1.1)
|
19
|
+
method_source (~> 1.0)
|
20
|
+
pry-byebug (3.9.0)
|
21
|
+
byebug (~> 11.0)
|
22
|
+
pry (~> 0.13.0)
|
14
23
|
rack (2.2.2)
|
15
24
|
rack-test (1.1.0)
|
16
25
|
rack (>= 1.0, < 3)
|
17
26
|
rake (12.3.3)
|
18
|
-
redis (4.
|
27
|
+
redis (4.2.5)
|
19
28
|
rspec (3.9.0)
|
20
29
|
rspec-core (~> 3.9.0)
|
21
30
|
rspec-expectations (~> 3.9.0)
|
@@ -36,10 +45,11 @@ PLATFORMS
|
|
36
45
|
DEPENDENCIES
|
37
46
|
bundler (~> 2.1.0)
|
38
47
|
lowkiq!
|
48
|
+
pry-byebug (~> 3.9.0)
|
39
49
|
rack-test (~> 1.1)
|
40
50
|
rake (~> 12.3.0)
|
41
51
|
rspec (~> 3.0)
|
42
52
|
rspec-mocks (~> 3.8)
|
43
53
|
|
44
54
|
BUNDLED WITH
|
45
|
-
2.1.
|
55
|
+
2.1.4
|
data/README.md
CHANGED
@@ -16,6 +16,7 @@ Ordered background jobs processing
|
|
16
16
|
* [Api](#api)
|
17
17
|
* [Ring app](#ring-app)
|
18
18
|
* [Configuration](#configuration)
|
19
|
+
* [Performance](#performance)
|
19
20
|
* [Execution](#execution)
|
20
21
|
* [Shutdown](#shutdown)
|
21
22
|
* [Debug](#debug)
|
@@ -27,13 +28,15 @@ Ordered background jobs processing
|
|
27
28
|
* [Recommendations on configuration](#recommendations-on-configuration)
|
28
29
|
+ [`SomeWorker.shards_count`](#someworkershards_count)
|
29
30
|
+ [`SomeWorker.max_retry_count`](#someworkermax_retry_count)
|
31
|
+
* [Changing of worker's shards amount](#changing-of-workers-shards-amount)
|
32
|
+
* [Extended error info](#extended-error-info)
|
30
33
|
|
31
34
|
## Rationale
|
32
35
|
|
33
36
|
We've faced some problems using Sidekiq while processing messages from a side system.
|
34
|
-
For instance, the message is
|
35
|
-
The side system will send
|
36
|
-
Orders are frequently updated and a queue
|
37
|
+
For instance, the message is the data of an order at a particular time.
|
38
|
+
The side system will send new data of an order on every change.
|
39
|
+
Orders are frequently updated and a queue contains some closely located messages of the same order.
|
37
40
|
|
38
41
|
Sidekiq doesn't guarantee a strict message order, because a queue is processed by multiple threads.
|
39
42
|
For example, we've received 2 messages: M1 and M2.
|
@@ -42,8 +45,8 @@ so M2 can be processed before M1.
|
|
42
45
|
|
43
46
|
Parallel processing of such kind of messages can result in:
|
44
47
|
|
45
|
-
+
|
46
|
-
+ overwriting new data with old one
|
48
|
+
+ deadlocks
|
49
|
+
+ overwriting new data with an old one
|
47
50
|
|
48
51
|
Lowkiq has been created to eliminate such problems by avoiding parallel task processing within one entity.
|
49
52
|
|
@@ -51,17 +54,17 @@ Lowkiq has been created to eliminate such problems by avoiding parallel task pro
|
|
51
54
|
|
52
55
|
Lowkiq's queues are reliable i.e.,
|
53
56
|
Lowkiq saves information about a job being processed
|
54
|
-
and returns
|
57
|
+
and returns uncompleted jobs to the queue on startup.
|
55
58
|
|
56
59
|
Jobs in queues are ordered by preassigned execution time, so they are not FIFO queues.
|
57
60
|
|
58
|
-
Every job has
|
61
|
+
Every job has its identifier. Lowkiq guarantees that jobs with equal IDs are processed by the same thread.
|
59
62
|
|
60
63
|
Every queue is divided into a permanent set of shards.
|
61
|
-
A job is placed into particular shard based on an id of the job.
|
64
|
+
A job is placed into a particular shard based on an id of the job.
|
62
65
|
So jobs with the same id are always placed into the same shard.
|
63
66
|
All jobs of the shard are always processed with the same thread.
|
64
|
-
This guarantees the
|
67
|
+
This guarantees the sequential processing of jobs with the same ids and excludes the possibility of locks.
|
65
68
|
|
66
69
|
Besides the id, every job has a payload.
|
67
70
|
Payloads are accumulated for jobs with the same id.
|
@@ -70,8 +73,8 @@ It's useful when you need to process only the last message and drop all previous
|
|
70
73
|
|
71
74
|
A worker corresponds to a queue and contains a job processing logic.
|
72
75
|
|
73
|
-
|
74
|
-
Adding or removing queues or
|
76
|
+
The fixed number of threads is used to process all jobs of all queues.
|
77
|
+
Adding or removing queues or their shards won't affect the number of threads.
|
75
78
|
|
76
79
|
## Sidekiq comparison
|
77
80
|
|
@@ -82,45 +85,46 @@ But if you use plugins like
|
|
82
85
|
[sidekiq-merger](https://github.com/dtaniwaki/sidekiq-merger)
|
83
86
|
or implement your own lock system, you should look at Lowkiq.
|
84
87
|
|
85
|
-
For example, sidekiq-grouping accumulates a batch of jobs
|
86
|
-
With this approach queue can
|
88
|
+
For example, sidekiq-grouping accumulates a batch of jobs then enqueues it and accumulates the next batch.
|
89
|
+
With this approach, a queue can contain two batches with data of the same order.
|
87
90
|
These batches are parallel processed with different threads, so we come back to the initial problem.
|
88
91
|
|
89
|
-
Lowkiq was designed to avoid any
|
92
|
+
Lowkiq was designed to avoid any type of locking.
|
90
93
|
|
91
94
|
Furthermore, Lowkiq's queues are reliable. Only Sidekiq Pro or plugins can add such functionality.
|
92
95
|
|
93
|
-
This [benchmark](examples/benchmark) shows overhead on
|
94
|
-
|
96
|
+
This [benchmark](examples/benchmark) shows overhead on Redis usage.
|
97
|
+
These are the results for 5 threads, 100,000 blank jobs:
|
95
98
|
|
96
|
-
+ lowkiq:
|
97
|
-
+
|
99
|
+
+ lowkiq: 155 sec or 1.55 ms per job
|
100
|
+
+ lowkiq +hiredis: 80 sec or 0.80 ms per job
|
101
|
+
+ sidekiq: 15 sec or 0.15 ms per job
|
98
102
|
|
99
103
|
This difference is related to different queues structure.
|
100
104
|
Sidekiq uses one list for all workers and fetches the job entirely for O(1).
|
101
|
-
Lowkiq uses several data structures, including sorted sets for
|
105
|
+
Lowkiq uses several data structures, including sorted sets for keeping ids of jobs.
|
102
106
|
So fetching only an id of a job takes O(log(N)).
|
103
107
|
|
104
108
|
## Queue
|
105
109
|
|
106
110
|
Please, look at [the presentation](https://docs.google.com/presentation/d/e/2PACX-1vRdwA2Ck22r26KV1DbY__XcYpj2FdlnR-2G05w1YULErnJLB_JL1itYbBC6_JbLSPOHwJ0nwvnIHH2A/pub?start=false&loop=false&delayms=3000).
|
107
111
|
|
108
|
-
Every job has following attributes:
|
112
|
+
Every job has the following attributes:
|
109
113
|
|
110
114
|
+ `id` is a job identifier with string type.
|
111
|
-
+ `payloads` is a sorted set of payloads ordered by
|
112
|
-
+ `perform_in` is planned execution time. It's
|
115
|
+
+ `payloads` is a sorted set of payloads ordered by its score. A payload is an object. A score is a real number.
|
116
|
+
+ `perform_in` is planned execution time. It's a Unix timestamp with a real number type.
|
113
117
|
+ `retry_count` is amount of retries. It's a real number.
|
114
118
|
|
115
|
-
For example, `id` can be an identifier of replicated entity.
|
116
|
-
`payloads` is a sorted set ordered by score of payload and resulted by grouping a payload of job by
|
117
|
-
`payload` can be a ruby object
|
118
|
-
`score` can be `payload`'s creation date (
|
119
|
-
By default `score` and `perform_in` are current
|
119
|
+
For example, `id` can be an identifier of a replicated entity.
|
120
|
+
`payloads` is a sorted set ordered by a score of payload and resulted by grouping a payload of the job by its `id`.
|
121
|
+
`payload` can be a ruby object because it is serialized by `Marshal.dump`.
|
122
|
+
`score` can be `payload`'s creation date (Unix timestamp) or it's an incremental version number.
|
123
|
+
By default, `score` and `perform_in` are current Unix timestamp.
|
120
124
|
`retry_count` for new unprocessed job equals to `-1`,
|
121
125
|
for one-time failed is `0`, so the planned retries are counted, not the performed ones.
|
122
126
|
|
123
|
-
|
127
|
+
Job execution can be unsuccessful. In this case, its `retry_count` is incremented, the new `perform_in` is calculated with determined formula, and it moves back to a queue.
|
124
128
|
|
125
129
|
In case of `retry_count` is getting `>=` `max_retry_count` an element of `payloads` with less (oldest) score is moved to a morgue,
|
126
130
|
rest elements are moved back to the queue, wherein `retry_count` and `perform_in` are reset to `-1` and `now()` respectively.
|
@@ -144,14 +148,14 @@ If `max_retry_count = 1`, retries stop.
|
|
144
148
|
|
145
149
|
They are applied when:
|
146
150
|
|
147
|
-
+ a job
|
148
|
-
+ a job
|
149
|
-
+ a job from morgue
|
151
|
+
+ a job has been in a queue and a new one with the same id is added
|
152
|
+
+ a job is failed, but a new one with the same id has been added
|
153
|
+
+ a job from a morgue is moved back to a queue, but the queue has had a job with the same id
|
150
154
|
|
151
155
|
Algorithm:
|
152
156
|
|
153
|
-
+ payloads
|
154
|
-
+ if a new job and queued job is merged, `perform_in` and `retry_count` is taken from the
|
157
|
+
+ payloads are merged, the minimal score is chosen for equal payloads
|
158
|
+
+ if a new job and queued job is merged, `perform_in` and `retry_count` is taken from the job from the queue
|
155
159
|
+ if a failed job and queued job is merged, `perform_in` and `retry_count` is taken from the failed one
|
156
160
|
+ if morgue job and queued job is merged, `perform_in = now()`, `retry_count = -1`
|
157
161
|
|
@@ -170,8 +174,8 @@ Example:
|
|
170
174
|
{ id: "1", payloads: #{"v1": 1, "v2": 3, "v3": 4}, retry_count: 0, perform_in: 1536323288 }
|
171
175
|
```
|
172
176
|
|
173
|
-
|
174
|
-
A job in morgue has following attributes:
|
177
|
+
A morgue is a part of a queue. Jobs in a morgue are not processed.
|
178
|
+
A job in a morgue has the following attributes:
|
175
179
|
|
176
180
|
+ id is the job identifier
|
177
181
|
+ payloads
|
@@ -214,6 +218,12 @@ module ATestWorker
|
|
214
218
|
end
|
215
219
|
```
|
216
220
|
|
221
|
+
And then you have to add it to Lowkiq in your initializer file due to problems with autoloading:
|
222
|
+
|
223
|
+
```ruby
|
224
|
+
Lowkiq.workers = [ ATestWorker ]
|
225
|
+
```
|
226
|
+
|
217
227
|
Default values:
|
218
228
|
|
219
229
|
```ruby
|
@@ -232,10 +242,10 @@ end
|
|
232
242
|
ATestWorker.perform_async [
|
233
243
|
{ id: 0 },
|
234
244
|
{ id: 1, payload: { attr: 'v1' } },
|
235
|
-
{ id: 2, payload: { attr: 'v1' }, score: Time.now.
|
245
|
+
{ id: 2, payload: { attr: 'v1' }, score: Time.now.to_f, perform_in: Time.now.to_f },
|
236
246
|
]
|
237
247
|
# payload by default equals to ""
|
238
|
-
# score and perform_in by default equals to Time.now.
|
248
|
+
# score and perform_in by default equals to Time.now.to_f
|
239
249
|
```
|
240
250
|
|
241
251
|
It is possible to redefine `perform_async` and calculate `id`, `score` и `perform_in` in a worker code:
|
@@ -268,10 +278,11 @@ ATestWorker.perform_async 1000.times.map { |id| { payload: {id: id} } }
|
|
268
278
|
|
269
279
|
## Configuration
|
270
280
|
|
271
|
-
|
281
|
+
Options and their default values are:
|
272
282
|
|
283
|
+
+ `Lowkiq.workers = []`- list of workers to use. Since 1.1.0.
|
273
284
|
+ `Lowkiq.poll_interval = 1` - delay in seconds between queue polling for new jobs.
|
274
|
-
Used only if
|
285
|
+
Used only if a queue was empty in a previous cycle or an error occurred.
|
275
286
|
+ `Lowkiq.threads_per_node = 5` - threads per node.
|
276
287
|
+ `Lowkiq.redis = ->() { Redis.new url: ENV.fetch('REDIS_URL') }` - redis connection options
|
277
288
|
+ `Lowkiq.client_pool_size = 5` - redis pool size for queueing jobs
|
@@ -281,6 +292,12 @@ Default options and values are:
|
|
281
292
|
+ `Lowkiq.build_scheduler = ->() { Lowkiq.build_lag_scheduler }` is a scheduler
|
282
293
|
+ `Lowkiq.build_splitter = ->() { Lowkiq.build_default_splitter }` is a splitter
|
283
294
|
+ `Lowkiq.last_words = ->(ex) {}` is an exception handler of descendants of `StandardError` caused the process stop
|
295
|
+
+ `Lowkiq.dump_payload = Marshal.method :dump`
|
296
|
+
+ `Lowkiq.load_payload = Marshal.method :load`
|
297
|
+
|
298
|
+
+ `Lowkiq.format_error = -> (error) { error.message }` can be used to add error backtrace. Please see [Extended error info](#extended-error-info)
|
299
|
+
+ `Lowkiq.dump_error = -> (msg) { msg }` can be used to implement a custom compression logic for errors. Recommended when using `Lowkiq.format_error`.
|
300
|
+
+ `Lowkiq.load_error = -> (msg) { msg }` can be used to implement a custom decompression logic for errors.
|
284
301
|
|
285
302
|
```ruby
|
286
303
|
$logger = Logger.new(STDOUT)
|
@@ -301,13 +318,29 @@ Lowkiq.server_middlewares << -> (worker, batch, &block) do
|
|
301
318
|
end
|
302
319
|
```
|
303
320
|
|
321
|
+
## Performance
|
322
|
+
|
323
|
+
Use [hiredis](https://github.com/redis/hiredis-rb) for better performance.
|
324
|
+
|
325
|
+
```ruby
|
326
|
+
# Gemfile
|
327
|
+
|
328
|
+
gem "hiredis"
|
329
|
+
```
|
330
|
+
|
331
|
+
```ruby
|
332
|
+
# config
|
333
|
+
|
334
|
+
Lowkiq.redis = ->() { Redis.new url: ENV.fetch('REDIS_URL'), driver: :hiredis }
|
335
|
+
```
|
336
|
+
|
304
337
|
## Execution
|
305
338
|
|
306
339
|
`lowkiq -r ./path_to_app`
|
307
340
|
|
308
341
|
`path_to_app.rb` must load app. [Example](examples/dummy/lib/app.rb).
|
309
342
|
|
310
|
-
|
343
|
+
The lazy loading of worker modules is unacceptable.
|
311
344
|
For preliminarily loading modules use
|
312
345
|
`require`
|
313
346
|
or [`require_dependency`](https://api.rubyonrails.org/classes/ActiveSupport/Dependencies/Loadable.html#method-i-require_dependency)
|
@@ -315,15 +348,15 @@ for Ruby on Rails.
|
|
315
348
|
|
316
349
|
## Shutdown
|
317
350
|
|
318
|
-
Send TERM or INT signal to process (Ctrl-C).
|
319
|
-
|
351
|
+
Send TERM or INT signal to the process (Ctrl-C).
|
352
|
+
The process will wait for executed jobs to finish.
|
320
353
|
|
321
|
-
Note that if queue is empty, process sleeps `poll_interval` seconds,
|
354
|
+
Note that if a queue is empty, the process sleeps `poll_interval` seconds,
|
322
355
|
therefore, the process will not stop until the `poll_interval` seconds have passed.
|
323
356
|
|
324
357
|
## Debug
|
325
358
|
|
326
|
-
To get trace of all threads of app:
|
359
|
+
To get trace of all threads of an app:
|
327
360
|
|
328
361
|
```
|
329
362
|
kill -TTIN <pid>
|
@@ -337,11 +370,22 @@ docker-compose run --rm --service-port app bash
|
|
337
370
|
bundle
|
338
371
|
rspec
|
339
372
|
cd examples/dummy ; bundle exec ../../exe/lowkiq -r ./lib/app.rb
|
373
|
+
|
374
|
+
# open localhost:8080
|
375
|
+
```
|
376
|
+
|
377
|
+
```
|
378
|
+
docker-compose run --rm --service-port frontend bash
|
379
|
+
npm run dumb
|
380
|
+
# open localhost:8081
|
381
|
+
|
382
|
+
# npm run build
|
383
|
+
# npm run web-api
|
340
384
|
```
|
341
385
|
|
342
386
|
## Exceptions
|
343
387
|
|
344
|
-
`StandardError` thrown by worker are handled with middleware. Such exceptions
|
388
|
+
`StandardError` thrown by a worker are handled with middleware. Such exceptions don't lead to process stops.
|
345
389
|
|
346
390
|
All other exceptions cause the process to stop.
|
347
391
|
Lowkiq will wait for job execution by other threads.
|
@@ -364,15 +408,18 @@ end
|
|
364
408
|
```ruby
|
365
409
|
# config/initializers/lowkiq.rb
|
366
410
|
|
367
|
-
# loading all lowkiq workers
|
368
|
-
Dir["#{Rails.root}/app/lowkiq_workers/**/*.rb"].each { |file| require_dependency file }
|
369
|
-
|
370
411
|
# configuration:
|
371
412
|
# Lowkiq.redis = -> { Redis.new url: ENV.fetch('LOWKIQ_REDIS_URL') }
|
372
413
|
# Lowkiq.threads_per_node = ENV.fetch('LOWKIQ_THREADS_PER_NODE').to_i
|
373
414
|
# Lowkiq.client_pool_size = ENV.fetch('LOWKIQ_CLIENT_POOL_SIZE').to_i
|
374
415
|
# ...
|
375
416
|
|
417
|
+
# since 1.1.0
|
418
|
+
Lowkiq.workers = [
|
419
|
+
ATestWorker,
|
420
|
+
OtherCoolWorker
|
421
|
+
]
|
422
|
+
|
376
423
|
Lowkiq.server_middlewares << -> (worker, batch, &block) do
|
377
424
|
logger = Rails.logger
|
378
425
|
tag = "#{worker}-#{Thread.current.object_id}"
|
@@ -460,10 +507,10 @@ worker C: 0
|
|
460
507
|
worker D: 0, 1
|
461
508
|
```
|
462
509
|
|
463
|
-
Lowkiq uses fixed
|
510
|
+
Lowkiq uses a fixed number of threads for job processing, therefore it is necessary to distribute shards between threads.
|
464
511
|
Splitter does it.
|
465
512
|
|
466
|
-
To define a set of shards, which is being processed by thread,
|
513
|
+
To define a set of shards, which is being processed by a thread, let's move them to one list:
|
467
514
|
|
468
515
|
```
|
469
516
|
A0, A1, A2, B0, B1, B2, B3, C0, D0, D1
|
@@ -480,7 +527,7 @@ t1: A1, B1, C0
|
|
480
527
|
t2: A2, B2, D0
|
481
528
|
```
|
482
529
|
|
483
|
-
Besides Default Lowkiq has ByNode splitter. It allows
|
530
|
+
Besides Default Lowkiq has the ByNode splitter. It allows dividing the load by several processes (nodes).
|
484
531
|
|
485
532
|
```
|
486
533
|
Lowkiq.build_splitter = -> () do
|
@@ -491,7 +538,7 @@ Lowkiq.build_splitter = -> () do
|
|
491
538
|
end
|
492
539
|
```
|
493
540
|
|
494
|
-
So, instead of single process you need to execute multiple ones and to set environment variables up:
|
541
|
+
So, instead of a single process, you need to execute multiple ones and to set environment variables up:
|
495
542
|
|
496
543
|
```
|
497
544
|
# process 0
|
@@ -503,18 +550,18 @@ LOWKIQ_NUMBER_OF_NODES=2 LOWKIQ_NODE_NUMBER=1 bundle exec lowkiq -r ./lib/app.rb
|
|
503
550
|
|
504
551
|
Summary amount of threads are equal product of `ENV.fetch('LOWKIQ_NUMBER_OF_NODES')` and `Lowkiq.threads_per_node`.
|
505
552
|
|
506
|
-
You can also write your own splitter if your app needs extra distribution of shards between threads or nodes.
|
553
|
+
You can also write your own splitter if your app needs an extra distribution of shards between threads or nodes.
|
507
554
|
|
508
555
|
## Scheduler
|
509
556
|
|
510
|
-
Every thread processes a set of shards.
|
511
|
-
Every thread has
|
557
|
+
Every thread processes a set of shards. The scheduler selects shard for processing.
|
558
|
+
Every thread has its own instance of the scheduler.
|
512
559
|
|
513
560
|
Lowkiq has 2 schedulers for your choice.
|
514
|
-
`Seq`
|
561
|
+
`Seq` sequentially looks over shards.
|
515
562
|
`Lag` chooses shard with the oldest job minimizing the lag. It's used by default.
|
516
563
|
|
517
|
-
|
564
|
+
The scheduler can be set up through settings:
|
518
565
|
|
519
566
|
```
|
520
567
|
Lowkiq.build_scheduler = ->() { Lowkiq.build_seq_scheduler }
|
@@ -527,13 +574,13 @@ Lowkiq.build_scheduler = ->() { Lowkiq.build_lag_scheduler }
|
|
527
574
|
### `SomeWorker.shards_count`
|
528
575
|
|
529
576
|
Sum of `shards_count` of all workers shouldn't be less than `Lowkiq.threads_per_node`
|
530
|
-
otherwise threads will stay idle.
|
577
|
+
otherwise, threads will stay idle.
|
531
578
|
|
532
579
|
Sum of `shards_count` of all workers can be equal to `Lowkiq.threads_per_node`.
|
533
|
-
In this case thread processes a single shard. This makes sense only with uniform queue load.
|
580
|
+
In this case, a thread processes a single shard. This makes sense only with a uniform queue load.
|
534
581
|
|
535
582
|
Sum of `shards_count` of all workers can be more than `Lowkiq.threads_per_node`.
|
536
|
-
In this case `shards_count` can be counted as a priority.
|
583
|
+
In this case, `shards_count` can be counted as a priority.
|
537
584
|
The larger it is, the more often the tasks of this queue will be processed.
|
538
585
|
|
539
586
|
There is no reason to set `shards_count` of one worker more than `Lowkiq.threads_per_node`,
|
@@ -541,8 +588,8 @@ because every thread will handle more than one shard from this queue, so it incr
|
|
541
588
|
|
542
589
|
### `SomeWorker.max_retry_count`
|
543
590
|
|
544
|
-
From `retry_in` and `max_retry_count`, you can calculate approximate time that payload of job will be in a queue.
|
545
|
-
After `max_retry_count` is reached
|
591
|
+
From `retry_in` and `max_retry_count`, you can calculate the approximate time that a payload of a job will be in a queue.
|
592
|
+
After `max_retry_count` is reached a payload with a minimal score will be moved to a morgue.
|
546
593
|
|
547
594
|
For default `retry_in` we receive the following table.
|
548
595
|
|
@@ -570,9 +617,9 @@ end
|
|
570
617
|
|
571
618
|
## Changing of worker's shards amount
|
572
619
|
|
573
|
-
Try to count
|
620
|
+
Try to count the number of shards right away and don't change it in the future.
|
574
621
|
|
575
|
-
If you can disable adding of new jobs, wait for queues to get empty and deploy the new version of code with changed amount of shards.
|
622
|
+
If you can disable adding of new jobs, wait for queues to get empty, and deploy the new version of code with a changed amount of shards.
|
576
623
|
|
577
624
|
If you can't do it, follow the next steps:
|
578
625
|
|
@@ -590,7 +637,7 @@ module ATestWorker
|
|
590
637
|
end
|
591
638
|
```
|
592
639
|
|
593
|
-
Set the number of shards and new queue name:
|
640
|
+
Set the number of shards and the new queue name:
|
594
641
|
|
595
642
|
```ruby
|
596
643
|
module ATestWorker
|
@@ -605,7 +652,7 @@ module ATestWorker
|
|
605
652
|
end
|
606
653
|
```
|
607
654
|
|
608
|
-
Add a worker moving jobs from the old queue to
|
655
|
+
Add a worker moving jobs from the old queue to the new one:
|
609
656
|
|
610
657
|
```ruby
|
611
658
|
module ATestMigrationWorker
|
@@ -625,3 +672,23 @@ module ATestMigrationWorker
|
|
625
672
|
end
|
626
673
|
end
|
627
674
|
```
|
675
|
+
|
676
|
+
## Extended error info
|
677
|
+
For failed jobs, lowkiq only stores `error.message` by default. This can be configured by using `Lowkiq.format_error` setting.
|
678
|
+
`Lowkiq.dump` and `Lowkiq.load_error` can be used to compress and decompress the error messages respectively.
|
679
|
+
Example:
|
680
|
+
```ruby
|
681
|
+
Lowkiq.format_error = -> (error) { error.full_message(highlight: false) }
|
682
|
+
|
683
|
+
Lowkiq.dump_error = Proc.new do |msg|
|
684
|
+
compressed = Zlib::Deflate.deflate(msg.to_s)
|
685
|
+
Base64.encode64(compressed)
|
686
|
+
end
|
687
|
+
|
688
|
+
Lowkiq.load_error = Proc.new do |input|
|
689
|
+
decoded = Base64.decode64(input)
|
690
|
+
Zlib::Inflate.inflate(decoded)
|
691
|
+
rescue
|
692
|
+
input
|
693
|
+
end
|
694
|
+
```
|
data/README.ru.md
CHANGED
@@ -247,10 +247,10 @@ end
|
|
247
247
|
ATestWorker.perform_async [
|
248
248
|
{ id: 0 },
|
249
249
|
{ id: 1, payload: { attr: 'v1' } },
|
250
|
-
{ id: 2, payload: { attr: 'v1' }, score: Time.now.
|
250
|
+
{ id: 2, payload: { attr: 'v1' }, score: Time.now.to_f, perform_in: Time.now.to_f },
|
251
251
|
]
|
252
252
|
# payload по умолчанию равен ""
|
253
|
-
# score и perform_in по умолчанию равны Time.now.
|
253
|
+
# score и perform_in по умолчанию равны Time.now.to_f
|
254
254
|
```
|
255
255
|
|
256
256
|
Вы можете переопределить `perform_async` и вычислять `id`, `score` и `perform_in` в воркере:
|