lowkiq 1.0.0 → 1.0.5
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Gemfile.lock +19 -19
- data/LICENSE.md +13 -3
- data/README.md +378 -308
- data/README.ru.md +645 -0
- data/docker-compose.yml +1 -1
- data/lib/lowkiq.rb +6 -2
- data/lib/lowkiq/extend_tracker.rb +1 -1
- data/lib/lowkiq/queue/fetch.rb +2 -2
- data/lib/lowkiq/queue/keys.rb +16 -4
- data/lib/lowkiq/queue/queue.rb +103 -55
- data/lib/lowkiq/script.rb +42 -0
- data/lib/lowkiq/server.rb +4 -0
- data/lib/lowkiq/shard_handler.rb +3 -3
- data/lib/lowkiq/version.rb +1 -1
- data/lowkiq.gemspec +3 -2
- metadata +12 -10
- data/lib/lowkiq/queue/marshal.rb +0 -23
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d0eae4c60bd2784dca216cc7f9cdc74a4cb115c337d943ef0d4abcf27f88704e
|
4
|
+
data.tar.gz: 8342bc74346bb6e1403d4a85d95a6242a87a1bd2c0029eb98e5368ef72010c15
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 74024d53c26c6cc27637138918e56e2ffc2badeae88da4d76ed510effc075a1ba6417c91896cbacc22bb22962b9aaef44fc5b0a482acc545e7e4497aa5b18cf0
|
7
|
+
data.tar.gz: 3db7be61db43c9b578b025d9aefd129bd9d2f5fc5bf4bcf904b9c424a2cf0d191aae0d1c97148abed28db1f7ffa78e840d9777a8fbd4cae6cf42d3601e270c8c
|
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
lowkiq (1.0.
|
4
|
+
lowkiq (1.0.4)
|
5
5
|
connection_pool (~> 2.2, >= 2.2.2)
|
6
6
|
rack (>= 1.5.0)
|
7
7
|
redis (>= 4.0.1, < 5)
|
@@ -9,37 +9,37 @@ PATH
|
|
9
9
|
GEM
|
10
10
|
remote: https://rubygems.org/
|
11
11
|
specs:
|
12
|
-
connection_pool (2.2.
|
12
|
+
connection_pool (2.2.3)
|
13
13
|
diff-lcs (1.3)
|
14
|
-
rack (2.
|
14
|
+
rack (2.2.2)
|
15
15
|
rack-test (1.1.0)
|
16
16
|
rack (>= 1.0, < 3)
|
17
|
-
rake (
|
18
|
-
redis (4.1
|
19
|
-
rspec (3.
|
20
|
-
rspec-core (~> 3.
|
21
|
-
rspec-expectations (~> 3.
|
22
|
-
rspec-mocks (~> 3.
|
23
|
-
rspec-core (3.
|
24
|
-
rspec-support (~> 3.
|
25
|
-
rspec-expectations (3.
|
17
|
+
rake (12.3.3)
|
18
|
+
redis (4.2.1)
|
19
|
+
rspec (3.9.0)
|
20
|
+
rspec-core (~> 3.9.0)
|
21
|
+
rspec-expectations (~> 3.9.0)
|
22
|
+
rspec-mocks (~> 3.9.0)
|
23
|
+
rspec-core (3.9.1)
|
24
|
+
rspec-support (~> 3.9.1)
|
25
|
+
rspec-expectations (3.9.0)
|
26
26
|
diff-lcs (>= 1.2.0, < 2.0)
|
27
|
-
rspec-support (~> 3.
|
28
|
-
rspec-mocks (3.
|
27
|
+
rspec-support (~> 3.9.0)
|
28
|
+
rspec-mocks (3.9.1)
|
29
29
|
diff-lcs (>= 1.2.0, < 2.0)
|
30
|
-
rspec-support (~> 3.
|
31
|
-
rspec-support (3.
|
30
|
+
rspec-support (~> 3.9.0)
|
31
|
+
rspec-support (3.9.2)
|
32
32
|
|
33
33
|
PLATFORMS
|
34
34
|
ruby
|
35
35
|
|
36
36
|
DEPENDENCIES
|
37
|
-
bundler (~> 1.
|
37
|
+
bundler (~> 2.1.0)
|
38
38
|
lowkiq!
|
39
39
|
rack-test (~> 1.1)
|
40
|
-
rake (~>
|
40
|
+
rake (~> 12.3.0)
|
41
41
|
rspec (~> 3.0)
|
42
42
|
rspec-mocks (~> 3.8)
|
43
43
|
|
44
44
|
BUNDLED WITH
|
45
|
-
1.
|
45
|
+
2.1.4
|
data/LICENSE.md
CHANGED
@@ -1,3 +1,13 @@
|
|
1
|
+
This software is dual-licensed under the LGPL version 3 or under the Licence Agreement.
|
2
|
+
Recipients can choose the terms under which they want to use or distribute the
|
3
|
+
software.
|
4
|
+
|
5
|
+
Copyright © BIA-Technologies Limited Liability Company (OOO)
|
6
|
+
|
7
|
+
# The LGPL Version 3 (LGPL-3.0)
|
8
|
+
|
9
|
+
https://www.gnu.org/licenses/lgpl-3.0.html
|
10
|
+
|
1
11
|
# Licence Agreement
|
2
12
|
On granting a non-exclusive right to use open source software
|
3
13
|
|
@@ -7,7 +17,7 @@ On granting a non-exclusive right to use open source software
|
|
7
17
|
|
8
18
|
1.1. The Licensor provides the Licensee, in the manner and on the terms set forth in this Agreement, the right to use (license) **the Lowkiq open source software** (hereinafter - the "Software").
|
9
19
|
|
10
|
-
1.2. The source code for the software is available on the website located in the Internet telecommunication network "Internet" at the address: https://github.com/bia-
|
20
|
+
1.2. The source code for the software is available on the website located in the Internet telecommunication network "Internet" at the address: https://github.com/bia-technologies/lowkiq.
|
11
21
|
|
12
22
|
1.3. Software characteristics, that individualize it as a unique result of intellectual activity:
|
13
23
|
|
@@ -129,5 +139,5 @@ TIN/ 7810385714
|
|
129
139
|
RRC/ 781001001
|
130
140
|
|
131
141
|
Name and email address of the representative:<br>
|
132
|
-
|
133
|
-
|
142
|
+
Mikhail Kuzmin<br>
|
143
|
+
Mihail.Kuzmin@bia-tech.ru<br>
|
data/README.md
CHANGED
@@ -1,163 +1,196 @@
|
|
1
|
+
[![Gem Version](https://badge.fury.io/rb/lowkiq.svg)](https://badge.fury.io/rb/lowkiq)
|
2
|
+
|
1
3
|
# Lowkiq
|
2
4
|
|
3
|
-
|
5
|
+
Ordered background jobs processing
|
4
6
|
|
5
7
|
![dashboard](doc/dashboard.png)
|
6
8
|
|
9
|
+
* [Rationale](#rationale)
|
10
|
+
* [Description](#description)
|
11
|
+
* [Sidekiq comparison](#sidekiq-comparison)
|
12
|
+
* [Queue](#queue)
|
13
|
+
+ [Calculation algorithm for `retry_count` and `perform_in`](#calculation-algorithm-for-retry_count-and-perform_in)
|
14
|
+
+ [Job merging rules](#job-merging-rules)
|
15
|
+
* [Install](#install)
|
16
|
+
* [Api](#api)
|
17
|
+
* [Ring app](#ring-app)
|
18
|
+
* [Configuration](#configuration)
|
19
|
+
* [Performance](#performance)
|
20
|
+
* [Execution](#execution)
|
21
|
+
* [Shutdown](#shutdown)
|
22
|
+
* [Debug](#debug)
|
23
|
+
* [Development](#development)
|
24
|
+
* [Exceptions](#exceptions)
|
25
|
+
* [Rails integration](#rails-integration)
|
26
|
+
* [Splitter](#splitter)
|
27
|
+
* [Scheduler](#scheduler)
|
28
|
+
* [Recommendations on configuration](#recommendations-on-configuration)
|
29
|
+
+ [`SomeWorker.shards_count`](#someworkershards_count)
|
30
|
+
+ [`SomeWorker.max_retry_count`](#someworkermax_retry_count)
|
31
|
+
|
7
32
|
## Rationale
|
8
33
|
|
9
|
-
|
34
|
+
We've faced some problems using Sidekiq while processing messages from a side system.
|
35
|
+
For instance, the message is a data of an order in particular time.
|
36
|
+
The side system will send a new data of an order on an every change.
|
37
|
+
Orders are frequently updated and a queue containts some closely located messages of the same order.
|
10
38
|
|
11
|
-
Sidekiq
|
12
|
-
|
13
|
-
Sidekiq
|
14
|
-
|
39
|
+
Sidekiq doesn't guarantee a strict message order, because a queue is processed by multiple threads.
|
40
|
+
For example, we've received 2 messages: M1 and M2.
|
41
|
+
Sidekiq handlers begin to process them parallel,
|
42
|
+
so M2 can be processed before M1.
|
15
43
|
|
16
|
-
|
17
|
-
Параллельная обработка таких сообщений приводит к:
|
44
|
+
Parallel processing of such kind of messages can result in:
|
18
45
|
|
19
46
|
+ dead locks
|
20
|
-
+
|
47
|
+
+ overwriting new data with old one
|
21
48
|
|
22
|
-
Lowkiq
|
49
|
+
Lowkiq has been created to eliminate such problems by avoiding parallel task processing within one entity.
|
23
50
|
|
24
51
|
## Description
|
25
52
|
|
26
|
-
|
27
|
-
|
53
|
+
Lowkiq's queues are reliable i.e.,
|
54
|
+
Lowkiq saves information about a job being processed
|
55
|
+
and returns incompleted jobs back to the queue on startup.
|
56
|
+
|
57
|
+
Jobs in queues are ordered by preassigned execution time, so they are not FIFO queues.
|
28
58
|
|
29
|
-
|
30
|
-
когда несколько потоков обрабатывают задачи с одинаковыми идентификаторами.
|
59
|
+
Every job has it's own identifier. Lowkiq guarantees that jobs with equal id are processed by the same thread.
|
31
60
|
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
61
|
+
Every queue is divided into a permanent set of shards.
|
62
|
+
A job is placed into particular shard based on an id of the job.
|
63
|
+
So jobs with the same id are always placed into the same shard.
|
64
|
+
All jobs of the shard are always processed with the same thread.
|
65
|
+
This guarantees the sequently processing of jobs with the same ids and excludes the possibility of locks.
|
37
66
|
|
38
|
-
|
39
|
-
|
40
|
-
|
67
|
+
Besides the id, every job has a payload.
|
68
|
+
Payloads are accumulated for jobs with the same id.
|
69
|
+
So all accumulated payloads will be processed together.
|
70
|
+
It's useful when you need to process only the last message and drop all previous ones.
|
41
71
|
|
42
|
-
|
43
|
-
Если задачи содержат снимки (версии) сущности, то обработчик может использовать только последнюю версию.
|
72
|
+
A worker corresponds to a queue and contains a job processing logic.
|
44
73
|
|
45
|
-
|
46
|
-
|
47
|
-
таким образом, добавление или удаление очереди/воркера не приводит к изменению числа тредов.
|
48
|
-
Нет смысла задавать кол-во шардов одного воркера больше, чем общее кол-во тредов.
|
74
|
+
Fixed amount of threads is used to process all job of all queues.
|
75
|
+
Adding or removing queues or it's shards won't affect the amount of threads.
|
49
76
|
|
50
|
-
##
|
77
|
+
## Sidekiq comparison
|
51
78
|
|
52
|
-
|
79
|
+
If Sidekiq is good for your tasks you should use it.
|
80
|
+
But if you use plugins like
|
81
|
+
[sidekiq-grouping](https://github.com/gzigzigzeo/sidekiq-grouping),
|
82
|
+
[sidekiq-unique-jobs](https://github.com/mhenrixon/sidekiq-unique-jobs),
|
83
|
+
[sidekiq-merger](https://github.com/dtaniwaki/sidekiq-merger)
|
84
|
+
or implement your own lock system, you should look at Lowkiq.
|
53
85
|
|
54
|
-
|
55
|
-
|
56
|
-
|
86
|
+
For example, sidekiq-grouping accumulates a batch of jobs than enqueues it and accumulates a next batch.
|
87
|
+
With this approach queue can contains two batches with a data of the same order.
|
88
|
+
These batches are parallel processed with different threads, so we come back to the initial problem.
|
57
89
|
|
58
|
-
|
90
|
+
Lowkiq was designed to avoid any types of locking.
|
59
91
|
|
60
|
-
|
92
|
+
Furthermore, Lowkiq's queues are reliable. Only Sidekiq Pro or plugins can add such functionality.
|
61
93
|
|
62
|
-
|
63
|
-
|
94
|
+
This [benchmark](examples/benchmark) shows overhead on redis usage.
|
95
|
+
This is the results for 5 threads, 100,000 blank jobs:
|
64
96
|
|
65
|
-
|
66
|
-
|
97
|
+
+ lowkiq: 155 sec or 1.55 ms per job
|
98
|
+
+ lowkiq +hiredis: 80 sec or 0.80 ms per job
|
99
|
+
+ sidekiq: 15 sec or 0.15 ms per job
|
67
100
|
|
68
|
-
|
101
|
+
This difference is related to different queues structure.
|
102
|
+
Sidekiq uses one list for all workers and fetches the job entirely for O(1).
|
103
|
+
Lowkiq uses several data structures, including sorted sets for storing ids of jobs.
|
104
|
+
So fetching only an id of a job takes O(log(N)).
|
69
105
|
|
70
|
-
|
106
|
+
## Queue
|
71
107
|
|
72
|
-
|
73
|
-
+ `payloads` - сортированное множество payload'ов (объекты) по их score (вещественное число)
|
74
|
-
+ `perform_in` - запланированное время начала иполнения задачи (unix timestamp, вещественное число)
|
75
|
-
+ `retry_count` - количество совершённых повторов задачи (вещественное число)
|
108
|
+
Please, look at [the presentation](https://docs.google.com/presentation/d/e/2PACX-1vRdwA2Ck22r26KV1DbY__XcYpj2FdlnR-2G05w1YULErnJLB_JL1itYbBC6_JbLSPOHwJ0nwvnIHH2A/pub?start=false&loop=false&delayms=3000).
|
76
109
|
|
77
|
-
|
78
|
-
`payloads` - множество,
|
79
|
-
получаемое в результате группировки полезной нагрузки задачи по `id` и отсортированное по ее `score`.
|
80
|
-
`payload` может быть объектом, т.к. сериализуется с помощью `Marshal.dump`.
|
81
|
-
`score` может быть датой (unix timestamp) создания `payload`
|
82
|
-
или ее монотонно увеличивающимся номером версии.
|
83
|
-
По умолчанию - текущий unix timestamp.
|
84
|
-
По умолчанию `perform_in` - текущий unix timestamp.
|
85
|
-
`retry_count` для новой необработанной задачи равен `-1`, для упавшей один раз - `0`,
|
86
|
-
т.е. считаются не совершённые, а запланированные повторы.
|
110
|
+
Every job has following attributes:
|
87
111
|
|
88
|
-
|
112
|
+
+ `id` is a job identifier with string type.
|
113
|
+
+ `payloads` is a sorted set of payloads ordered by it's score. Payload is an object. Score is a real number.
|
114
|
+
+ `perform_in` is planned execution time. It's unix timestamp with real number type.
|
115
|
+
+ `retry_count` is amount of retries. It's a real number.
|
89
116
|
|
90
|
-
|
117
|
+
For example, `id` can be an identifier of replicated entity.
|
118
|
+
`payloads` is a sorted set ordered by score of payload and resulted by grouping a payload of job by it's `id`.
|
119
|
+
`payload` can be a ruby object, because it is serialized by `Marshal.dump`.
|
120
|
+
`score` can be `payload`'s creation date (unix timestamp) or it's incremental version number.
|
121
|
+
By default `score` and `perform_in` are current unix timestamp.
|
122
|
+
`retry_count` for new unprocessed job equals to `-1`,
|
123
|
+
for one-time failed is `0`, so the planned retries are counted, not the performed ones.
|
91
124
|
|
92
|
-
|
93
|
-
В этом случае ее `retry_count` инкрементируется и по заданной формуле вычисляется новый `perform_at`,
|
94
|
-
и она ставится обратно в очередь.
|
125
|
+
A job execution can be unsuccessful. In this case, its `retry_count` is incremented, new `perform_in` is calculated with determined formula and it moves back to a queue.
|
95
126
|
|
96
|
-
|
97
|
-
|
98
|
-
а оставшиеся элементы помещаются обратно в очередь, при этом
|
99
|
-
`retry_count` и `perform_at` сбрасываются в `-1` и `now()` соответственно.
|
127
|
+
In case of `retry_count` is getting `>=` `max_retry_count` an element of `payloads` with less (oldest) score is moved to a morgue,
|
128
|
+
rest elements are moved back to the queue, wherein `retry_count` and `perform_in` are reset to `-1` and `now()` respectively.
|
100
129
|
|
101
|
-
###
|
130
|
+
### Calculation algorithm for `retry_count` and `perform_in`
|
102
131
|
|
103
|
-
0.
|
132
|
+
0. a job's been executed and failed
|
104
133
|
1. `retry_count++`
|
105
|
-
2. `perform_in = now + retry_in(try_count)`
|
106
|
-
3. `if retry_count >= max_retry_count`
|
134
|
+
2. `perform_in = now + retry_in (try_count)`
|
135
|
+
3. `if retry_count >= max_retry_count` the job will be moved to a morgue.
|
107
136
|
|
108
|
-
|
|
109
|
-
| ---
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
137
|
+
| type | `retry_count` | `perform_in` |
|
138
|
+
| --- | --- | --- |
|
139
|
+
| new haven't been executed | -1 | set or `now()` |
|
140
|
+
| new failed | 0 | `now() + retry_in(0)` |
|
141
|
+
| retry failed | 1 | `now() + retry_in(1)` |
|
113
142
|
|
114
|
-
|
143
|
+
If `max_retry_count = 1`, retries stop.
|
115
144
|
|
116
|
-
###
|
145
|
+
### Job merging rules
|
117
146
|
|
118
|
-
|
147
|
+
They are applied when:
|
119
148
|
|
120
|
-
+
|
121
|
-
+
|
122
|
-
+
|
149
|
+
+ a job had been in a queue and a new one with the same id was added
|
150
|
+
+ a job was failed, but a new one with the same id had been added
|
151
|
+
+ a job from morgue was moved back to queue, but queue had had a job with the same id
|
123
152
|
|
124
|
-
|
153
|
+
Algorithm:
|
125
154
|
|
126
|
-
+ payloads
|
127
|
-
|
128
|
-
+
|
129
|
-
|
130
|
-
+ если объединяется упавшая задача и задача из очереди,
|
131
|
-
то `perform_at` и `retry_count` берутся из упавшей
|
132
|
-
+ если объединяется задача из морга и задача из очереди,
|
133
|
-
то `perform_at = now()`, `retry_count = -1`
|
155
|
+
+ payloads is merged, minimal score is chosen for equal payloads
|
156
|
+
+ if a new job and queued job is merged, `perform_in` and `retry_count` is taken from the the job from the queue
|
157
|
+
+ if a failed job and queued job is merged, `perform_in` and `retry_count` is taken from the failed one
|
158
|
+
+ if morgue job and queued job is merged, `perform_in = now()`, `retry_count = -1`
|
134
159
|
|
135
|
-
|
160
|
+
Example:
|
136
161
|
|
137
162
|
```
|
138
|
-
# v1
|
139
|
-
# #{"v1": 1}
|
163
|
+
# v1 is the first version and v2 is the second
|
164
|
+
# #{"v1": 1} is a sorted set of a single element, the payload is "v1", the score is 1
|
140
165
|
|
141
|
-
#
|
142
|
-
{ id: "1", payloads: #{"v1": 1, "v2": 2}, retry_count: 0,
|
143
|
-
#
|
144
|
-
{ id: "1", payloads: #{"v2": 3, "v3": 4}, retry_count: -1,
|
166
|
+
# a job is in a queue
|
167
|
+
{ id: "1", payloads: #{"v1": 1, "v2": 2}, retry_count: 0, perform_in: 1536323288 }
|
168
|
+
# a job which is being added
|
169
|
+
{ id: "1", payloads: #{"v2": 3, "v3": 4}, retry_count: -1, perform_in: 1536323290 }
|
145
170
|
|
146
|
-
#
|
147
|
-
{ id: "1", payloads: #{"v1": 1, "v2": 3, "v3": 4}, retry_count: 0,
|
171
|
+
# a resulted job in the queue
|
172
|
+
{ id: "1", payloads: #{"v1": 1, "v2": 3, "v3": 4}, retry_count: 0, perform_in: 1536323288 }
|
148
173
|
```
|
149
174
|
|
150
|
-
|
151
|
-
|
175
|
+
Morgue is a part of the queue. Jobs in morgue are not processed.
|
176
|
+
A job in morgue has following attributes:
|
152
177
|
|
153
|
-
+ id
|
154
|
-
+ payloads
|
178
|
+
+ id is the job identifier
|
179
|
+
+ payloads
|
155
180
|
|
156
|
-
|
181
|
+
A job from morgue can be moved back to the queue, `retry_count` = 0 and `perform_in = now()` would be set.
|
157
182
|
|
158
|
-
|
183
|
+
## Install
|
159
184
|
|
160
|
-
|
185
|
+
```
|
186
|
+
# Gemfile
|
187
|
+
|
188
|
+
gem 'lowkiq'
|
189
|
+
```
|
190
|
+
|
191
|
+
Redis >= 3.2
|
192
|
+
|
193
|
+
## Api
|
161
194
|
|
162
195
|
```ruby
|
163
196
|
module ATestWorker
|
@@ -171,11 +204,10 @@ module ATestWorker
|
|
171
204
|
10 * (count + 1) # (i.e. 10, 20, 30, 40, 50)
|
172
205
|
end
|
173
206
|
|
174
|
-
def self.perform(
|
175
|
-
# payloads_by_id
|
207
|
+
def self.perform(payloads_by_id)
|
208
|
+
# payloads_by_id is a hash map
|
176
209
|
payloads_by_id.each do |id, payloads|
|
177
|
-
#
|
178
|
-
# payloads отсортированы по score, от старых к новым (от минимальных к максимальным)
|
210
|
+
# payloads are sorted by score, from old to new (min to max)
|
179
211
|
payloads.each do |payload|
|
180
212
|
do_some_work(id, payload)
|
181
213
|
end
|
@@ -184,7 +216,7 @@ module ATestWorker
|
|
184
216
|
end
|
185
217
|
```
|
186
218
|
|
187
|
-
|
219
|
+
Default values:
|
188
220
|
|
189
221
|
```ruby
|
190
222
|
self.shards_count = 5
|
@@ -204,11 +236,11 @@ ATestWorker.perform_async [
|
|
204
236
|
{ id: 1, payload: { attr: 'v1' } },
|
205
237
|
{ id: 2, payload: { attr: 'v1' }, score: Time.now.to_i, perform_in: Time.now.to_i },
|
206
238
|
]
|
207
|
-
# payload
|
208
|
-
# score
|
239
|
+
# payload by default equals to ""
|
240
|
+
# score and perform_in by default equals to Time.now.to_i
|
209
241
|
```
|
210
242
|
|
211
|
-
|
243
|
+
It is possible to redefine `perform_async` and calculate `id`, `score` и `perform_in` in a worker code:
|
212
244
|
|
213
245
|
```ruby
|
214
246
|
module ATestWorker
|
@@ -229,56 +261,30 @@ end
|
|
229
261
|
ATestWorker.perform_async 1000.times.map { |id| { payload: {id: id} } }
|
230
262
|
```
|
231
263
|
|
232
|
-
### Max retry count
|
233
|
-
|
234
|
-
Исходя из `retry_in` и `max_retry_count`,
|
235
|
-
можно вычислить примерное время, которая задача проведет в очереди.
|
236
|
-
|
237
|
-
Для `retry_in`, заданного по умолчанию получается следующая таблица:
|
238
|
-
|
239
|
-
```ruby
|
240
|
-
def retry_in(retry_count)
|
241
|
-
(retry_count ** 4) + 15 + (rand(30) * (retry_count + 1))
|
242
|
-
end
|
243
|
-
```
|
244
|
-
|
245
|
-
| `max_retry_count` | кол-во дней жизни задачи |
|
246
|
-
| --- | --- |
|
247
|
-
| 14 | 1 |
|
248
|
-
| 16 | 2 |
|
249
|
-
| 18 | 3 |
|
250
|
-
| 19 | 5 |
|
251
|
-
| 20 | 6 |
|
252
|
-
| 21 | 8 |
|
253
|
-
| 22 | 10 |
|
254
|
-
| 23 | 13 |
|
255
|
-
| 24 | 16 |
|
256
|
-
| 25 | 20 |
|
257
|
-
|
258
|
-
`(0...25).map{ |c| retry_in c }.sum / 60 / 60 / 24`
|
259
|
-
|
260
264
|
## Ring app
|
261
265
|
|
262
|
-
`Lowkiq::Web` - ring app.
|
266
|
+
`Lowkiq::Web` - a ring app.
|
263
267
|
|
264
|
-
+ `/` - dashboard
|
265
|
-
+ `/api/v1/stats` -
|
268
|
+
+ `/` - a dashboard
|
269
|
+
+ `/api/v1/stats` - queue length, morgue length, lag for each worker and total result
|
266
270
|
|
267
|
-
##
|
271
|
+
## Configuration
|
268
272
|
|
269
|
-
|
273
|
+
Options and their default values are:
|
270
274
|
|
271
|
-
+ `Lowkiq.poll_interval = 1` -
|
272
|
-
|
273
|
-
+ `Lowkiq.threads_per_node = 5` -
|
274
|
-
+ `Lowkiq.redis = ->() { Redis.new url: ENV.fetch('REDIS_URL') }` -
|
275
|
-
+ `Lowkiq.client_pool_size = 5` -
|
276
|
-
+ `Lowkiq.pool_timeout = 5` -
|
277
|
-
+ `Lowkiq.server_middlewares = []` -
|
278
|
-
+ `Lowkiq.on_server_init = ->() {}` -
|
279
|
-
+ `Lowkiq.build_scheduler = ->() { Lowkiq.build_lag_scheduler }`
|
280
|
-
+ `Lowkiq.build_splitter = ->() { Lowkiq.build_default_splitter }`
|
281
|
-
+ `Lowkiq.last_words = ->(ex) {}`
|
275
|
+
+ `Lowkiq.poll_interval = 1` - delay in seconds between queue polling for new jobs.
|
276
|
+
Used only if the queue was empty at previous cycle or error was occured.
|
277
|
+
+ `Lowkiq.threads_per_node = 5` - threads per node.
|
278
|
+
+ `Lowkiq.redis = ->() { Redis.new url: ENV.fetch('REDIS_URL') }` - redis connection options
|
279
|
+
+ `Lowkiq.client_pool_size = 5` - redis pool size for queueing jobs
|
280
|
+
+ `Lowkiq.pool_timeout = 5` - client and server redis pool timeout
|
281
|
+
+ `Lowkiq.server_middlewares = []` - a middleware list, used for worker wrapping
|
282
|
+
+ `Lowkiq.on_server_init = ->() {}` - a lambda is being executed when server inits
|
283
|
+
+ `Lowkiq.build_scheduler = ->() { Lowkiq.build_lag_scheduler }` is a scheduler
|
284
|
+
+ `Lowkiq.build_splitter = ->() { Lowkiq.build_default_splitter }` is a splitter
|
285
|
+
+ `Lowkiq.last_words = ->(ex) {}` is an exception handler of descendants of `StandardError` caused the process stop
|
286
|
+
+ `Lowkiq.dump_payload = Marshal.method :dump`
|
287
|
+
+ `Lowkiq.load_payload = Marshal.method :load`
|
282
288
|
|
283
289
|
```ruby
|
284
290
|
$logger = Logger.new(STDOUT)
|
@@ -299,184 +305,69 @@ Lowkiq.server_middlewares << -> (worker, batch, &block) do
|
|
299
305
|
end
|
300
306
|
```
|
301
307
|
|
302
|
-
##
|
303
|
-
|
304
|
-
У каждого воркера есть несколько шардов:
|
305
|
-
|
306
|
-
```
|
307
|
-
# worker: shard ids
|
308
|
-
worker A: 0, 1, 2
|
309
|
-
worker B: 0, 1, 2, 3
|
310
|
-
worker C: 0
|
311
|
-
worker D: 0, 1
|
312
|
-
```
|
313
|
-
|
314
|
-
Lowkiq использует фиксированное кол-во тредов для обработки задач, следовательно нужно распределить шарды
|
315
|
-
между тредами. Этим занимается Splitter.
|
316
|
-
|
317
|
-
Чтобы определить набор шардов, которые будет обрабатывать тред, поместим их в один список:
|
318
|
-
|
319
|
-
```
|
320
|
-
A0, A1, A2, B0, B1, B2, B3, C0, D0, D1
|
321
|
-
```
|
322
|
-
|
323
|
-
Рассмотрим Default splitter, который равномерно распределяет шарды по тредам единственной ноды.
|
324
|
-
|
325
|
-
Если `threads_per_node` установлено в 3, то распределение будет таким:
|
326
|
-
|
327
|
-
```
|
328
|
-
# thread id: shards
|
329
|
-
t0: A0, B0, B3, D1
|
330
|
-
t1: A1, B1, C0
|
331
|
-
t2: A2, B2, D0
|
332
|
-
```
|
333
|
-
|
334
|
-
Помимо Default есть ByNode splitter. Он позволяет распределить нагрузку по нескольким процессам (нодам).
|
335
|
-
|
336
|
-
|
337
|
-
```
|
338
|
-
Lowkiq.build_splitter = -> () do
|
339
|
-
Lowkiq.build_by_node_splitter(
|
340
|
-
ENV.fetch('LOWKIQ_NUMBER_OF_NODES').to_i,
|
341
|
-
ENV.fetch('LOWKIQ_NODE_NUMBER').to_i
|
342
|
-
)
|
343
|
-
end
|
344
|
-
```
|
345
|
-
|
346
|
-
Таким образом, вместо одного процесса нужно запустить несколько и указать переменные окружения:
|
347
|
-
|
348
|
-
```
|
349
|
-
# process 0
|
350
|
-
LOWKIQ_NUMBER_OF_NODES=2 LOWKIQ_NODE_NUMBER=0 bundle exec lowkiq -r ./lib/app.rb
|
351
|
-
|
352
|
-
# process 1
|
353
|
-
LOWKIQ_NUMBER_OF_NODES=2 LOWKIQ_NODE_NUMBER=1 bundle exec lowkiq -r ./lib/app.rb
|
354
|
-
```
|
308
|
+
## Performance
|
355
309
|
|
356
|
-
|
357
|
-
|
358
|
-
Вы можете написать свой сплиттер, если ваше приложение требует особого распределения шардов между тредами или нодами.
|
359
|
-
|
360
|
-
## Scheduler
|
361
|
-
|
362
|
-
Каждый тред обрабатывает набор шардов. За выбор шарда для обработки отвечает планировщик.
|
363
|
-
Каждый поток имеет свой собственный экземпляр планировщика.
|
364
|
-
|
365
|
-
Lowkiq имеет 2 планировщика на выбор.
|
366
|
-
Первый, `Seq` - последовательно перебирает шарды.
|
367
|
-
Второй, `Lag` - выбирает шард с самой старой задачей, т.е. стремится минимизировать лаг.
|
368
|
-
Используется по умолчанию.
|
369
|
-
|
370
|
-
Планировщик задается через настройки:
|
371
|
-
|
372
|
-
```
|
373
|
-
Lowkiq.build_scheduler = ->() { Lowkiq.build_seq_scheduler }
|
374
|
-
# или
|
375
|
-
Lowkiq.build_scheduler = ->() { Lowkiq.build_lag_scheduler }
|
376
|
-
```
|
377
|
-
|
378
|
-
## Исключения
|
379
|
-
|
380
|
-
`StandardError` выброшенные воркером обрабатываются с помощью middleware.
|
381
|
-
Такие исключения не приводят к остановке процесса.
|
382
|
-
|
383
|
-
Все прочие исключения приводят к остановке процесса.
|
384
|
-
При этом Lowkiq дожидается выполнения задач другими тредами.
|
385
|
-
|
386
|
-
`StandardError` выброшенные вне воркера передаются в `Lowkiq.last_words`.
|
387
|
-
Например это происходит при потере соединения к Redis или при ошибке в коде Lowkiq.
|
388
|
-
|
389
|
-
## Изменение количества шардов воркера
|
390
|
-
|
391
|
-
Старайтесь не менять кол-во шардов.
|
392
|
-
|
393
|
-
Если вы можете отключить добавление новых заданий,
|
394
|
-
то дождитесь опустошения очередей и выкатите новую версию кода с измененным кол-вом шардов.
|
395
|
-
|
396
|
-
Если такой возможности нет, воспользуйтесь следующим сценарием.
|
397
|
-
|
398
|
-
Например, есть воркер:
|
310
|
+
Use [hiredis](https://github.com/redis/hiredis-rb) for better performance.
|
399
311
|
|
400
312
|
```ruby
|
401
|
-
|
402
|
-
extend Lowkiq::Worker
|
403
|
-
|
404
|
-
self.shards_count = 5
|
313
|
+
# Gemfile
|
405
314
|
|
406
|
-
|
407
|
-
some_code
|
408
|
-
end
|
409
|
-
end
|
315
|
+
gem "hiredis"
|
410
316
|
```
|
411
317
|
|
412
|
-
Теперь нужно указать новое кол-во шардов и задать новое имя очереди:
|
413
|
-
|
414
318
|
```ruby
|
415
|
-
|
416
|
-
extend Lowkiq::Worker
|
319
|
+
# config
|
417
320
|
|
418
|
-
|
419
|
-
self.queue_name = "#{self.name}_V2"
|
420
|
-
|
421
|
-
def self.perform(payloads_by_id)
|
422
|
-
some_code
|
423
|
-
end
|
424
|
-
end
|
321
|
+
Lowkiq.redis = ->() { Redis.new url: ENV.fetch('REDIS_URL'), driver: :hiredis }
|
425
322
|
```
|
426
323
|
|
427
|
-
|
428
|
-
|
429
|
-
```ruby
|
430
|
-
module ATestMigrationWorker
|
431
|
-
extend Lowkiq::Worker
|
324
|
+
## Execution
|
432
325
|
|
433
|
-
|
434
|
-
self.queue_name = "ATestWorker"
|
326
|
+
`lowkiq -r ./path_to_app`
|
435
327
|
|
436
|
-
|
437
|
-
jobs = payloads_by_id.each_with_object([]) do |(id, payloads), acc|
|
438
|
-
payloads.each do |payload|
|
439
|
-
acc << { id: id, payload: payload }
|
440
|
-
end
|
441
|
-
end
|
328
|
+
`path_to_app.rb` must load app. [Example](examples/dummy/lib/app.rb).
|
442
329
|
|
443
|
-
|
444
|
-
|
445
|
-
|
446
|
-
|
330
|
+
Lazy loading of workers modules is unacceptable.
|
331
|
+
For preliminarily loading modules use
|
332
|
+
`require`
|
333
|
+
or [`require_dependency`](https://api.rubyonrails.org/classes/ActiveSupport/Dependencies/Loadable.html#method-i-require_dependency)
|
334
|
+
for Ruby on Rails.
|
447
335
|
|
448
|
-
##
|
336
|
+
## Shutdown
|
449
337
|
|
450
|
-
|
338
|
+
Send TERM or INT signal to process (Ctrl-C).
|
339
|
+
Process will wait for executed jobs to finish.
|
451
340
|
|
452
|
-
`
|
453
|
-
|
341
|
+
Note that if queue is empty, process sleeps `poll_interval` seconds,
|
342
|
+
therefore, the process will not stop until the `poll_interval` seconds have passed.
|
454
343
|
|
455
|
-
|
344
|
+
## Debug
|
456
345
|
|
457
|
-
|
346
|
+
To get trace of all threads of app:
|
458
347
|
|
459
|
-
|
460
|
-
|
461
|
-
|
348
|
+
```
|
349
|
+
kill -TTIN <pid>
|
350
|
+
cat /tmp/lowkiq_ttin.txt
|
351
|
+
```
|
462
352
|
|
463
353
|
## Development
|
464
354
|
|
465
355
|
```
|
466
356
|
docker-compose run --rm --service-port app bash
|
467
|
-
|
357
|
+
bundle
|
468
358
|
rspec
|
469
359
|
cd examples/dummy ; bundle exec ../../exe/lowkiq -r ./lib/app.rb
|
470
360
|
```
|
471
361
|
|
472
|
-
##
|
362
|
+
## Exceptions
|
473
363
|
|
474
|
-
|
364
|
+
`StandardError` thrown by worker are handled with middleware. Such exceptions doesn't lead to process stop.
|
475
365
|
|
476
|
-
|
477
|
-
|
478
|
-
|
479
|
-
|
366
|
+
All other exceptions cause the process to stop.
|
367
|
+
Lowkiq will wait for job execution by other threads.
|
368
|
+
|
369
|
+
`StandardError` thrown outside of worker are passed to `Lowkiq.last_words`.
|
370
|
+
For example, it can happen when Redis connection is lost or when Lowkiq's code has a bug.
|
480
371
|
|
481
372
|
## Rails integration
|
482
373
|
|
@@ -493,10 +384,10 @@ end
|
|
493
384
|
```ruby
|
494
385
|
# config/initializers/lowkiq.rb
|
495
386
|
|
496
|
-
#
|
387
|
+
# loading all lowkiq workers
|
497
388
|
Dir["#{Rails.root}/app/lowkiq_workers/**/*.rb"].each { |file| require_dependency file }
|
498
389
|
|
499
|
-
#
|
390
|
+
# configuration:
|
500
391
|
# Lowkiq.redis = -> { Redis.new url: ENV.fetch('LOWKIQ_REDIS_URL') }
|
501
392
|
# Lowkiq.threads_per_node = ENV.fetch('LOWKIQ_THREADS_PER_NODE').to_i
|
502
393
|
# Lowkiq.client_pool_size = ENV.fetch('LOWKIQ_CLIENT_POOL_SIZE').to_i
|
@@ -558,7 +449,7 @@ if defined? NewRelic
|
|
558
449
|
Lowkiq.server_middlewares << NewRelicLowkiqMiddleware.new
|
559
450
|
end
|
560
451
|
|
561
|
-
# Rails reloader,
|
452
|
+
# Rails reloader, responsible for cleaning of ActiveRecord connections
|
562
453
|
Lowkiq.server_middlewares << -> (worker, batch, &block) do
|
563
454
|
Rails.application.reloader.wrap do
|
564
455
|
block.call
|
@@ -574,4 +465,183 @@ Lowkiq.on_server_init = ->() do
|
|
574
465
|
end
|
575
466
|
```
|
576
467
|
|
577
|
-
|
468
|
+
Execution: `bundle exec lowkiq -r ./config/environment.rb`
|
469
|
+
|
470
|
+
|
471
|
+
## Splitter
|
472
|
+
|
473
|
+
Each worker has several shards:
|
474
|
+
|
475
|
+
```
|
476
|
+
# worker: shard ids
|
477
|
+
worker A: 0, 1, 2
|
478
|
+
worker B: 0, 1, 2, 3
|
479
|
+
worker C: 0
|
480
|
+
worker D: 0, 1
|
481
|
+
```
|
482
|
+
|
483
|
+
Lowkiq uses fixed amount of threads for job processing, therefore it is necessary to distribute shards between threads.
|
484
|
+
Splitter does it.
|
485
|
+
|
486
|
+
To define a set of shards, which is being processed by thread, lets move them to one list:
|
487
|
+
|
488
|
+
```
|
489
|
+
A0, A1, A2, B0, B1, B2, B3, C0, D0, D1
|
490
|
+
```
|
491
|
+
|
492
|
+
Default splitter evenly distributes shards by threads of a single node.
|
493
|
+
|
494
|
+
If `threads_per_node` is set to 3, the distribution will be:
|
495
|
+
|
496
|
+
```
|
497
|
+
# thread id: shards
|
498
|
+
t0: A0, B0, B3, D1
|
499
|
+
t1: A1, B1, C0
|
500
|
+
t2: A2, B2, D0
|
501
|
+
```
|
502
|
+
|
503
|
+
Besides Default Lowkiq has ByNode splitter. It allows to divide the load by several processes (nodes).
|
504
|
+
|
505
|
+
```
|
506
|
+
Lowkiq.build_splitter = -> () do
|
507
|
+
Lowkiq.build_by_node_splitter(
|
508
|
+
ENV.fetch('LOWKIQ_NUMBER_OF_NODES').to_i,
|
509
|
+
ENV.fetch('LOWKIQ_NODE_NUMBER').to_i
|
510
|
+
)
|
511
|
+
end
|
512
|
+
```
|
513
|
+
|
514
|
+
So, instead of single process you need to execute multiple ones and to set environment variables up:
|
515
|
+
|
516
|
+
```
|
517
|
+
# process 0
|
518
|
+
LOWKIQ_NUMBER_OF_NODES=2 LOWKIQ_NODE_NUMBER=0 bundle exec lowkiq -r ./lib/app.rb
|
519
|
+
|
520
|
+
# process 1
|
521
|
+
LOWKIQ_NUMBER_OF_NODES=2 LOWKIQ_NODE_NUMBER=1 bundle exec lowkiq -r ./lib/app.rb
|
522
|
+
```
|
523
|
+
|
524
|
+
Summary amount of threads are equal product of `ENV.fetch('LOWKIQ_NUMBER_OF_NODES')` and `Lowkiq.threads_per_node`.
|
525
|
+
|
526
|
+
You can also write your own splitter if your app needs extra distribution of shards between threads or nodes.
|
527
|
+
|
528
|
+
## Scheduler
|
529
|
+
|
530
|
+
Every thread processes a set of shards. Scheduler select shard for processing.
|
531
|
+
Every thread has it's own instance of scheduler.
|
532
|
+
|
533
|
+
Lowkiq has 2 schedulers for your choice.
|
534
|
+
`Seq` sequentally looks over shards.
|
535
|
+
`Lag` chooses shard with the oldest job minimizing the lag. It's used by default.
|
536
|
+
|
537
|
+
Scheduler can be set up through settings:
|
538
|
+
|
539
|
+
```
|
540
|
+
Lowkiq.build_scheduler = ->() { Lowkiq.build_seq_scheduler }
|
541
|
+
# or
|
542
|
+
Lowkiq.build_scheduler = ->() { Lowkiq.build_lag_scheduler }
|
543
|
+
```
|
544
|
+
|
545
|
+
## Recommendations on configuration
|
546
|
+
|
547
|
+
### `SomeWorker.shards_count`
|
548
|
+
|
549
|
+
Sum of `shards_count` of all workers shouldn't be less than `Lowkiq.threads_per_node`
|
550
|
+
otherwise threads will stay idle.
|
551
|
+
|
552
|
+
Sum of `shards_count` of all workers can be equal to `Lowkiq.threads_per_node`.
|
553
|
+
In this case thread processes a single shard. This makes sense only with uniform queue load.
|
554
|
+
|
555
|
+
Sum of `shards_count` of all workers can be more than `Lowkiq.threads_per_node`.
|
556
|
+
In this case `shards_count` can be counted as a priority.
|
557
|
+
The larger it is, the more often the tasks of this queue will be processed.
|
558
|
+
|
559
|
+
There is no reason to set `shards_count` of one worker more than `Lowkiq.threads_per_node`,
|
560
|
+
because every thread will handle more than one shard from this queue, so it increases the overhead.
|
561
|
+
|
562
|
+
### `SomeWorker.max_retry_count`
|
563
|
+
|
564
|
+
From `retry_in` and `max_retry_count`, you can calculate approximate time that payload of job will be in a queue.
|
565
|
+
After `max_retry_count` is reached the payload with a minimal score will be moved to a morgue.
|
566
|
+
|
567
|
+
For default `retry_in` we receive the following table.
|
568
|
+
|
569
|
+
```ruby
|
570
|
+
def retry_in(retry_count)
|
571
|
+
(retry_count ** 4) + 15 + (rand(30) * (retry_count + 1))
|
572
|
+
end
|
573
|
+
```
|
574
|
+
|
575
|
+
| `max_retry_count` | amount of days of job's life |
|
576
|
+
| --- | --- |
|
577
|
+
| 14 | 1 |
|
578
|
+
| 16 | 2 |
|
579
|
+
| 18 | 3 |
|
580
|
+
| 19 | 5 |
|
581
|
+
| 20 | 6 |
|
582
|
+
| 21 | 8 |
|
583
|
+
| 22 | 10 |
|
584
|
+
| 23 | 13 |
|
585
|
+
| 24 | 16 |
|
586
|
+
| 25 | 20 |
|
587
|
+
|
588
|
+
`(0...25).map{ |c| retry_in c }.sum / 60 / 60 / 24`
|
589
|
+
|
590
|
+
|
591
|
+
## Changing of worker's shards amount
|
592
|
+
|
593
|
+
Try to count amount of shards right away and don't change it in future.
|
594
|
+
|
595
|
+
If you can disable adding of new jobs, wait for queues to get empty and deploy the new version of code with changed amount of shards.
|
596
|
+
|
597
|
+
If you can't do it, follow the next steps:
|
598
|
+
|
599
|
+
A worker example:
|
600
|
+
|
601
|
+
```ruby
|
602
|
+
module ATestWorker
|
603
|
+
extend Lowkiq::Worker
|
604
|
+
|
605
|
+
self.shards_count = 5
|
606
|
+
|
607
|
+
def self.perform(payloads_by_id)
|
608
|
+
some_code
|
609
|
+
end
|
610
|
+
end
|
611
|
+
```
|
612
|
+
|
613
|
+
Set the number of shards and new queue name:
|
614
|
+
|
615
|
+
```ruby
|
616
|
+
module ATestWorker
|
617
|
+
extend Lowkiq::Worker
|
618
|
+
|
619
|
+
self.shards_count = 10
|
620
|
+
self.queue_name = "#{self.name}_V2"
|
621
|
+
|
622
|
+
def self.perform(payloads_by_id)
|
623
|
+
some_code
|
624
|
+
end
|
625
|
+
end
|
626
|
+
```
|
627
|
+
|
628
|
+
Add a worker moving jobs from the old queue to a new one:
|
629
|
+
|
630
|
+
```ruby
|
631
|
+
module ATestMigrationWorker
|
632
|
+
extend Lowkiq::Worker
|
633
|
+
|
634
|
+
self.shards_count = 5
|
635
|
+
self.queue_name = "ATestWorker"
|
636
|
+
|
637
|
+
def self.perform(payloads_by_id)
|
638
|
+
jobs = payloads_by_id.each_with_object([]) do |(id, payloads), acc|
|
639
|
+
payloads.each do |payload|
|
640
|
+
acc << { id: id, payload: payload }
|
641
|
+
end
|
642
|
+
end
|
643
|
+
|
644
|
+
ATestWorker.perform_async jobs
|
645
|
+
end
|
646
|
+
end
|
647
|
+
```
|