jiggler 0.1.0.rc4 → 0.1.0.rc6
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +38 -200
- data/bin/jiggler +1 -1
- data/lib/jiggler/at_least_once/acknowledger.rb +43 -0
- data/lib/jiggler/at_least_once/fetcher.rb +93 -0
- data/lib/jiggler/at_most_once/acknowledger.rb +21 -0
- data/lib/jiggler/at_most_once/fetcher.rb +46 -0
- data/lib/jiggler/base_acknowledger.rb +11 -0
- data/lib/jiggler/base_fetcher.rb +14 -0
- data/lib/jiggler/cleaner.rb +14 -5
- data/lib/jiggler/cli.rb +3 -2
- data/lib/jiggler/config.rb +48 -4
- data/lib/jiggler/launcher.rb +9 -3
- data/lib/jiggler/manager.rb +26 -2
- data/lib/jiggler/retrier.rb +8 -10
- data/lib/jiggler/scheduled/enqueuer.rb +1 -1
- data/lib/jiggler/scheduled/poller.rb +38 -26
- data/lib/jiggler/scheduled/requeuer.rb +57 -0
- data/lib/jiggler/server.rb +7 -0
- data/lib/jiggler/stats/collection.rb +3 -2
- data/lib/jiggler/stats/monitor.rb +2 -2
- data/lib/jiggler/summary.rb +8 -1
- data/lib/jiggler/support/helper.rb +17 -3
- data/lib/jiggler/version.rb +1 -1
- data/lib/jiggler/web.rb +0 -2
- data/lib/jiggler/worker.rb +21 -36
- data/spec/examples.txt +96 -79
- data/spec/fixtures/config/jiggler.yml +2 -2
- data/spec/jiggler/at_least_once/acknowledger_spec.rb +30 -0
- data/spec/jiggler/at_least_once/fetcher_spec.rb +69 -0
- data/spec/jiggler/at_most_once/fetcher_spec.rb +33 -0
- data/spec/jiggler/cleaner_spec.rb +11 -1
- data/spec/jiggler/cli_spec.rb +5 -7
- data/spec/jiggler/config_spec.rb +45 -3
- data/spec/jiggler/core_spec.rb +2 -0
- data/spec/jiggler/job_spec.rb +4 -4
- data/spec/jiggler/launcher_spec.rb +60 -54
- data/spec/jiggler/manager_spec.rb +50 -41
- data/spec/jiggler/retrier_spec.rb +3 -1
- data/spec/jiggler/scheduled/requeuer_spec.rb +57 -0
- data/spec/jiggler/stats/monitor_spec.rb +3 -2
- data/spec/jiggler/summary_spec.rb +19 -5
- data/spec/jiggler/worker_spec.rb +11 -15
- data/spec/spec_helper.rb +7 -0
- metadata +38 -9
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f7cfe745f2f20d4e838c4aee98ec59671703eeeb65541fcc7805c4d9382e3971
|
4
|
+
data.tar.gz: 39ebc55a5835241419b6769ccd31bbeeba6144a5dcc61ae2206c0771c558df5e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 502627650d22839f2f9555dbc926d016e44173ec5d2f5f9195bb5a8f28ac1f4c2402d223b735e4a49b5b5e960d47037de900838565aa555caed3ee3c660d99b6
|
7
|
+
data.tar.gz: 88d50a8f353b50fd85ef14d8979c1623b5b5d1cee67cc3b57e15f026af86108a5386ca8efc01ec203f40ead1a14a90db45491df806bf2fb80d517210760b447b
|
data/README.md
CHANGED
@@ -7,7 +7,7 @@ Jiggler is a [Sidekiq](https://github.com/mperham/sidekiq)-inspired background j
|
|
7
7
|
|
8
8
|
*Jiggler is based on Sidekiq implementation, and re-uses most of its concepts and ideas.*
|
9
9
|
|
10
|
-
**NOTE**:
|
10
|
+
**NOTE**: Jiggler is a small gem made purely for fun and to gain some hand-on experience with async and fibers. It isn't tested with production projects and might have not-yet-discovered issues. Use at your own risk. \
|
11
11
|
However, it's good to play around and/or to try it in the name of science.
|
12
12
|
|
13
13
|
### Installation
|
@@ -17,6 +17,11 @@ Install the gem:
|
|
17
17
|
gem install jiggler
|
18
18
|
```
|
19
19
|
|
20
|
+
Use `--pre` for release candidates:
|
21
|
+
```
|
22
|
+
gem install jiggler --pre
|
23
|
+
```
|
24
|
+
|
20
25
|
Start Jiggler server as a separate process with bin command:
|
21
26
|
```
|
22
27
|
jiggler -r <FILE_PATH>
|
@@ -28,204 +33,8 @@ Run `jiggler --help` to see the list of command line arguments.
|
|
28
33
|
|
29
34
|
### Performance
|
30
35
|
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
Ruby `3.2.0` \
|
35
|
-
Redis `7.0.7` \
|
36
|
-
Poller interval `5s` \
|
37
|
-
Monitoring interval `10s` \
|
38
|
-
Logging level `WARN`
|
39
|
-
|
40
|
-
#### Noop task measures
|
41
|
-
|
42
|
-
```ruby
|
43
|
-
def perform
|
44
|
-
# just an empty job doing nothing
|
45
|
-
end
|
46
|
-
```
|
47
|
-
|
48
|
-
The parent process enqueues the jobs, starts the monitoring, and then forks the child job-processor-process. Thus, `RSS` value is affected by the number of jobs uploaded in the parent process. See `bin/jigglerload` to see the load test structure and measuring. \
|
49
|
-
1_000_000 jobs were enqueued in 100k batches x10.
|
50
|
-
|
51
|
-
| Job Processor | Concurrency | Jobs | Time | Start RSS | Finish RSS |
|
52
|
-
|------------------|-------------|-----------|-----------|------------|---------------|
|
53
|
-
| Sidekiq 7.0.2 | 5 | 100_000 | 20.01 sec | 132_080 kb | 103_168 kb (GC) |
|
54
|
-
| Jiggler 0.1.0 | 5 | 100_000 | 14.25 sec | 87_532 kb | 91_464 kb |
|
55
|
-
| - | | | | | |
|
56
|
-
| Sidekiq 7.0.2 | 10 | 100_000 | 20.49 sec | 132_164 kb | 125_768 kb (GC) |
|
57
|
-
| Jiggler 0.1.0 | 10 | 100_000 | 13.25 sec | 87_688 kb | 91_625 kb |
|
58
|
-
| - | | | | | |
|
59
|
-
| Sidekiq 7.0.2 | 5 | 1_000_000 | 186.90 sec | 159_712 kb | 186_224 kb |
|
60
|
-
| Jiggler 0.1.0 | 5 | 1_000_000 | 123.13 sec | 113_212 kb | 116_336 kb |
|
61
|
-
| - | | | | | |
|
62
|
-
| Sidekiq 7.0.2 | 10 | 1_000_000 | 186.94 sec | 159_000 kb | 192_780 kb |
|
63
|
-
| Jiggler 0.1.0 | 10 | 1_000_000 | 115.56 sec | 113_656 kb | 116_896 kb |
|
64
|
-
|
65
|
-
|
66
|
-
#### IO tests
|
67
|
-
|
68
|
-
The idea of the next tests is to simulate jobs with different kinds of IO tasks. \
|
69
|
-
Ruby 3 has introduced fiber scheduler interface, which allows to implement hooks related to IO/blocking operations.\
|
70
|
-
The context switching won't work well in case IO is performed by C-extentions which are not aware of Ruby scheduler.
|
71
|
-
|
72
|
-
##### NET/HTTP requests
|
73
|
-
|
74
|
-
Spin-up a local sinatra app to exclude network issues while testing HTTP requests (it uses `falcon` web server).
|
75
|
-
|
76
|
-
```ruby
|
77
|
-
require "sinatra"
|
78
|
-
|
79
|
-
class MyApp < Sinatra::Base
|
80
|
-
get "/hello" do
|
81
|
-
sleep(0.2)
|
82
|
-
"Hello World!"
|
83
|
-
end
|
84
|
-
end
|
85
|
-
```
|
86
|
-
|
87
|
-
Then, the code which is going to be performed within the workers should make a `net/http` request to the local endpoint.
|
88
|
-
|
89
|
-
```ruby
|
90
|
-
# a single job takes ~0.21s to perform
|
91
|
-
def perform
|
92
|
-
uri = URI("http://127.0.0.1:9292/hello")
|
93
|
-
res = Net::HTTP.get_response(uri)
|
94
|
-
puts "Request Error!!!" unless res.is_a?(Net::HTTPSuccess)
|
95
|
-
end
|
96
|
-
```
|
97
|
-
|
98
|
-
It's not recommended to run sidekiq with high concurrency values, setting it for the sake of test. \
|
99
|
-
The time difference for these samples is small-ish, however the memory consumption is less with the fibers. \
|
100
|
-
Since fibers have relatively small memory foot-print and context switching is also relatively cheap, it's possible to set concurrency to higher values within Jiggler without too much trade-offs.
|
101
|
-
|
102
|
-
| Job Processor | Concurrency | Jobs | Time to complete | Start RSS | Finish RSS | %CPU |
|
103
|
-
|------------------|-------------|-------|-------------------|-----------|------------|------|
|
104
|
-
| Sidekiq 7.0.2 | 5 | 1_000 | 43.74 sec | 30_444 kb | 45_124 kb | 5.9 |
|
105
|
-
| Jiggler 0.1.0 | 5 | 1_000 | 43.65 sec | 33_476 kb | 34_144 kb | 2.9 |
|
106
|
-
| - | | | | | | |
|
107
|
-
| Sidekiq 7.0.2 | 10 | 1_000 | 23.05 sec | 30_604 kb | 50_292 kb | 10.93 |
|
108
|
-
| Jiggler 0.1.0 | 10 | 1_000 | 22.86 sec | 32_416 kb | 34_128 kb | 5.69 |
|
109
|
-
| - | | | | | | |
|
110
|
-
| Sidekiq 7.0.2 | 15 | 1_000 | 16.17 sec | 30_636 kb | 55_144 kb | 16.47 |
|
111
|
-
| Jiggler 0.1.0 | 15 | 1_000 | 15.87 sec | 33_328 kb | 34_548 kb | 8.25 |
|
112
|
-
|
113
|
-
**NOTE**: Jiggler has more dependencies, so with small load `start RSS` takes more space.
|
114
|
-
|
115
|
-
##### PostgreSQL connection/queries
|
116
|
-
|
117
|
-
`pg` gem supports Ruby's `Fiber.scheduler` starting from 1.3.0 version. Make sure yours DB-adapter supports it.
|
118
|
-
|
119
|
-
```ruby
|
120
|
-
### global namespace
|
121
|
-
require "pg"
|
122
|
-
|
123
|
-
$pg_pool = ConnectionPool.new(size: CONCURRENCY) do
|
124
|
-
PG.connect({ dbname: "test", password: "test", user: "test" })
|
125
|
-
end
|
126
|
-
|
127
|
-
### worker context
|
128
|
-
# a single job takes ~0.102s to perform
|
129
|
-
def perform
|
130
|
-
$pg_pool.with do |conn|
|
131
|
-
conn.exec("SELECT pg_sleep(0.1)")
|
132
|
-
end
|
133
|
-
end
|
134
|
-
```
|
135
|
-
|
136
|
-
| Job Processor | Concurrency | Jobs | Time | Start RSS | Finish RSS | %CPU |
|
137
|
-
|------------------|-------------|-------|-----------|-----------|------------|------|
|
138
|
-
| Sidekiq 7.0.2 | 5 | 1_000 | 23.44 sec | 31_436 kb | 48_856 kb | 7.56 |
|
139
|
-
| Jiggler 0.1.0 | 5 | 1_000 | 23.20 sec | 35_312 kb | 38_592 kb | 2.91 |
|
140
|
-
| - | | | | | | |
|
141
|
-
| Sidekiq 7.0.2 | 10 | 1_000 | 13.15 sec | 31_272 kb | 52_808 kb | 13.76 |
|
142
|
-
| Jiggler 0.1.0 | 10 | 1_000 | 12.65 sec | 35_296 kb | 38_784 kb | 6.11 |
|
143
|
-
| - | | | | | | |
|
144
|
-
| Sidekiq 7.0.2 | 15 | 1_000 | 9.63 sec | 31_016 kb | 59_868 kb | 20.32 |
|
145
|
-
| Jiggler 0.1.0 | 15 | 1_000 | 9.17 sec | 35_188 kb | 38_948 kb | 9.26 |
|
146
|
-
|
147
|
-
##### File IO
|
148
|
-
|
149
|
-
```ruby
|
150
|
-
def perform(file_name, id)
|
151
|
-
File.open(file_name, "a") { |f| f.write("#{id}\n") }
|
152
|
-
end
|
153
|
-
```
|
154
|
-
|
155
|
-
| Job Processor | Concurrency | Jobs | Time | Start RSS | Finish RSS | %CPU |
|
156
|
-
|------------------|-------------|--------|-----------|--------------|------------|-------|
|
157
|
-
| Sidekiq 7.0.2 | 5 | 30_000 | 11.94 sec | 61_944 kb | 71_948 kb | 94.34 |
|
158
|
-
| Jiggler 0.1.0 | 5 | 30_000 | 7.87 sec | 50_140 kb | 51_272 kb | 61.7 |
|
159
|
-
| - | | | | | | |
|
160
|
-
| Sidekiq 7.0.2 | 10 | 30_000 | 11.6 sec | 62_020 kb | 78_952 kb | 94.44 |
|
161
|
-
| Jiggler 0.1.0 | 10 | 30_000 | 7.17 sec | 50_060 kb | 51_464 kb | 69.25 |
|
162
|
-
| - | | | | | | |
|
163
|
-
| Sidekiq 7.0.2 | 15 | 30_000 | 11.24 sec | 62_016 kb | 83_808 kb | 94.16 |
|
164
|
-
| Jiggler 0.1.0 | 15 | 30_000 | 7.02 sec | 49_988 kb | 51_428 kb | 70.3 |
|
165
|
-
|
166
|
-
|
167
|
-
Jiggler is effective only for tasks with a lot of IO. You must test the concurrency setting with your jobs to find out what configuration works best for your payload.
|
168
|
-
|
169
|
-
#### Simulate CPU-only job
|
170
|
-
|
171
|
-
With CPU-heavy jobs Jiggler has poor performance. Just to make sure it's generally able to work with CPU-only payloads:
|
172
|
-
|
173
|
-
```ruby
|
174
|
-
def fib(n)
|
175
|
-
if n <= 1
|
176
|
-
1
|
177
|
-
else
|
178
|
-
(fib(n-1) + fib(n-2))
|
179
|
-
end
|
180
|
-
end
|
181
|
-
|
182
|
-
# a single job takes ~0.035s to perform
|
183
|
-
def perform(_idx)
|
184
|
-
fib(25)
|
185
|
-
end
|
186
|
-
```
|
187
|
-
|
188
|
-
| Job Processor | Concurrency | Jobs | Time | Start RSS | Finish RSS |
|
189
|
-
|------------------|-------------|------|----------|------------|------------|
|
190
|
-
| Sidekiq 7.0.2 | 5 | 100 | 5.81 sec | 27_792 kb | 42_464 kb |
|
191
|
-
| Jiggler 0.1.0 | 5 | 100 | 5.29 sec | 31_304 kb | 32_320 kb |
|
192
|
-
| - | | | | | |
|
193
|
-
| Sidekiq 7.0.2 | 10 | 100 | 5.63 sec | 28_044 kb | 47_640 kb |
|
194
|
-
| Jiggler 0.1.0 | 10 | 100 | 5.43 sec | 32_316 kb | 32_548 kb |
|
195
|
-
|
196
|
-
#### IO Event selector
|
197
|
-
|
198
|
-
`IO_EVENT_SELECTOR` is an env variable which allows to specify the event selector used by the Ruby scheduler. \
|
199
|
-
On default it uses `Epoll` (`IO_EVENT_SELECTOR=EPoll`). \
|
200
|
-
Another available option is `URing` (`IO_EVENT_SELECTOR=URing`). Underneath it uses `io_uring` library. It is a Linux kernel library that provides a high-performance interface for asynchronous I/O operations. It was introduced in Linux kernel version 5.1 and aims to address some of the limitations and scalability issues of the existing AIO (Asynchronous I/O) interface.
|
201
|
-
In the future it might bring a lot of performance boost into Ruby fibers world (once `async` project fully adopts it), but at the moment in the most cases its performance is similar to `EPoll`, yet it could give some boost with File IO.
|
202
|
-
|
203
|
-
#### Socketry stack
|
204
|
-
|
205
|
-
The gem allows to use libs from `socketry` stack (https://github.com/socketry) within workers. \
|
206
|
-
F.e. when making HTTP requests using `async/http/internet` to the Sinatra app described above:
|
207
|
-
|
208
|
-
```ruby
|
209
|
-
### global namespace
|
210
|
-
require "async/http/internet"
|
211
|
-
$internet = Async::HTTP::Internet.new
|
212
|
-
|
213
|
-
### worker context
|
214
|
-
def perform
|
215
|
-
uri = "https://127.0.0.1/hello"
|
216
|
-
res = $internet.get(uri)
|
217
|
-
res.finish
|
218
|
-
puts "Request Error!!!" unless res.status == 200
|
219
|
-
end
|
220
|
-
```
|
221
|
-
|
222
|
-
| Job Processor | Concurrency | Jobs | Time | Start RSS | Finish RSS | %CPU |
|
223
|
-
|------------------|-------------|-------|-----------|-----------|------------|------|
|
224
|
-
| Jiggler 0.1.0 | 5 | 1_000 | 43.23 sec | 34_340 kb | 38_488 kb | 1.51 |
|
225
|
-
| - | | | | | | |
|
226
|
-
| Jiggler 0.1.0 | 10 | 1_000 | 22.67 sec | 34_552 kb | 38_600 kb | 2.75 |
|
227
|
-
| - | | | | | | |
|
228
|
-
| Jiggler 0.1.0 | 15 | 1_000 | 15.88 sec | 34_332 kb | 38_544 kb | 4.06 |
|
36
|
+
[Jiggler 0.1.0rc4 performance results (at most once delivery)](/docs/perf_results_0.1.0rc4.md)
|
37
|
+
[Jiggler 0.1.0 performance results](/docs/perf_results_0.1.0.md)
|
229
38
|
|
230
39
|
### Getting Started
|
231
40
|
|
@@ -255,9 +64,34 @@ Jiggler.configure do |config|
|
|
255
64
|
config[:redis_url] = ENV["REDIS_URL"] # On default fetches the value from ENV["REDIS_URL"]
|
256
65
|
config[:queues] = ["shippers"] # An array of queue names the server is going to listen to. On default uses ["default"]
|
257
66
|
config[:config_file] = "./jiggler.yml" # .yml file with Jiggler settings
|
67
|
+
config[:mode] = :at_most_once # at_most_once and at_least_once modes supported. Defaults to :at_least_once
|
68
|
+
end
|
69
|
+
```
|
70
|
+
|
71
|
+
`at_least_once` mode grants reliability for the regular enqueued jobs which are going to be executed by workers. The scheduled jobs (the ones planned to be executed at a specific time, or which failed and going to be retried) still support only `at_most_once` strategy. The support for them is going to be added in the upcoming versions.
|
72
|
+
|
73
|
+
On default all queues have the same priority (equals to 0). Higher number means higher prio. \
|
74
|
+
It's possible to specify custom priorities as follows:
|
75
|
+
|
76
|
+
```ruby
|
77
|
+
Jiggler.configure do |config|
|
78
|
+
config[:queues] = [["shippers", 0], ["shipments", 1], ["delivery", 2]]
|
258
79
|
end
|
259
80
|
```
|
260
81
|
|
82
|
+
#### IO Event selector
|
83
|
+
|
84
|
+
`IO_EVENT_SELECTOR` is an env variable which allows to specify the event selector used by the Ruby scheduler. \
|
85
|
+
On default it uses `Epoll` (`IO_EVENT_SELECTOR=EPoll`). \
|
86
|
+
Another available option is `URing` (`IO_EVENT_SELECTOR=URing`). Underneath it uses `io_uring` library. It is a Linux kernel library that provides a high-performance interface for asynchronous I/O operations. It was introduced in Linux kernel version 5.1 and aims to address some of the limitations and scalability issues of the existing AIO (Asynchronous I/O) interface.
|
87
|
+
In the future it might bring a lot of performance boost into Ruby fibers world (once `async` project fully adopts it), but at the moment in the most cases its performance is similar to `EPoll`, yet it could give some boost with File IO.
|
88
|
+
|
89
|
+
#### Socketry stack
|
90
|
+
|
91
|
+
The gem allows to use libs from `socketry` stack (https://github.com/socketry) within workers.
|
92
|
+
|
93
|
+
#### Core concepts
|
94
|
+
|
261
95
|
Internally Jiggler server consists of 3 parts: `Manager`, `Poller`, `Monitor`. \
|
262
96
|
`Manager` is responsible for workers. \
|
263
97
|
`Poller` fetches data for retries and scheduled jobs. \
|
@@ -377,7 +211,7 @@ Jiggler.configure_client do |config|
|
|
377
211
|
config[:client_redis_pool] = my_async_redis_pool
|
378
212
|
end
|
379
213
|
|
380
|
-
# or use
|
214
|
+
# or use built-in async pool with
|
381
215
|
require "async/pool"
|
382
216
|
|
383
217
|
Jiggler.configure_client do |config|
|
@@ -416,3 +250,7 @@ docker-compose run --rm web -- bundle exec rspec
|
|
416
250
|
```
|
417
251
|
|
418
252
|
To run the load tests modify the `docker-compose.yml` to point to `bin/jigglerload`
|
253
|
+
|
254
|
+
### Contributing
|
255
|
+
|
256
|
+
Fork & Pull Request.
|
data/bin/jiggler
CHANGED
@@ -9,7 +9,7 @@ begin
|
|
9
9
|
cli.parse_and_init
|
10
10
|
|
11
11
|
cli.config.logger.info("Jiggler is starting in #{Jiggler.config[:environment].upcase} ✯⸜(*❛‿❛)⸝✯")
|
12
|
-
cli.config.logger.info("Jiggler version=#{Jiggler::VERSION} pid=#{Process.pid} concurrency=#{cli.config[:concurrency]} queues=#{cli.config
|
12
|
+
cli.config.logger.info("Jiggler version=#{Jiggler::VERSION} pid=#{Process.pid} concurrency=#{cli.config[:concurrency]} queues=#{cli.config.sorted_queues.join(',')}")
|
13
13
|
|
14
14
|
cli.start
|
15
15
|
rescue => e
|
@@ -0,0 +1,43 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Jiggler
|
4
|
+
module AtLeastOnce
|
5
|
+
class Acknowledger < BaseAcknowledger
|
6
|
+
def initialize(config)
|
7
|
+
super
|
8
|
+
@runners = []
|
9
|
+
@queue = Queue.new
|
10
|
+
end
|
11
|
+
|
12
|
+
def ack(job)
|
13
|
+
@queue.push(job)
|
14
|
+
end
|
15
|
+
|
16
|
+
def start
|
17
|
+
@config[:concurrency].times do
|
18
|
+
@runners << safe_async('Acknowledger') do
|
19
|
+
while (job = @queue.pop) != nil
|
20
|
+
begin
|
21
|
+
job.ack
|
22
|
+
rescue StandardError => err
|
23
|
+
log_error(err, context: '\'Could not acknowledge a job\'', job: job)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
logger.debug('Acknowledger exits')
|
27
|
+
rescue Async::Stop
|
28
|
+
logger.debug('Acknowledger received stop signal')
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
def wait
|
34
|
+
@runners.each(&:wait)
|
35
|
+
end
|
36
|
+
|
37
|
+
def terminate
|
38
|
+
logger.debug('Suspending the acknowledger')
|
39
|
+
@queue.close
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,93 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'fc'
|
4
|
+
|
5
|
+
module Jiggler
|
6
|
+
module AtLeastOnce
|
7
|
+
class Fetcher < BaseFetcher
|
8
|
+
TIMEOUT = 2.0 # 2 seconds of waiting for brpoplpush
|
9
|
+
RESERVE_QUEUE_SUFFIX = 'in_progress'
|
10
|
+
|
11
|
+
attr_reader :producers
|
12
|
+
|
13
|
+
def initialize(config, collection)
|
14
|
+
super
|
15
|
+
@tasks_queue = FastContainers::PriorityQueue.new(:min)
|
16
|
+
@condition = Async::Notification.new
|
17
|
+
@consumers_queue = Queue.new
|
18
|
+
end
|
19
|
+
|
20
|
+
CurrentJob = Struct.new(:queue, :args, :reserve_queue, :config, keyword_init: true) do
|
21
|
+
def ack
|
22
|
+
config.with_sync_redis do |conn|
|
23
|
+
conn.call('LREM', reserve_queue, 1, args)
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
def start
|
29
|
+
config.sorted_queues_data.each do |queue, data|
|
30
|
+
config[:fetchers_concurrency].times do
|
31
|
+
safe_async("'Fetcher for #{queue}'") do
|
32
|
+
list = data[:list]
|
33
|
+
rlist = in_process_queue(list)
|
34
|
+
loop do
|
35
|
+
if @consumers_queue.num_waiting.zero? && !@done
|
36
|
+
@condition.wait # supposed to block here until consumers notify
|
37
|
+
end
|
38
|
+
break if @done
|
39
|
+
|
40
|
+
args = config.with_sync_redis do |conn|
|
41
|
+
conn.blocking_call(false, 'BRPOPLPUSH', list, rlist, TIMEOUT)
|
42
|
+
end
|
43
|
+
# no requeue logic rn as we expect monitor to handle
|
44
|
+
# in-process-tasks list for this process
|
45
|
+
break if @done
|
46
|
+
|
47
|
+
next if args.nil?
|
48
|
+
|
49
|
+
@tasks_queue.push(job(list, args, rlist), data[:priority])
|
50
|
+
@consumers_queue.push('') # to unblock any waiting consumer
|
51
|
+
end
|
52
|
+
logger.debug("Fetcher for #{queue} stopped")
|
53
|
+
rescue Async::Stop
|
54
|
+
logger.debug("Fetcher for #{queue} received stop signal")
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
|
60
|
+
def fetch
|
61
|
+
@condition.signal if signal?
|
62
|
+
return :done if @consumers_queue.pop.nil?
|
63
|
+
|
64
|
+
@tasks_queue.pop
|
65
|
+
end
|
66
|
+
|
67
|
+
def suspend
|
68
|
+
logger.debug("Suspending the fetcher")
|
69
|
+
@done = true
|
70
|
+
@condition.signal
|
71
|
+
@consumers_queue.close # unblocks awaiting consumers
|
72
|
+
end
|
73
|
+
|
74
|
+
private
|
75
|
+
|
76
|
+
def queue_signaling_limit
|
77
|
+
@queue_signaling_limit ||= [config[:concurrency] / 2, 1].max
|
78
|
+
end
|
79
|
+
|
80
|
+
def signal?
|
81
|
+
@consumers_queue.size < config[:concurrency]
|
82
|
+
end
|
83
|
+
|
84
|
+
def in_process_queue(queue)
|
85
|
+
"#{queue}:#{RESERVE_QUEUE_SUFFIX}:#{collection.uuid}"
|
86
|
+
end
|
87
|
+
|
88
|
+
def job(queue, args, reserve_queue)
|
89
|
+
CurrentJob.new(queue:, args:, reserve_queue:, config:)
|
90
|
+
end
|
91
|
+
end
|
92
|
+
end
|
93
|
+
end
|
@@ -0,0 +1,46 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Jiggler
|
4
|
+
module AtMostOnce
|
5
|
+
class Fetcher < BaseFetcher
|
6
|
+
TIMEOUT = 2.0 # 2 seconds of waiting for brpop
|
7
|
+
|
8
|
+
CurrentJob = Struct.new(:queue, :args)
|
9
|
+
|
10
|
+
def start
|
11
|
+
# noop, we just block directly during the fetch
|
12
|
+
end
|
13
|
+
|
14
|
+
def fetch
|
15
|
+
return :done if @done
|
16
|
+
|
17
|
+
q, args = config.with_sync_redis do |conn|
|
18
|
+
conn.blocking_call(false, 'BRPOP', *config.sorted_lists, TIMEOUT)
|
19
|
+
end
|
20
|
+
|
21
|
+
if @done
|
22
|
+
requeue(q, args) unless q.nil?
|
23
|
+
return :done
|
24
|
+
end
|
25
|
+
|
26
|
+
job(q, args) unless q.nil?
|
27
|
+
end
|
28
|
+
|
29
|
+
def suspend
|
30
|
+
@done = true
|
31
|
+
end
|
32
|
+
|
33
|
+
private
|
34
|
+
|
35
|
+
def requeue(queue, args)
|
36
|
+
config.with_sync_redis do |conn|
|
37
|
+
conn.call('RPUSH', queue, args)
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
def job(queue, args)
|
42
|
+
CurrentJob.new(queue, args)
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
data/lib/jiggler/cleaner.rb
CHANGED
@@ -108,15 +108,24 @@ module Jiggler
|
|
108
108
|
def prn_dead_set(conn)
|
109
109
|
conn.call('DEL', config.dead_set)
|
110
110
|
end
|
111
|
-
|
111
|
+
|
112
|
+
# it deletes in_progress queues as well
|
112
113
|
def prn_all_queues(conn)
|
113
|
-
|
114
|
-
|
114
|
+
cursor = '0'
|
115
|
+
loop do
|
116
|
+
cursor, queues = conn.call('SCAN', cursor, 'MATCH', config.queue_scan_key)
|
117
|
+
conn.call('DEL', *queues) unless queues.empty?
|
118
|
+
break if cursor == '0'
|
119
|
+
end
|
115
120
|
end
|
116
121
|
|
117
122
|
def prn_all_processes(conn)
|
118
|
-
|
119
|
-
|
123
|
+
cursor = '0'
|
124
|
+
loop do
|
125
|
+
cursor, processes = conn.call('SCAN', cursor, 'MATCH', config.process_scan_key)
|
126
|
+
conn.call('DEL', *processes) unless processes.empty?
|
127
|
+
break if cursor == '0'
|
128
|
+
end
|
120
129
|
end
|
121
130
|
|
122
131
|
def prn_failures_counter(conn)
|
data/lib/jiggler/cli.rb
CHANGED
@@ -3,10 +3,11 @@
|
|
3
3
|
require 'singleton'
|
4
4
|
require 'optparse'
|
5
5
|
require 'yaml'
|
6
|
-
require '
|
6
|
+
require 'erb'
|
7
7
|
require 'async'
|
8
8
|
require 'async/io/trap'
|
9
9
|
require 'async/pool'
|
10
|
+
require 'debug'
|
10
11
|
|
11
12
|
module Jiggler
|
12
13
|
class CLI
|
@@ -45,7 +46,7 @@ module Jiggler
|
|
45
46
|
def start
|
46
47
|
return unless ping_redis
|
47
48
|
@cond = Async::Condition.new
|
48
|
-
Async do
|
49
|
+
Async do |task|
|
49
50
|
setup_signal_handlers
|
50
51
|
patch_scheduler
|
51
52
|
@launcher = Launcher.new(config)
|
data/lib/jiggler/config.rb
CHANGED
@@ -25,11 +25,15 @@ module Jiggler
|
|
25
25
|
stats_interval: 10,
|
26
26
|
poller_enabled: true,
|
27
27
|
poll_interval: 5,
|
28
|
+
# used in scheduled/requeuer
|
29
|
+
in_process_interval: 120,
|
28
30
|
dead_timeout: 180 * 24 * 60 * 60, # 6 months in seconds
|
31
|
+
mode: :at_least_once,
|
29
32
|
# client settings
|
30
33
|
client_concurrency: 10,
|
31
34
|
client_redis_pool: nil,
|
32
|
-
|
35
|
+
fetchers_concurrency: 1,
|
36
|
+
client_async: false
|
33
37
|
}
|
34
38
|
|
35
39
|
def initialize(options = {})
|
@@ -80,11 +84,42 @@ module Jiggler
|
|
80
84
|
@queue_scan_key ||= "#{queue_prefix}*"
|
81
85
|
end
|
82
86
|
|
83
|
-
def
|
84
|
-
@
|
85
|
-
|
87
|
+
def at_least_once?
|
88
|
+
@options[:mode] == :at_least_once
|
89
|
+
end
|
90
|
+
|
91
|
+
def queues_data
|
92
|
+
@queues_data ||= begin
|
93
|
+
queues = {}
|
94
|
+
|
95
|
+
@options[:queues].each do |queue|
|
96
|
+
name, priority = queue
|
97
|
+
# by default all queues have the same priority
|
98
|
+
priority ||= 0
|
99
|
+
|
100
|
+
queues[name] = {
|
101
|
+
priority: priority,
|
102
|
+
# list is a redis list key for a queue
|
103
|
+
list: "#{QUEUE_PREFIX}#{name}",
|
104
|
+
}
|
105
|
+
end
|
106
|
+
|
107
|
+
queues
|
86
108
|
end
|
87
109
|
end
|
110
|
+
|
111
|
+
# sort in descending order (higher priority first)
|
112
|
+
def sorted_queues_data
|
113
|
+
@sorted_queues_data ||= queues_data.sort_by { |_, v| -v[:priority] }
|
114
|
+
end
|
115
|
+
|
116
|
+
def sorted_lists
|
117
|
+
@sorted_lists ||= sorted_queues_data.map { |_, v| v[:list] }
|
118
|
+
end
|
119
|
+
|
120
|
+
def sorted_queues
|
121
|
+
@sorted_queues ||= sorted_queues_data.map { |k, _| k }
|
122
|
+
end
|
88
123
|
|
89
124
|
def with_async_redis
|
90
125
|
Async do
|
@@ -109,6 +144,15 @@ module Jiggler
|
|
109
144
|
:redis_url
|
110
145
|
)
|
111
146
|
|
147
|
+
if at_least_once?
|
148
|
+
# for acknowledgers
|
149
|
+
opts[:concurrency] *= 2
|
150
|
+
# for queue fetchers
|
151
|
+
opts[:concurrency] += @options[:fetchers_concurrency] * sorted_queues.count
|
152
|
+
# extra poller task to cleanup leftover in-process queues
|
153
|
+
opts[:concurrency] += 1
|
154
|
+
end
|
155
|
+
|
112
156
|
opts[:concurrency] += 2 # monitor + safety margin
|
113
157
|
opts[:concurrency] += 1 if @options[:poller_enabled]
|
114
158
|
opts[:async] = true
|