wurk 1.12.0 → 1.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +2 -2
- data/config/routes.rb +8 -0
- data/exe/sidekiq +8 -0
- data/exe/wurk +1 -1
- data/lib/generators/wurk/install/install_generator.rb +56 -0
- data/lib/wurk/boot_checks.rb +80 -0
- data/lib/wurk/cli.rb +55 -17
- data/lib/wurk/configuration.rb +27 -6
- data/lib/wurk/fetcher/orphan_reclaim.rb +48 -11
- data/lib/wurk/{middleware → fetcher}/poison_pill.rb +3 -1
- data/lib/wurk/fetcher/reaper.rb +28 -9
- data/lib/wurk/fetcher/reliable.rb +4 -5
- data/lib/wurk/fetcher/unit_of_work.rb +1 -1
- data/lib/wurk/job_logger.rb +4 -2
- data/lib/wurk/job_util.rb +14 -0
- data/lib/wurk/launcher.rb +30 -3
- data/lib/wurk/leader.rb +1 -8
- data/lib/wurk/manager.rb +27 -1
- data/lib/wurk/middleware/timeout.rb +6 -6
- data/lib/wurk/preflight/app_scan.rb +182 -0
- data/lib/wurk/preflight/command.rb +21 -5
- data/lib/wurk/preflight/redis_guard.rb +98 -0
- data/lib/wurk/preflight.rb +9 -5
- data/lib/wurk/rails_boot.rb +7 -2
- data/lib/wurk/redis_options.rb +5 -0
- data/lib/wurk/redis_pool.rb +24 -1
- data/lib/wurk/sentry.rb +11 -0
- data/lib/wurk/swarm.rb +63 -14
- data/lib/wurk/version.rb +1 -1
- data/lib/wurk/web/rack_app.rb +8 -6
- data/lib/wurk/web/stats_routes.rb +93 -0
- data/lib/wurk.rb +1 -1
- data/vendor/assets/dashboard/wurk-manifest.json +2 -2
- metadata +11 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b0817941e1b8f3ff6a7195567e31ac9d22c5039c338d708f9c329ff87f0bbcec
|
|
4
|
+
data.tar.gz: 966125b576e3a775f868ad39224a70ffe6baddf6cbd35eb8b76fc81d63e6b762
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c265be3c09774f76f6a9d560a59434a6800d56c06eadd471c6391a1cad589eb709ba51fdfbf672c14c5ef39c482d2e40a24a9a3a9ad8861d917a2bb5daa01d14
|
|
7
|
+
data.tar.gz: 998f43d63f710cd96b6e413fe20b6c4783149113c2bc1128d74afa3c33a225e75189b65d2515b4cd8351b8d14f02bfd4bfc390a164d9c01914147161e1532559
|
data/README.md
CHANGED
|
@@ -303,9 +303,9 @@ Wurk is MIT and stays that way. If what you need is a commercial support contrac
|
|
|
303
303
|
+ gem "wurk"
|
|
304
304
|
```
|
|
305
305
|
|
|
306
|
-
`bundle install && restart`. Wurk reads and writes the same Redis schema, so there is no data migration: Wurk picks up the queues, schedules, retries, batches and periodic loops Sidekiq left behind. Running Sidekiq and Wurk workers against one Redis *at the same time* is not tested yet, so a live production move is a **drain cutover** (stop Sidekiq's workers, confirm nothing is in flight, start Wurk), with a matching rollback; the [production cutover](https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md#9-production-cutover) section has every step and command. The upstream suites of sidekiq-cron, sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq run against Wurk in the [`ecosystem` CI job](https://github.com/developerz-ai/wurk/blob/main/.github/workflows/ecosystem.yml) (see [`test/ecosystem/`](https://github.com/developerz-ai/wurk/tree/main/test/ecosystem)). sidekiq-scheduler, sidekiq-status and sidekiq-throttled are target additions tracked in [docs/idea/14-ecosystem-compat.md](https://github.com/developerz-ai/wurk/blob/main/docs/idea/14-ecosystem-compat.md).
|
|
306
|
+
`bundle install && restart`. The worker command stays as it was: Wurk ships `sidekiq` and `sidekiqswarm` executables, so `bundle exec sidekiq -C config/sidekiq.yml` in a Procfile or manifest keeps working. Wurk reads and writes the same Redis schema, so there is no data migration: Wurk picks up the queues, schedules, retries, batches and periodic loops Sidekiq left behind. Running Sidekiq and Wurk workers against one Redis *at the same time* is not tested yet, so a live production move is a **drain cutover** (stop Sidekiq's workers, confirm nothing is in flight, start Wurk), with a matching rollback; the [production cutover](https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md#9-production-cutover) section has every step and command. The upstream suites of sidekiq-cron, sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq run against Wurk in the [`ecosystem` CI job](https://github.com/developerz-ai/wurk/blob/main/.github/workflows/ecosystem.yml) (see [`test/ecosystem/`](https://github.com/developerz-ai/wurk/tree/main/test/ecosystem)). sidekiq-scheduler, sidekiq-status and sidekiq-throttled are target additions tracked in [docs/idea/14-ecosystem-compat.md](https://github.com/developerz-ai/wurk/blob/main/docs/idea/14-ecosystem-compat.md).
|
|
307
307
|
|
|
308
|
-
**Keeping a `sidekiq-*` add-on gem?** Those gems declare `add_dependency "sidekiq"`, which would reinstall real Sidekiq next to Wurk. Satisfy it with Wurk's git-only [`sidekiq` shim gem](https://github.com/developerz-ai/wurk/tree/main/ecosystem/sidekiq-shim) instead: `gem "sidekiq", github: "developerz-ai/wurk", glob: "ecosystem/sidekiq-shim/*.gemspec"`. Only sidekiq-cron, sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq are exercised that way, by the `ecosystem` workflow; treat the others as untested.
|
|
308
|
+
**Keeping a `sidekiq-*` add-on gem?** Those gems declare `add_dependency "sidekiq"`, which would reinstall real Sidekiq next to Wurk. Satisfy it with Wurk's git-only [`sidekiq` shim gem](https://github.com/developerz-ai/wurk/tree/main/ecosystem/sidekiq-shim) instead, pinned to the tag of the Wurk release you run: `gem "sidekiq", github: "developerz-ai/wurk", tag: "v1.13.0", glob: "ecosystem/sidekiq-shim/*.gemspec"`. Only sidekiq-cron, sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq are exercised that way, by the `ecosystem` workflow; treat the others as untested.
|
|
309
309
|
|
|
310
310
|
Full walkthrough — config side-by-side, the Redis key/`sidekiq_options` mapping, known incompatibilities, moving sidekiq-cron schedules over (`rake wurk:import:cron`), and the production cutover and rollback playbook: **[docs/migrate-from-sidekiq.md](https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md)**. On the night, keep the **[incident runbook](https://github.com/developerz-ai/wurk/blob/main/docs/runbook.md)** open.
|
|
311
311
|
|
data/config/routes.rb
CHANGED
|
@@ -1,6 +1,14 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
Wurk::Engine.routes.draw do
|
|
4
|
+
# Sidekiq Web's machine-readable routes (free §25.4), upstream's shapes, for
|
|
5
|
+
# uptime checks and scripts. `HEAD /` is declared before `root`, which would
|
|
6
|
+
# otherwise answer it with the SPA shell and never touch Redis.
|
|
7
|
+
match '/', to: Wurk::Web::StatsRoutes, via: :head
|
|
8
|
+
get 'stats', to: Wurk::Web::StatsRoutes, format: false
|
|
9
|
+
get 'stats/queues', to: Wurk::Web::StatsRoutes, format: false
|
|
10
|
+
get 'dashboard/stats', to: Wurk::Web::StatsRoutes, format: false
|
|
11
|
+
|
|
4
12
|
root to: 'dashboard#index'
|
|
5
13
|
|
|
6
14
|
# JSON APIs consumed by the SPA. Nested under whatever mount the host chose.
|
data/exe/sidekiq
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
# Drop-in alias for `wurk`. Sidekiq apps start workers with `bundle exec sidekiq
|
|
5
|
+
# -C config/sidekiq.yml` in a Procfile, systemd unit or k8s manifest; this keeps
|
|
6
|
+
# that exact command working after a one-line gem swap. Spec:
|
|
7
|
+
# docs/target/sidekiq-free.md §21.
|
|
8
|
+
load File.expand_path('wurk', __dir__)
|
data/exe/wurk
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
# Usage: `exe/wurk -C config/wurk.yml` — worker: one process, thread pool
|
|
6
6
|
# `exe/wurk api --port 7433` — machine HTTP API, runs no jobs
|
|
7
7
|
# `exe/wurk preflight` / `verify` — read-only cutover checks, run no jobs
|
|
8
|
-
# `exe/wurk boot-check -r .` — boot the app as wurkswarm would, then exit (CI)
|
|
8
|
+
# `exe/wurk boot-check -r .` — boot the app as wurkswarm (or `--runner wurk`) would, then exit (CI)
|
|
9
9
|
|
|
10
10
|
$stdout.sync = true
|
|
11
11
|
$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
|
|
@@ -1,15 +1,23 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require 'rails/generators'
|
|
4
|
+
require 'bundler'
|
|
5
|
+
require 'wurk/version'
|
|
4
6
|
|
|
5
7
|
module Wurk
|
|
6
8
|
module Generators
|
|
7
9
|
# `bin/rails g wurk:install`
|
|
8
10
|
# Writes a config initializer and adds a commented-out mount line to routes.
|
|
9
11
|
# The host app picks the mount path; the generator suggests /wurk.
|
|
12
|
+
# Reads Gemfile.lock and Procfile to print what a Sidekiq app must change;
|
|
13
|
+
# it edits neither.
|
|
10
14
|
class InstallGenerator < ::Rails::Generators::Base
|
|
11
15
|
source_root File.expand_path('templates', __dir__)
|
|
12
16
|
|
|
17
|
+
# Sidekiq's own tiers depend on `sidekiq` too, but the swap removes them.
|
|
18
|
+
SIDEKIQ_TIERS = %w[sidekiq sidekiq-pro sidekiq-ent].freeze
|
|
19
|
+
SIDEKIQ_COMMAND = /\bsidekiq(?:swarm)?\b/
|
|
20
|
+
|
|
13
21
|
def copy_initializer
|
|
14
22
|
template 'wurk.rb', 'config/initializers/wurk.rb'
|
|
15
23
|
end
|
|
@@ -18,6 +26,27 @@ module Wurk
|
|
|
18
26
|
route(%(# mount Wurk::Engine => "/wurk" # uncomment to expose the Wurk dashboard))
|
|
19
27
|
end
|
|
20
28
|
|
|
29
|
+
def show_shim_line
|
|
30
|
+
dependents = sidekiq_dependents
|
|
31
|
+
return if dependents.empty?
|
|
32
|
+
|
|
33
|
+
say <<~SHIM
|
|
34
|
+
|
|
35
|
+
#{dependents.join(', ')} #{dependents.size == 1 ? 'depends' : 'depend'} on the sidekiq gem, which would load real Sidekiq next to Wurk.
|
|
36
|
+
Keep #{dependents.size == 1 ? 'it' : 'them'} by adding the shim to the Gemfile, in place of gem "sidekiq":
|
|
37
|
+
gem "sidekiq", github: "developerz-ai/wurk", tag: "v#{Wurk::VERSION}", glob: "ecosystem/sidekiq-shim/*.gemspec"
|
|
38
|
+
SHIM
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def show_procfile_runner
|
|
42
|
+
lines = procfile_sidekiq_lines
|
|
43
|
+
return if lines.empty?
|
|
44
|
+
|
|
45
|
+
say "\nProcfile worker lines keep working: wurk ships the sidekiq and sidekiqswarm commands."
|
|
46
|
+
lines.each { |line| say " #{line}" }
|
|
47
|
+
say 'For one process per core, run `bundle exec wurkswarm` with the same flags.'
|
|
48
|
+
end
|
|
49
|
+
|
|
21
50
|
def show_next_steps
|
|
22
51
|
say <<~STEPS
|
|
23
52
|
|
|
@@ -29,6 +58,33 @@ module Wurk
|
|
|
29
58
|
docs/authentication.md in the wurk gem's repository
|
|
30
59
|
STEPS
|
|
31
60
|
end
|
|
61
|
+
|
|
62
|
+
private
|
|
63
|
+
|
|
64
|
+
def sidekiq_dependents
|
|
65
|
+
specs = locked_specs
|
|
66
|
+
# The shim is the `sidekiq` spec that depends on wurk: already in place.
|
|
67
|
+
return [] if specs.any? { |spec| spec.name == 'sidekiq' && depends_on?(spec, 'wurk') }
|
|
68
|
+
|
|
69
|
+
specs.select { |spec| depends_on?(spec, 'sidekiq') }.map(&:name).uniq
|
|
70
|
+
.reject { |name| SIDEKIQ_TIERS.include?(name) }.sort
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def locked_specs
|
|
74
|
+
lockfile = File.join(destination_root, 'Gemfile.lock')
|
|
75
|
+
File.file?(lockfile) ? ::Bundler::LockfileParser.new(File.read(lockfile)).specs : []
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
def depends_on?(spec, name)
|
|
79
|
+
spec.dependencies.any? { |dep| dep.name == name }
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
def procfile_sidekiq_lines
|
|
83
|
+
procfile = File.join(destination_root, 'Procfile')
|
|
84
|
+
return [] unless File.file?(procfile)
|
|
85
|
+
|
|
86
|
+
File.readlines(procfile, chomp: true).grep(SIDEKIQ_COMMAND)
|
|
87
|
+
end
|
|
32
88
|
end
|
|
33
89
|
end
|
|
34
90
|
end
|
data/lib/wurk/boot_checks.rb
CHANGED
|
@@ -25,9 +25,11 @@ module Wurk
|
|
|
25
25
|
if info
|
|
26
26
|
log_backend(config.logger, info)
|
|
27
27
|
warn_on_foreign_processes(config)
|
|
28
|
+
warn_on_unconsumed_framework_queues(config)
|
|
28
29
|
end
|
|
29
30
|
warn_on_ignored_locks(config.logger)
|
|
30
31
|
warn_on_ignored_unique(config.logger)
|
|
32
|
+
warn_on_client_config_only(config)
|
|
31
33
|
rescue StandardError => e
|
|
32
34
|
config.logger.warn { "wurk boot checks skipped: #{e.class}: #{e.message}" }
|
|
33
35
|
end
|
|
@@ -97,6 +99,84 @@ module Wurk
|
|
|
97
99
|
end
|
|
98
100
|
end
|
|
99
101
|
|
|
102
|
+
# Stock Sidekiq skips a configure_client block in a server without a word,
|
|
103
|
+
# and so does Configuration: an app normally configures both roles. A
|
|
104
|
+
# server where only client blocks were written ran none of them, so the
|
|
105
|
+
# Redis settings, middleware and handlers they hold are all absent here.
|
|
106
|
+
def warn_on_client_config_only(config)
|
|
107
|
+
return unless config.client_config_only?
|
|
108
|
+
|
|
109
|
+
config.logger.warn do
|
|
110
|
+
'This is a server process, so its configure_client blocks were skipped, and no configure_server block ran: ' \
|
|
111
|
+
'whatever those blocks set (config.redis, middleware, handlers) does not apply to these workers. Set it ' \
|
|
112
|
+
'in a configure_server block too, or outside both.'
|
|
113
|
+
end
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
# The topology only shapes what a swarm forks; `wurk` and an embedded
|
|
117
|
+
# worker run one process with the capsules' threads, so a declared one
|
|
118
|
+
# silently does nothing there.
|
|
119
|
+
def warn_on_ignored_topology(config, runner)
|
|
120
|
+
return unless config.topology_assigned?
|
|
121
|
+
|
|
122
|
+
config.logger.warn do
|
|
123
|
+
"config.topology is set, but #{runner} runs a single process and ignores it: its slots' queues and " \
|
|
124
|
+
'concurrency are not used here. Run wurkswarm to fork it, or set the queues and concurrency on the capsules.'
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
# Rails enqueues its own jobs (mail delivery, Active Storage analysis and
|
|
129
|
+
# purge, Action Mailbox routing) to queues an app never names itself, so
|
|
130
|
+
# they are the ones that go unlistened. Read from Redis, so only with it.
|
|
131
|
+
def warn_on_unconsumed_framework_queues(config)
|
|
132
|
+
wanted = framework_queues
|
|
133
|
+
return if wanted.empty?
|
|
134
|
+
|
|
135
|
+
listened = own_queues(config) | config.redis(idempotent: true) { |conn| QueueConsumers.counts(conn).keys }
|
|
136
|
+
missing = wanted.except(*listened)
|
|
137
|
+
return if missing.empty?
|
|
138
|
+
|
|
139
|
+
config.logger.warn do
|
|
140
|
+
"Rails enqueues to #{listing(missing.map { |queue, what| "#{queue} (#{what})" })}, which neither this " \
|
|
141
|
+
'process nor any live one listens to, so those jobs wait forever. Add the queues to a worker.'
|
|
142
|
+
end
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
# `{ queue => what Rails sends there }`, for apps whose ActiveJob adapter is Wurk.
|
|
146
|
+
def framework_queues
|
|
147
|
+
return {} unless wurk_active_job?
|
|
148
|
+
|
|
149
|
+
named = {}
|
|
150
|
+
named['mail delivery'] = ::ActionMailer::Base.deliver_later_queue_name if defined?(::ActionMailer::Base)
|
|
151
|
+
named.merge!(framework_queue_set('Active Storage', defined?(::ActiveStorage) && ::ActiveStorage))
|
|
152
|
+
named.merge!(framework_queue_set('Action Mailbox', defined?(::ActionMailbox) && ::ActionMailbox))
|
|
153
|
+
named.compact.to_h { |what, queue| [active_job_queue_name(queue), what] }
|
|
154
|
+
end
|
|
155
|
+
|
|
156
|
+
def framework_queue_set(framework, mod)
|
|
157
|
+
return {} unless mod.respond_to?(:queues) && mod.queues.respond_to?(:to_h)
|
|
158
|
+
|
|
159
|
+
mod.queues.to_h.transform_keys { |kind| "#{framework} #{kind}" }
|
|
160
|
+
end
|
|
161
|
+
|
|
162
|
+
def wurk_active_job?
|
|
163
|
+
return false unless defined?(::ActiveJob::Base)
|
|
164
|
+
|
|
165
|
+
::ActiveJob::Base.queue_adapter.class.name.to_s.match?(/Wurk|Sidekiq/)
|
|
166
|
+
end
|
|
167
|
+
|
|
168
|
+
def active_job_queue_name(queue)
|
|
169
|
+
prefix = ::ActiveJob::Base.queue_name_prefix
|
|
170
|
+
prefix ? "#{prefix}#{::ActiveJob::Base.queue_name_delimiter}#{queue}" : queue.to_s
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
def own_queues(config)
|
|
174
|
+
queues = config.capsules.each_value.flat_map { |cap| cap.weights.keys }
|
|
175
|
+
return queues unless config.topology_assigned?
|
|
176
|
+
|
|
177
|
+
queues | config.topology.slots.flat_map { |slot| Array(slot.queues).map { |entry| entry.to_s.split(',').first } }
|
|
178
|
+
end
|
|
179
|
+
|
|
100
180
|
def locked_job_classes = job_classes_setting(%w[lock])
|
|
101
181
|
|
|
102
182
|
# Workers, and ActiveJob classes that take `sidekiq_options` (Wurk::Job::Options).
|
data/lib/wurk/cli.rb
CHANGED
|
@@ -112,8 +112,10 @@ module Wurk
|
|
|
112
112
|
].freeze
|
|
113
113
|
|
|
114
114
|
BOOT_CHECK_OPTION_FLAGS = [
|
|
115
|
-
['-R', '--redis', :boot_check_redis, 'Also read Redis: version, eviction policy, live peers (read-only)']
|
|
115
|
+
['-R', '--redis', :boot_check_redis, 'Also read Redis: version, eviction policy, live peers (read-only)'],
|
|
116
|
+
['-n', '--runner NAME', :boot_check_runner, 'Boot as wurkswarm (default) or wurk, the runner you deploy']
|
|
116
117
|
].freeze
|
|
118
|
+
BOOT_CHECK_RUNNERS = %w[wurkswarm wurk].freeze
|
|
117
119
|
|
|
118
120
|
HEALTH_OPTION_FLAGS = [
|
|
119
121
|
['-w', '--window SECONDS', :health_window, 'A heartbeat older than this does not count (default 60)', :to_i],
|
|
@@ -123,7 +125,7 @@ module Wurk
|
|
|
123
125
|
].freeze
|
|
124
126
|
|
|
125
127
|
HELP_COMMANDS = "\nCommands:\n " \
|
|
126
|
-
"boot-check Boot
|
|
128
|
+
"boot-check Boot as wurkswarm (or --runner wurk) would, then exit; no fork, no fetch (CI)\n " \
|
|
127
129
|
"health Exit 0 if this host's workers are heartbeating\n " \
|
|
128
130
|
"preflight Read a live Redis: what will carry over to Wurk (read-only)\n " \
|
|
129
131
|
"verify Post-cutover check of a live fleet (read-only)\n\nOptions:"
|
|
@@ -215,6 +217,7 @@ module Wurk
|
|
|
215
217
|
validate_redis!
|
|
216
218
|
validate_pool_sizes!
|
|
217
219
|
BootChecks.run(@config, redis_info)
|
|
220
|
+
BootChecks.warn_on_ignored_topology(@config, 'wurk')
|
|
218
221
|
BootChecks.log_process_role(@config, 'wurk')
|
|
219
222
|
@config[:identity] = identity
|
|
220
223
|
# Force lazy server-middleware chain so worker threads don't race
|
|
@@ -273,25 +276,25 @@ module Wurk
|
|
|
273
276
|
exit(1) if @swarm.crash_looped?
|
|
274
277
|
end
|
|
275
278
|
|
|
276
|
-
# `wurk boot-check` —
|
|
277
|
-
#
|
|
278
|
-
#
|
|
279
|
-
#
|
|
280
|
-
#
|
|
279
|
+
# `wurk boot-check` — the boot of the runner named by --runner up to where it
|
|
280
|
+
# would fork or fetch: #run_swarm's (the default) or #run's, with the same
|
|
281
|
+
# require order, config file and checks, plus what a worker reaches only later
|
|
282
|
+
# (every constant, the routes), so a CI step finds what would crash-loop a
|
|
283
|
+
# deploy. `configure_server` blocks run, since their load errors are the point;
|
|
284
|
+
# no lifecycle event fires (`:startup` included) and periodic loops are never
|
|
285
|
+
# applied. Wurk's Redis pools refuse every command without --redis and every
|
|
286
|
+
# write with it (Preflight::RedisGuard), so boot-time app code cannot touch a
|
|
287
|
+
# real server. Returns the exit code; a boot that raises is the binary's
|
|
281
288
|
# failure report.
|
|
282
289
|
def run_boot_check(boot_app: true)
|
|
290
|
+
require_relative 'preflight/redis_guard'
|
|
291
|
+
Preflight::RedisGuard.install(@config[:boot_check_redis] ? :read_only : :closed, label: 'wurk boot-check')
|
|
283
292
|
enter_server_mode
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
boot_swarm_application
|
|
287
|
-
eager_load_rails
|
|
288
|
-
draw_routes
|
|
289
|
-
end
|
|
290
|
-
validate_pool_sizes!
|
|
291
|
-
BootChecks.run(@config, boot_check_redis_info)
|
|
292
|
-
log_swarm_role
|
|
293
|
-
logger.info { "wurk boot-check: #{@config[:require]} booted as wurkswarm would" }
|
|
293
|
+
boot_check_runner == 'wurk' ? boot_check_as_wurk(boot_app) : boot_check_as_swarm(boot_app)
|
|
294
|
+
logger.info { "wurk boot-check: #{@config[:require]} booted as #{boot_check_runner} would" }
|
|
294
295
|
0
|
|
296
|
+
ensure
|
|
297
|
+
Preflight::RedisGuard.uninstall
|
|
295
298
|
end
|
|
296
299
|
|
|
297
300
|
# `wurk health` — see Wurk::HealthProbe. Boots nothing; returns the exit code.
|
|
@@ -467,6 +470,36 @@ module Wurk
|
|
|
467
470
|
end
|
|
468
471
|
end
|
|
469
472
|
|
|
473
|
+
def boot_check_runner = @config[:boot_check_runner] || BOOT_CHECK_RUNNERS.first
|
|
474
|
+
|
|
475
|
+
def boot_check_as_swarm(boot_app)
|
|
476
|
+
Health.options_from_env
|
|
477
|
+
if boot_app
|
|
478
|
+
boot_swarm_application
|
|
479
|
+
boot_check_reach
|
|
480
|
+
end
|
|
481
|
+
validate_pool_sizes!
|
|
482
|
+
BootChecks.run(@config, boot_check_redis_info)
|
|
483
|
+
log_swarm_role
|
|
484
|
+
end
|
|
485
|
+
|
|
486
|
+
def boot_check_as_wurk(boot_app)
|
|
487
|
+
if boot_app
|
|
488
|
+
boot_application
|
|
489
|
+
boot_check_reach
|
|
490
|
+
end
|
|
491
|
+
validate_pool_sizes!
|
|
492
|
+
BootChecks.run(@config, boot_check_redis_info)
|
|
493
|
+
BootChecks.warn_on_ignored_topology(@config, 'wurk')
|
|
494
|
+
BootChecks.log_process_role(@config, 'boot-check (as wurk)')
|
|
495
|
+
end
|
|
496
|
+
|
|
497
|
+
# What a worker loads only later: every constant, and the routes.
|
|
498
|
+
def boot_check_reach
|
|
499
|
+
eager_load_rails
|
|
500
|
+
draw_routes
|
|
501
|
+
end
|
|
502
|
+
|
|
470
503
|
def boot_check_redis_info
|
|
471
504
|
return unless @config[:boot_check_redis]
|
|
472
505
|
|
|
@@ -809,6 +842,11 @@ module Wurk
|
|
|
809
842
|
reason = require_path_problem(@config[:require].to_s)
|
|
810
843
|
return print_help_and_die(reason) if reason
|
|
811
844
|
|
|
845
|
+
runner = @config[:boot_check_runner]
|
|
846
|
+
if runner && !BOOT_CHECK_RUNNERS.include?(runner)
|
|
847
|
+
raise StartupError, "--runner: #{runner.inspect} is not one of #{BOOT_CHECK_RUNNERS.join(', ')}"
|
|
848
|
+
end
|
|
849
|
+
|
|
812
850
|
%i[concurrency timeout].each do |opt|
|
|
813
851
|
raise StartupError, "#{opt}: #{@config[opt]} is not a valid value" if @config[opt].to_i <= 0
|
|
814
852
|
end
|
data/lib/wurk/configuration.rb
CHANGED
|
@@ -877,7 +877,7 @@ module Wurk
|
|
|
877
877
|
# and the like silently absent from a process that does run workers.
|
|
878
878
|
def configure_server(&block)
|
|
879
879
|
return unless block
|
|
880
|
-
return
|
|
880
|
+
return run_server_block(block) if server?
|
|
881
881
|
|
|
882
882
|
@deferred_server_blocks << block
|
|
883
883
|
return if @configure_server_skip_warned
|
|
@@ -889,23 +889,34 @@ module Wurk
|
|
|
889
889
|
|
|
890
890
|
def enter_server_mode
|
|
891
891
|
self[:server] = true
|
|
892
|
-
@deferred_server_blocks.shift
|
|
892
|
+
run_server_block(@deferred_server_blocks.shift) until @deferred_server_blocks.empty?
|
|
893
893
|
end
|
|
894
894
|
|
|
895
895
|
# A web process that also boots the swarm (Wurk::RailsBoot) is flagged
|
|
896
896
|
# server, yet every enqueue it makes from a request is client work, so
|
|
897
|
-
# `client_in_server` lets its configure_client blocks run too.
|
|
898
|
-
#
|
|
897
|
+
# `client_in_server` lets its configure_client blocks run too.
|
|
898
|
+
#
|
|
899
|
+
# In a server, skipping the client block is the Sidekiq contract and the
|
|
900
|
+
# usual case: an app configures both roles. So the skip itself is DEBUG,
|
|
901
|
+
# and BootChecks warns at boot only for the case worth one, a server
|
|
902
|
+
# process in which no configure_server block ran at all (#816).
|
|
899
903
|
def configure_client
|
|
900
904
|
return unless block_given?
|
|
901
905
|
return yield self if !server? || @options[:client_in_server]
|
|
906
|
+
|
|
902
907
|
return if @configure_client_skip_warned
|
|
903
908
|
|
|
904
909
|
@configure_client_skip_warned = true
|
|
905
|
-
logger.
|
|
910
|
+
logger.debug { 'configure_client block skipped: server? is true and client_in_server is false or nil' }
|
|
906
911
|
nil
|
|
907
912
|
end
|
|
908
913
|
|
|
914
|
+
# True once a configure_client block was skipped and no configure_server
|
|
915
|
+
# block ran: the server got none of the settings the app made for a role.
|
|
916
|
+
def client_config_only?
|
|
917
|
+
@configure_client_skip_warned == true && !@configure_server_ran
|
|
918
|
+
end
|
|
919
|
+
|
|
909
920
|
def server?
|
|
910
921
|
@options[:server] == true
|
|
911
922
|
end
|
|
@@ -920,9 +931,14 @@ module Wurk
|
|
|
920
931
|
|
|
921
932
|
def topology=(value)
|
|
922
933
|
guard_frozen!
|
|
934
|
+
@topology_assigned = true
|
|
923
935
|
@topology = value
|
|
924
936
|
end
|
|
925
937
|
|
|
938
|
+
# Whether the app declared a topology, as opposed to the default one
|
|
939
|
+
# #topology builds. Only a swarm reads it.
|
|
940
|
+
def topology_assigned? = @topology_assigned == true
|
|
941
|
+
|
|
926
942
|
# Memory-based child recycling (Sidekiq Ent §7.5): the swarm parent TERMs
|
|
927
943
|
# (and respawns) any child whose RSS exceeds this many MB. Set in code or
|
|
928
944
|
# via SIDEKIQ_MAXMEM_MB (WURK_MAXMEM_MB is the native alias); an explicit
|
|
@@ -1001,6 +1017,11 @@ module Wurk
|
|
|
1001
1017
|
|
|
1002
1018
|
private
|
|
1003
1019
|
|
|
1020
|
+
def run_server_block(block)
|
|
1021
|
+
@configure_server_ran = true
|
|
1022
|
+
block.call(self)
|
|
1023
|
+
end
|
|
1024
|
+
|
|
1004
1025
|
# Pools read the hash once, when built, so the later assignment silently does
|
|
1005
1026
|
# nothing to this process's connections until something rebuilds them.
|
|
1006
1027
|
def warn_on_late_redis_assignment
|
|
@@ -1079,7 +1100,7 @@ module Wurk
|
|
|
1079
1100
|
# as throttling or memory pressure.
|
|
1080
1101
|
def default_child_count
|
|
1081
1102
|
count, source = child_count_and_source
|
|
1082
|
-
logger.info { "Wurk swarm:
|
|
1103
|
+
logger.info { "Wurk swarm size: #{count} #{count == 1 ? 'child' : 'children'} (#{source})" }
|
|
1083
1104
|
count
|
|
1084
1105
|
end
|
|
1085
1106
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative '../keys'
|
|
4
4
|
require_relative '../lua'
|
|
5
|
-
require_relative '
|
|
5
|
+
require_relative 'poison_pill'
|
|
6
6
|
require_relative '../metrics/process_counters'
|
|
7
7
|
require_relative 'oom_drain'
|
|
8
8
|
|
|
@@ -28,13 +28,15 @@ module Wurk
|
|
|
28
28
|
# Returns true when this call took the payload off the private list, false
|
|
29
29
|
# when it had already left (another reaper, an ACK).
|
|
30
30
|
#
|
|
31
|
-
#
|
|
32
|
-
#
|
|
33
|
-
#
|
|
34
|
-
# the
|
|
31
|
+
# Sent once, never re-sent on a fresh socket: a replay of a script that
|
|
32
|
+
# already ran finds the payload gone and answers "someone else moved it",
|
|
33
|
+
# which skipped the death handlers of a kill (#908), and a list holding two
|
|
34
|
+
# identical copies would move the second. A reply lost after the script ran
|
|
35
|
+
# is settled by reading back what it wrote instead (#settle_lost_reply).
|
|
35
36
|
def call(private_list, public_q, payload)
|
|
36
37
|
job = Middleware::PoisonPill.parse(payload)
|
|
37
|
-
|
|
38
|
+
keys = move_keys(private_list, public_q, job)
|
|
39
|
+
count = moved_count(keys, payload, now)
|
|
38
40
|
return false unless count
|
|
39
41
|
|
|
40
42
|
Metrics::ProcessCounters.increment(:reaper_reclaimed)
|
|
@@ -47,15 +49,50 @@ module Wurk
|
|
|
47
49
|
|
|
48
50
|
private
|
|
49
51
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
52
|
+
def moved_count(keys, payload, score)
|
|
53
|
+
@config.redis { |conn| move(conn, keys, payload, score) }
|
|
54
|
+
rescue RedisClient::ConnectionError => e
|
|
55
|
+
settle_lost_reply(keys, payload, score, e)
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def move_keys(private_list, public_q, job)
|
|
53
59
|
keys = [private_list, public_q, Keys::DEAD]
|
|
54
60
|
jid = job && job['jid']
|
|
55
61
|
keys << Middleware::PoisonPill.counter_key(jid) if jid.is_a?(String) && !jid.empty?
|
|
56
|
-
|
|
62
|
+
keys
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
# Past maxmemory the counter cannot be written, nor the job killed: it goes
|
|
66
|
+
# back uncounted rather than waiting on the memory it would help free.
|
|
67
|
+
def move(conn, keys, payload, score)
|
|
68
|
+
Wurk::Lua::Loader.eval_once(conn, :reclaim_orphan, keys: keys, argv: [payload, THRESHOLD, TTL, score])
|
|
57
69
|
rescue RedisClient::OutOfMemoryError => e
|
|
58
|
-
:uncounted if OomDrain.requeue(conn, e,
|
|
70
|
+
:uncounted if OomDrain.requeue(conn, e, keys[0], keys[1], payload) == 1
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
# The connection dropped around the script, so it ran or it did not. Only a
|
|
74
|
+
# kill needs settling: the job is gone from the private list, so no later
|
|
75
|
+
# sweep would find it, and the death handlers (a batch's `:death`, a unique
|
|
76
|
+
# lock's release) would never fire. The script wrote the dead-set member
|
|
77
|
+
# at the score this call chose, so that exact score proves the kill was
|
|
78
|
+
# this call's. Returns the recovery count for a kill this call made; nil
|
|
79
|
+
# for a requeue or a move someone else made, which need nothing more;
|
|
80
|
+
# raises the original error when the payload is still in the private list,
|
|
81
|
+
# which is the next pass's to move.
|
|
82
|
+
def settle_lost_reply(keys, payload, score, error)
|
|
83
|
+
private_list, _public_q, dead, counter = keys
|
|
84
|
+
still_there, dead_score, count = @config.redis(idempotent: true) do |conn|
|
|
85
|
+
conn.pipelined do |pipe|
|
|
86
|
+
pipe.call('LPOS', private_list, payload, 'RANK', -1)
|
|
87
|
+
pipe.call('ZSCORE', dead, payload)
|
|
88
|
+
pipe.call('GET', counter) if counter
|
|
89
|
+
end
|
|
90
|
+
end
|
|
91
|
+
# Exact on purpose: the same double, written by the script and read back.
|
|
92
|
+
return count.to_i if dead_score && Float(dead_score) == Float(score) # rubocop:disable Lint/FloatComparison
|
|
93
|
+
raise error if still_there
|
|
94
|
+
|
|
95
|
+
nil
|
|
59
96
|
end
|
|
60
97
|
|
|
61
98
|
# DeadSet#kill's score format, so the entry sorts with every other kill.
|
|
@@ -19,7 +19,9 @@ module Wurk
|
|
|
19
19
|
# Counter key TTL is wire-compat with Sidekiq Pro — third-party tooling
|
|
20
20
|
# that watches `super_fetch:recovered:*` expects 72h.
|
|
21
21
|
#
|
|
22
|
-
#
|
|
22
|
+
# Not a chain middleware, so the file lives with the fetcher; the constant
|
|
23
|
+
# keeps its Middleware path because Pro initializers name it. No
|
|
24
|
+
# server-middleware registration: the producer is Fetcher::OrphanReclaim,
|
|
23
25
|
# whose script bumps the counter and takes the requeue-or-kill decision in
|
|
24
26
|
# the same step as the move, then hands the outcome to {recovered!} for the
|
|
25
27
|
# notifications. The consumer is the ACK —
|
data/lib/wurk/fetcher/reaper.rb
CHANGED
|
@@ -9,6 +9,7 @@ require_relative 'orphan_grace'
|
|
|
9
9
|
require_relative 'oom_drain'
|
|
10
10
|
require_relative 'orphan_reclaim'
|
|
11
11
|
require_relative 'self_reconcile'
|
|
12
|
+
require_relative 'reliable'
|
|
12
13
|
|
|
13
14
|
module Wurk
|
|
14
15
|
class Fetcher
|
|
@@ -28,9 +29,10 @@ module Wurk
|
|
|
28
29
|
# ignores a stale `processes` SET member — the SET carries no TTL, so
|
|
29
30
|
# the member lingers after its heartbeat hash expires (60s) until
|
|
30
31
|
# `ProcessSet#cleanup` prunes it. A `kill -9`ed sibling is reclaimed
|
|
31
|
-
# the moment the supervisor reaps it, not 60s later.
|
|
32
|
-
# unrelated process
|
|
33
|
-
#
|
|
32
|
+
# the moment the supervisor reaps it, not 60s later. Pid reuse by an
|
|
33
|
+
# unrelated process reads a dead owner as live; a swarm child does not
|
|
34
|
+
# depend on this check, since its parent drains its lists by exact key
|
|
35
|
+
# the moment it reaps it (#reclaim_exited), before the pid can be reused.
|
|
34
36
|
# * any other incarnation — its pid means nothing in our namespace, so we
|
|
35
37
|
# trust the heartbeat: the owner is alive iff its identity's `info`
|
|
36
38
|
# hash still exists. Such reclaim waits out the 60s heartbeat TTL, the
|
|
@@ -172,13 +174,30 @@ module Wurk
|
|
|
172
174
|
sweep.tap { wake_for_settling }
|
|
173
175
|
end
|
|
174
176
|
|
|
175
|
-
#
|
|
176
|
-
#
|
|
177
|
-
#
|
|
178
|
-
#
|
|
179
|
-
#
|
|
177
|
+
# The sweep a booting process runs, under the same lock as the periodic one:
|
|
178
|
+
# a rolling restart or a deploy of N processes is one keyspace pass, not N.
|
|
179
|
+
# A replacement's sweep is no longer what recovers the swarm child it
|
|
180
|
+
# replaces; the parent did that by exact key when it reaped the child
|
|
181
|
+
# (#reclaim_exited). What this pass still finds is other incarnations'
|
|
182
|
+
# lists, which wait on the heartbeat TTL and the grace either way.
|
|
180
183
|
def boot_reclaim!
|
|
181
|
-
|
|
184
|
+
reap
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
# Drains the private lists of a process known to be dead: a swarm child its
|
|
188
|
+
# parent has just reaped. The parent knows the pid, the queues the child
|
|
189
|
+
# served and, since the nonce is inherited across fork, the rest of each
|
|
190
|
+
# list's name, so neither a SCAN nor a liveness check is needed. One
|
|
191
|
+
# pipelined LLEN finds the lists holding anything (none, after a drained
|
|
192
|
+
# exit) and raises rather than paying the pool's backoff once per list
|
|
193
|
+
# when Redis is away. Returns the jobs moved.
|
|
194
|
+
def reclaim_exited(pid, queue_names)
|
|
195
|
+
lists = queue_names.to_h do |name|
|
|
196
|
+
public_q = Keys.queue(name)
|
|
197
|
+
[Reliable.private_queue_name(public_q, pid: pid), public_q]
|
|
198
|
+
end
|
|
199
|
+
lengths = redis(idempotent: true) { |c| c.pipelined { |p| lists.each_key { |l| p.call('LLEN', l) } } }
|
|
200
|
+
lists.zip(lengths).sum { |(list, public_q), len| len.positive? ? drain(list, public_q) : 0 }
|
|
182
201
|
end
|
|
183
202
|
|
|
184
203
|
# Lists whose grace has run out, whichever sweep started it. Runs on every
|
|
@@ -1,6 +1,5 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require 'socket'
|
|
4
3
|
require_relative '../component'
|
|
5
4
|
require_relative '../pid_cache'
|
|
6
5
|
require_relative '../keys'
|
|
@@ -80,10 +79,10 @@ module Wurk
|
|
|
80
79
|
# reaper's `kill(0)` liveness check would read a dead owner's list as
|
|
81
80
|
# live (jobs stranded) or a live owner's as dead (job run twice). Keys
|
|
82
81
|
# written before the nonce existed stay reclaimable — PrivateListKey
|
|
83
|
-
# accepts both shapes.
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
"#{public_queue}|#{
|
|
82
|
+
# accepts both shapes. `pid:` lets a swarm parent name a reaped child's
|
|
83
|
+
# lists (Reaper#reclaim_exited).
|
|
84
|
+
def self.private_queue_name(public_queue, index = 0, pid: ::Process.pid)
|
|
85
|
+
"#{public_queue}|#{Component.hostname}|#{pid}|#{Component::PROCESS_NONCE}|#{index}"
|
|
87
86
|
end
|
|
88
87
|
|
|
89
88
|
# Guards the generation bump only. Fetchers read the counter without it —
|