wurk 1.11.1 → 1.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +1 -1
- data/exe/wurk +4 -2
- data/lib/generators/wurk/install/templates/wurk.rb +5 -3
- data/lib/wurk/boot_checks.rb +130 -5
- data/lib/wurk/cli.rb +72 -10
- data/lib/wurk/configuration.rb +1 -1
- data/lib/wurk/fetcher/oom_drain.rb +0 -5
- data/lib/wurk/fetcher/orphan_reclaim.rb +71 -0
- data/lib/wurk/fetcher/reaper.rb +18 -56
- data/lib/wurk/fetcher/self_reconcile.rb +8 -14
- data/lib/wurk/job_util.rb +18 -2
- data/lib/wurk/lua/reclaim_orphan.lua +31 -0
- data/lib/wurk/lua.rb +5 -20
- data/lib/wurk/middleware/poison_pill.rb +31 -47
- data/lib/wurk/rails_boot.rb +40 -11
- data/lib/wurk/unique.rb +12 -2
- data/lib/wurk/version.rb +1 -1
- data/lib/wurk.rb +4 -0
- data/vendor/assets/dashboard/assets/{ArgsValue-CmaXArv8.js → ArgsValue-BJol9S0s.js} +1 -1
- data/vendor/assets/dashboard/assets/{BatchDetail-nc5yePZW.js → BatchDetail-D_zGQ9Sd.js} +1 -1
- data/vendor/assets/dashboard/assets/{Batches-DgEcyOZ_.js → Batches-Dt_xC660.js} +1 -1
- data/vendor/assets/dashboard/assets/{Busy-CUZncvCy.js → Busy-CWRthsmc.js} +1 -1
- data/vendor/assets/dashboard/assets/{Cron-CIZbaaZs.js → Cron-qeGr2YAm.js} +1 -1
- data/vendor/assets/dashboard/assets/Dashboard-DQWaAwPA.js +1 -0
- data/vendor/assets/dashboard/assets/Dead-D9gOPHtK.js +1 -0
- data/vendor/assets/dashboard/assets/{Extension-Doy4bJoA.js → Extension-DONXVLdA.js} +1 -1
- data/vendor/assets/dashboard/assets/{FilterBox-C4bO26_d.js → FilterBox-BbB6gXHp.js} +1 -1
- data/vendor/assets/dashboard/assets/{FlowDetail-Dh3SJdFh.js → FlowDetail-CrmnVRp0.js} +1 -1
- data/vendor/assets/dashboard/assets/{Flows-C5SMRuhq.js → Flows-g4GC8KmE.js} +1 -1
- data/vendor/assets/dashboard/assets/{JobDetailModal-Cg6sAsJB.js → JobDetailModal-C86k_1wj.js} +1 -1
- data/vendor/assets/dashboard/assets/{Limiters-BIxwDOJM.js → Limiters-zyO-iMlK.js} +1 -1
- data/vendor/assets/dashboard/assets/{Metrics-Qp9q0nF8.js → Metrics-Ckq5fNBv.js} +1 -1
- data/vendor/assets/dashboard/assets/{NotFound-DwjPxFjU.js → NotFound-C8NnyjlY.js} +1 -1
- data/vendor/assets/dashboard/assets/{Profiles-7dabYVe3.js → Profiles-DRl-FXPA.js} +1 -1
- data/vendor/assets/dashboard/assets/Queues-DkXWrihA.js +1 -0
- data/vendor/assets/dashboard/assets/Retries-CmlojYIy.js +1 -0
- data/vendor/assets/dashboard/assets/Scheduled-Cbbpp3xw.js +1 -0
- data/vendor/assets/dashboard/assets/{Search-BAsVE2qw.js → Search-LaEpj2fM.js} +1 -1
- data/vendor/assets/dashboard/assets/{charts-DZs138BV.js → charts-D_PQLoyc.js} +1 -1
- data/vendor/assets/dashboard/assets/index-FBKGtysy.js +141 -0
- data/vendor/assets/dashboard/assets/{useResetPageOnEmpty-DxkLZCIm.js → useResetPageOnEmpty-BiwhHqgh.js} +1 -1
- data/vendor/assets/dashboard/assets/{utils-B0kQFNE8.js → utils-CnWZzW2L.js} +1 -1
- data/vendor/assets/dashboard/index.html +1 -1
- data/vendor/assets/dashboard/wurk-manifest.json +2 -2
- metadata +27 -25
- data/vendor/assets/dashboard/assets/Dashboard-BWmfAook.js +0 -1
- data/vendor/assets/dashboard/assets/Dead-CGKsTZHq.js +0 -1
- data/vendor/assets/dashboard/assets/Queues-CMg-gyP8.js +0 -1
- data/vendor/assets/dashboard/assets/Retries-CNX8ZtTH.js +0 -1
- data/vendor/assets/dashboard/assets/Scheduled-ClVBj0nc.js +0 -1
- data/vendor/assets/dashboard/assets/index-CMzPFxNE.js +0 -141
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: efda965ecab6b73c86eddfd4f0b8acc5f0942bcba46d933aa60e922aae453c1f
|
|
4
|
+
data.tar.gz: 404c2f3b9055edeafedd0c1a98f1933799c89830d0c2f70d08fb4018c183fa4f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1d466f7764855b501fe06a0e3a2586bf4908b6bd65274e74fa5739b77255613801bc6625fdee52f4a4771cc4c1ea49e6f9b171fa364e8fd5fda95997b788da0c
|
|
7
|
+
data.tar.gz: 1333a3f95fc34da5a126caae9ff041a404bb80c221f425177e9034bf16bb4a16abc11fe9ad8aa1a8e18470f8066cc8aba0d9c45de2dea6ee46bbb38795b83a66
|
data/README.md
CHANGED
|
@@ -22,7 +22,7 @@
|
|
|
22
22
|
|
|
23
23
|
Wurk is wire-compatible with Sidekiq — same Redis keys, same job JSON, same Ruby DSL. Swap one line in your `Gemfile` and your existing jobs, batches, limiters, cron entries, and live Redis data keep working untouched. The Pro and Enterprise feature sets ship in the same free gem, with no license check and no tiers.
|
|
24
24
|
|
|
25
|
-
**In production:** Wurk runs background work at partner deployments, across many servers, on the fork-based swarm described below. It is not a preview.
|
|
25
|
+
**In production:** Wurk runs background work at partner deployments, across many servers, on the fork-based swarm described below. It is not a preview. Teams running billions of jobs a month on Wurk in production recommend it. Those volumes are user-reported; no production throughput numbers are published yet, and what has been measured is in [docs/benchmarks.md](docs/benchmarks.md).
|
|
26
26
|
|
|
27
27
|
**At scale:** Wurk is built for fleets, not just for one box. Kubernetes `/live` + `/ready` probes are a config line, not a sidecar; a bearer-scoped [HTTP API](docs/api-http.md) lets non-Ruby services enqueue and inspect; [OpenTelemetry](docs/telemetry.md) traces propagate client → server; per-queue [global concurrency caps](docs/rate-limiting.md) hold cluster-wide limits; and monitoring is the dashboard you already mount — live SSE, charts, no separate stack to run. See [Wurk extras](#wurk-extras).
|
|
28
28
|
|
data/exe/wurk
CHANGED
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
# Usage: `exe/wurk -C config/wurk.yml` — worker: one process, thread pool
|
|
6
6
|
# `exe/wurk api --port 7433` — machine HTTP API, runs no jobs
|
|
7
7
|
# `exe/wurk preflight` / `verify` — read-only cutover checks, run no jobs
|
|
8
|
+
# `exe/wurk boot-check -r .` — boot the app as wurkswarm would, then exit (CI)
|
|
8
9
|
|
|
9
10
|
$stdout.sync = true
|
|
10
11
|
$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
|
|
@@ -20,8 +21,9 @@ begin
|
|
|
20
21
|
cli = Wurk::CLI.instance
|
|
21
22
|
cli.parse
|
|
22
23
|
case cli.command
|
|
23
|
-
when 'api'
|
|
24
|
-
|
|
24
|
+
when 'api' then cli.run_api
|
|
25
|
+
when 'boot-check' then exit(cli.run_boot_check)
|
|
26
|
+
else cli.run
|
|
25
27
|
end
|
|
26
28
|
rescue StandardError => e
|
|
27
29
|
raise e if $DEBUG
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
# Wurk configuration. Generated by `bin/rails g wurk:install`.
|
|
4
|
-
#
|
|
5
|
-
#
|
|
4
|
+
# Every option: https://github.com/developerz-ai/wurk/blob/main/docs/configuration.md
|
|
5
|
+
# Coming from Sidekiq? https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md
|
|
6
6
|
|
|
7
7
|
Wurk.configure_server do |config|
|
|
8
8
|
# config.redis = { url: ENV.fetch("REDIS_URL", "redis://localhost:6379/0") }
|
|
@@ -19,7 +19,9 @@ Wurk.configure_server do |config|
|
|
|
19
19
|
# By default Wurk forks one worker process per usable CPU (cgroup quota aware) (set WURK_COUNT, or
|
|
20
20
|
# SIDEKIQ_COUNT, to override). For full control declare a topology — `flat`
|
|
21
21
|
# spawns N identical forks; use `slot`s for dedicated queues. Total in-flight
|
|
22
|
-
# jobs = forks × concurrency.
|
|
22
|
+
# jobs = forks × concurrency. Topology and sizing:
|
|
23
|
+
# https://github.com/developerz-ai/wurk/blob/main/docs/configuration.md#topology
|
|
24
|
+
# https://github.com/developerz-ai/wurk/blob/main/docs/deployment.md#capacity-and-sizing
|
|
23
25
|
# config.topology = Wurk::Topology.flat(count: 2, queues: %w[critical default low], concurrency: 10)
|
|
24
26
|
end
|
|
25
27
|
|
data/lib/wurk/boot_checks.rb
CHANGED
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative 'keys'
|
|
4
4
|
require_relative 'queue_consumers'
|
|
5
|
+
require_relative 'capsule'
|
|
5
6
|
|
|
6
7
|
module Wurk
|
|
7
8
|
# What a standalone runner says once at boot, after the app loaded. Log only,
|
|
@@ -13,13 +14,20 @@ module Wurk
|
|
|
13
14
|
MAX_LISTED = 20
|
|
14
15
|
SUBCLASS = ::Module.instance_method(:<)
|
|
15
16
|
NAME = ::Module.instance_method(:name)
|
|
17
|
+
UNIQUE_OPTIONS = %w[unique_for unique_until].freeze
|
|
18
|
+
STRICT_ENV = 'WURK_STRICT'
|
|
16
19
|
|
|
17
20
|
module_function
|
|
18
21
|
|
|
22
|
+
# `info` is nil when there is no Redis to ask (`wurk boot-check` without
|
|
23
|
+
# `--redis`): only the checks that read the loaded app run then.
|
|
19
24
|
def run(config, info)
|
|
20
|
-
|
|
21
|
-
|
|
25
|
+
if info
|
|
26
|
+
log_backend(config.logger, info)
|
|
27
|
+
warn_on_foreign_processes(config)
|
|
28
|
+
end
|
|
22
29
|
warn_on_ignored_locks(config.logger)
|
|
30
|
+
warn_on_ignored_unique(config.logger)
|
|
23
31
|
rescue StandardError => e
|
|
24
32
|
config.logger.warn { "wurk boot checks skipped: #{e.class}: #{e.message}" }
|
|
25
33
|
end
|
|
@@ -73,18 +81,135 @@ module Wurk
|
|
|
73
81
|
end
|
|
74
82
|
end
|
|
75
83
|
|
|
84
|
+
# Only the middleware `unique!` installs reads these options, so without it
|
|
85
|
+
# every push enqueues. Turning it on for the app is a human call, not a
|
|
86
|
+
# boot check's: this only says so.
|
|
87
|
+
def warn_on_ignored_unique(logger)
|
|
88
|
+
return if Wurk::Unique.enabled?
|
|
89
|
+
|
|
90
|
+
classes = job_classes_setting(UNIQUE_OPTIONS)
|
|
91
|
+
return if classes.empty?
|
|
92
|
+
|
|
93
|
+
logger.warn do
|
|
94
|
+
'These jobs set `unique_for:` or `unique_until:` but unique jobs are not enabled, so the options do ' \
|
|
95
|
+
"nothing and duplicates enqueue: #{listing(classes)}. Enable them once at boot, in an initializer: " \
|
|
96
|
+
'`Sidekiq::Enterprise.unique!`.'
|
|
97
|
+
end
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def locked_job_classes = job_classes_setting(%w[lock])
|
|
101
|
+
|
|
102
|
+
# Workers, and ActiveJob classes that take `sidekiq_options` (Wurk::Job::Options).
|
|
76
103
|
# A walk over every class meets classes that redefine these with other arguments
|
|
77
104
|
# (Faker::Travel::Airport.name takes required keywords), and one such class must
|
|
78
105
|
# not cost the app this check.
|
|
79
|
-
def
|
|
106
|
+
def job_classes_setting(options)
|
|
80
107
|
::ObjectSpace.each_object(::Class).filter_map do |klass|
|
|
81
|
-
next unless SUBCLASS.bind_call(klass, Wurk::Worker)
|
|
108
|
+
next unless SUBCLASS.bind_call(klass, Wurk::Worker) || SUBCLASS.bind_call(klass, Wurk::Job::Options)
|
|
82
109
|
|
|
83
110
|
name = NAME.bind_call(klass)
|
|
84
|
-
name if name && klass.get_sidekiq_options.
|
|
111
|
+
name if name && klass.get_sidekiq_options.values_at(*options).any?
|
|
85
112
|
end.sort
|
|
86
113
|
end
|
|
87
114
|
|
|
115
|
+
# The real gem next to Wurk loads both: `Sidekiq::*` is then partly theirs,
|
|
116
|
+
# partly ours. Gems that depend on `sidekiq` pull it in unless the shim gem,
|
|
117
|
+
# which depends on Wurk instead, satisfies the dependency. Runs as Wurk loads.
|
|
118
|
+
def flag_real_sidekiq_gem(logger, specs: ::Gem.loaded_specs, strict: ENV[STRICT_ENV] == '1')
|
|
119
|
+
message = real_sidekiq_message(specs, logger)
|
|
120
|
+
return unless message
|
|
121
|
+
raise Wurk::Error, message if strict
|
|
122
|
+
|
|
123
|
+
logger.error { "#{message} Set #{STRICT_ENV}=1 to refuse to boot instead." }
|
|
124
|
+
end
|
|
125
|
+
|
|
126
|
+
def real_sidekiq_message(specs, logger)
|
|
127
|
+
real = specs['sidekiq']
|
|
128
|
+
return if real.nil? || real.dependencies.any? { |dep| dep.name == 'wurk' }
|
|
129
|
+
|
|
130
|
+
"The bundle holds the real sidekiq #{real.version} gem, so it loads next to Wurk and the process runs " \
|
|
131
|
+
"half on each. Pulled in by: #{sidekiq_dependents(specs)}. Replace it with the shim in the Gemfile: " \
|
|
132
|
+
"gem 'sidekiq', github: 'developerz-ai/wurk', tag: 'v#{Wurk::VERSION}', " \
|
|
133
|
+
"glob: 'ecosystem/sidekiq-shim/*.gemspec'."
|
|
134
|
+
rescue StandardError => e
|
|
135
|
+
logger.warn { "wurk: could not check the bundle for the real sidekiq gem: #{e.class}: #{e.message}" }
|
|
136
|
+
nil
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
def sidekiq_dependents(specs)
|
|
140
|
+
names = specs.each_value.select { |spec| spec.dependencies.any? { |dep| dep.name == 'sidekiq' } }.map(&:name)
|
|
141
|
+
names.empty? ? 'the Gemfile itself' : listing(names.sort)
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
# The role line of a process that forks the topology's children and runs no
|
|
145
|
+
# jobs itself. Its children each run the threads of one slot.
|
|
146
|
+
def log_swarm_role(config, role, ar_pool: active_record_pool)
|
|
147
|
+
slots = config.topology.slots.map do |slot|
|
|
148
|
+
["#{slot.count} #{slot.count == 1 ? 'process' : 'processes'}", slot.concurrency]
|
|
149
|
+
end
|
|
150
|
+
log_role(config, role, slots, per_process: slots.map(&:last).max, ar_pool: ar_pool)
|
|
151
|
+
rescue StandardError => e
|
|
152
|
+
role_skipped(config.logger, e)
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
# The role line of a process that runs every capsule's threads itself.
|
|
156
|
+
def log_process_role(config, role, ar_pool: active_record_pool)
|
|
157
|
+
capsules = config.capsules.values
|
|
158
|
+
slots = capsules.map { |cap| [capsules.size > 1 ? "capsule #{cap.name}" : '1 process', cap.concurrency] }
|
|
159
|
+
log_role(config, role, slots, per_process: slots.sum(&:last), ar_pool: ar_pool)
|
|
160
|
+
rescue StandardError => e
|
|
161
|
+
role_skipped(config.logger, e)
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
def role_skipped(logger, error)
|
|
165
|
+
logger.warn { "wurk role line skipped: #{error.class}: #{error.message}" }
|
|
166
|
+
end
|
|
167
|
+
|
|
168
|
+
# One line per process: what it is, and how big. `per_process` is the most
|
|
169
|
+
# job threads one process runs, which is what its Active Record pool covers.
|
|
170
|
+
def log_role(config, role, slots, per_process:, ar_pool:)
|
|
171
|
+
sizes = slots.map do |label, threads|
|
|
172
|
+
"#{label} × #{threads} threads (redis pool #{redis_pool_size(config, threads)}, fetch #{threads})"
|
|
173
|
+
end
|
|
174
|
+
config.logger.info { "wurk: #{role}; #{sizes.join(' + ')}#{"; AR pool #{ar_pool}" if ar_pool}" }
|
|
175
|
+
warn_on_small_ar_pool(config.logger, ar_pool, per_process)
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
# DEBUG, not INFO: Wurk logs to stdout, and every `rails runner`, rake task
|
|
179
|
+
# and `wurk preflight --app` boots the app; their stdout is someone's output.
|
|
180
|
+
def log_client_only(logger, reason)
|
|
181
|
+
logger.debug { "wurk: client only (#{reason}); this process enqueues and runs no jobs" }
|
|
182
|
+
end
|
|
183
|
+
|
|
184
|
+
# Capsule#main_pool_size, without building a pool: a swarm parent describes
|
|
185
|
+
# slots whose pools only its children build.
|
|
186
|
+
def redis_pool_size(config, threads)
|
|
187
|
+
config.redis_config[:size] || [threads + Capsule::POOL_HEADROOM, Capsule::MIN_POOL_SIZE].max
|
|
188
|
+
end
|
|
189
|
+
|
|
190
|
+
# Every job thread that touches the database holds a connection; with fewer
|
|
191
|
+
# in the pool the rest wait, then fail with ActiveRecord::ConnectionTimeoutError.
|
|
192
|
+
def warn_on_small_ar_pool(logger, ar_pool, threads)
|
|
193
|
+
return unless ar_pool && ar_pool < threads
|
|
194
|
+
|
|
195
|
+
logger.warn do
|
|
196
|
+
"The Active Record pool holds #{ar_pool} connections but this process runs up to #{threads} job threads, " \
|
|
197
|
+
'so jobs that use the database wait for a connection and then fail on a timeout. Set `pool:` in ' \
|
|
198
|
+
"config/database.yml to at least #{threads} (plus the web server's threads when jobs run in the web process)."
|
|
199
|
+
end
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
# The pool size the app configured, read without opening a connection; nil
|
|
203
|
+
# without Active Record or a configured database. Rails 8.1 renamed `pool`.
|
|
204
|
+
def active_record_pool
|
|
205
|
+
return unless defined?(::ActiveRecord::Base)
|
|
206
|
+
|
|
207
|
+
db = ::ActiveRecord::Base.connection_db_config
|
|
208
|
+
db.respond_to?(:max_connections) ? db.max_connections : db.pool
|
|
209
|
+
rescue ::ActiveRecord::ActiveRecordError
|
|
210
|
+
nil
|
|
211
|
+
end
|
|
212
|
+
|
|
88
213
|
def listing(names)
|
|
89
214
|
extra = names.size - MAX_LISTED
|
|
90
215
|
names.first(MAX_LISTED).join(', ') + (extra.positive? ? " and #{extra} more" : '')
|
data/lib/wurk/cli.rb
CHANGED
|
@@ -32,12 +32,13 @@ module Wurk
|
|
|
32
32
|
# Subcommands. Bare `wurk` runs the worker — the historical shape, and the
|
|
33
33
|
# only one the swarm binaries accept — so a subcommand is never anything
|
|
34
34
|
# but the first argument, and every argument after it is that command's.
|
|
35
|
-
COMMANDS = %w[api health].freeze
|
|
35
|
+
COMMANDS = %w[api boot-check health].freeze
|
|
36
36
|
# Set only across the one re-exec of #load_rails_framework, and removed on arrival.
|
|
37
37
|
PRELOAD_FIRST_ENV = 'WURK_BOOT_PRELOAD_FIRST'
|
|
38
38
|
# `bundle exec` loads the binstub in-process, so argv0 is Bundler's executable, and
|
|
39
39
|
# `ruby -e "load Gem.bin_path(...)"` has `-e` there: the running program can't name it.
|
|
40
40
|
SWARM_EXECUTABLE = ::File.expand_path('../../exe/wurkswarm', __dir__)
|
|
41
|
+
WURK_EXECUTABLE = ::File.expand_path('../../exe/wurk', __dir__)
|
|
41
42
|
|
|
42
43
|
# A refusal this file raises itself, with nothing a trace would add.
|
|
43
44
|
class StartupError < ArgumentError; end
|
|
@@ -110,6 +111,10 @@ module Wurk
|
|
|
110
111
|
['-s', '--server NAME', :api_server, 'Rack handler to serve with (default: the first installed)']
|
|
111
112
|
].freeze
|
|
112
113
|
|
|
114
|
+
BOOT_CHECK_OPTION_FLAGS = [
|
|
115
|
+
['-R', '--redis', :boot_check_redis, 'Also read Redis: version, eviction policy, live peers (read-only)']
|
|
116
|
+
].freeze
|
|
117
|
+
|
|
113
118
|
HEALTH_OPTION_FLAGS = [
|
|
114
119
|
['-w', '--window SECONDS', :health_window, 'A heartbeat older than this does not count (default 60)', :to_i],
|
|
115
120
|
['-m', '--min COUNT', :health_min, 'Processes on this host that must be beating (default 1)', :to_i],
|
|
@@ -118,6 +123,7 @@ module Wurk
|
|
|
118
123
|
].freeze
|
|
119
124
|
|
|
120
125
|
HELP_COMMANDS = "\nCommands:\n " \
|
|
126
|
+
"boot-check Boot the app the way wurkswarm does, then exit; no fork, no fetch (CI)\n " \
|
|
121
127
|
"health Exit 0 if this host's workers are heartbeating\n " \
|
|
122
128
|
"preflight Read a live Redis: what will carry over to Wurk (read-only)\n " \
|
|
123
129
|
"verify Post-cutover check of a live fleet (read-only)\n\nOptions:"
|
|
@@ -209,6 +215,7 @@ module Wurk
|
|
|
209
215
|
validate_redis!
|
|
210
216
|
validate_pool_sizes!
|
|
211
217
|
BootChecks.run(@config, redis_info)
|
|
218
|
+
BootChecks.log_process_role(@config, 'wurk')
|
|
212
219
|
@config[:identity] = identity
|
|
213
220
|
# Force lazy server-middleware chain so worker threads don't race
|
|
214
221
|
# against each other constructing it. Spec: Sidekiq::CLI line 104.
|
|
@@ -245,15 +252,11 @@ module Wurk
|
|
|
245
252
|
# Only the children read it. Rejected here, a typo fails this boot instead of
|
|
246
253
|
# crash-looping the fleet until the crash-loop exit.
|
|
247
254
|
Health.options_from_env
|
|
248
|
-
if boot_app
|
|
249
|
-
load_rails_framework
|
|
250
|
-
preload_bundler_groups
|
|
251
|
-
boot_application
|
|
252
|
-
eager_load_application
|
|
253
|
-
end
|
|
255
|
+
boot_swarm_application if boot_app
|
|
254
256
|
validate_redis!
|
|
255
257
|
validate_pool_sizes!
|
|
256
258
|
BootChecks.run(@config, redis_info)
|
|
259
|
+
log_swarm_role
|
|
257
260
|
@config[:identity] = identity
|
|
258
261
|
warm_up_process if warmup
|
|
259
262
|
@swarm = Wurk::Swarm.new(topology: @config.topology, config: @config, exit_on_crash_loop: true,
|
|
@@ -270,6 +273,27 @@ module Wurk
|
|
|
270
273
|
exit(1) if @swarm.crash_looped?
|
|
271
274
|
end
|
|
272
275
|
|
|
276
|
+
# `wurk boot-check` — #run_swarm's boot up to the fork: the same require
|
|
277
|
+
# order, config file and checks, plus what a worker reaches only later (every
|
|
278
|
+
# constant, the routes), so a CI step finds what would crash-loop a deploy.
|
|
279
|
+
# Forks nothing, fetches nothing, fires no lifecycle event, and reads Redis
|
|
280
|
+
# only with --redis. Returns the exit code; a boot that raises is the binary's
|
|
281
|
+
# failure report.
|
|
282
|
+
def run_boot_check(boot_app: true)
|
|
283
|
+
enter_server_mode
|
|
284
|
+
Health.options_from_env
|
|
285
|
+
if boot_app
|
|
286
|
+
boot_swarm_application
|
|
287
|
+
eager_load_rails
|
|
288
|
+
draw_routes
|
|
289
|
+
end
|
|
290
|
+
validate_pool_sizes!
|
|
291
|
+
BootChecks.run(@config, boot_check_redis_info)
|
|
292
|
+
log_swarm_role
|
|
293
|
+
logger.info { "wurk boot-check: #{@config[:require]} booted as wurkswarm would" }
|
|
294
|
+
0
|
|
295
|
+
end
|
|
296
|
+
|
|
273
297
|
# `wurk health` — see Wurk::HealthProbe. Boots nothing; returns the exit code.
|
|
274
298
|
def run_health
|
|
275
299
|
require_relative 'health_probe'
|
|
@@ -434,10 +458,26 @@ module Wurk
|
|
|
434
458
|
|
|
435
459
|
def validate_pool_sizes!
|
|
436
460
|
@config.capsules.each_pair do |name, cap|
|
|
437
|
-
|
|
461
|
+
size = cap.redis_pool.size
|
|
462
|
+
next if size >= cap.concurrency
|
|
463
|
+
|
|
464
|
+
raise StartupError, "the Redis pool for capsule #{name} holds #{size} connections, fewer than its " \
|
|
465
|
+
"concurrency of #{cap.concurrency}. Set `config.redis = { size: #{cap.concurrency} }` " \
|
|
466
|
+
'or more, or drop `size:` and Wurk sizes the pool itself (concurrency + 5).'
|
|
438
467
|
end
|
|
439
468
|
end
|
|
440
469
|
|
|
470
|
+
def boot_check_redis_info
|
|
471
|
+
return unless @config[:boot_check_redis]
|
|
472
|
+
|
|
473
|
+
validate_redis!
|
|
474
|
+
redis_info
|
|
475
|
+
end
|
|
476
|
+
|
|
477
|
+
def log_swarm_role
|
|
478
|
+
BootChecks.log_swarm_role(@config, @command ? "#{@command} (as wurkswarm parent)" : 'wurkswarm parent')
|
|
479
|
+
end
|
|
480
|
+
|
|
441
481
|
# --- options + config-file --------------------------------------------
|
|
442
482
|
|
|
443
483
|
# Pulled off before OptionParser sees argv, so the banner and the flag
|
|
@@ -448,6 +488,20 @@ module Wurk
|
|
|
448
488
|
COMMANDS.include?(args.first) ? args.shift : nil
|
|
449
489
|
end
|
|
450
490
|
|
|
491
|
+
def boot_swarm_application
|
|
492
|
+
load_rails_framework
|
|
493
|
+
preload_bundler_groups
|
|
494
|
+
boot_application
|
|
495
|
+
eager_load_application
|
|
496
|
+
end
|
|
497
|
+
|
|
498
|
+
# Rails may draw routes only on the first request; a `mount` of a constant
|
|
499
|
+
# only the web process loads must fail here, not on that request.
|
|
500
|
+
def draw_routes
|
|
501
|
+
app = rails_application
|
|
502
|
+
app.reload_routes! if app.respond_to?(:reload_routes!)
|
|
503
|
+
end
|
|
504
|
+
|
|
451
505
|
def parse_health_options(args)
|
|
452
506
|
@health_options = {}
|
|
453
507
|
parser = option_parser(@health_options)
|
|
@@ -547,6 +601,7 @@ module Wurk
|
|
|
547
601
|
o.separator help_intro if help_intro
|
|
548
602
|
define_value_flags(o, opts, @command == 'health' ? HEALTH_OPTION_FLAGS : OPTION_FLAGS)
|
|
549
603
|
define_value_flags(o, opts, API_OPTION_FLAGS) if @command == 'api'
|
|
604
|
+
define_value_flags(o, opts, BOOT_CHECK_OPTION_FLAGS) if @command == 'boot-check'
|
|
550
605
|
define_meta_flags(o, opts)
|
|
551
606
|
end
|
|
552
607
|
end
|
|
@@ -627,7 +682,7 @@ module Wurk
|
|
|
627
682
|
end
|
|
628
683
|
|
|
629
684
|
def restart_state
|
|
630
|
-
{ env: ::ENV.to_h, argv:
|
|
685
|
+
{ env: ::ENV.to_h, argv: [restart_executable, *::ARGV], dir: ::Dir.pwd }
|
|
631
686
|
end
|
|
632
687
|
|
|
633
688
|
def note_preload_first(first_error)
|
|
@@ -644,11 +699,14 @@ module Wurk
|
|
|
644
699
|
env = state[:env].merge(PRELOAD_FIRST_ENV => error.message.lines.first.to_s.strip[0, 300])
|
|
645
700
|
env['BUNDLE_GEMFILE'] ||= ::Bundler.default_gemfile.to_s if defined?(::Bundler)
|
|
646
701
|
ruby = [::RbConfig.ruby, *('-rbundler/setup' if defined?(::Bundler))]
|
|
647
|
-
::Process.exec(env, *ruby,
|
|
702
|
+
::Process.exec(env, *ruby, *state[:argv], unsetenv_others: true, chdir: state[:dir])
|
|
648
703
|
rescue ::SystemCallError
|
|
649
704
|
raise error
|
|
650
705
|
end
|
|
651
706
|
|
|
707
|
+
# The binary that was run: only `wurk boot-check` reaches this with a command.
|
|
708
|
+
def restart_executable = @command ? WURK_EXECUTABLE : SWARM_EXECUTABLE
|
|
709
|
+
|
|
652
710
|
# The app root for a `-r` that is the app directory or its environment file.
|
|
653
711
|
def rails_app_root(require_path)
|
|
654
712
|
path = ::File.expand_path(require_path.to_s)
|
|
@@ -688,6 +746,10 @@ module Wurk
|
|
|
688
746
|
def eager_load_application
|
|
689
747
|
return unless preload_app?
|
|
690
748
|
|
|
749
|
+
eager_load_rails
|
|
750
|
+
end
|
|
751
|
+
|
|
752
|
+
def eager_load_rails
|
|
691
753
|
app = rails_application
|
|
692
754
|
app.eager_load! if app.respond_to?(:eager_load!)
|
|
693
755
|
end
|
data/lib/wurk/configuration.rb
CHANGED
|
@@ -452,7 +452,7 @@ module Wurk
|
|
|
452
452
|
# The optional block is Pro's recovery callback: `|jobstr, pill|`, fired
|
|
453
453
|
# once per orphan recovery (`pill` nil) and once on a poison kill (`pill`
|
|
454
454
|
# responds to .jid/.klass/.count/.queue). The reaper drives it via
|
|
455
|
-
# Wurk::Middleware::PoisonPill.
|
|
455
|
+
# Wurk::Middleware::PoisonPill.recovered!. Spec: docs/target/sidekiq-pro.md §3.1.
|
|
456
456
|
def super_fetch!(*, &block)
|
|
457
457
|
@super_fetch_callback = block if block
|
|
458
458
|
nil
|
|
@@ -50,7 +50,6 @@ module Wurk
|
|
|
50
50
|
return nil
|
|
51
51
|
LUA
|
|
52
52
|
|
|
53
|
-
RECLAIM = (HEADER + Lua::RELIABLE_RECLAIM).freeze
|
|
54
53
|
REQUEUE = (HEADER + Lua::RELIABLE_REQUEUE).freeze
|
|
55
54
|
|
|
56
55
|
# KEYS = [lock], ARGV = [ttl_ms]. The reaper's sweep lock: refused, no sweep
|
|
@@ -72,10 +71,6 @@ module Wurk
|
|
|
72
71
|
run(conn, error, CLAIM, [Keys::PROCESSES, Component.identity, *keys], [info, beat, HEARTBEAT_TTL])
|
|
73
72
|
end
|
|
74
73
|
|
|
75
|
-
def reclaim(conn, error, source, destination)
|
|
76
|
-
run(conn, error, RECLAIM, [source, destination], [])
|
|
77
|
-
end
|
|
78
|
-
|
|
79
74
|
# Returns 1 when moved, 0 when the job has left the private list since.
|
|
80
75
|
def requeue(conn, error, private_list, public_q, job)
|
|
81
76
|
run(conn, error, REQUEUE, [private_list, public_q], [job])
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative '../keys'
|
|
4
|
+
require_relative '../lua'
|
|
5
|
+
require_relative '../middleware/poison_pill'
|
|
6
|
+
require_relative '../metrics/process_counters'
|
|
7
|
+
require_relative 'oom_drain'
|
|
8
|
+
|
|
9
|
+
module Wurk
|
|
10
|
+
class Fetcher
|
|
11
|
+
# Puts one payload from a private list back where it belongs, for both ways
|
|
12
|
+
# an orphan is found: the Reaper draining a dead owner's list, and
|
|
13
|
+
# SelfReconcile returning a payload its own live process lost track of.
|
|
14
|
+
#
|
|
15
|
+
# The poison-pill verdict is taken in the same script that moves the job
|
|
16
|
+
# (`lua/reclaim_orphan.lua`), so a job past the cap goes straight from the
|
|
17
|
+
# private list to the dead set and is never visible on its public queue. The
|
|
18
|
+
# notifications (death handlers, `super_fetch!`, `on_poison`) follow once
|
|
19
|
+
# the script has returned, for a job no worker can be running.
|
|
20
|
+
class OrphanReclaim
|
|
21
|
+
THRESHOLD = Middleware::PoisonPill::RECOVERY_THRESHOLD
|
|
22
|
+
TTL = Middleware::PoisonPill::RECOVERY_TTL
|
|
23
|
+
|
|
24
|
+
def initialize(config)
|
|
25
|
+
@config = config
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Returns true when this call took the payload off the private list, false
|
|
29
|
+
# when it had already left (another reaper, an ACK).
|
|
30
|
+
#
|
|
31
|
+
# Unclaimed apply-safety: the move is guarded, but a reply lost after a
|
|
32
|
+
# poison kill would leave the death handlers unfired, and a list holding
|
|
33
|
+
# two identical copies would move the second on a replay. A raise leaves
|
|
34
|
+
# the payload for the next pass instead.
|
|
35
|
+
def call(private_list, public_q, payload)
|
|
36
|
+
job = Middleware::PoisonPill.parse(payload)
|
|
37
|
+
count = @config.redis { |conn| move(conn, private_list, public_q, payload, job) }
|
|
38
|
+
return false unless count
|
|
39
|
+
|
|
40
|
+
Metrics::ProcessCounters.increment(:reaper_reclaimed)
|
|
41
|
+
unless count == :uncounted
|
|
42
|
+
Middleware::PoisonPill.recovered!(payload, job, count: count, queue: queue_name(public_q),
|
|
43
|
+
config: @config)
|
|
44
|
+
end
|
|
45
|
+
true
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
private
|
|
49
|
+
|
|
50
|
+
# Past maxmemory the counter cannot be written, nor the job killed: it goes
|
|
51
|
+
# back uncounted rather than waiting on the memory it would help free.
|
|
52
|
+
def move(conn, private_list, public_q, payload, job)
|
|
53
|
+
keys = [private_list, public_q, Keys::DEAD]
|
|
54
|
+
jid = job && job['jid']
|
|
55
|
+
keys << Middleware::PoisonPill.counter_key(jid) if jid.is_a?(String) && !jid.empty?
|
|
56
|
+
Wurk::Lua::Loader.eval_cached(conn, :reclaim_orphan, keys: keys, argv: [payload, THRESHOLD, TTL, now])
|
|
57
|
+
rescue RedisClient::OutOfMemoryError => e
|
|
58
|
+
:uncounted if OomDrain.requeue(conn, e, private_list, public_q, payload) == 1
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
# DeadSet#kill's score format, so the entry sorts with every other kill.
|
|
62
|
+
def now
|
|
63
|
+
::Process.clock_gettime(::Process::CLOCK_REALTIME).to_s
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def queue_name(public_q)
|
|
67
|
+
public_q.delete_prefix(Keys::QUEUE_PREFIX)
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
|
71
|
+
end
|
data/lib/wurk/fetcher/reaper.rb
CHANGED
|
@@ -2,13 +2,12 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative '../component'
|
|
4
4
|
require_relative '../keys'
|
|
5
|
-
require_relative '../middleware/poison_pill'
|
|
6
|
-
require_relative '../metrics/process_counters'
|
|
7
5
|
require_relative '../timer_loop'
|
|
8
6
|
require_relative 'private_list_key'
|
|
9
7
|
require_relative 'unparseable_keys'
|
|
10
8
|
require_relative 'orphan_grace'
|
|
11
9
|
require_relative 'oom_drain'
|
|
10
|
+
require_relative 'orphan_reclaim'
|
|
12
11
|
require_relative 'self_reconcile'
|
|
13
12
|
|
|
14
13
|
module Wurk
|
|
@@ -45,10 +44,10 @@ module Wurk
|
|
|
45
44
|
# process's own clock (OrphanGrace), and its identity must still have no
|
|
46
45
|
# `info` hash. Our own incarnation skips both: kill(0) is not a snapshot.
|
|
47
46
|
#
|
|
48
|
-
#
|
|
49
|
-
#
|
|
50
|
-
#
|
|
51
|
-
# its worker every time can't loop forever.
|
|
47
|
+
# Every move goes through OrphanReclaim, which caps a job at
|
|
48
|
+
# Middleware::PoisonPill::RECOVERY_THRESHOLD recoveries within 72h: past the
|
|
49
|
+
# cap the job goes from the private list straight to the dead set instead of
|
|
50
|
+
# its queue, so a job that crashes its worker every time can't loop forever.
|
|
52
51
|
#
|
|
53
52
|
# One sweep an interval: one SCAN of `queue:*|*` acting on every list, gated
|
|
54
53
|
# by a cluster lock so about one process sweeps per interval. Every list, and
|
|
@@ -62,7 +61,7 @@ module Wurk
|
|
|
62
61
|
# of a job.
|
|
63
62
|
#
|
|
64
63
|
# Spec: docs/target/sidekiq-pro.md §3.2.
|
|
65
|
-
class Reaper
|
|
64
|
+
class Reaper
|
|
66
65
|
include Component
|
|
67
66
|
|
|
68
67
|
# Sweep cadence in seconds. 60s matches the heartbeat TTL — the floor below
|
|
@@ -103,8 +102,8 @@ module Wurk
|
|
|
103
102
|
@sleeper = ::ConditionVariable.new
|
|
104
103
|
@unparseable = UnparseableKeys.new(config)
|
|
105
104
|
@next_sweep_at = 0.0
|
|
106
|
-
@
|
|
107
|
-
|
|
105
|
+
@orphan_reclaim = OrphanReclaim.new(config)
|
|
106
|
+
@self_reconcile = SelfReconcile.new(config, confirm: reconcile_confirm, stop: -> { done? })
|
|
108
107
|
end
|
|
109
108
|
|
|
110
109
|
# Spawns the sweep loop. Idempotent. The loop waits one interval before
|
|
@@ -313,27 +312,19 @@ module Wurk
|
|
|
313
312
|
identity.split(':')[0..1].join(':')
|
|
314
313
|
end
|
|
315
314
|
|
|
316
|
-
# Drain one orphaned private list back to its public queue
|
|
317
|
-
#
|
|
318
|
-
#
|
|
319
|
-
#
|
|
320
|
-
#
|
|
315
|
+
# Drain one orphaned private list back to its public queue, a job at a
|
|
316
|
+
# time from the tail: peek it, then OrphanReclaim moves it, to the public
|
|
317
|
+
# tail or, past the poison cap, to the dead set. The decision is taken
|
|
318
|
+
# before the job is visible, so a worker parked in BLMOVE can never take a
|
|
319
|
+
# job that is being killed. A peeked job that has left by the time it is
|
|
320
|
+
# moved (another reaper drained it) costs one more peek.
|
|
321
321
|
#
|
|
322
|
-
#
|
|
323
|
-
#
|
|
324
|
-
# one already on the public tail, so a job that kills its worker every time
|
|
325
|
-
# would get a free recovery past the cap. A raise instead lands in the
|
|
326
|
-
# rescue below, and the next sweep re-drains what's left.
|
|
322
|
+
# A raise lands in the rescue below; the job stays in the private list
|
|
323
|
+
# and the next sweep re-drains what's left.
|
|
327
324
|
def drain(private_list, public_q)
|
|
328
|
-
queue_name = public_q.delete_prefix(Keys::QUEUE_PREFIX)
|
|
329
325
|
count = 0
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
break unless job
|
|
333
|
-
|
|
334
|
-
count += 1
|
|
335
|
-
Metrics::ProcessCounters.increment(:reaper_reclaimed)
|
|
336
|
-
poison_off(public_q, job, queue_name)
|
|
326
|
+
while (job = redis(idempotent: true) { |c| c.call('LINDEX', private_list, -1) })
|
|
327
|
+
count += 1 if @orphan_reclaim.call(private_list, public_q, job)
|
|
337
328
|
end
|
|
338
329
|
count
|
|
339
330
|
rescue StandardError => e
|
|
@@ -341,35 +332,6 @@ module Wurk
|
|
|
341
332
|
count
|
|
342
333
|
end
|
|
343
334
|
|
|
344
|
-
# A full Redis must not hold a dead worker's jobs back from the workers
|
|
345
|
-
# still able to drain the queue.
|
|
346
|
-
def reclaim_one(conn, private_list, public_q)
|
|
347
|
-
Wurk::Lua::Loader.eval_cached(conn, :reliable_reclaim, keys: [private_list, public_q], argv: [])
|
|
348
|
-
rescue RedisClient::OutOfMemoryError => e
|
|
349
|
-
OomDrain.reclaim(conn, e, private_list, public_q)
|
|
350
|
-
end
|
|
351
|
-
|
|
352
|
-
# SelfReconcile put a payload back: the same accounting as a reclaim, so a
|
|
353
|
-
# job that kills its processor thread every time still hits the poison cap.
|
|
354
|
-
def recovered(public_q, job)
|
|
355
|
-
Metrics::ProcessCounters.increment(:reaper_reclaimed)
|
|
356
|
-
poison_off(public_q, job, public_q.delete_prefix(Keys::QUEUE_PREFIX))
|
|
357
|
-
end
|
|
358
|
-
|
|
359
|
-
# The job is back on its queue whatever happens here. Past maxmemory the
|
|
360
|
-
# count can't be taken (nor the job killed, leaving it where it is): raising
|
|
361
|
-
# would end the drain at one job per pass, with the rest of the list waiting
|
|
362
|
-
# on memory they would help free.
|
|
363
|
-
def poison_off(public_q, job, queue_name)
|
|
364
|
-
return unless Middleware::PoisonPill.track!(job, queue: queue_name, config: @config) == :poison
|
|
365
|
-
|
|
366
|
-
# track! already ZADDed the payload to the dead set; pull the copy we
|
|
367
|
-
# just moved onto the public tail so it isn't also re-run.
|
|
368
|
-
redis { |c| c.call('LREM', public_q, -1, job) }
|
|
369
|
-
rescue RedisClient::OutOfMemoryError => e
|
|
370
|
-
report(e)
|
|
371
|
-
end
|
|
372
|
-
|
|
373
335
|
def acquire_lock?
|
|
374
336
|
ttl = (@interval * LOCK_TTL_RATIO * 1000).ceil
|
|
375
337
|
redis do |c|
|