wurk 1.11.1 → 1.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +1 -1
  3. data/exe/wurk +4 -2
  4. data/lib/generators/wurk/install/templates/wurk.rb +5 -3
  5. data/lib/wurk/boot_checks.rb +130 -5
  6. data/lib/wurk/cli.rb +72 -10
  7. data/lib/wurk/configuration.rb +1 -1
  8. data/lib/wurk/fetcher/oom_drain.rb +0 -5
  9. data/lib/wurk/fetcher/orphan_reclaim.rb +71 -0
  10. data/lib/wurk/fetcher/reaper.rb +18 -56
  11. data/lib/wurk/fetcher/self_reconcile.rb +8 -14
  12. data/lib/wurk/job_util.rb +18 -2
  13. data/lib/wurk/lua/reclaim_orphan.lua +31 -0
  14. data/lib/wurk/lua.rb +5 -20
  15. data/lib/wurk/middleware/poison_pill.rb +31 -47
  16. data/lib/wurk/rails_boot.rb +40 -11
  17. data/lib/wurk/unique.rb +12 -2
  18. data/lib/wurk/version.rb +1 -1
  19. data/lib/wurk.rb +4 -0
  20. data/vendor/assets/dashboard/assets/{ArgsValue-CmaXArv8.js → ArgsValue-BJol9S0s.js} +1 -1
  21. data/vendor/assets/dashboard/assets/{BatchDetail-nc5yePZW.js → BatchDetail-D_zGQ9Sd.js} +1 -1
  22. data/vendor/assets/dashboard/assets/{Batches-DgEcyOZ_.js → Batches-Dt_xC660.js} +1 -1
  23. data/vendor/assets/dashboard/assets/{Busy-CUZncvCy.js → Busy-CWRthsmc.js} +1 -1
  24. data/vendor/assets/dashboard/assets/{Cron-CIZbaaZs.js → Cron-qeGr2YAm.js} +1 -1
  25. data/vendor/assets/dashboard/assets/Dashboard-DQWaAwPA.js +1 -0
  26. data/vendor/assets/dashboard/assets/Dead-D9gOPHtK.js +1 -0
  27. data/vendor/assets/dashboard/assets/{Extension-Doy4bJoA.js → Extension-DONXVLdA.js} +1 -1
  28. data/vendor/assets/dashboard/assets/{FilterBox-C4bO26_d.js → FilterBox-BbB6gXHp.js} +1 -1
  29. data/vendor/assets/dashboard/assets/{FlowDetail-Dh3SJdFh.js → FlowDetail-CrmnVRp0.js} +1 -1
  30. data/vendor/assets/dashboard/assets/{Flows-C5SMRuhq.js → Flows-g4GC8KmE.js} +1 -1
  31. data/vendor/assets/dashboard/assets/{JobDetailModal-Cg6sAsJB.js → JobDetailModal-C86k_1wj.js} +1 -1
  32. data/vendor/assets/dashboard/assets/{Limiters-BIxwDOJM.js → Limiters-zyO-iMlK.js} +1 -1
  33. data/vendor/assets/dashboard/assets/{Metrics-Qp9q0nF8.js → Metrics-Ckq5fNBv.js} +1 -1
  34. data/vendor/assets/dashboard/assets/{NotFound-DwjPxFjU.js → NotFound-C8NnyjlY.js} +1 -1
  35. data/vendor/assets/dashboard/assets/{Profiles-7dabYVe3.js → Profiles-DRl-FXPA.js} +1 -1
  36. data/vendor/assets/dashboard/assets/Queues-DkXWrihA.js +1 -0
  37. data/vendor/assets/dashboard/assets/Retries-CmlojYIy.js +1 -0
  38. data/vendor/assets/dashboard/assets/Scheduled-Cbbpp3xw.js +1 -0
  39. data/vendor/assets/dashboard/assets/{Search-BAsVE2qw.js → Search-LaEpj2fM.js} +1 -1
  40. data/vendor/assets/dashboard/assets/{charts-DZs138BV.js → charts-D_PQLoyc.js} +1 -1
  41. data/vendor/assets/dashboard/assets/index-FBKGtysy.js +141 -0
  42. data/vendor/assets/dashboard/assets/{useResetPageOnEmpty-DxkLZCIm.js → useResetPageOnEmpty-BiwhHqgh.js} +1 -1
  43. data/vendor/assets/dashboard/assets/{utils-B0kQFNE8.js → utils-CnWZzW2L.js} +1 -1
  44. data/vendor/assets/dashboard/index.html +1 -1
  45. data/vendor/assets/dashboard/wurk-manifest.json +2 -2
  46. metadata +27 -25
  47. data/vendor/assets/dashboard/assets/Dashboard-BWmfAook.js +0 -1
  48. data/vendor/assets/dashboard/assets/Dead-CGKsTZHq.js +0 -1
  49. data/vendor/assets/dashboard/assets/Queues-CMg-gyP8.js +0 -1
  50. data/vendor/assets/dashboard/assets/Retries-CNX8ZtTH.js +0 -1
  51. data/vendor/assets/dashboard/assets/Scheduled-ClVBj0nc.js +0 -1
  52. data/vendor/assets/dashboard/assets/index-CMzPFxNE.js +0 -141
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8153a2b51dcd55ed7857e54b5cd2d533ca0e37f7d443e93f5d2bc33e5be171c4
4
- data.tar.gz: 85ee27f3ee4bb24004aa8d50372f0a555312e3a9607f8b1f644155d24722114e
3
+ metadata.gz: efda965ecab6b73c86eddfd4f0b8acc5f0942bcba46d933aa60e922aae453c1f
4
+ data.tar.gz: 404c2f3b9055edeafedd0c1a98f1933799c89830d0c2f70d08fb4018c183fa4f
5
5
  SHA512:
6
- metadata.gz: 4153f8cfbcd87743ec7b13a9433ffd2fa2238fbe7a4e5aeae3041d1f14fbea8c74a009287b418e934273938bed6ec619c1671bb411ae0de8c979532a0f7f4fca
7
- data.tar.gz: ae57460f22da77be15b74a36a2b870630eeec2c694dab6e25e3e28bdfe4b48cf6264703e0167b6a47cf5d1258b2d96c168b57370a9172a6dbc34dde0ec2d53fe
6
+ metadata.gz: 1d466f7764855b501fe06a0e3a2586bf4908b6bd65274e74fa5739b77255613801bc6625fdee52f4a4771cc4c1ea49e6f9b171fa364e8fd5fda95997b788da0c
7
+ data.tar.gz: 1333a3f95fc34da5a126caae9ff041a404bb80c221f425177e9034bf16bb4a16abc11fe9ad8aa1a8e18470f8066cc8aba0d9c45de2dea6ee46bbb38795b83a66
data/README.md CHANGED
@@ -22,7 +22,7 @@
22
22
 
23
23
  Wurk is wire-compatible with Sidekiq — same Redis keys, same job JSON, same Ruby DSL. Swap one line in your `Gemfile` and your existing jobs, batches, limiters, cron entries, and live Redis data keep working untouched. The Pro and Enterprise feature sets ship in the same free gem, with no license check and no tiers.
24
24
 
25
- **In production:** Wurk runs background work at partner deployments, across many servers, on the fork-based swarm described below. It is not a preview. Throughput numbers from those deployments are not published yet; what has been measured is in [docs/benchmarks.md](docs/benchmarks.md).
25
+ **In production:** Wurk runs background work at partner deployments, across many servers, on the fork-based swarm described below. It is not a preview. Teams running billions of jobs a month on Wurk in production recommend it. Those volumes are user-reported; no production throughput numbers are published yet, and what has been measured is in [docs/benchmarks.md](docs/benchmarks.md).
26
26
 
27
27
  **At scale:** Wurk is built for fleets, not just for one box. Kubernetes `/live` + `/ready` probes are a config line, not a sidecar; a bearer-scoped [HTTP API](docs/api-http.md) lets non-Ruby services enqueue and inspect; [OpenTelemetry](docs/telemetry.md) traces propagate client → server; per-queue [global concurrency caps](docs/rate-limiting.md) hold cluster-wide limits; and monitoring is the dashboard you already mount — live SSE, charts, no separate stack to run. See [Wurk extras](#wurk-extras).
28
28
 
data/exe/wurk CHANGED
@@ -5,6 +5,7 @@
5
5
  # Usage: `exe/wurk -C config/wurk.yml` — worker: one process, thread pool
6
6
  # `exe/wurk api --port 7433` — machine HTTP API, runs no jobs
7
7
  # `exe/wurk preflight` / `verify` — read-only cutover checks, run no jobs
8
+ # `exe/wurk boot-check -r .` — boot the app as wurkswarm would, then exit (CI)
8
9
 
9
10
  $stdout.sync = true
10
11
  $LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
@@ -20,8 +21,9 @@ begin
20
21
  cli = Wurk::CLI.instance
21
22
  cli.parse
22
23
  case cli.command
23
- when 'api' then cli.run_api
24
- else cli.run
24
+ when 'api' then cli.run_api
25
+ when 'boot-check' then exit(cli.run_boot_check)
26
+ else cli.run
25
27
  end
26
28
  rescue StandardError => e
27
29
  raise e if $DEBUG
@@ -1,8 +1,8 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  # Wurk configuration. Generated by `bin/rails g wurk:install`.
4
- # Full surface area: docs/target/sidekiq-free.md. Coming from Sidekiq?
5
- # See docs/migrate-from-sidekiq.md.
4
+ # Every option: https://github.com/developerz-ai/wurk/blob/main/docs/configuration.md
5
+ # Coming from Sidekiq? https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md
6
6
 
7
7
  Wurk.configure_server do |config|
8
8
  # config.redis = { url: ENV.fetch("REDIS_URL", "redis://localhost:6379/0") }
@@ -19,7 +19,9 @@ Wurk.configure_server do |config|
19
19
  # By default Wurk forks one worker process per usable CPU (cgroup quota aware) (set WURK_COUNT, or
20
20
  # SIDEKIQ_COUNT, to override). For full control declare a topology — `flat`
21
21
  # spawns N identical forks; use `slot`s for dedicated queues. Total in-flight
22
- # jobs = forks × concurrency. See docs/idea/03-process-model.md.
22
+ # jobs = forks × concurrency. Topology and sizing:
23
+ # https://github.com/developerz-ai/wurk/blob/main/docs/configuration.md#topology
24
+ # https://github.com/developerz-ai/wurk/blob/main/docs/deployment.md#capacity-and-sizing
23
25
  # config.topology = Wurk::Topology.flat(count: 2, queues: %w[critical default low], concurrency: 10)
24
26
  end
25
27
 
@@ -2,6 +2,7 @@
2
2
 
3
3
  require_relative 'keys'
4
4
  require_relative 'queue_consumers'
5
+ require_relative 'capsule'
5
6
 
6
7
  module Wurk
7
8
  # What a standalone runner says once at boot, after the app loaded. Log only,
@@ -13,13 +14,20 @@ module Wurk
13
14
  MAX_LISTED = 20
14
15
  SUBCLASS = ::Module.instance_method(:<)
15
16
  NAME = ::Module.instance_method(:name)
17
+ UNIQUE_OPTIONS = %w[unique_for unique_until].freeze
18
+ STRICT_ENV = 'WURK_STRICT'
16
19
 
17
20
  module_function
18
21
 
22
+ # `info` is nil when there is no Redis to ask (`wurk boot-check` without
23
+ # `--redis`): only the checks that read the loaded app run then.
19
24
  def run(config, info)
20
- log_backend(config.logger, info)
21
- warn_on_foreign_processes(config)
25
+ if info
26
+ log_backend(config.logger, info)
27
+ warn_on_foreign_processes(config)
28
+ end
22
29
  warn_on_ignored_locks(config.logger)
30
+ warn_on_ignored_unique(config.logger)
23
31
  rescue StandardError => e
24
32
  config.logger.warn { "wurk boot checks skipped: #{e.class}: #{e.message}" }
25
33
  end
@@ -73,18 +81,135 @@ module Wurk
73
81
  end
74
82
  end
75
83
 
84
+ # Only the middleware `unique!` installs reads these options, so without it
85
+ # every push enqueues. Turning it on for the app is a human call, not a
86
+ # boot check's: this only says so.
87
+ def warn_on_ignored_unique(logger)
88
+ return if Wurk::Unique.enabled?
89
+
90
+ classes = job_classes_setting(UNIQUE_OPTIONS)
91
+ return if classes.empty?
92
+
93
+ logger.warn do
94
+ 'These jobs set `unique_for:` or `unique_until:` but unique jobs are not enabled, so the options do ' \
95
+ "nothing and duplicates enqueue: #{listing(classes)}. Enable them once at boot, in an initializer: " \
96
+ '`Sidekiq::Enterprise.unique!`.'
97
+ end
98
+ end
99
+
100
+ def locked_job_classes = job_classes_setting(%w[lock])
101
+
102
+ # Workers, and ActiveJob classes that take `sidekiq_options` (Wurk::Job::Options).
76
103
  # A walk over every class meets classes that redefine these with other arguments
77
104
  # (Faker::Travel::Airport.name takes required keywords), and one such class must
78
105
  # not cost the app this check.
79
- def locked_job_classes
106
+ def job_classes_setting(options)
80
107
  ::ObjectSpace.each_object(::Class).filter_map do |klass|
81
- next unless SUBCLASS.bind_call(klass, Wurk::Worker)
108
+ next unless SUBCLASS.bind_call(klass, Wurk::Worker) || SUBCLASS.bind_call(klass, Wurk::Job::Options)
82
109
 
83
110
  name = NAME.bind_call(klass)
84
- name if name && klass.get_sidekiq_options.key?('lock')
111
+ name if name && klass.get_sidekiq_options.values_at(*options).any?
85
112
  end.sort
86
113
  end
87
114
 
115
+ # The real gem next to Wurk loads both: `Sidekiq::*` is then partly theirs,
116
+ # partly ours. Gems that depend on `sidekiq` pull it in unless the shim gem,
117
+ # which depends on Wurk instead, satisfies the dependency. Runs as Wurk loads.
118
+ def flag_real_sidekiq_gem(logger, specs: ::Gem.loaded_specs, strict: ENV[STRICT_ENV] == '1')
119
+ message = real_sidekiq_message(specs, logger)
120
+ return unless message
121
+ raise Wurk::Error, message if strict
122
+
123
+ logger.error { "#{message} Set #{STRICT_ENV}=1 to refuse to boot instead." }
124
+ end
125
+
126
+ def real_sidekiq_message(specs, logger)
127
+ real = specs['sidekiq']
128
+ return if real.nil? || real.dependencies.any? { |dep| dep.name == 'wurk' }
129
+
130
+ "The bundle holds the real sidekiq #{real.version} gem, so it loads next to Wurk and the process runs " \
131
+ "half on each. Pulled in by: #{sidekiq_dependents(specs)}. Replace it with the shim in the Gemfile: " \
132
+ "gem 'sidekiq', github: 'developerz-ai/wurk', tag: 'v#{Wurk::VERSION}', " \
133
+ "glob: 'ecosystem/sidekiq-shim/*.gemspec'."
134
+ rescue StandardError => e
135
+ logger.warn { "wurk: could not check the bundle for the real sidekiq gem: #{e.class}: #{e.message}" }
136
+ nil
137
+ end
138
+
139
+ def sidekiq_dependents(specs)
140
+ names = specs.each_value.select { |spec| spec.dependencies.any? { |dep| dep.name == 'sidekiq' } }.map(&:name)
141
+ names.empty? ? 'the Gemfile itself' : listing(names.sort)
142
+ end
143
+
144
+ # The role line of a process that forks the topology's children and runs no
145
+ # jobs itself. Its children each run the threads of one slot.
146
+ def log_swarm_role(config, role, ar_pool: active_record_pool)
147
+ slots = config.topology.slots.map do |slot|
148
+ ["#{slot.count} #{slot.count == 1 ? 'process' : 'processes'}", slot.concurrency]
149
+ end
150
+ log_role(config, role, slots, per_process: slots.map(&:last).max, ar_pool: ar_pool)
151
+ rescue StandardError => e
152
+ role_skipped(config.logger, e)
153
+ end
154
+
155
+ # The role line of a process that runs every capsule's threads itself.
156
+ def log_process_role(config, role, ar_pool: active_record_pool)
157
+ capsules = config.capsules.values
158
+ slots = capsules.map { |cap| [capsules.size > 1 ? "capsule #{cap.name}" : '1 process', cap.concurrency] }
159
+ log_role(config, role, slots, per_process: slots.sum(&:last), ar_pool: ar_pool)
160
+ rescue StandardError => e
161
+ role_skipped(config.logger, e)
162
+ end
163
+
164
+ def role_skipped(logger, error)
165
+ logger.warn { "wurk role line skipped: #{error.class}: #{error.message}" }
166
+ end
167
+
168
+ # One line per process: what it is, and how big. `per_process` is the most
169
+ # job threads one process runs, which is what its Active Record pool covers.
170
+ def log_role(config, role, slots, per_process:, ar_pool:)
171
+ sizes = slots.map do |label, threads|
172
+ "#{label} × #{threads} threads (redis pool #{redis_pool_size(config, threads)}, fetch #{threads})"
173
+ end
174
+ config.logger.info { "wurk: #{role}; #{sizes.join(' + ')}#{"; AR pool #{ar_pool}" if ar_pool}" }
175
+ warn_on_small_ar_pool(config.logger, ar_pool, per_process)
176
+ end
177
+
178
+ # DEBUG, not INFO: Wurk logs to stdout, and every `rails runner`, rake task
179
+ # and `wurk preflight --app` boots the app; their stdout is someone's output.
180
+ def log_client_only(logger, reason)
181
+ logger.debug { "wurk: client only (#{reason}); this process enqueues and runs no jobs" }
182
+ end
183
+
184
+ # Capsule#main_pool_size, without building a pool: a swarm parent describes
185
+ # slots whose pools only its children build.
186
+ def redis_pool_size(config, threads)
187
+ config.redis_config[:size] || [threads + Capsule::POOL_HEADROOM, Capsule::MIN_POOL_SIZE].max
188
+ end
189
+
190
+ # Every job thread that touches the database holds a connection; with fewer
191
+ # in the pool the rest wait, then fail with ActiveRecord::ConnectionTimeoutError.
192
+ def warn_on_small_ar_pool(logger, ar_pool, threads)
193
+ return unless ar_pool && ar_pool < threads
194
+
195
+ logger.warn do
196
+ "The Active Record pool holds #{ar_pool} connections but this process runs up to #{threads} job threads, " \
197
+ 'so jobs that use the database wait for a connection and then fail on a timeout. Set `pool:` in ' \
198
+ "config/database.yml to at least #{threads} (plus the web server's threads when jobs run in the web process)."
199
+ end
200
+ end
201
+
202
+ # The pool size the app configured, read without opening a connection; nil
203
+ # without Active Record or a configured database. Rails 8.1 renamed `pool`.
204
+ def active_record_pool
205
+ return unless defined?(::ActiveRecord::Base)
206
+
207
+ db = ::ActiveRecord::Base.connection_db_config
208
+ db.respond_to?(:max_connections) ? db.max_connections : db.pool
209
+ rescue ::ActiveRecord::ActiveRecordError
210
+ nil
211
+ end
212
+
88
213
  def listing(names)
89
214
  extra = names.size - MAX_LISTED
90
215
  names.first(MAX_LISTED).join(', ') + (extra.positive? ? " and #{extra} more" : '')
data/lib/wurk/cli.rb CHANGED
@@ -32,12 +32,13 @@ module Wurk
32
32
  # Subcommands. Bare `wurk` runs the worker — the historical shape, and the
33
33
  # only one the swarm binaries accept — so a subcommand is never anything
34
34
  # but the first argument, and every argument after it is that command's.
35
- COMMANDS = %w[api health].freeze
35
+ COMMANDS = %w[api boot-check health].freeze
36
36
  # Set only across the one re-exec of #load_rails_framework, and removed on arrival.
37
37
  PRELOAD_FIRST_ENV = 'WURK_BOOT_PRELOAD_FIRST'
38
38
  # `bundle exec` loads the binstub in-process, so argv0 is Bundler's executable, and
39
39
  # `ruby -e "load Gem.bin_path(...)"` has `-e` there: the running program can't name it.
40
40
  SWARM_EXECUTABLE = ::File.expand_path('../../exe/wurkswarm', __dir__)
41
+ WURK_EXECUTABLE = ::File.expand_path('../../exe/wurk', __dir__)
41
42
 
42
43
  # A refusal this file raises itself, with nothing a trace would add.
43
44
  class StartupError < ArgumentError; end
@@ -110,6 +111,10 @@ module Wurk
110
111
  ['-s', '--server NAME', :api_server, 'Rack handler to serve with (default: the first installed)']
111
112
  ].freeze
112
113
 
114
+ BOOT_CHECK_OPTION_FLAGS = [
115
+ ['-R', '--redis', :boot_check_redis, 'Also read Redis: version, eviction policy, live peers (read-only)']
116
+ ].freeze
117
+
113
118
  HEALTH_OPTION_FLAGS = [
114
119
  ['-w', '--window SECONDS', :health_window, 'A heartbeat older than this does not count (default 60)', :to_i],
115
120
  ['-m', '--min COUNT', :health_min, 'Processes on this host that must be beating (default 1)', :to_i],
@@ -118,6 +123,7 @@ module Wurk
118
123
  ].freeze
119
124
 
120
125
  HELP_COMMANDS = "\nCommands:\n " \
126
+ "boot-check Boot the app the way wurkswarm does, then exit; no fork, no fetch (CI)\n " \
121
127
  "health Exit 0 if this host's workers are heartbeating\n " \
122
128
  "preflight Read a live Redis: what will carry over to Wurk (read-only)\n " \
123
129
  "verify Post-cutover check of a live fleet (read-only)\n\nOptions:"
@@ -209,6 +215,7 @@ module Wurk
209
215
  validate_redis!
210
216
  validate_pool_sizes!
211
217
  BootChecks.run(@config, redis_info)
218
+ BootChecks.log_process_role(@config, 'wurk')
212
219
  @config[:identity] = identity
213
220
  # Force lazy server-middleware chain so worker threads don't race
214
221
  # against each other constructing it. Spec: Sidekiq::CLI line 104.
@@ -245,15 +252,11 @@ module Wurk
245
252
  # Only the children read it. Rejected here, a typo fails this boot instead of
246
253
  # crash-looping the fleet until the crash-loop exit.
247
254
  Health.options_from_env
248
- if boot_app
249
- load_rails_framework
250
- preload_bundler_groups
251
- boot_application
252
- eager_load_application
253
- end
255
+ boot_swarm_application if boot_app
254
256
  validate_redis!
255
257
  validate_pool_sizes!
256
258
  BootChecks.run(@config, redis_info)
259
+ log_swarm_role
257
260
  @config[:identity] = identity
258
261
  warm_up_process if warmup
259
262
  @swarm = Wurk::Swarm.new(topology: @config.topology, config: @config, exit_on_crash_loop: true,
@@ -270,6 +273,27 @@ module Wurk
270
273
  exit(1) if @swarm.crash_looped?
271
274
  end
272
275
 
276
+ # `wurk boot-check` — #run_swarm's boot up to the fork: the same require
277
+ # order, config file and checks, plus what a worker reaches only later (every
278
+ # constant, the routes), so a CI step finds what would crash-loop a deploy.
279
+ # Forks nothing, fetches nothing, fires no lifecycle event, and reads Redis
280
+ # only with --redis. Returns the exit code; a boot that raises is the binary's
281
+ # failure report.
282
+ def run_boot_check(boot_app: true)
283
+ enter_server_mode
284
+ Health.options_from_env
285
+ if boot_app
286
+ boot_swarm_application
287
+ eager_load_rails
288
+ draw_routes
289
+ end
290
+ validate_pool_sizes!
291
+ BootChecks.run(@config, boot_check_redis_info)
292
+ log_swarm_role
293
+ logger.info { "wurk boot-check: #{@config[:require]} booted as wurkswarm would" }
294
+ 0
295
+ end
296
+
273
297
  # `wurk health` — see Wurk::HealthProbe. Boots nothing; returns the exit code.
274
298
  def run_health
275
299
  require_relative 'health_probe'
@@ -434,10 +458,26 @@ module Wurk
434
458
 
435
459
  def validate_pool_sizes!
436
460
  @config.capsules.each_pair do |name, cap|
437
- raise StartupError, "Pool size too small for #{name}" if cap.redis_pool.size < cap.concurrency
461
+ size = cap.redis_pool.size
462
+ next if size >= cap.concurrency
463
+
464
+ raise StartupError, "the Redis pool for capsule #{name} holds #{size} connections, fewer than its " \
465
+ "concurrency of #{cap.concurrency}. Set `config.redis = { size: #{cap.concurrency} }` " \
466
+ 'or more, or drop `size:` and Wurk sizes the pool itself (concurrency + 5).'
438
467
  end
439
468
  end
440
469
 
470
+ def boot_check_redis_info
471
+ return unless @config[:boot_check_redis]
472
+
473
+ validate_redis!
474
+ redis_info
475
+ end
476
+
477
+ def log_swarm_role
478
+ BootChecks.log_swarm_role(@config, @command ? "#{@command} (as wurkswarm parent)" : 'wurkswarm parent')
479
+ end
480
+
441
481
  # --- options + config-file --------------------------------------------
442
482
 
443
483
  # Pulled off before OptionParser sees argv, so the banner and the flag
@@ -448,6 +488,20 @@ module Wurk
448
488
  COMMANDS.include?(args.first) ? args.shift : nil
449
489
  end
450
490
 
491
+ def boot_swarm_application
492
+ load_rails_framework
493
+ preload_bundler_groups
494
+ boot_application
495
+ eager_load_application
496
+ end
497
+
498
+ # Rails may draw routes only on the first request; a `mount` of a constant
499
+ # only the web process loads must fail here, not on that request.
500
+ def draw_routes
501
+ app = rails_application
502
+ app.reload_routes! if app.respond_to?(:reload_routes!)
503
+ end
504
+
451
505
  def parse_health_options(args)
452
506
  @health_options = {}
453
507
  parser = option_parser(@health_options)
@@ -547,6 +601,7 @@ module Wurk
547
601
  o.separator help_intro if help_intro
548
602
  define_value_flags(o, opts, @command == 'health' ? HEALTH_OPTION_FLAGS : OPTION_FLAGS)
549
603
  define_value_flags(o, opts, API_OPTION_FLAGS) if @command == 'api'
604
+ define_value_flags(o, opts, BOOT_CHECK_OPTION_FLAGS) if @command == 'boot-check'
550
605
  define_meta_flags(o, opts)
551
606
  end
552
607
  end
@@ -627,7 +682,7 @@ module Wurk
627
682
  end
628
683
 
629
684
  def restart_state
630
- { env: ::ENV.to_h, argv: ::ARGV.dup, dir: ::Dir.pwd }
685
+ { env: ::ENV.to_h, argv: [restart_executable, *::ARGV], dir: ::Dir.pwd }
631
686
  end
632
687
 
633
688
  def note_preload_first(first_error)
@@ -644,11 +699,14 @@ module Wurk
644
699
  env = state[:env].merge(PRELOAD_FIRST_ENV => error.message.lines.first.to_s.strip[0, 300])
645
700
  env['BUNDLE_GEMFILE'] ||= ::Bundler.default_gemfile.to_s if defined?(::Bundler)
646
701
  ruby = [::RbConfig.ruby, *('-rbundler/setup' if defined?(::Bundler))]
647
- ::Process.exec(env, *ruby, SWARM_EXECUTABLE, *state[:argv], unsetenv_others: true, chdir: state[:dir])
702
+ ::Process.exec(env, *ruby, *state[:argv], unsetenv_others: true, chdir: state[:dir])
648
703
  rescue ::SystemCallError
649
704
  raise error
650
705
  end
651
706
 
707
+ # The binary that was run: only `wurk boot-check` reaches this with a command.
708
+ def restart_executable = @command ? WURK_EXECUTABLE : SWARM_EXECUTABLE
709
+
652
710
  # The app root for a `-r` that is the app directory or its environment file.
653
711
  def rails_app_root(require_path)
654
712
  path = ::File.expand_path(require_path.to_s)
@@ -688,6 +746,10 @@ module Wurk
688
746
  def eager_load_application
689
747
  return unless preload_app?
690
748
 
749
+ eager_load_rails
750
+ end
751
+
752
+ def eager_load_rails
691
753
  app = rails_application
692
754
  app.eager_load! if app.respond_to?(:eager_load!)
693
755
  end
@@ -452,7 +452,7 @@ module Wurk
452
452
  # The optional block is Pro's recovery callback: `|jobstr, pill|`, fired
453
453
  # once per orphan recovery (`pill` nil) and once on a poison kill (`pill`
454
454
  # responds to .jid/.klass/.count/.queue). The reaper drives it via
455
- # Wurk::Middleware::PoisonPill.track!. Spec: docs/target/sidekiq-pro.md §3.1.
455
+ # Wurk::Middleware::PoisonPill.recovered!. Spec: docs/target/sidekiq-pro.md §3.1.
456
456
  def super_fetch!(*, &block)
457
457
  @super_fetch_callback = block if block
458
458
  nil
@@ -50,7 +50,6 @@ module Wurk
50
50
  return nil
51
51
  LUA
52
52
 
53
- RECLAIM = (HEADER + Lua::RELIABLE_RECLAIM).freeze
54
53
  REQUEUE = (HEADER + Lua::RELIABLE_REQUEUE).freeze
55
54
 
56
55
  # KEYS = [lock], ARGV = [ttl_ms]. The reaper's sweep lock: refused, no sweep
@@ -72,10 +71,6 @@ module Wurk
72
71
  run(conn, error, CLAIM, [Keys::PROCESSES, Component.identity, *keys], [info, beat, HEARTBEAT_TTL])
73
72
  end
74
73
 
75
- def reclaim(conn, error, source, destination)
76
- run(conn, error, RECLAIM, [source, destination], [])
77
- end
78
-
79
74
  # Returns 1 when moved, 0 when the job has left the private list since.
80
75
  def requeue(conn, error, private_list, public_q, job)
81
76
  run(conn, error, REQUEUE, [private_list, public_q], [job])
@@ -0,0 +1,71 @@
1
+ # frozen_string_literal: true
2
+
3
+ require_relative '../keys'
4
+ require_relative '../lua'
5
+ require_relative '../middleware/poison_pill'
6
+ require_relative '../metrics/process_counters'
7
+ require_relative 'oom_drain'
8
+
9
+ module Wurk
10
+ class Fetcher
11
+ # Puts one payload from a private list back where it belongs, for both ways
12
+ # an orphan is found: the Reaper draining a dead owner's list, and
13
+ # SelfReconcile returning a payload its own live process lost track of.
14
+ #
15
+ # The poison-pill verdict is taken in the same script that moves the job
16
+ # (`lua/reclaim_orphan.lua`), so a job past the cap goes straight from the
17
+ # private list to the dead set and is never visible on its public queue. The
18
+ # notifications (death handlers, `super_fetch!`, `on_poison`) follow once
19
+ # the script has returned, for a job no worker can be running.
20
+ class OrphanReclaim
21
+ THRESHOLD = Middleware::PoisonPill::RECOVERY_THRESHOLD
22
+ TTL = Middleware::PoisonPill::RECOVERY_TTL
23
+
24
+ def initialize(config)
25
+ @config = config
26
+ end
27
+
28
+ # Returns true when this call took the payload off the private list, false
29
+ # when it had already left (another reaper, an ACK).
30
+ #
31
+ # Unclaimed apply-safety: the move is guarded, but a reply lost after a
32
+ # poison kill would leave the death handlers unfired, and a list holding
33
+ # two identical copies would move the second on a replay. A raise leaves
34
+ # the payload for the next pass instead.
35
+ def call(private_list, public_q, payload)
36
+ job = Middleware::PoisonPill.parse(payload)
37
+ count = @config.redis { |conn| move(conn, private_list, public_q, payload, job) }
38
+ return false unless count
39
+
40
+ Metrics::ProcessCounters.increment(:reaper_reclaimed)
41
+ unless count == :uncounted
42
+ Middleware::PoisonPill.recovered!(payload, job, count: count, queue: queue_name(public_q),
43
+ config: @config)
44
+ end
45
+ true
46
+ end
47
+
48
+ private
49
+
50
+ # Past maxmemory the counter cannot be written, nor the job killed: it goes
51
+ # back uncounted rather than waiting on the memory it would help free.
52
+ def move(conn, private_list, public_q, payload, job)
53
+ keys = [private_list, public_q, Keys::DEAD]
54
+ jid = job && job['jid']
55
+ keys << Middleware::PoisonPill.counter_key(jid) if jid.is_a?(String) && !jid.empty?
56
+ Wurk::Lua::Loader.eval_cached(conn, :reclaim_orphan, keys: keys, argv: [payload, THRESHOLD, TTL, now])
57
+ rescue RedisClient::OutOfMemoryError => e
58
+ :uncounted if OomDrain.requeue(conn, e, private_list, public_q, payload) == 1
59
+ end
60
+
61
+ # DeadSet#kill's score format, so the entry sorts with every other kill.
62
+ def now
63
+ ::Process.clock_gettime(::Process::CLOCK_REALTIME).to_s
64
+ end
65
+
66
+ def queue_name(public_q)
67
+ public_q.delete_prefix(Keys::QUEUE_PREFIX)
68
+ end
69
+ end
70
+ end
71
+ end
@@ -2,13 +2,12 @@
2
2
 
3
3
  require_relative '../component'
4
4
  require_relative '../keys'
5
- require_relative '../middleware/poison_pill'
6
- require_relative '../metrics/process_counters'
7
5
  require_relative '../timer_loop'
8
6
  require_relative 'private_list_key'
9
7
  require_relative 'unparseable_keys'
10
8
  require_relative 'orphan_grace'
11
9
  require_relative 'oom_drain'
10
+ require_relative 'orphan_reclaim'
12
11
  require_relative 'self_reconcile'
13
12
 
14
13
  module Wurk
@@ -45,10 +44,10 @@ module Wurk
45
44
  # process's own clock (OrphanGrace), and its identity must still have no
46
45
  # `info` hash. Our own incarnation skips both: kill(0) is not a snapshot.
47
46
  #
48
- # Re-pushed jobs run through Wurk::Middleware::PoisonPill, which caps a
49
- # job at RECOVERY_THRESHOLD recoveries within 72h: past the cap the job
50
- # is killed into the dead set instead of re-queued, so a job that crashes
51
- # its worker every time can't loop forever.
47
+ # Every move goes through OrphanReclaim, which caps a job at
48
+ # Middleware::PoisonPill::RECOVERY_THRESHOLD recoveries within 72h: past the
49
+ # cap the job goes from the private list straight to the dead set instead of
50
+ # its queue, so a job that crashes its worker every time can't loop forever.
52
51
  #
53
52
  # One sweep an interval: one SCAN of `queue:*|*` acting on every list, gated
54
53
  # by a cluster lock so about one process sweeps per interval. Every list, and
@@ -62,7 +61,7 @@ module Wurk
62
61
  # of a job.
63
62
  #
64
63
  # Spec: docs/target/sidekiq-pro.md §3.2.
65
- class Reaper # rubocop:disable Metrics/ClassLength
64
+ class Reaper
66
65
  include Component
67
66
 
68
67
  # Sweep cadence in seconds. 60s matches the heartbeat TTL — the floor below
@@ -103,8 +102,8 @@ module Wurk
103
102
  @sleeper = ::ConditionVariable.new
104
103
  @unparseable = UnparseableKeys.new(config)
105
104
  @next_sweep_at = 0.0
106
- @self_reconcile = SelfReconcile.new(config, confirm: reconcile_confirm, stop: -> { done? },
107
- on_moved: method(:recovered))
105
+ @orphan_reclaim = OrphanReclaim.new(config)
106
+ @self_reconcile = SelfReconcile.new(config, confirm: reconcile_confirm, stop: -> { done? })
108
107
  end
109
108
 
110
109
  # Spawns the sweep loop. Idempotent. The loop waits one interval before
@@ -313,27 +312,19 @@ module Wurk
313
312
  identity.split(':')[0..1].join(':')
314
313
  end
315
314
 
316
- # Drain one orphaned private list back to its public queue. Each job is
317
- # moved with an atomic LMOVE (private tail → public tail) BEFORE the
318
- # poison check, so a crash mid-drain leaves the job safely in the public
319
- # queue (at-least-once), never lost. Poison jobs are killed to the dead
320
- # set by PoisonPill.track! and then LREM'd out of the public queue.
315
+ # Drain one orphaned private list back to its public queue, a job at a
316
+ # time from the tail: peek it, then OrphanReclaim moves it, to the public
317
+ # tail or, past the poison cap, to the dead set. The decision is taken
318
+ # before the job is visible, so a worker parked in BLMOVE can never take a
319
+ # job that is being killed. A peeked job that has left by the time it is
320
+ # moved (another reaper drained it) costs one more peek.
321
321
  #
322
- # Unlike the fetcher's LMOVE, this one does *not* claim apply-safety: a
323
- # replay after a lost reply moves the next job and never poison-checks the
324
- # one already on the public tail, so a job that kills its worker every time
325
- # would get a free recovery past the cap. A raise instead lands in the
326
- # rescue below, and the next sweep re-drains what's left.
322
+ # A raise lands in the rescue below; the job stays in the private list
323
+ # and the next sweep re-drains what's left.
327
324
  def drain(private_list, public_q)
328
- queue_name = public_q.delete_prefix(Keys::QUEUE_PREFIX)
329
325
  count = 0
330
- loop do
331
- job = redis { |c| reclaim_one(c, private_list, public_q) }
332
- break unless job
333
-
334
- count += 1
335
- Metrics::ProcessCounters.increment(:reaper_reclaimed)
336
- poison_off(public_q, job, queue_name)
326
+ while (job = redis(idempotent: true) { |c| c.call('LINDEX', private_list, -1) })
327
+ count += 1 if @orphan_reclaim.call(private_list, public_q, job)
337
328
  end
338
329
  count
339
330
  rescue StandardError => e
@@ -341,35 +332,6 @@ module Wurk
341
332
  count
342
333
  end
343
334
 
344
- # A full Redis must not hold a dead worker's jobs back from the workers
345
- # still able to drain the queue.
346
- def reclaim_one(conn, private_list, public_q)
347
- Wurk::Lua::Loader.eval_cached(conn, :reliable_reclaim, keys: [private_list, public_q], argv: [])
348
- rescue RedisClient::OutOfMemoryError => e
349
- OomDrain.reclaim(conn, e, private_list, public_q)
350
- end
351
-
352
- # SelfReconcile put a payload back: the same accounting as a reclaim, so a
353
- # job that kills its processor thread every time still hits the poison cap.
354
- def recovered(public_q, job)
355
- Metrics::ProcessCounters.increment(:reaper_reclaimed)
356
- poison_off(public_q, job, public_q.delete_prefix(Keys::QUEUE_PREFIX))
357
- end
358
-
359
- # The job is back on its queue whatever happens here. Past maxmemory the
360
- # count can't be taken (nor the job killed, leaving it where it is): raising
361
- # would end the drain at one job per pass, with the rest of the list waiting
362
- # on memory they would help free.
363
- def poison_off(public_q, job, queue_name)
364
- return unless Middleware::PoisonPill.track!(job, queue: queue_name, config: @config) == :poison
365
-
366
- # track! already ZADDed the payload to the dead set; pull the copy we
367
- # just moved onto the public tail so it isn't also re-run.
368
- redis { |c| c.call('LREM', public_q, -1, job) }
369
- rescue RedisClient::OutOfMemoryError => e
370
- report(e)
371
- end
372
-
373
335
  def acquire_lock?
374
336
  ttl = (@interval * LOCK_TTL_RATIO * 1000).ceil
375
337
  redis do |c|