wurk 1.12.0 → 1.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: efda965ecab6b73c86eddfd4f0b8acc5f0942bcba46d933aa60e922aae453c1f
4
- data.tar.gz: 404c2f3b9055edeafedd0c1a98f1933799c89830d0c2f70d08fb4018c183fa4f
3
+ metadata.gz: b0817941e1b8f3ff6a7195567e31ac9d22c5039c338d708f9c329ff87f0bbcec
4
+ data.tar.gz: 966125b576e3a775f868ad39224a70ffe6baddf6cbd35eb8b76fc81d63e6b762
5
5
  SHA512:
6
- metadata.gz: 1d466f7764855b501fe06a0e3a2586bf4908b6bd65274e74fa5739b77255613801bc6625fdee52f4a4771cc4c1ea49e6f9b171fa364e8fd5fda95997b788da0c
7
- data.tar.gz: 1333a3f95fc34da5a126caae9ff041a404bb80c221f425177e9034bf16bb4a16abc11fe9ad8aa1a8e18470f8066cc8aba0d9c45de2dea6ee46bbb38795b83a66
6
+ metadata.gz: c265be3c09774f76f6a9d560a59434a6800d56c06eadd471c6391a1cad589eb709ba51fdfbf672c14c5ef39c482d2e40a24a9a3a9ad8861d917a2bb5daa01d14
7
+ data.tar.gz: 998f43d63f710cd96b6e413fe20b6c4783149113c2bc1128d74afa3c33a225e75189b65d2515b4cd8351b8d14f02bfd4bfc390a164d9c01914147161e1532559
data/README.md CHANGED
@@ -303,9 +303,9 @@ Wurk is MIT and stays that way. If what you need is a commercial support contrac
303
303
  + gem "wurk"
304
304
  ```
305
305
 
306
- `bundle install && restart`. Wurk reads and writes the same Redis schema, so there is no data migration: Wurk picks up the queues, schedules, retries, batches and periodic loops Sidekiq left behind. Running Sidekiq and Wurk workers against one Redis *at the same time* is not tested yet, so a live production move is a **drain cutover** (stop Sidekiq's workers, confirm nothing is in flight, start Wurk), with a matching rollback; the [production cutover](https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md#9-production-cutover) section has every step and command. The upstream suites of sidekiq-cron, sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq run against Wurk in the [`ecosystem` CI job](https://github.com/developerz-ai/wurk/blob/main/.github/workflows/ecosystem.yml) (see [`test/ecosystem/`](https://github.com/developerz-ai/wurk/tree/main/test/ecosystem)). sidekiq-scheduler, sidekiq-status and sidekiq-throttled are target additions tracked in [docs/idea/14-ecosystem-compat.md](https://github.com/developerz-ai/wurk/blob/main/docs/idea/14-ecosystem-compat.md).
306
+ `bundle install && restart`. The worker command stays as it was: Wurk ships `sidekiq` and `sidekiqswarm` executables, so `bundle exec sidekiq -C config/sidekiq.yml` in a Procfile or manifest keeps working. Wurk reads and writes the same Redis schema, so there is no data migration: Wurk picks up the queues, schedules, retries, batches and periodic loops Sidekiq left behind. Running Sidekiq and Wurk workers against one Redis *at the same time* is not tested yet, so a live production move is a **drain cutover** (stop Sidekiq's workers, confirm nothing is in flight, start Wurk), with a matching rollback; the [production cutover](https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md#9-production-cutover) section has every step and command. The upstream suites of sidekiq-cron, sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq run against Wurk in the [`ecosystem` CI job](https://github.com/developerz-ai/wurk/blob/main/.github/workflows/ecosystem.yml) (see [`test/ecosystem/`](https://github.com/developerz-ai/wurk/tree/main/test/ecosystem)). sidekiq-scheduler, sidekiq-status and sidekiq-throttled are target additions tracked in [docs/idea/14-ecosystem-compat.md](https://github.com/developerz-ai/wurk/blob/main/docs/idea/14-ecosystem-compat.md).
307
307
 
308
- **Keeping a `sidekiq-*` add-on gem?** Those gems declare `add_dependency "sidekiq"`, which would reinstall real Sidekiq next to Wurk. Satisfy it with Wurk's git-only [`sidekiq` shim gem](https://github.com/developerz-ai/wurk/tree/main/ecosystem/sidekiq-shim) instead: `gem "sidekiq", github: "developerz-ai/wurk", glob: "ecosystem/sidekiq-shim/*.gemspec"`. Only sidekiq-cron, sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq are exercised that way, by the `ecosystem` workflow; treat the others as untested.
308
+ **Keeping a `sidekiq-*` add-on gem?** Those gems declare `add_dependency "sidekiq"`, which would reinstall real Sidekiq next to Wurk. Satisfy it with Wurk's git-only [`sidekiq` shim gem](https://github.com/developerz-ai/wurk/tree/main/ecosystem/sidekiq-shim) instead, pinned to the tag of the Wurk release you run: `gem "sidekiq", github: "developerz-ai/wurk", tag: "v1.13.0", glob: "ecosystem/sidekiq-shim/*.gemspec"`. Only sidekiq-cron, sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq are exercised that way, by the `ecosystem` workflow; treat the others as untested.
309
309
 
310
310
  Full walkthrough — config side-by-side, the Redis key/`sidekiq_options` mapping, known incompatibilities, moving sidekiq-cron schedules over (`rake wurk:import:cron`), and the production cutover and rollback playbook: **[docs/migrate-from-sidekiq.md](https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md)**. On the night, keep the **[incident runbook](https://github.com/developerz-ai/wurk/blob/main/docs/runbook.md)** open.
311
311
 
data/config/routes.rb CHANGED
@@ -1,6 +1,14 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  Wurk::Engine.routes.draw do
4
+ # Sidekiq Web's machine-readable routes (free §25.4), upstream's shapes, for
5
+ # uptime checks and scripts. `HEAD /` is declared before `root`, which would
6
+ # otherwise answer it with the SPA shell and never touch Redis.
7
+ match '/', to: Wurk::Web::StatsRoutes, via: :head
8
+ get 'stats', to: Wurk::Web::StatsRoutes, format: false
9
+ get 'stats/queues', to: Wurk::Web::StatsRoutes, format: false
10
+ get 'dashboard/stats', to: Wurk::Web::StatsRoutes, format: false
11
+
4
12
  root to: 'dashboard#index'
5
13
 
6
14
  # JSON APIs consumed by the SPA. Nested under whatever mount the host chose.
data/exe/sidekiq ADDED
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ # Drop-in alias for `wurk`. Sidekiq apps start workers with `bundle exec sidekiq
5
+ # -C config/sidekiq.yml` in a Procfile, systemd unit or k8s manifest; this keeps
6
+ # that exact command working after a one-line gem swap. Spec:
7
+ # docs/target/sidekiq-free.md §21.
8
+ load File.expand_path('wurk', __dir__)
data/exe/wurk CHANGED
@@ -5,7 +5,7 @@
5
5
  # Usage: `exe/wurk -C config/wurk.yml` — worker: one process, thread pool
6
6
  # `exe/wurk api --port 7433` — machine HTTP API, runs no jobs
7
7
  # `exe/wurk preflight` / `verify` — read-only cutover checks, run no jobs
8
- # `exe/wurk boot-check -r .` — boot the app as wurkswarm would, then exit (CI)
8
+ # `exe/wurk boot-check -r .` — boot the app as wurkswarm (or `--runner wurk`) would, then exit (CI)
9
9
 
10
10
  $stdout.sync = true
11
11
  $LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
@@ -1,15 +1,23 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require 'rails/generators'
4
+ require 'bundler'
5
+ require 'wurk/version'
4
6
 
5
7
  module Wurk
6
8
  module Generators
7
9
  # `bin/rails g wurk:install`
8
10
  # Writes a config initializer and adds a commented-out mount line to routes.
9
11
  # The host app picks the mount path; the generator suggests /wurk.
12
+ # Reads Gemfile.lock and Procfile to print what a Sidekiq app must change;
13
+ # it edits neither.
10
14
  class InstallGenerator < ::Rails::Generators::Base
11
15
  source_root File.expand_path('templates', __dir__)
12
16
 
17
+ # Sidekiq's own tiers depend on `sidekiq` too, but the swap removes them.
18
+ SIDEKIQ_TIERS = %w[sidekiq sidekiq-pro sidekiq-ent].freeze
19
+ SIDEKIQ_COMMAND = /\bsidekiq(?:swarm)?\b/
20
+
13
21
  def copy_initializer
14
22
  template 'wurk.rb', 'config/initializers/wurk.rb'
15
23
  end
@@ -18,6 +26,27 @@ module Wurk
18
26
  route(%(# mount Wurk::Engine => "/wurk" # uncomment to expose the Wurk dashboard))
19
27
  end
20
28
 
29
+ def show_shim_line
30
+ dependents = sidekiq_dependents
31
+ return if dependents.empty?
32
+
33
+ say <<~SHIM
34
+
35
+ #{dependents.join(', ')} #{dependents.size == 1 ? 'depends' : 'depend'} on the sidekiq gem, which would load real Sidekiq next to Wurk.
36
+ Keep #{dependents.size == 1 ? 'it' : 'them'} by adding the shim to the Gemfile, in place of gem "sidekiq":
37
+ gem "sidekiq", github: "developerz-ai/wurk", tag: "v#{Wurk::VERSION}", glob: "ecosystem/sidekiq-shim/*.gemspec"
38
+ SHIM
39
+ end
40
+
41
+ def show_procfile_runner
42
+ lines = procfile_sidekiq_lines
43
+ return if lines.empty?
44
+
45
+ say "\nProcfile worker lines keep working: wurk ships the sidekiq and sidekiqswarm commands."
46
+ lines.each { |line| say " #{line}" }
47
+ say 'For one process per core, run `bundle exec wurkswarm` with the same flags.'
48
+ end
49
+
21
50
  def show_next_steps
22
51
  say <<~STEPS
23
52
 
@@ -29,6 +58,33 @@ module Wurk
29
58
  docs/authentication.md in the wurk gem's repository
30
59
  STEPS
31
60
  end
61
+
62
+ private
63
+
64
+ def sidekiq_dependents
65
+ specs = locked_specs
66
+ # The shim is the `sidekiq` spec that depends on wurk: already in place.
67
+ return [] if specs.any? { |spec| spec.name == 'sidekiq' && depends_on?(spec, 'wurk') }
68
+
69
+ specs.select { |spec| depends_on?(spec, 'sidekiq') }.map(&:name).uniq
70
+ .reject { |name| SIDEKIQ_TIERS.include?(name) }.sort
71
+ end
72
+
73
+ def locked_specs
74
+ lockfile = File.join(destination_root, 'Gemfile.lock')
75
+ File.file?(lockfile) ? ::Bundler::LockfileParser.new(File.read(lockfile)).specs : []
76
+ end
77
+
78
+ def depends_on?(spec, name)
79
+ spec.dependencies.any? { |dep| dep.name == name }
80
+ end
81
+
82
+ def procfile_sidekiq_lines
83
+ procfile = File.join(destination_root, 'Procfile')
84
+ return [] unless File.file?(procfile)
85
+
86
+ File.readlines(procfile, chomp: true).grep(SIDEKIQ_COMMAND)
87
+ end
32
88
  end
33
89
  end
34
90
  end
@@ -25,9 +25,11 @@ module Wurk
25
25
  if info
26
26
  log_backend(config.logger, info)
27
27
  warn_on_foreign_processes(config)
28
+ warn_on_unconsumed_framework_queues(config)
28
29
  end
29
30
  warn_on_ignored_locks(config.logger)
30
31
  warn_on_ignored_unique(config.logger)
32
+ warn_on_client_config_only(config)
31
33
  rescue StandardError => e
32
34
  config.logger.warn { "wurk boot checks skipped: #{e.class}: #{e.message}" }
33
35
  end
@@ -97,6 +99,84 @@ module Wurk
97
99
  end
98
100
  end
99
101
 
102
+ # Stock Sidekiq skips a configure_client block in a server without a word,
103
+ # and so does Configuration: an app normally configures both roles. A
104
+ # server where only client blocks were written ran none of them, so the
105
+ # Redis settings, middleware and handlers they hold are all absent here.
106
+ def warn_on_client_config_only(config)
107
+ return unless config.client_config_only?
108
+
109
+ config.logger.warn do
110
+ 'This is a server process, so its configure_client blocks were skipped, and no configure_server block ran: ' \
111
+ 'whatever those blocks set (config.redis, middleware, handlers) does not apply to these workers. Set it ' \
112
+ 'in a configure_server block too, or outside both.'
113
+ end
114
+ end
115
+
116
+ # The topology only shapes what a swarm forks; `wurk` and an embedded
117
+ # worker run one process with the capsules' threads, so a declared one
118
+ # silently does nothing there.
119
+ def warn_on_ignored_topology(config, runner)
120
+ return unless config.topology_assigned?
121
+
122
+ config.logger.warn do
123
+ "config.topology is set, but #{runner} runs a single process and ignores it: its slots' queues and " \
124
+ 'concurrency are not used here. Run wurkswarm to fork it, or set the queues and concurrency on the capsules.'
125
+ end
126
+ end
127
+
128
+ # Rails enqueues its own jobs (mail delivery, Active Storage analysis and
129
+ # purge, Action Mailbox routing) to queues an app never names itself, so
130
+ # they are the ones that go unlistened. Read from Redis, so only with it.
131
+ def warn_on_unconsumed_framework_queues(config)
132
+ wanted = framework_queues
133
+ return if wanted.empty?
134
+
135
+ listened = own_queues(config) | config.redis(idempotent: true) { |conn| QueueConsumers.counts(conn).keys }
136
+ missing = wanted.except(*listened)
137
+ return if missing.empty?
138
+
139
+ config.logger.warn do
140
+ "Rails enqueues to #{listing(missing.map { |queue, what| "#{queue} (#{what})" })}, which neither this " \
141
+ 'process nor any live one listens to, so those jobs wait forever. Add the queues to a worker.'
142
+ end
143
+ end
144
+
145
+ # `{ queue => what Rails sends there }`, for apps whose ActiveJob adapter is Wurk.
146
+ def framework_queues
147
+ return {} unless wurk_active_job?
148
+
149
+ named = {}
150
+ named['mail delivery'] = ::ActionMailer::Base.deliver_later_queue_name if defined?(::ActionMailer::Base)
151
+ named.merge!(framework_queue_set('Active Storage', defined?(::ActiveStorage) && ::ActiveStorage))
152
+ named.merge!(framework_queue_set('Action Mailbox', defined?(::ActionMailbox) && ::ActionMailbox))
153
+ named.compact.to_h { |what, queue| [active_job_queue_name(queue), what] }
154
+ end
155
+
156
+ def framework_queue_set(framework, mod)
157
+ return {} unless mod.respond_to?(:queues) && mod.queues.respond_to?(:to_h)
158
+
159
+ mod.queues.to_h.transform_keys { |kind| "#{framework} #{kind}" }
160
+ end
161
+
162
+ def wurk_active_job?
163
+ return false unless defined?(::ActiveJob::Base)
164
+
165
+ ::ActiveJob::Base.queue_adapter.class.name.to_s.match?(/Wurk|Sidekiq/)
166
+ end
167
+
168
+ def active_job_queue_name(queue)
169
+ prefix = ::ActiveJob::Base.queue_name_prefix
170
+ prefix ? "#{prefix}#{::ActiveJob::Base.queue_name_delimiter}#{queue}" : queue.to_s
171
+ end
172
+
173
+ def own_queues(config)
174
+ queues = config.capsules.each_value.flat_map { |cap| cap.weights.keys }
175
+ return queues unless config.topology_assigned?
176
+
177
+ queues | config.topology.slots.flat_map { |slot| Array(slot.queues).map { |entry| entry.to_s.split(',').first } }
178
+ end
179
+
100
180
  def locked_job_classes = job_classes_setting(%w[lock])
101
181
 
102
182
  # Workers, and ActiveJob classes that take `sidekiq_options` (Wurk::Job::Options).
data/lib/wurk/cli.rb CHANGED
@@ -112,8 +112,10 @@ module Wurk
112
112
  ].freeze
113
113
 
114
114
  BOOT_CHECK_OPTION_FLAGS = [
115
- ['-R', '--redis', :boot_check_redis, 'Also read Redis: version, eviction policy, live peers (read-only)']
115
+ ['-R', '--redis', :boot_check_redis, 'Also read Redis: version, eviction policy, live peers (read-only)'],
116
+ ['-n', '--runner NAME', :boot_check_runner, 'Boot as wurkswarm (default) or wurk, the runner you deploy']
116
117
  ].freeze
118
+ BOOT_CHECK_RUNNERS = %w[wurkswarm wurk].freeze
117
119
 
118
120
  HEALTH_OPTION_FLAGS = [
119
121
  ['-w', '--window SECONDS', :health_window, 'A heartbeat older than this does not count (default 60)', :to_i],
@@ -123,7 +125,7 @@ module Wurk
123
125
  ].freeze
124
126
 
125
127
  HELP_COMMANDS = "\nCommands:\n " \
126
- "boot-check Boot the app the way wurkswarm does, then exit; no fork, no fetch (CI)\n " \
128
+ "boot-check Boot as wurkswarm (or --runner wurk) would, then exit; no fork, no fetch (CI)\n " \
127
129
  "health Exit 0 if this host's workers are heartbeating\n " \
128
130
  "preflight Read a live Redis: what will carry over to Wurk (read-only)\n " \
129
131
  "verify Post-cutover check of a live fleet (read-only)\n\nOptions:"
@@ -215,6 +217,7 @@ module Wurk
215
217
  validate_redis!
216
218
  validate_pool_sizes!
217
219
  BootChecks.run(@config, redis_info)
220
+ BootChecks.warn_on_ignored_topology(@config, 'wurk')
218
221
  BootChecks.log_process_role(@config, 'wurk')
219
222
  @config[:identity] = identity
220
223
  # Force lazy server-middleware chain so worker threads don't race
@@ -273,25 +276,25 @@ module Wurk
273
276
  exit(1) if @swarm.crash_looped?
274
277
  end
275
278
 
276
- # `wurk boot-check` — #run_swarm's boot up to the fork: the same require
277
- # order, config file and checks, plus what a worker reaches only later (every
278
- # constant, the routes), so a CI step finds what would crash-loop a deploy.
279
- # Forks nothing, fetches nothing, fires no lifecycle event, and reads Redis
280
- # only with --redis. Returns the exit code; a boot that raises is the binary's
279
+ # `wurk boot-check` — the boot of the runner named by --runner up to where it
280
+ # would fork or fetch: #run_swarm's (the default) or #run's, with the same
281
+ # require order, config file and checks, plus what a worker reaches only later
282
+ # (every constant, the routes), so a CI step finds what would crash-loop a
283
+ # deploy. `configure_server` blocks run, since their load errors are the point;
284
+ # no lifecycle event fires (`:startup` included) and periodic loops are never
285
+ # applied. Wurk's Redis pools refuse every command without --redis and every
286
+ # write with it (Preflight::RedisGuard), so boot-time app code cannot touch a
287
+ # real server. Returns the exit code; a boot that raises is the binary's
281
288
  # failure report.
282
289
  def run_boot_check(boot_app: true)
290
+ require_relative 'preflight/redis_guard'
291
+ Preflight::RedisGuard.install(@config[:boot_check_redis] ? :read_only : :closed, label: 'wurk boot-check')
283
292
  enter_server_mode
284
- Health.options_from_env
285
- if boot_app
286
- boot_swarm_application
287
- eager_load_rails
288
- draw_routes
289
- end
290
- validate_pool_sizes!
291
- BootChecks.run(@config, boot_check_redis_info)
292
- log_swarm_role
293
- logger.info { "wurk boot-check: #{@config[:require]} booted as wurkswarm would" }
293
+ boot_check_runner == 'wurk' ? boot_check_as_wurk(boot_app) : boot_check_as_swarm(boot_app)
294
+ logger.info { "wurk boot-check: #{@config[:require]} booted as #{boot_check_runner} would" }
294
295
  0
296
+ ensure
297
+ Preflight::RedisGuard.uninstall
295
298
  end
296
299
 
297
300
  # `wurk health` — see Wurk::HealthProbe. Boots nothing; returns the exit code.
@@ -467,6 +470,36 @@ module Wurk
467
470
  end
468
471
  end
469
472
 
473
+ def boot_check_runner = @config[:boot_check_runner] || BOOT_CHECK_RUNNERS.first
474
+
475
+ def boot_check_as_swarm(boot_app)
476
+ Health.options_from_env
477
+ if boot_app
478
+ boot_swarm_application
479
+ boot_check_reach
480
+ end
481
+ validate_pool_sizes!
482
+ BootChecks.run(@config, boot_check_redis_info)
483
+ log_swarm_role
484
+ end
485
+
486
+ def boot_check_as_wurk(boot_app)
487
+ if boot_app
488
+ boot_application
489
+ boot_check_reach
490
+ end
491
+ validate_pool_sizes!
492
+ BootChecks.run(@config, boot_check_redis_info)
493
+ BootChecks.warn_on_ignored_topology(@config, 'wurk')
494
+ BootChecks.log_process_role(@config, 'boot-check (as wurk)')
495
+ end
496
+
497
+ # What a worker loads only later: every constant, and the routes.
498
+ def boot_check_reach
499
+ eager_load_rails
500
+ draw_routes
501
+ end
502
+
470
503
  def boot_check_redis_info
471
504
  return unless @config[:boot_check_redis]
472
505
 
@@ -809,6 +842,11 @@ module Wurk
809
842
  reason = require_path_problem(@config[:require].to_s)
810
843
  return print_help_and_die(reason) if reason
811
844
 
845
+ runner = @config[:boot_check_runner]
846
+ if runner && !BOOT_CHECK_RUNNERS.include?(runner)
847
+ raise StartupError, "--runner: #{runner.inspect} is not one of #{BOOT_CHECK_RUNNERS.join(', ')}"
848
+ end
849
+
812
850
  %i[concurrency timeout].each do |opt|
813
851
  raise StartupError, "#{opt}: #{@config[opt]} is not a valid value" if @config[opt].to_i <= 0
814
852
  end
@@ -877,7 +877,7 @@ module Wurk
877
877
  # and the like silently absent from a process that does run workers.
878
878
  def configure_server(&block)
879
879
  return unless block
880
- return yield self if server?
880
+ return run_server_block(block) if server?
881
881
 
882
882
  @deferred_server_blocks << block
883
883
  return if @configure_server_skip_warned
@@ -889,23 +889,34 @@ module Wurk
889
889
 
890
890
  def enter_server_mode
891
891
  self[:server] = true
892
- @deferred_server_blocks.shift.call(self) until @deferred_server_blocks.empty?
892
+ run_server_block(@deferred_server_blocks.shift) until @deferred_server_blocks.empty?
893
893
  end
894
894
 
895
895
  # A web process that also boots the swarm (Wurk::RailsBoot) is flagged
896
896
  # server, yet every enqueue it makes from a request is client work, so
897
- # `client_in_server` lets its configure_client blocks run too. Same
898
- # one-warning-per-gate contract as configure_server.
897
+ # `client_in_server` lets its configure_client blocks run too.
898
+ #
899
+ # In a server, skipping the client block is the Sidekiq contract and the
900
+ # usual case: an app configures both roles. So the skip itself is DEBUG,
901
+ # and BootChecks warns at boot only for the case worth one, a server
902
+ # process in which no configure_server block ran at all (#816).
899
903
  def configure_client
900
904
  return unless block_given?
901
905
  return yield self if !server? || @options[:client_in_server]
906
+
902
907
  return if @configure_client_skip_warned
903
908
 
904
909
  @configure_client_skip_warned = true
905
- logger.warn { 'configure_client block skipped: server? is true and client_in_server is false or nil' }
910
+ logger.debug { 'configure_client block skipped: server? is true and client_in_server is false or nil' }
906
911
  nil
907
912
  end
908
913
 
914
+ # True once a configure_client block was skipped and no configure_server
915
+ # block ran: the server got none of the settings the app made for a role.
916
+ def client_config_only?
917
+ @configure_client_skip_warned == true && !@configure_server_ran
918
+ end
919
+
909
920
  def server?
910
921
  @options[:server] == true
911
922
  end
@@ -920,9 +931,14 @@ module Wurk
920
931
 
921
932
  def topology=(value)
922
933
  guard_frozen!
934
+ @topology_assigned = true
923
935
  @topology = value
924
936
  end
925
937
 
938
+ # Whether the app declared a topology, as opposed to the default one
939
+ # #topology builds. Only a swarm reads it.
940
+ def topology_assigned? = @topology_assigned == true
941
+
926
942
  # Memory-based child recycling (Sidekiq Ent §7.5): the swarm parent TERMs
927
943
  # (and respawns) any child whose RSS exceeds this many MB. Set in code or
928
944
  # via SIDEKIQ_MAXMEM_MB (WURK_MAXMEM_MB is the native alias); an explicit
@@ -1001,6 +1017,11 @@ module Wurk
1001
1017
 
1002
1018
  private
1003
1019
 
1020
+ def run_server_block(block)
1021
+ @configure_server_ran = true
1022
+ block.call(self)
1023
+ end
1024
+
1004
1025
  # Pools read the hash once, when built, so the later assignment silently does
1005
1026
  # nothing to this process's connections until something rebuilds them.
1006
1027
  def warn_on_late_redis_assignment
@@ -1079,7 +1100,7 @@ module Wurk
1079
1100
  # as throttling or memory pressure.
1080
1101
  def default_child_count
1081
1102
  count, source = child_count_and_source
1082
- logger.info { "Wurk swarm: forking #{count} #{count == 1 ? 'child' : 'children'} (#{source})" }
1103
+ logger.info { "Wurk swarm size: #{count} #{count == 1 ? 'child' : 'children'} (#{source})" }
1083
1104
  count
1084
1105
  end
1085
1106
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  require_relative '../keys'
4
4
  require_relative '../lua'
5
- require_relative '../middleware/poison_pill'
5
+ require_relative 'poison_pill'
6
6
  require_relative '../metrics/process_counters'
7
7
  require_relative 'oom_drain'
8
8
 
@@ -28,13 +28,15 @@ module Wurk
28
28
  # Returns true when this call took the payload off the private list, false
29
29
  # when it had already left (another reaper, an ACK).
30
30
  #
31
- # Unclaimed apply-safety: the move is guarded, but a reply lost after a
32
- # poison kill would leave the death handlers unfired, and a list holding
33
- # two identical copies would move the second on a replay. A raise leaves
34
- # the payload for the next pass instead.
31
+ # Sent once, never re-sent on a fresh socket: a replay of a script that
32
+ # already ran finds the payload gone and answers "someone else moved it",
33
+ # which skipped the death handlers of a kill (#908), and a list holding two
34
+ # identical copies would move the second. A reply lost after the script ran
35
+ # is settled by reading back what it wrote instead (#settle_lost_reply).
35
36
  def call(private_list, public_q, payload)
36
37
  job = Middleware::PoisonPill.parse(payload)
37
- count = @config.redis { |conn| move(conn, private_list, public_q, payload, job) }
38
+ keys = move_keys(private_list, public_q, job)
39
+ count = moved_count(keys, payload, now)
38
40
  return false unless count
39
41
 
40
42
  Metrics::ProcessCounters.increment(:reaper_reclaimed)
@@ -47,15 +49,50 @@ module Wurk
47
49
 
48
50
  private
49
51
 
50
- # Past maxmemory the counter cannot be written, nor the job killed: it goes
51
- # back uncounted rather than waiting on the memory it would help free.
52
- def move(conn, private_list, public_q, payload, job)
52
+ def moved_count(keys, payload, score)
53
+ @config.redis { |conn| move(conn, keys, payload, score) }
54
+ rescue RedisClient::ConnectionError => e
55
+ settle_lost_reply(keys, payload, score, e)
56
+ end
57
+
58
+ def move_keys(private_list, public_q, job)
53
59
  keys = [private_list, public_q, Keys::DEAD]
54
60
  jid = job && job['jid']
55
61
  keys << Middleware::PoisonPill.counter_key(jid) if jid.is_a?(String) && !jid.empty?
56
- Wurk::Lua::Loader.eval_cached(conn, :reclaim_orphan, keys: keys, argv: [payload, THRESHOLD, TTL, now])
62
+ keys
63
+ end
64
+
65
+ # Past maxmemory the counter cannot be written, nor the job killed: it goes
66
+ # back uncounted rather than waiting on the memory it would help free.
67
+ def move(conn, keys, payload, score)
68
+ Wurk::Lua::Loader.eval_once(conn, :reclaim_orphan, keys: keys, argv: [payload, THRESHOLD, TTL, score])
57
69
  rescue RedisClient::OutOfMemoryError => e
58
- :uncounted if OomDrain.requeue(conn, e, private_list, public_q, payload) == 1
70
+ :uncounted if OomDrain.requeue(conn, e, keys[0], keys[1], payload) == 1
71
+ end
72
+
73
+ # The connection dropped around the script, so it ran or it did not. Only a
74
+ # kill needs settling: the job is gone from the private list, so no later
75
+ # sweep would find it, and the death handlers (a batch's `:death`, a unique
76
+ # lock's release) would never fire. The script wrote the dead-set member
77
+ # at the score this call chose, so that exact score proves the kill was
78
+ # this call's. Returns the recovery count for a kill this call made; nil
79
+ # for a requeue or a move someone else made, which need nothing more;
80
+ # raises the original error when the payload is still in the private list,
81
+ # which is the next pass's to move.
82
+ def settle_lost_reply(keys, payload, score, error)
83
+ private_list, _public_q, dead, counter = keys
84
+ still_there, dead_score, count = @config.redis(idempotent: true) do |conn|
85
+ conn.pipelined do |pipe|
86
+ pipe.call('LPOS', private_list, payload, 'RANK', -1)
87
+ pipe.call('ZSCORE', dead, payload)
88
+ pipe.call('GET', counter) if counter
89
+ end
90
+ end
91
+ # Exact on purpose: the same double, written by the script and read back.
92
+ return count.to_i if dead_score && Float(dead_score) == Float(score) # rubocop:disable Lint/FloatComparison
93
+ raise error if still_there
94
+
95
+ nil
59
96
  end
60
97
 
61
98
  # DeadSet#kill's score format, so the entry sorts with every other kill.
@@ -19,7 +19,9 @@ module Wurk
19
19
  # Counter key TTL is wire-compat with Sidekiq Pro — third-party tooling
20
20
  # that watches `super_fetch:recovered:*` expects 72h.
21
21
  #
22
- # No server-middleware registration: the producer is Fetcher::OrphanReclaim,
22
+ # Not a chain middleware, so the file lives with the fetcher; the constant
23
+ # keeps its Middleware path because Pro initializers name it. No
24
+ # server-middleware registration: the producer is Fetcher::OrphanReclaim,
23
25
  # whose script bumps the counter and takes the requeue-or-kill decision in
24
26
  # the same step as the move, then hands the outcome to {recovered!} for the
25
27
  # notifications. The consumer is the ACK —
@@ -9,6 +9,7 @@ require_relative 'orphan_grace'
9
9
  require_relative 'oom_drain'
10
10
  require_relative 'orphan_reclaim'
11
11
  require_relative 'self_reconcile'
12
+ require_relative 'reliable'
12
13
 
13
14
  module Wurk
14
15
  class Fetcher
@@ -28,9 +29,10 @@ module Wurk
28
29
  # ignores a stale `processes` SET member — the SET carries no TTL, so
29
30
  # the member lingers after its heartbeat hash expires (60s) until
30
31
  # `ProcessSet#cleanup` prunes it. A `kill -9`ed sibling is reclaimed
31
- # the moment the supervisor reaps it, not 60s later. (Pid reuse by an
32
- # unrelated process in the same tree is the one blind spot — the
33
- # supervisor respawns with a fresh pid, so it does not arise in practice.)
32
+ # the moment the supervisor reaps it, not 60s later. Pid reuse by an
33
+ # unrelated process reads a dead owner as live; a swarm child does not
34
+ # depend on this check, since its parent drains its lists by exact key
35
+ # the moment it reaps it (#reclaim_exited), before the pid can be reused.
34
36
  # * any other incarnation — its pid means nothing in our namespace, so we
35
37
  # trust the heartbeat: the owner is alive iff its identity's `info`
36
38
  # hash still exists. Such reclaim waits out the 60s heartbeat TTL, the
@@ -172,13 +174,30 @@ module Wurk
172
174
  sweep.tap { wake_for_settling }
173
175
  end
174
176
 
175
- # #reclaim! for a booting process. Deliberately under no lock, not even a
176
- # short one: a replacement's sweep is what recovers the sibling it replaces,
177
- # and the sibling may have died a moment after another process booted (a
178
- # 2s lock turned a child killed right after boot from 1s into 61s). One
179
- # pass per process costs keyspace / SCAN_COUNT calls.
177
+ # The sweep a booting process runs, under the same lock as the periodic one:
178
+ # a rolling restart or a deploy of N processes is one keyspace pass, not N.
179
+ # A replacement's sweep is no longer what recovers the swarm child it
180
+ # replaces; the parent did that by exact key when it reaped the child
181
+ # (#reclaim_exited). What this pass still finds is other incarnations'
182
+ # lists, which wait on the heartbeat TTL and the grace either way.
180
183
  def boot_reclaim!
181
- reclaim!
184
+ reap
185
+ end
186
+
187
+ # Drains the private lists of a process known to be dead: a swarm child its
188
+ # parent has just reaped. The parent knows the pid, the queues the child
189
+ # served and, since the nonce is inherited across fork, the rest of each
190
+ # list's name, so neither a SCAN nor a liveness check is needed. One
191
+ # pipelined LLEN finds the lists holding anything (none, after a drained
192
+ # exit) and raises rather than paying the pool's backoff once per list
193
+ # when Redis is away. Returns the jobs moved.
194
+ def reclaim_exited(pid, queue_names)
195
+ lists = queue_names.to_h do |name|
196
+ public_q = Keys.queue(name)
197
+ [Reliable.private_queue_name(public_q, pid: pid), public_q]
198
+ end
199
+ lengths = redis(idempotent: true) { |c| c.pipelined { |p| lists.each_key { |l| p.call('LLEN', l) } } }
200
+ lists.zip(lengths).sum { |(list, public_q), len| len.positive? ? drain(list, public_q) : 0 }
182
201
  end
183
202
 
184
203
  # Lists whose grace has run out, whichever sweep started it. Runs on every
@@ -1,6 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require 'socket'
4
3
  require_relative '../component'
5
4
  require_relative '../pid_cache'
6
5
  require_relative '../keys'
@@ -80,10 +79,10 @@ module Wurk
80
79
  # reaper's `kill(0)` liveness check would read a dead owner's list as
81
80
  # live (jobs stranded) or a live owner's as dead (job run twice). Keys
82
81
  # written before the nonce existed stay reclaimable — PrivateListKey
83
- # accepts both shapes.
84
- def self.private_queue_name(public_queue, index = 0)
85
- host = ENV['DYNO'] || Socket.gethostname
86
- "#{public_queue}|#{host}|#{::Process.pid}|#{Component::PROCESS_NONCE}|#{index}"
82
+ # accepts both shapes. `pid:` lets a swarm parent name a reaped child's
83
+ # lists (Reaper#reclaim_exited).
84
+ def self.private_queue_name(public_queue, index = 0, pid: ::Process.pid)
85
+ "#{public_queue}|#{Component.hostname}|#{pid}|#{Component::PROCESS_NONCE}|#{index}"
87
86
  end
88
87
 
89
88
  # Guards the generation bump only. Fetchers read the counter without it —
@@ -1,7 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require_relative '../queue_slot'
4
- require_relative '../middleware/poison_pill'
4
+ require_relative 'poison_pill'
5
5
 
6
6
  module Wurk
7
7
  class Fetcher