stablemates-workhorse 0.10.0 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 35dc8eaadac2ae86c30ad3cf2a688ffe571b7c7a8cd564ee00844685fbeb4ea6
4
- data.tar.gz: 34dfeb5010846d31ea3f7b5751e14bf7ec4ed4783a9223d4e2f483534b419669
3
+ metadata.gz: 1686623ff5e225f372245830a169d9b248db6bd195ed3f446679e01098bc9610
4
+ data.tar.gz: bde5f7e6517d2a7a3621b437237db776673156693b22fc1a46f42aa28305b563
5
5
  SHA512:
6
- metadata.gz: 99884c5d12e4501d537c66272a668be206f796ef0f8ac7663d863898caa3f0e4a530af7a79902d81983e671b76f146bc3e54f529a7cf7d6170d44ecaadc96d5f
7
- data.tar.gz: dee85878fb8e6862120d5863d8b4cd34cdd18d6e4281e1e5976f9215af12dc0c2b58aef3090e3b42e439868fe367d63f515e66a8968a73d33903c626cb5247c4
6
+ metadata.gz: eef44a6c8715cc1d6c0f7983fb42b3662332ad1302fd6d705e60eca9f2d4ed099d32fd53d1303a2ee1a0cdfa9ee08c3624c57516c967c22cb62f0366180bdf4b
7
+ data.tar.gz: e9137fa4c8b8f6a5f20f800cb4abc482e6c750721758fb5432f407c229816b4a4186700e882eeb8859a8d21f8bcfa2f2be6be1ae734ef574a003227a8db82a94
data/CHANGELOG.md CHANGED
@@ -3,6 +3,213 @@
3
3
  `stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
4
4
  other SDKs carry, because every tag names one release of all of them.
5
5
 
6
+ ## 0.11.0 — 2026-10-28
7
+
8
+ The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
9
+ packages release from one source commit.
10
+
11
+ Requires **schema v76**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
12
+
13
+ ### Upgrade steps
14
+
15
+ **Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
16
+ process of this release starts, producers too. Migrations 0088, 0089, and 0090 move the schema
17
+ from version 86 to version **89**. They replace function bodies and change no stored rows. The SDK
18
+ compatibility floor stays at **76**. A 0.10 process keeps working on version 89, because a
19
+ migration inside a major line only adds.
20
+
21
+ **Give each worker a pool with a known size of at least 3.** `Worker.new` no longer accepts
22
+ `shared_heartbeats:`. Remove the option. If a worker pool reports a size below 3, or reports no
23
+ size, set its size to 3 or more. Each worker process needs 3 connections: the listener, the
24
+ heartbeat connection, and one statement connection. A `polling_only: true` worker opens no
25
+ listener, but it still needs a pool of 3 (SM-1901).
26
+
27
+ ### Breaking changes
28
+
29
+ - **`shared_heartbeats:` is removed.** Every worker now reserves a dedicated heartbeat connection
30
+ on its pool (ADR 0071). Before, `shared_heartbeats: true` let heartbeat rounds borrow ordinary
31
+ pool connections and let a pool below 3 run. Then a heartbeat could wait behind claims and
32
+ settlements, so a lease could expire and a task could run twice. Now `Worker.new` raises
33
+ `ArgumentError` when it gets `shared_heartbeats:`, and the message names the removal. A worker
34
+ on a pool with a size below 3, or with no reported size, raises `ArgumentError` at construction.
35
+ The worker no longer logs `workhorse.worker.polling_only`, because its pool always has a
36
+ connection for the listener (SM-1901).
37
+
38
+ ### Behavior changes
39
+
40
+ These changes can make code that worked on 0.10 act differently:
41
+
42
+ 1. An idempotency TTL of exactly 0 still selects the default. The queue now raises
43
+ `ArgumentError` for a TTL that is not 0 and rounds to 0 ms (SM-1857). See "An idempotency
44
+ TTL that rounds to 0 ms is refused".
45
+ 2. The TypeScript `Queue`, `Admin`, and `Worker.runOnce()` now check the schema before their first
46
+ statement (SM-1864). This SDK already did, so it does not change.
47
+ 3. After a lease lapses while a handler runs, a stale failure or release now settles a due
48
+ deadline or attempt timeout at once. The worker reports `deadline_exceeded` or `timeout`, not
49
+ `lease_lost` (SM-1884). See "A stale failure or release after a lapsed lease settles its
50
+ cause".
51
+ 4. Every SDK removes the heartbeat opt-out (SM-1899, SM-1901, SM-1902). In this SDK,
52
+ `Worker.new` raises `ArgumentError` for `shared_heartbeats:`, and for a pool with a size below
53
+ 3 or with no reported size. See "Breaking changes".
54
+
55
+ ### A step of the wall clock no longer moves the expiry request
56
+
57
+ A worker asks PostgreSQL to end an attempt when its deadline or its attempt timeout passes. Before
58
+ this change, the worker read the wall clock again after each wait on the lease window (SM-1869). If
59
+ the host clock stepped back 10 minutes, the first expiry request came 10 minutes late. If the clock
60
+ stepped forward, the request came early and got `not_due`. Each retry then locked the task row until
61
+ the real due time.
62
+
63
+ Now the worker converts the expiry to a monotonic due time once, when its expiry
64
+ watch starts. After that, it reads only the monotonic clock.
65
+
66
+ ### Row retention lag skips a member of an unfinished task group
67
+
68
+ Terminal cleanup keeps each member of a task group that has not finished, because the group needs
69
+ its member outcomes when it finishes. Before this change, queue health counted such a member as row
70
+ retention lag after it passed its windows (SM-1851). An open or sealed group can stay unfinished for
71
+ any length of time. Thus health reported "Retention cleanup is late for task records, finished
72
+ results", and the lag grew without end. The public demo keeps one open and one sealed group on
73
+ purpose, so its Health page showed the report every day.
74
+
75
+ Migration 0088 (`0088-count-only-prunable-rows-as-row-retention-lag.sql`) moves the schema to
76
+ version 87. Now `queue_health_v1` skips a member of an unfinished group when it measures task
77
+ identity and terminal outcome lag. For a group member, lag counts from the latest of window
78
+ expiry, history release, and group finish. The health document keeps its keys, and the schema floor
79
+ does not change.
80
+ Every process of every SDK version gets the new result when the migration commits.
81
+
82
+ ### A stale failure or release after a lapsed lease settles its cause
83
+
84
+ A lease can lapse while a handler runs, with no change to the fence. Then `fail_v1` and
85
+ `release_owned_v1` answer `stale`, but `expire_owned_v1` can still settle a due deadline or attempt
86
+ timeout under that fence. Before this change, the worker settled nothing, and lease recovery
87
+ settled the task later (SM-1884).
88
+
89
+ Now the worker calls `expire_owned_v1` at once and settles the due boundary, as the Go and Rust
90
+ workers do:
91
+
92
+ - After a lapsed lease with a due deadline, the handler outcome was `lease_lost`. Now it is
93
+ `deadline_exceeded`, and the task fails at once.
94
+ - After a lapsed lease with a due attempt timeout, the handler outcome was `lease_lost`. Now it is
95
+ `timeout`, and PostgreSQL applies the retry policy at once.
96
+ - After a lapsed lease with no due boundary, the outcome stays `lease_lost`.
97
+
98
+ The handler span records `deadline_exceeded` or `timeout_exceeded`. Before this change, it
99
+ recorded `stale`.
100
+
101
+ If `fail_v1` or `release_owned_v1` answers `cancel_requested` and `acknowledge_cancel_v1` refuses,
102
+ the worker settles nothing. After a failed cancellation acknowledgement, the handler outcome was
103
+ `unknown`. Now it is `lease_lost`. The handler span records `stale`. Before this change, it recorded
104
+ `unknown`. A late heartbeat round can also deliver the cancellation request, after a failure, a
105
+ release, or a refused completion. If the acknowledgement then refuses, the handler outcome was
106
+ `canceled`. Now it is `lease_lost`.
107
+
108
+ ### The dashboard bundle uses Mantine 9.7.4
109
+
110
+ The Rack dashboard backend serves the shared dashboard browser bundle. After 0.10.0, a dependency
111
+ update moved the dashboard to Mantine 9.7.4 and rebuilt that bundle (SM-1839). The gem now carries
112
+ the rebuilt `read-surface-1.tar.gz`. Its `bundle.json` records the sha256
113
+ `a7d61ed1d0433e38a8d227cbcc8f98a0bf668f2d3bb1d47391b1882fd69bf795`.
114
+
115
+ The `dashboard/v1` procedures and the read surface version do not change (SM-1858).
116
+
117
+ ### A worker no longer keeps the tier entries of a task that the executor refuses
118
+
119
+ The worker records each claimed task in its dispatch state. A fast-tier claim records the task ID
120
+ in `@fast_task_ids`. A full-tier claim records the task ID and a deferred tier read in
121
+ `@tier_reads`. Only the end of a handler run removed these entries.
122
+
123
+ Before this change, the handler executor could refuse a task, for example after its shutdown. The
124
+ task then never ran, and the worker kept its tier entry for the life of the `Worker` object
125
+ (SM-1853). Each refusal kept the entry of at most one task.
126
+
127
+ Now the worker removes the task from both collections when the executor refuses the task.
128
+
129
+ ### The dashboard page tells each cache not to store it
130
+
131
+ The dashboard page holds `window.workhorseDashboard`, which contains the `auditActor` of the
132
+ operator. Before this change, the Rack dashboard backend sent the page with only `Content-Type`. A
133
+ shared cache in front of the application could store the page by heuristic. A later visitor could
134
+ then get the stored page and see the actor name of an earlier operator. A browser could also show
135
+ the page from its cache after logout (SM-1870).
136
+
137
+ Now the Rack dashboard backend sends the page with `Cache-Control: no-store`, as the TypeScript host
138
+ does. Each asset keeps `Cache-Control: public, max-age=31536000, immutable`.
139
+
140
+ ### An idempotency TTL that rounds to 0 ms is refused
141
+
142
+ The queue rounds `Idempotency#ttl` to the nearest millisecond. Before this change, a positive TTL
143
+ below 0.0005 s, or a small negative TTL such as -0.0004 s, rounded to 0. The queue then kept the key
144
+ for the default 24 hours without a warning (SM-1857).
145
+
146
+ Now the queue raises `ArgumentError` for a TTL that is not 0 and rounds to 0 ms. An exact 0 still
147
+ selects the default. ADR 0115 records the rule for all seven SDKs.
148
+
149
+ ### A stalled registration refresh no longer delays a stop
150
+
151
+ A worker refreshes its `worker_registry` row at each registry interval. Before this change, the
152
+ dispatch loop sent that refresh itself, with no time limit. If the statement stalled, for example on
153
+ a lock, `stop` could not start the stop path until the statement returned. Also, the draining
154
+ registration and the deregistration ran on two threads in no order. If the registration committed
155
+ last, the row of the stopped worker came back until the registry prune removed it (SM-1868).
156
+
157
+ Now each registry write runs on its own thread, after the write before it. The dispatch loop starts
158
+ a periodic refresh only when no registry write still runs, and it does not wait for the refresh.
159
+ Thus `run` returns within `shutdown_grace` plus the unwind window, and the deregistration runs
160
+ after every registration write of the run.
161
+
162
+ ### Recovery no longer locks a timed-out attempt that waits for its cancellation
163
+
164
+ An operator cancels a running task after its execution timeout passed, and its lease is still live.
165
+ Before migration 0089, each maintenance tick locked that attempt in its timeout scan, but settled
166
+ nothing, because the cancellation wins over the timeout. While the tick ran, the heartbeat of the
167
+ owning worker reported the lease busy. The locked rows also used the recovery limit, so a real
168
+ timeout behind them waited for their leases to expire (SM-1866).
169
+
170
+ Now the full-tier timeout scan skips such an attempt while its lease is live and its deadline is
171
+ ahead, as the fast tier does. If the deadline passed after the timeout, recovery closes the task
172
+ `canceled` at once.
173
+
174
+ ### Fast-tier recovery closes a canceled attempt when its deadline passes
175
+
176
+ A fast-tier task has a pending cancellation and a live lease. Its execution timeout passes, and
177
+ later its deadline passes. Before migration 0090, each maintenance tick locked that attempt and
178
+ settled nothing until the lease expired. The cancellation blocks the timeout, and recovery took the
179
+ deadline only when the deadline was not later than the timeout (SM-1881).
180
+
181
+ Now fast-tier recovery closes the task `canceled` when its deadline passes, as the full tier does.
182
+
183
+ ### The dashboard Schedules page reads a bounded number of occurrences
184
+
185
+ Before migration 0089, `dashboard_cron_v1` counted every retained occurrence of each listed
186
+ schedule. A schedule that fires each second keeps about 1.2 million occurrences in the default
187
+ retention of 14 days. Each poll of the Schedules page read all of them (SM-1866).
188
+
189
+ Now the function reads at most 1,001 occurrences for each listed schedule. `occurrenceCount` stops
190
+ at 1,000, and the new optional key `occurrenceCountCapped` is `true` when more occurrences are
191
+ retained. The bundled Schedules page then shows `1,000+`. `lastFiredAt` is the fire time of the
192
+ newest occurrence. The `dashboard/v1` change is additive. A database below schema 88 returns an
193
+ exact count and no `occurrenceCountCapped`. The page change rebuilds the shared dashboard bundle
194
+ again. The sha256 in "The dashboard bundle uses Mantine 9.7.4" is the sha256 of that bundle.
195
+
196
+ ### The worker registry prune finds expired kept pauses through their index
197
+
198
+ Before migration 0089, `prune_worker_registry_v1` deleted expired kept pauses with a condition that
199
+ the planner could not match to `worker_pause_tombstone_expires_idx`. Each maintenance pass read the
200
+ whole table. Now the delete reads only the expired rows through the index (SM-1866).
201
+
202
+ ### A child set and a task group check every request before their first statement
203
+
204
+ Before this change, `HandlerContext#run_children` and `#run_children_all` checked the options of
205
+ each child while they serialized it. Thus the first child could read its payload contract before
206
+ the SDK refused the second child. `Queue#create_task_group` and `Queue#task_group` checked the
207
+ callbacks after the compatibility query and the member contract reads. A member add through
208
+ `HandlerContext#task_group` read the membership of its task first (SM-1883).
209
+
210
+ Now these calls check the options of every request before their first statement, also before the
211
+ deferred tier read of a child set. The errors do not change. Only their time changes.
212
+
6
213
  ## 0.10.0 — 2026-10-26
7
214
 
8
215
  The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
data/README.md CHANGED
@@ -30,7 +30,7 @@ bundle add connection_pool
30
30
  Install the schema once, as a deployment step. The application never installs or migrates it.
31
31
 
32
32
  ```bash
33
- npx --package @stablemates/workhorse@0.10.0 workhorse schema install
33
+ npx --package @stablemates/workhorse@0.11.0 workhorse schema install
34
34
  ```
35
35
 
36
36
  The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
@@ -108,7 +108,8 @@ The gem is `stablemates-workhorse`, and its namespace is `Stablemates::Workhorse
108
108
  that was `:non_replaceable` also carries a `reason`: `:incompatible_key_mode`, `:not_pending`, or
109
109
  `:window_elapsed_pending`.
110
110
  - `Worker` takes a pool, registers handlers by task type, and runs them under a lease with a shared
111
- heartbeat connection. `run_worker_process` drains it on `TERM` or `INT`.
111
+ heartbeat connection. The pool must report a maximum size of at least 3 connections, or the
112
+ constructor raises `ArgumentError`. `run_worker_process` drains the worker on `TERM` or `INT`.
112
113
  - `HandlerContext` offers checkpoints, durable sleeps, signal and human waits, child tasks, and
113
114
  progress. PostgreSQL owns every durable decision, and an unresolved wait suspends the task.
114
115
  - `Admin` lists, inspects, and repairs tasks, dead letters, waits, workers, and queues.
@@ -63,6 +63,10 @@ module Stablemates
63
63
 
64
64
  def submit(outcome) = @outcome.compare_and_set(nil, outcome)
65
65
 
66
+ # Replaces a cancellation that PostgreSQL refused to acknowledge with a lost lease. A
67
+ # heartbeat round can submit the cancellation request first, but the worker settled nothing.
68
+ def refuse_cancellation = @outcome.compare_and_set(:cancelled, :lease_expired)
69
+
66
70
  # Records that a fenced durable write released the task for the suspension +outcome+. The
67
71
  # attempt then suspends whichever outcome won, because the heartbeat reports that release as a
68
72
  # lost lease.
@@ -353,6 +357,8 @@ module Stablemates
353
357
  children.each { |child| child_name(child.name) }
354
358
  raise ArgumentError, "child names must be unique" unless children.map(&:name).uniq.length == children.length
355
359
 
360
+ # Every child is checked before the deferred tier read and the first contract read.
361
+ children.each { |child| @queue.check_child_request(@task, child.task_type, child.payload, child.options) }
356
362
  durable!("child tasks")
357
363
 
358
364
  build = lambda do |versions = {}|
@@ -2,5 +2,5 @@
2
2
  "formatVersion": 1,
3
3
  "readSurfaceVersion": 1,
4
4
  "archive": "read-surface-1.tar.gz",
5
- "sha256": "7a0bff093b8feb58861e351ea92e4d43939da90bcbd6905099b52cc61cf4a868"
5
+ "sha256": "a7d61ed1d0433e38a8d227cbcc8f98a0bf668f2d3bb1d47391b1882fd69bf795"
6
6
  }
@@ -322,7 +322,7 @@ module Stablemates
322
322
  "window.workhorseDashboard = #{JSON.generate(config).gsub(/[<>&

]/, SCRIPT_ESCAPES)}"
323
323
  }
324
324
  .sub("<!--__WORKHORSE_BROWSER_MODULES__-->") { modules }
325
- [200, {"content-type" => "text/html; charset=utf-8"}, [html]]
325
+ [200, {"content-type" => "text/html; charset=utf-8", "cache-control" => "no-store"}, [html]]
326
326
  end
327
327
 
328
328
  def asset(name)
@@ -298,9 +298,9 @@ module Stablemates
298
298
  end
299
299
  end
300
300
 
301
- # A worker with the dedicated heartbeat connection could not reserve that connection, or could
302
- # not borrow a connection for its readiness check, within its startup budget. The message
303
- # names the pool, the budget, and the +shared_heartbeats+ opt-out. A pool error is the +cause+.
301
+ # A worker could not reserve its dedicated heartbeat connection, or could not borrow a
302
+ # connection for its readiness check, within its startup budget. The message names the pool and
303
+ # the budget. A pool error is the +cause+.
304
304
  class WorkerStartupError < Error; end
305
305
 
306
306
  # Any other PG::Error. The original is the +cause+, and +sqlstate+ is its SQLSTATE or nil.
@@ -120,6 +120,8 @@ module Stablemates
120
120
  # SDK.
121
121
  def add_to_task_group_as_member(task, worker_id, requests) # :nodoc:
122
122
  TaskGroups.validate_add(requests)
123
+ # Every request is checked before the membership read.
124
+ request_inputs(requests)
123
125
  link = find_task_group(task.id)
124
126
  raise ArgumentError, "Task #{task.id} is not a member of a task group" unless link&.role == :member
125
127
 
@@ -338,6 +340,32 @@ module Stablemates
338
340
  # queue of +parent+. Internal to the SDK.
339
341
  def serialize_child_request(parent, task_type, payload, options, contracts = {}, version: :current, # :nodoc:
340
342
  stored_queue: nil)
343
+ request, input = child_input(parent, task_type, payload, options, stored_queue)
344
+ key = (version == :current) ? request.task_type : [request.task_type, version]
345
+ contract = contracts.fetch(key) do
346
+ contracts[key] = if version != :current
347
+ version && Contracts.load(@executor, request.task_type, version)
348
+ else
349
+ Contracts.load(@executor, request.task_type).tap do |loaded|
350
+ @lock.synchronize { @contracts[request.task_type] = loaded }
351
+ end
352
+ end
353
+ end
354
+ stamp_contract(input, request, contract)
355
+ input
356
+ end
357
+
358
+ # Checks a child request with no round trip. A child set checks every child with this before
359
+ # its first contract read. Internal to the SDK.
360
+ def check_child_request(parent, task_type, payload, options) # :nodoc:
361
+ child_input(parent, task_type, payload, options, nil)
362
+ nil
363
+ end
364
+
365
+ private
366
+
367
+ # The request and the input of a child, without its contract stamp.
368
+ def child_input(parent, task_type, payload, options, stored_queue)
341
369
  request = EnqueueRequest.new(task_type: task_type, payload: payload, **options)
342
370
  if request.idempotency || request.debounce || request.throttle || request.dependencies
343
371
  raise ArgumentError, "Child tasks cannot use coalescing or dependency enqueue options"
@@ -354,22 +382,9 @@ module Stablemates
354
382
  input["tags"] = tags(request.tags)
355
383
  input["traceContext"] = parent.trace_context unless parent.trace_context.nil?
356
384
  input["runAt"] = Values.timestamp(request.run_at, "run at") if request.run_at
357
- key = (version == :current) ? request.task_type : [request.task_type, version]
358
- contract = contracts.fetch(key) do
359
- contracts[key] = if version != :current
360
- version && Contracts.load(@executor, request.task_type, version)
361
- else
362
- Contracts.load(@executor, request.task_type).tap do |loaded|
363
- @lock.synchronize { @contracts[request.task_type] = loaded }
364
- end
365
- end
366
- end
367
- stamp_contract(input, request, contract)
368
- input
385
+ [request, input]
369
386
  end
370
387
 
371
- private
372
-
373
388
  # Serializes +requests+, passes their JSON document to the block, and returns the rows the block
374
389
  # returns. On a contract mismatch, PostgreSQL enqueued nothing: the client reloads the named
375
390
  # contracts and calls the block one more time.
@@ -388,20 +403,25 @@ module Stablemates
388
403
  end
389
404
 
390
405
  def serialize_requests(requests)
391
- now = Time.now
392
406
  trace_context = Telemetry.inject_trace_context
393
- inputs = requests.each_with_index.map do |request, index|
394
- input = serialize_request(request, now)
395
- input["traceContext"] = trace_context if trace_context
396
- input
397
- rescue ArgumentError => e
398
- raise ArgumentError, "enqueue request #{index + 1}: #{e.message}"
399
- end
407
+ inputs = request_inputs(requests)
408
+ inputs.each { |input| input["traceContext"] = trace_context } if trace_context
400
409
  assert_compatible
401
410
  inputs.zip(requests) { |input, request| apply_contract(input, request) }
402
411
  JSON.generate(inputs)
403
412
  end
404
413
 
414
+ # The inputs of +requests+, without a round trip, so a batch refuses an invalid option of any
415
+ # request before its first statement.
416
+ def request_inputs(requests)
417
+ now = Time.now
418
+ requests.each_with_index.map do |request, index|
419
+ serialize_request(request, now)
420
+ rescue ArgumentError => e
421
+ raise ArgumentError, "enqueue request #{index + 1}: #{e.message}"
422
+ end
423
+ end
424
+
405
425
  # Creates a group with +requests+ as its member tasks, and seals it when +seal+ is true. A
406
426
  # contract mismatch of a member or a callback creates nothing, and the retry serializes the
407
427
  # callbacks again under the contracts the mismatch reloaded.
@@ -413,6 +433,11 @@ module Stablemates
413
433
  raise ArgumentError, "task group callbacks cannot use enqueue idempotency, debounce, or throttle"
414
434
  end
415
435
  end
436
+ # The callbacks are serialized after the member contracts are read, so their options are
437
+ # checked here, before the first statement.
438
+ now = Time.now
439
+ callbacks.each { |kind, request| callback_input(kind, request, now) unless request.nil? }
440
+ request_inputs(requests)
416
441
  group_id = nil
417
442
  rows = submit_requests(requests) do |document|
418
443
  created = @executor.rows(SqlCatalogue::CREATE_TASK_GROUP_V1,
@@ -435,11 +460,7 @@ module Stablemates
435
460
  document = callbacks.each_with_object({}) do |(kind, request), entries|
436
461
  next if request.nil?
437
462
 
438
- input = begin
439
- serialize_request(request, now)
440
- rescue ArgumentError => e
441
- raise ArgumentError, "#{kind} callback: #{e.message}"
442
- end
463
+ input = callback_input(kind, request, now)
443
464
  input.delete("runAt") if request.run_at.nil?
444
465
  input["traceContext"] = trace_context if trace_context
445
466
  apply_contract(input, request)
@@ -448,6 +469,12 @@ module Stablemates
448
469
  JSON.generate(document)
449
470
  end
450
471
 
472
+ def callback_input(kind, request, now)
473
+ serialize_request(request, now)
474
+ rescue ArgumentError => e
475
+ raise ArgumentError, "#{kind} callback: #{e.message}"
476
+ end
477
+
451
478
  # +member+ is the task ID, worker ID, and fence token of a running member task, or three nils.
452
479
  def add_task_group_members(group_id, requests, member)
453
480
  TaskGroups.validate_add(requests)
@@ -545,14 +572,21 @@ module Stablemates
545
572
  def idempotency_document(idempotency)
546
573
  raise ArgumentError, "idempotency must be an Idempotency" unless idempotency.is_a?(Idempotency)
547
574
 
548
- ttl = Values.milliseconds(idempotency.ttl, "idempotency TTL", 0..MAX_DURATION_MS)
549
575
  {
550
576
  "key" => required_bytes(idempotency.key, MAX_KEY_BYTES, "idempotency key"),
551
577
  "scope" => scope(idempotency.scope),
552
- "ttlMs" => ttl.zero? ? DEFAULT_IDEMPOTENCY_TTL_MS : ttl
578
+ "ttlMs" => idempotency_ttl(idempotency.ttl)
553
579
  }
554
580
  end
555
581
 
582
+ # The protocol value of an idempotency TTL. An exact zero takes the default. A nonzero value
583
+ # that rounds to 0 ms is refused, so a unit mistake cannot select the default.
584
+ def idempotency_ttl(value)
585
+ return DEFAULT_IDEMPOTENCY_TTL_MS if value.is_a?(Numeric) && value.zero?
586
+
587
+ Values.milliseconds(value, "idempotency TTL", 1..MAX_DURATION_MS)
588
+ end
589
+
556
590
  def debounce_document(debounce)
557
591
  raise ArgumentError, "debounce must be a Debounce" unless debounce.is_a?(Debounce)
558
592
 
@@ -11,7 +11,7 @@ module Stablemates
11
11
  MINIMUM_PROTOCOL_VERSION = 5
12
12
  MAXIMUM_PROTOCOL_VERSION = 5
13
13
  MINIMUM_SCHEMA_VERSION = 76
14
- MAXIMUM_SCHEMA_VERSION = 86
14
+ MAXIMUM_SCHEMA_VERSION = 89
15
15
  # PostgreSQL's atomic enqueue batch limit.
16
16
  MAX_ENQUEUE_BATCH_SIZE = 1000
17
17
  DEFAULT_TASK_VALUE_MAX_BYTES = 1048576
@@ -163,11 +163,11 @@ module Stablemates
163
163
  # errors of a socket), or SocketError (DNS); or a SQLSTATE that starts with 08, or 57P01,
164
164
  # 57P02, or 57P03 (a server shutdown). IO::TimeoutError is an IOError, so a socket timeout
165
165
  # is a connection error too.
166
- # - "connection" also covers a pool error that CONNECTION_ERROR_NAMES names. Under
167
- # +shared_heartbeats: true+ the round borrows a connection through the pool's +with+, so a
168
- # wait that ends at the pool's own limit reaches the round: ConnectionPool::TimeoutError,
169
- # Sequel::PoolTimeout, or ActiveRecord::ConnectionTimeoutError (an
170
- # ActiveRecord::ConnectionNotEstablished).
166
+ # - "connection" also covers a pool error that CONNECTION_ERROR_NAMES names: a wait that ends at
167
+ # the pool's own limit, ConnectionPool::TimeoutError, Sequel::PoolTimeout, or
168
+ # ActiveRecord::ConnectionTimeoutError (an ActiveRecord::ConnectionNotEstablished). A round
169
+ # runs on the dedicated heartbeat connection and borrows none from the pool. The classifier
170
+ # keeps these names, so a pool error in a round still counts as "connection".
171
171
  # - "database": any other SQLSTATE.
172
172
  # - "other": everything else. A Timeout::Error that is not one of these, for example from
173
173
  # Timeout.timeout, is "other": only the round deadline is "timeout".
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Stablemates
4
4
  module Workhorse
5
- VERSION = "0.10.0"
5
+ VERSION = "0.11.0"
6
6
  end
7
7
  end
@@ -11,9 +11,7 @@ module Stablemates
11
11
  # never delay a lease round. A checkout thread borrows that connection (+Acquisition+), and the
12
12
  # heartbeat thread waits for it at most one round bound. The heartbeat replaces a lost
13
13
  # connection at once, with a backoff capped at the interval. A connection that arrives late
14
- # has one owner, which hands it over or returns it to the pool. With
15
- # +shared_heartbeats: true+ a worker has its own heartbeat, which borrows a connection from
16
- # the executor for each round instead.
14
+ # has one owner, which hands it over or returns it to the pool.
17
15
  class Heartbeat # :nodoc:
18
16
  # One running attempt. +renew+ takes the monotonic send time of an accepted round;
19
17
  # +settle+ takes a status other than +accepted+ and +busy+; +fail+ takes an error +settle+
@@ -175,14 +173,11 @@ module Stablemates
175
173
  # statement, plus a margin for the payload, the wait for the connection, and BEGIN.
176
174
  BUSY_RETRY_MARGIN = 0.012
177
175
  BUSY_RETRY_MINIMUM = MINIMUM_REMAINING + BUSY_RETRY_MARGIN
178
- # The wait after a shared round raised an unexpected error, in seconds, so that an error that
179
- # repeats cannot keep the thread busy.
180
- FAILED_ROUND_DELAY = 0.1
181
176
  # heartbeat_many_v2 raises an error for more leases than this in one call, so a batch of
182
177
  # heartbeat_pool_v1 holds at most this many leases too.
183
178
  STATEMENT_LEASE_LIMIT = 100
184
179
  private_constant :STRINGS, :ROUND_TIMEOUTS, :MINIMUM_REMAINING, :REACQUIRE_BASE, :PROBE_INTERVAL, :RELEASE_WAIT,
185
- :BUSY_RETRY_BASE, :BUSY_RETRY_MARGIN, :BUSY_RETRY_MINIMUM, :FAILED_ROUND_DELAY, :STATEMENT_LEASE_LIMIT
180
+ :BUSY_RETRY_BASE, :BUSY_RETRY_MARGIN, :BUSY_RETRY_MINIMUM, :STATEMENT_LEASE_LIMIT
186
181
 
187
182
  # The deadline of one round, in seconds, for an +interval+ in seconds. It is three quarters
188
183
  # of the interval, so that one timed-out round leaves a renewal margin when 2 × interval +
@@ -213,12 +208,11 @@ module Stablemates
213
208
 
214
209
  # The heartbeat every worker on +pool+ shares.
215
210
  def self.dedicated(pool)
216
- @dedicated_lock.synchronize { @dedicated[pool] ||= new(pool, dedicated: true) }
211
+ @dedicated_lock.synchronize { @dedicated[pool] ||= new(pool) }
217
212
  end
218
213
 
219
- def initialize(source, dedicated:)
214
+ def initialize(source)
220
215
  @source = source
221
- @dedicated = dedicated
222
216
  @lock = Mutex.new
223
217
  @members = {}
224
218
  @users = {}.compare_by_identity
@@ -344,7 +338,7 @@ module Stablemates
344
338
  # failed round does not start a new schedule.
345
339
  @previous = Process.clock_gettime(Process::CLOCK_MONOTONIC)
346
340
  @busy = nil
347
- @dedicated ? run_dedicated : run_shared
341
+ run_dedicated
348
342
  ensure
349
343
  lost
350
344
  end
@@ -369,21 +363,6 @@ module Stablemates
369
363
  probe
370
364
  end
371
365
 
372
- # Runs the rounds of a borrowing heartbeat. An unexpected error does not end the thread while
373
- # a member remains: the thread ends only through +stop_if_idle+, so a register that races
374
- # the end starts a new thread.
375
- def run_shared
376
- loop do
377
- return if rounds { |sql, params, _timeout| Executor.for(@source).rows(sql, params) }
378
- rescue
379
- @busy = nil
380
- return if stop_if_idle
381
-
382
- @wake.wait(FAILED_ROUND_DELAY)
383
- @wake.reset
384
- end
385
- end
386
-
387
366
  # Holds one connection across rounds, also while no member remains and a worker still
388
367
  # runs. Each attempt waits at most one round bound for its checkout (+acquire+). A round
389
368
  # that fails or overruns discards the connection, and so does a probe that finds it unusable
@@ -597,7 +576,7 @@ module Stablemates
597
576
  # member interval after the previous regular round started; a wake recomputes that from the
598
577
  # current members. A round that ends after its next slot starts one overdue round at once,
599
578
  # and the interval after it counts from that start, so missed slots are never replayed. A
600
- # failed round raises only from a dedicated connection, so the caller can replace it.
579
+ # failed round raises, so the caller can replace the connection.
601
580
  #
602
581
  # Between two regular rounds, a busy retry (+take_busy_retry+) sends only the members that
603
582
  # the latest round reported busy. It runs on the same thread, so it is a round for the
@@ -642,7 +621,7 @@ module Stablemates
642
621
  deadline = retry_deadline(selection, interval, regular_at, started)
643
622
  # A bounded retry with too little time left would fail before its heartbeat statement.
644
623
  # It sends nothing, and the next regular round renews every lease, the busy ones too.
645
- next if @dedicated && deadline - started < BUSY_RETRY_MINIMUM
624
+ next if deadline - started < BUSY_RETRY_MINIMUM
646
625
 
647
626
  send_round(selection, deadline, interval, &)
648
627
  next
@@ -665,9 +644,7 @@ module Stablemates
665
644
  #
666
645
  # On the dedicated connection, one heartbeat_pool_v1 statement in one transaction renews
667
646
  # every batch of every worker, by the one +deadline+ (+send_pooled+). The round is
668
- # all-or-nothing: if the statement or COMMIT fails, no lease of the round counts. A
669
- # borrowing heartbeat has no deadline, and sends one heartbeat_many_v2 statement for each
670
- # batch (+send_each+).
647
+ # all-or-nothing: if the statement or COMMIT fails, no lease of the round counts.
671
648
  #
672
649
  # A round replaces the pending busy retry. If one or more members are busy, it schedules one
673
650
  # retry for them. A round that raises or expires schedules none, and the next regular round
@@ -675,7 +652,7 @@ module Stablemates
675
652
  def send_round(members, deadline, interval, &)
676
653
  @busy = nil
677
654
  batches = batches(members)
678
- busy = @dedicated ? send_pooled(batches, deadline, &) : send_each(batches, deadline, &)
655
+ busy = send_pooled(batches, deadline, &)
679
656
  return if busy.empty?
680
657
 
681
658
  @busy = busy
@@ -720,29 +697,7 @@ module Stablemates
720
697
  batches.each_with_index.flat_map { |(_worker_id, group), index| settle(group, statuses[index], sent_at) }
721
698
  end
722
699
 
723
- # Sends one heartbeat_many_v2 statement for each batch through the executor, and returns the
724
- # busy members. A batch whose statement fails gets nothing, and the other batches still go.
725
- # A round with one or more failed statements counts once in
726
- # +workhorse.worker.heartbeat.round_failure+, with the reason of its first failure. The count
727
- # occurs when that failure occurs, so a later batch that stalls does not delay it.
728
- def send_each(batches, deadline)
729
- failed = false
730
- batches.flat_map do |worker_id, group|
731
- sent_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
732
- params = [worker_id, JSON.generate(leases(group))]
733
- rows = begin
734
- yield SqlCatalogue::HEARTBEAT_MANY_V2, params, deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
735
- rescue => e
736
- # A skipped round sends no log record: the lease watchdog handles rounds that keep failing.
737
- Telemetry.record_heartbeat_round_failure(e) unless failed
738
- failed = true
739
- next []
740
- end
741
- settle(group, rows.to_h { |row| [row["task_id"], row["status"]] }, sent_at)
742
- end
743
- end
744
-
745
- # The lease objects of heartbeat_many_v2 for the members of one batch.
700
+ # The lease objects of one batch of heartbeat_pool_v1.
746
701
  def leases(group)
747
702
  group.map do |member|
748
703
  {"taskId" => member.task.id, "fenceToken" => member.task.fence_token.to_s, "leaseMs" => member.lease_ms}
@@ -768,8 +723,7 @@ module Stablemates
768
723
 
769
724
  # A busy retry ends by its own round bound, by the next regular slot, and by the end of the
770
725
  # earliest local lease window in it, so it never renews a lease that the worker has already
771
- # given up. A borrowing heartbeat ignores the deadline, as in its regular rounds. +started+ is
772
- # the monotonic start of the retry.
726
+ # given up. +started+ is the monotonic start of the retry.
773
727
  def retry_deadline(selection, interval, regular_at, started)
774
728
  ends = selection.map { |member| member.lease_ends_at.call || started }
775
729
  [started + round_bound(interval), regular_at, *ends].min
@@ -826,43 +780,25 @@ module Stablemates
826
780
  @lock.synchronize { @members[[member.worker_id, member.task.id]].equal?(member) }
827
781
  end
828
782
 
829
- # Ends a borrowing heartbeat thread when no member and no worker remains, and returns true
830
- # then. A register that races this starts a new one.
831
- def stop_if_idle
832
- @lock.synchronize do
833
- unused = @members.empty? && @users.empty?
834
- if unused
835
- @thread = nil
836
- clear_mark
837
- end
838
- unused
839
- end
840
- end
841
-
842
783
  # The shortest member interval; an Idle with the shortest worker interval when only workers
843
- # remain; or nil when nothing remains. Nil also ends a borrowing heartbeat thread. The read
844
- # and the end take one lock, so a register between them cannot find a thread that is about
845
- # to end. A dedicated thread ends later, in +interval_or_stop+, after the pool has its
846
- # connection back.
784
+ # remain; or nil when nothing remains. The thread does not end here: it ends later, in
785
+ # +interval_or_stop+, after the pool has its connection back.
847
786
  def shortest_interval_or_stop
848
787
  @lock.synchronize do
849
788
  next @members.each_value.map(&:interval).min unless @members.empty?
850
789
  next Idle.new(@users.each_key.map(&:interval).min) unless @users.empty?
851
790
 
852
- @thread = nil unless @dedicated
853
791
  nil
854
792
  end
855
793
  end
856
794
 
857
795
  # The current members; an empty Array when only workers remain; or nil when nothing
858
- # remains. Nil also ends a borrowing heartbeat thread, under the same lock as the read
859
- # (+shortest_interval_or_stop+).
796
+ # remains (+shortest_interval_or_stop+).
860
797
  def members_or_stop
861
798
  @lock.synchronize do
862
799
  next @members.values unless @members.empty?
863
800
  next [] unless @users.empty?
864
801
 
865
- @thread = nil unless @dedicated
866
802
  nil
867
803
  end
868
804
  end
@@ -13,7 +13,8 @@ module Stablemates
13
13
  # smallest supported +heartbeat+ is 0.1. The worker accepts a shorter value, but on a loaded
14
14
  # host it can renew no lease.
15
15
  #
16
- # Unless +shared_heartbeats+ is set, each run first reserves the heartbeat connection that the
16
+ # The pool must report a maximum +size+ of at least 3: the heartbeat connection, the listener,
17
+ # and one connection for claims. Each run first reserves the heartbeat connection that the
17
18
  # workers on the pool share, and then borrows one other connection for the compatibility query.
18
19
  # Both steps must end within max(+heartbeat+, 5 s), or the run raises WorkerStartupError. The
19
20
  # worker claims nothing while the pool has no usable heartbeat connection.
@@ -207,20 +208,13 @@ module Stablemates
207
208
  :gate) # :nodoc:
208
209
  private_constant :PendingCompletion
209
210
 
210
- # The claim gate of a run under shared_heartbeats, which never closes. A run with the
211
- # dedicated heartbeat connection uses its Heartbeat::Reservation as its gate.
212
- OPEN_GATE = Object.new.tap do |gate|
213
- gate.define_singleton_method(:ready?) { true }
214
- gate.define_singleton_method(:enter) { true }
215
- gate.define_singleton_method(:leave) { nil }
216
- end.freeze
217
- # The gate of a fused completion that no running dispatch admitted.
211
+ # A run uses its Heartbeat::Reservation as its claim gate. The gate of a fused completion that no running dispatch admitted.
218
212
  CLOSED_GATE = Object.new.tap do |gate|
219
213
  gate.define_singleton_method(:ready?) { false }
220
214
  gate.define_singleton_method(:enter) { false }
221
215
  gate.define_singleton_method(:leave) { nil }
222
216
  end.freeze
223
- private_constant :OPEN_GATE, :CLOSED_GATE
217
+ private_constant :CLOSED_GATE
224
218
 
225
219
  # Raised before a claim statement when the claim gate is closed at send time. The claim sends
226
220
  # nothing.
@@ -233,6 +227,14 @@ module Stablemates
233
227
  end
234
228
  private_constant :Borrowed
235
229
 
230
+ POOL_SIZE_REFUSAL = "Worker needs a pool with a known maximum size of at least #{MIN_POOL_SIZE} connections " \
231
+ "(found %s). Give the worker a pool with a maximum size of at least #{MIN_POOL_SIZE}."
232
+ # Every worker reserves a dedicated heartbeat connection since 0.11.0 (ADR 0071, SM-1901).
233
+ SHARED_HEARTBEATS_REMOVED = "shared_heartbeats was removed in 0.11.0. Every worker now reserves a dedicated " \
234
+ "heartbeat connection. Remove the option, and give the worker a pool with a known maximum size of at least " \
235
+ "#{MIN_POOL_SIZE}."
236
+ private_constant :POOL_SIZE_REFUSAL, :SHARED_HEARTBEATS_REMOVED
237
+
236
238
  RETRY_DELAY_RANGE = (0..2_147_483_647)
237
239
  RETRY_DELAY_REFUSAL = "retry_delay must be a finite Numeric count of seconds between 0 and 2147483.647"
238
240
  private_constant :RETRY_DELAY_RANGE, :RETRY_DELAY_REFUSAL
@@ -247,9 +249,10 @@ module Stablemates
247
249
  def initialize(pool, queues: ["default"], worker_id: nil, concurrency: 1, lease: 30, heartbeat: nil,
248
250
  poll_interval: nil, polling_only: false, maintenance_interval: 1, maintenance_routine_interval: 60,
249
251
  registry_interval: 5, disable_registry: false, schedule_namespaces: [], schedule_catchup_limit: 100,
250
- shutdown_grace: 25, shared_heartbeats: false, retry_delay: nil, on_registration_error: nil,
251
- on_notification_error: nil, cohorts: nil, logger: nil)
252
- @executor = worker_executor(pool, shared_heartbeats)
252
+ shutdown_grace: 25, retry_delay: nil, on_registration_error: nil, on_notification_error: nil,
253
+ cohorts: nil, logger: nil, **removed)
254
+ refuse_removed_options(removed)
255
+ @executor = worker_executor(pool)
253
256
  @pool = pool
254
257
  @queues = unique_names(queues, "queues")
255
258
  raise ArgumentError, "queues must contain at least one non-empty queue name" if @queues.empty?
@@ -258,7 +261,7 @@ module Stablemates
258
261
  concurrency.is_a?(Integer) && concurrency.between?(1, 100)
259
262
 
260
263
  @concurrency = concurrency
261
- @cohorts = dispatch_cohorts(cohorts, pool, shared_heartbeats, polling_only)
264
+ @cohorts = dispatch_cohorts(cohorts, pool, polling_only)
262
265
  @worker_id = worker_id || "#{Socket.gethostname}-#{Process.pid}-#{SecureRandom.hex(4)}"
263
266
  raise ArgumentError, "worker_id must be a non-empty String" unless @worker_id.is_a?(String) && !@worker_id.empty?
264
267
 
@@ -292,8 +295,7 @@ module Stablemates
292
295
  @reporter_failures_lock = Mutex.new
293
296
  @logger = logger
294
297
  @queue = Queue.new(@executor, default_queue: @queues.first)
295
- @heartbeat = shared_heartbeats ? Heartbeat.new(@executor, dedicated: false) : Heartbeat.dedicated(pool)
296
- @shared_heartbeats = shared_heartbeats ? true : false
298
+ @heartbeat = Heartbeat.dedicated(pool)
297
299
  @handlers = Concurrent::Map.new
298
300
  @batch_coordinators = Concurrent::Map.new
299
301
  @contracts = Concurrent::Map.new
@@ -310,6 +312,11 @@ module Stablemates
310
312
  @last_routine_at = -Float::INFINITY
311
313
  @last_registry_at = -Float::INFINITY
312
314
  @registered = Concurrent::AtomicBoolean.new(false)
315
+ # The newest registry write thread (+registry_step+). Each write waits for the one before it.
316
+ @registry_lock = Mutex.new
317
+ @registry_write = nil
318
+ # The first error of the run that ended the body of a registry write (+registry_step+).
319
+ @registry_failure = Concurrent::AtomicReference.new
313
320
  @instance_id = nil
314
321
  @draining = false
315
322
  # Guards the dispatch slots, the queue tiers, and the claim order that handler threads share
@@ -386,14 +393,25 @@ module Stablemates
386
393
 
387
394
  private
388
395
 
389
- def worker_executor(pool, shared_heartbeats)
396
+ # Raises the ArgumentError of a keyword that +initialize+ does not take. Any keyword other
397
+ # than the removed +shared_heartbeats+ gets the message Ruby gives for an unknown keyword, and
398
+ # it comes first, so the removal message never hides a typo. The removed option alone gets a
399
+ # message that names its replacement.
400
+ def refuse_removed_options(removed)
401
+ unknown = removed.keys - [:shared_heartbeats]
402
+ unless unknown.empty?
403
+ raise ArgumentError, "unknown keyword#{"s" if unknown.size > 1}: #{unknown.map(&:inspect).join(", ")}"
404
+ end
405
+ raise ArgumentError, SHARED_HEARTBEATS_REMOVED unless removed.empty?
406
+ end
407
+
408
+ def worker_executor(pool)
390
409
  raise ArgumentError, "Worker pool must be a ConnectionPool, not a PG::Connection" if pool.is_a?(PG::Connection)
391
410
  raise ArgumentError, "Worker pool must respond to with" unless pool.respond_to?(:with)
392
411
 
393
412
  capacity = pool.respond_to?(:size) ? pool.size : nil
394
- if !shared_heartbeats && !(capacity.is_a?(Integer) && capacity >= MIN_POOL_SIZE)
395
- raise ArgumentError, "Worker pool capacity must be at least #{MIN_POOL_SIZE} " \
396
- "(found #{capacity.nil? ? "unknown" : capacity}); set shared_heartbeats: true to opt out"
413
+ unless capacity.is_a?(Integer) && capacity >= MIN_POOL_SIZE
414
+ raise ArgumentError, format(POOL_SIZE_REFUSAL, capacity.is_a?(Integer) ? capacity : "unknown")
397
415
  end
398
416
  Executor.for(pool)
399
417
  end
@@ -402,7 +420,7 @@ module Stablemates
402
420
  # pooled connection per cohort after the listener and the heartbeat connection, which every
403
421
  # worker on the pool shares. A polling-only worker opens no listener, so it keeps that
404
422
  # connection for its cohorts.
405
- def dispatch_cohorts(cohorts, pool, shared_heartbeats, polling_only)
423
+ def dispatch_cohorts(cohorts, pool, polling_only)
406
424
  unless cohorts.nil?
407
425
  raise ArgumentError, "cohorts must be an integer between 1 and concurrency" unless
408
426
  cohorts.is_a?(Integer) && cohorts.between?(1, @concurrency)
@@ -413,7 +431,7 @@ module Stablemates
413
431
  capacity = pool.respond_to?(:size) ? pool.size : nil
414
432
  return default unless capacity.is_a?(Integer)
415
433
 
416
- spare = capacity - (polling_only ? 0 : 1) - (shared_heartbeats ? 0 : 1)
434
+ spare = capacity - (polling_only ? 0 : 1) - 1
417
435
  spare.clamp(1, default)
418
436
  end
419
437
 
@@ -507,13 +525,13 @@ module Stablemates
507
525
  return false if reservation == :stopped
508
526
 
509
527
  begin
510
- run_reserved(continuous, version, reservation || OPEN_GATE)
528
+ run_reserved(continuous, version, reservation)
511
529
  ensure
512
530
  # The run's members renew until the drain ends, so the reservation outlives it. Its
513
531
  # release waits only within the stop deadline of the run.
514
532
  final = @final_deadline || monotonic
515
533
  @final_deadline = nil
516
- reservation&.close([final - monotonic, 0].max)
534
+ reservation.close([final - monotonic, 0].max)
517
535
  end
518
536
  end
519
537
 
@@ -521,22 +539,47 @@ module Stablemates
521
539
  # at most the shutdown grace plus the unwind window in total.
522
540
  def remaining(deadline) = [deadline - monotonic, 0].max
523
541
 
524
- # Runs +block+ on its own thread named +name+ and waits for it until +deadline+. A block that
525
- # is still running then finishes on its own thread, which keeps its connection until its
526
- # statement ends. The stop path sends its registration statements this way. The next run of
527
- # this worker waits for these threads (+await_previous_stop+).
528
- def within(deadline, name, &block)
529
- thread = Thread.new do
530
- Thread.current.name = name
531
- block.call
532
- rescue
533
- nil
534
- end
542
+ # Waits for the registry write +thread+ (+registry_step+) until +deadline+. A write that is
543
+ # still running then finishes on its own thread, which keeps its connection until its statement
544
+ # ends. The stop path sends its registration statements this way. The next run of this worker
545
+ # waits for these threads (+await_previous_stop+). Each write waits for the write before it,
546
+ # so these threads also outlast every periodic refresh of the run.
547
+ def within(deadline, thread)
535
548
  @stop_statements << thread
536
549
  thread.join(remaining(deadline))
537
550
  nil
538
551
  end
539
552
 
553
+ # Starts a registry write on its own thread named +name+ and returns the thread. The write
554
+ # first waits for the write before it, so the registrations and the deregistration of a run
555
+ # reach PostgreSQL in that order. The thread takes the earlier write out of its holder before
556
+ # it waits. Thus no write keeps each earlier write reachable (SM-1846, SM-1847).
557
+ #
558
+ # The thread never ends with an error. Otherwise the wait of the next write, and +within+,
559
+ # would raise it, and the chain and the stop path would end there. The write records the
560
+ # first error of the run in @registry_failure instead, and the run raises it
561
+ # (+raise_registry_failure+). Only an error that is not a StandardError gets there: the
562
+ # registration logs and reports a StandardError, and the deregistration ignores one.
563
+ def registry_step(name, &body)
564
+ @registry_lock.synchronize do
565
+ before = [@registry_write]
566
+ @registry_write = Thread.new do
567
+ Thread.current.name = name
568
+ before.pop&.join
569
+ body.call
570
+ rescue Exception => e # rubocop:disable Lint/RescueException
571
+ @registry_failure.compare_and_set(nil, e)
572
+ nil
573
+ end
574
+ end
575
+ end
576
+
577
+ # Raises the first error that ended the body of a registry write of this run, if one did.
578
+ def raise_registry_failure
579
+ failure = @registry_failure.get
580
+ raise failure if failure
581
+ end
582
+
540
583
  # Waits until every stop statement of the previous run of this worker has ended, at most
541
584
  # until +deadline+. deregister_worker_v2 deletes only the row of its own run, but a late
542
585
  # registration write of the previous run would still replace the registry row of this run, and
@@ -566,6 +609,9 @@ module Stablemates
566
609
  def run_reserved(continuous, version, gate)
567
610
  @instance_id = SecureRandom.uuid
568
611
  @registered.make_false
612
+ # Every registry write of the previous run has ended (+await_previous_stop+), so a failure
613
+ # that one of them recorded after its run ended does not reach this run.
614
+ @registry_failure.set(nil)
569
615
  @draining = false
570
616
  run_errors = Concurrent::Array.new
571
617
  maintenance_errors = Concurrent::Array.new
@@ -574,11 +620,12 @@ module Stablemates
574
620
  startup = nil
575
621
  claimed_any = false
576
622
  begin
577
- if continuous && !@polling_only && pool_lends_listener?
623
+ if continuous && !@polling_only
578
624
  listener = Listener.shared(@pool).subscribe(@queues, on_wake: method(:notify),
579
625
  on_error: ->(error) { call_reporter(:on_notification_error, @on_notification_error, error) })
580
626
  end
581
- refresh_registration(force: true)
627
+ registry_step("workhorse-registration") { refresh_registration(force: true) }.join
628
+ raise_registry_failure
582
629
  log(:info, "workhorse.worker.started", "Worker started",
583
630
  "workhorse.worker.concurrency" => @concurrency, "workhorse.worker.queues" => @queues)
584
631
  if continuous && !stop_requested?(version)
@@ -604,15 +651,18 @@ module Stablemates
604
651
  startup&.join(remaining(stop_deadline))
605
652
  listener&.close(remaining(stop_deadline))
606
653
  @draining = true
607
- within(stop_deadline, "workhorse-stop-registration") { refresh_registration(force: true) }
654
+ within(stop_deadline, registry_step("workhorse-stop-registration") { refresh_registration(force: true) })
608
655
  abandoned = drain(handlers, stop_deadline, final_deadline)
609
656
  # A handler's completion may admit tasks until the drain ends, so the slots outlive it.
610
657
  @state_lock.synchronize { @dispatch_slots = nil }
611
658
  instance_id = @instance_id
612
- within(final_deadline, "workhorse-deregistration") { deregister(instance_id) }
659
+ within(final_deadline, registry_step("workhorse-deregistration") { deregister(instance_id) })
613
660
  log(:info, "workhorse.worker.stopped", "Worker stopped",
614
661
  "workhorse.worker.active_slots" => abandoned, "workhorse.worker.queues" => @queues)
615
662
  end
663
+ # A registry write that failed after the last dispatch pass raises only now, so the drain and
664
+ # the deregistration still ran. A write that has not ended by the final deadline is not known.
665
+ raise_registry_failure
616
666
  raise ShutdownIncompleteError.new(abandoned) if abandoned.positive?
617
667
  raise maintenance_errors.first unless maintenance_errors.empty?
618
668
  raise run_errors.first unless run_errors.empty?
@@ -622,15 +672,10 @@ module Stablemates
622
672
 
623
673
  # Reserves the heartbeat connection of the pool, then borrows one other connection for the
624
674
  # compatibility query and returns it at once, both within the startup budget. Returns the
625
- # Heartbeat::Reservation, or nil under shared_heartbeats. A failure releases the reservation
626
- # and raises WorkerStartupError, or CompatibilityError for an incompatible schema. A stop
627
- # issued after +version+ ends the wait for the reservation, closes it, and returns :stopped.
675
+ # Heartbeat::Reservation. A failure releases the reservation and raises WorkerStartupError, or
676
+ # CompatibilityError for an incompatible schema. A stop issued after +version+ ends the wait
677
+ # for the reservation, closes it, and returns :stopped.
628
678
  def start_heartbeat(version, deadline, budget)
629
- if @shared_heartbeats
630
- @queue.assert_compatible
631
- return nil
632
- end
633
-
634
679
  reservation = @heartbeat.reserve(@heartbeat_ms / 1000.0, method(:wake_dispatcher))
635
680
  begin
636
681
  loop do
@@ -673,25 +718,12 @@ module Stablemates
673
718
  raise startup_error("could not borrow a connection for its readiness check", budget), cause: probe.error
674
719
  end
675
720
 
676
- # Whether the pool can lend the shared listener a connection (SM-1585). The listener holds one
677
- # pooled connection while it listens. With the opt-out, a pool smaller than MIN_POOL_SIZE is
678
- # allowed, so the listener starts only on a pool that reports a size of at least 2. Otherwise
679
- # it could hold the only connection, and the worker polls instead. The run logs that once.
680
- def pool_lends_listener?
681
- size = @pool.respond_to?(:size) ? @pool.size : nil
682
- return true if size.is_a?(Integer) && size >= 2
683
-
684
- log(:warn, "workhorse.worker.polling_only", "Worker has no notification listener and dispatches by polling",
685
- "workhorse.worker.queues" => @queues, "workhorse.worker.pool_max_size" => size.is_a?(Integer) ? size : "unknown")
686
- false
687
- end
688
-
689
721
  def startup_error(failure, budget)
690
722
  name = @pool.name if @pool.respond_to?(:name)
691
723
  size = @pool.respond_to?(:size) ? @pool.size : "unknown"
692
724
  pool = "#{@pool.class.name}#{" #{name.inspect}" if name} (size #{size})"
693
725
  WorkerStartupError.new("Worker #{failure} from pool #{pool} within the startup budget of " \
694
- "#{budget.round(3)} s; set shared_heartbeats: true to opt out")
726
+ "#{budget.round(3)} s; grow the pool, or check that the database accepts connections")
695
727
  end
696
728
 
697
729
  # Keeps the slots full without one serial claim round trip per task (ADR 0076). A claim
@@ -701,9 +733,8 @@ module Stablemates
701
733
  # fast-tier completion claims too, from the same accounting. On the fast tier the slots split
702
734
  # into cohorts, and plain claims leave one at a time, each for the roomiest cohort.
703
735
  #
704
- # +gate+ is the claim gate of this run. Under shared_heartbeats it is always open. Otherwise it
705
- # is the run's Heartbeat::Reservation: open while the run holds it and the pool holds a
706
- # usable heartbeat connection. A busy or rejected lease does not close it.
736
+ # +gate+ is the claim gate of this run, the run's Heartbeat::Reservation: open while the run
737
+ # holds it and the pool holds a usable heartbeat connection. A busy or rejected lease does not close it.
707
738
  def dispatch(continuous, version, handlers, listener, startup, run_errors, maintenance_errors, gate)
708
739
  slots = DispatchSlots.new(@concurrency, @cohorts, listener, version, run_errors, handlers, gate)
709
740
  @state_lock.synchronize { @dispatch_slots = slots }
@@ -735,7 +766,7 @@ module Stablemates
735
766
  loop do
736
767
  @wake.reset
737
768
  settle.call(results.pop) until results.empty?
738
- refresh_registration
769
+ refresh_registration_if_due
739
770
  claim_error, pass_ended, empty, empty_wait, wake_version = @state_lock.synchronize do
740
771
  [slots.claim_error, slots.pass_ended, slots.empty, slots.empty_wait, @wake_version]
741
772
  end
@@ -1300,6 +1331,8 @@ module Stablemates
1300
1331
  # A task the executor refused never runs, so the drain must not wait for it.
1301
1332
  @active.delete(task.id)
1302
1333
  @dispatch_order.delete(task.id)
1334
+ @fast_task_ids.delete(task.id)
1335
+ @tier_reads.delete(task.id)
1303
1336
  slots.cohort_active[cohort] -= 1 if slots.thread_cohorts.delete(task.id)
1304
1337
  raise
1305
1338
  end
@@ -1379,6 +1412,20 @@ module Stablemates
1379
1412
 
1380
1413
  # --- Registration and maintenance -------------------------------------------------------
1381
1414
 
1415
+ # Starts the periodic registration refresh on its own thread when its interval elapsed and no
1416
+ # registry write still runs. Thus at most one registry statement is in flight, and a refresh
1417
+ # that stalls blocks neither dispatch nor the stop path. Only the dispatch loop calls it. The
1418
+ # error that ended an earlier refresh raises here, so it ends the run as it did on the dispatch
1419
+ # thread.
1420
+ def refresh_registration_if_due
1421
+ return if @registry_ms.zero?
1422
+
1423
+ raise_registry_failure
1424
+ return if monotonic - @last_registry_at < @registry_ms / 1000.0 || @registry_write&.alive?
1425
+
1426
+ registry_step("workhorse-registration") { refresh_registration }
1427
+ end
1428
+
1382
1429
  def refresh_registration(force: false)
1383
1430
  return if @registry_ms.zero?
1384
1431
 
@@ -1582,21 +1629,20 @@ module Stablemates
1582
1629
  # span.
1583
1630
  def release(task, arbiter, attributes)
1584
1631
  log(:warn, "workhorse.handler.missing", "No handler registered for the claimed task type", attributes)
1585
- release_owned(task, arbiter, attributes)
1632
+ release_owned(task, arbiter, attributes, nil)
1586
1633
  end
1587
1634
 
1588
1635
  # Hands an owned task back to PostgreSQL, which makes it claimable again without charging the
1589
- # attempt. Returns the release status.
1590
- def release_owned(task, arbiter, attributes)
1636
+ # attempt. Returns the release status, or nil after a reconciled stale answer.
1637
+ def release_owned(task, arbiter, attributes, ownership)
1591
1638
  status = fenced_row(SqlCatalogue::RELEASE_OWNED_V1, task)["status"]
1592
1639
  log(:info, "workhorse.task.release_processed", "Owned task release processed",
1593
1640
  attributes.merge("workhorse.release.status" => status))
1594
1641
  if status == "released"
1595
1642
  arbiter.submit(:released)
1596
- else
1597
- settle_status(task, status, arbiter, "release", ["not_due"], :lease_expired)
1643
+ return status
1598
1644
  end
1599
- status
1645
+ settle_status(task, status, arbiter, "release", ["not_due"], :lease_expired, ownership)
1600
1646
  end
1601
1647
 
1602
1648
  def run_handler(task, handler, claim_started_at, active, arbiter, state)
@@ -1621,11 +1667,11 @@ module Stablemates
1621
1667
 
1622
1668
  # A handler that stopped for shutdown returns the task to the queue without an attempt.
1623
1669
  if cancellation.reason == :shutdown && shutdown_cancellation?(e)
1624
- state[:span_outcome] = release_owned(task, arbiter, Telemetry.task_span_attributes(task))
1670
+ state[:span_outcome] = release_owned(task, arbiter, Telemetry.task_span_attributes(task), ownership)
1625
1671
  return
1626
1672
  end
1627
1673
 
1628
- state[:span_outcome] = settle_failure(task, e, arbiter)
1674
+ state[:span_outcome] = settle_failure(task, e, arbiter, ownership)
1629
1675
  state[:errors] << (task.redact_error_details ? REDACTED_NAME : error_name(e))
1630
1676
  return
1631
1677
  ensure
@@ -1705,15 +1751,39 @@ module Stablemates
1705
1751
  # nothing. The heartbeat thread delivers statuses inline, so the lock is not held during the
1706
1752
  # statements here.
1707
1753
  def reconcile_rejected_completion(task, ownership, arbiter)
1708
- decided = ownership.settlement.synchronize { !arbiter.outcome.nil? }
1709
- return if decided && finish_lifecycle(task, arbiter)
1754
+ reconcile_refusal(task, ownership, arbiter, "complete")
1755
+ end
1710
1756
 
1711
- outcome = outcome_for_status(expire_owned(task, Telemetry.current_context), ["not_due"], "complete")
1757
+ # Like reconcile_rejected_completion, for the fenced write of +operation+: the completion, or a
1758
+ # failure or release that PostgreSQL answered stale (SM-1884). Without +ownership+, no
1759
+ # heartbeat round can settle the attempt, so the reconciliation does not wait.
1760
+ def reconcile_refusal(task, ownership, arbiter, operation)
1761
+ decided = ownership&.settlement&.synchronize { !arbiter.outcome.nil? }
1762
+ if decided
1763
+ begin
1764
+ return if finish_lifecycle(task, arbiter)
1765
+ rescue LeaseLostError
1766
+ # A heartbeat round submitted a cancellation that PostgreSQL refused to acknowledge.
1767
+ arbiter.refuse_cancellation
1768
+ raise
1769
+ end
1770
+ end
1771
+
1772
+ settle_refusal(task, arbiter, operation)
1773
+ end
1774
+
1775
+ # Settles the cause of a fenced write that PostgreSQL refused, under the attempt's own fence.
1776
+ # expire_owned_v1 does not read expires_at, so a due deadline or attempt timeout settles also
1777
+ # after the lease lapsed (SM-1884). A requested cancellation settles when acknowledge_cancel_v1
1778
+ # accepts it. Any other answer submits a lost lease and raises LeaseLostError.
1779
+ def settle_refusal(task, arbiter, operation)
1780
+ outcome = outcome_for_status(expire_owned(task, Telemetry.current_context), ["not_due"], operation)
1712
1781
  return arbiter.submit(:cancelled) if outcome == :cancelled && acknowledge_cancel(task)
1713
1782
  return arbiter.submit(outcome) if %i[deadline_exceeded attempt_timeout].include?(outcome)
1714
1783
 
1715
1784
  arbiter.submit(:lease_expired)
1716
- raise LeaseLostError.new(task.id, "complete")
1785
+ arbiter.refuse_cancellation
1786
+ raise LeaseLostError.new(task.id, operation)
1717
1787
  end
1718
1788
 
1719
1789
  # Registers the attempt with the heartbeat and starts its lease watchdog. The watchdog's
@@ -1756,17 +1826,18 @@ module Stablemates
1756
1826
  # deadline or an attempt timeout when one passes. While PostgreSQL answers not_due, the first
1757
1827
  # retry waits EXPIRATION_RETRY and each later one waits twice as long, up to
1758
1828
  # EXPIRATION_RETRY_MAX. Each request locks the task row, so a short fixed retry makes the
1759
- # heartbeat of the attempt answer busy (SM-1701).
1829
+ # heartbeat of the attempt answer busy (SM-1701). The expiry is a database timestamp, so the
1830
+ # watch converts it to a monotonic due time once. Every wait reads only the monotonic clock,
1831
+ # so a step of the wall clock does not move it (SM-1869).
1760
1832
  def watch_expiration(task, ownership, arbiter, cancellation, parent, fail_attempt)
1761
1833
  expiration_at = [task.deadline_at, task.attempt_timeout_at].compact.min
1834
+ # A past expiry is due now.
1835
+ expiration_due = monotonic + [expiration_at - Time.now, 0].max if expiration_at
1762
1836
  retry_at = nil
1763
1837
  retry_delay = EXPIRATION_RETRY
1764
1838
  loop do
1765
- delay = if retry_at
1766
- retry_at - monotonic
1767
- elsif expiration_at
1768
- expiration_at - Time.now
1769
- end
1839
+ due = retry_at || expiration_due
1840
+ delay = due && (due - monotonic)
1770
1841
  lease_delay = ownership.lease_remaining
1771
1842
  if delay.nil? || lease_delay < delay
1772
1843
  if lease_delay.positive?
@@ -1839,18 +1910,31 @@ module Stablemates
1839
1910
  STATUS_OUTCOMES.fetch(status) { raise UnexpectedStatusError.new(operation, status) }
1840
1911
  end
1841
1912
 
1842
- # Submits the outcome for a status the worker's own write returned. A cancel request is
1843
- # acknowledged; a lost lease raises LeaseLostError. A neutral status submits +neutral_outcome+.
1844
- def settle_status(task, status, arbiter, operation, neutral, neutral_outcome)
1913
+ # Submits the outcome for a status the worker's own write returned, and returns the status
1914
+ # for the handler span. A cancel request is acknowledged. A stale answer settles its cause, as
1915
+ # settle_refusal describes, and returns nil, so the span takes its outcome from that cause. A
1916
+ # lost lease submits :lease_expired and raises LeaseLostError. A neutral status submits
1917
+ # +neutral_outcome+. The reconciliation waits for a heartbeat round of +ownership+, as
1918
+ # reconcile_refusal describes.
1919
+ def settle_status(task, status, arbiter, operation, neutral, neutral_outcome, ownership)
1845
1920
  outcome = outcome_for_status(status, neutral, operation)
1846
1921
  case outcome
1847
- when nil then return arbiter.submit(neutral_outcome)
1848
- when :cancelled then raise LeaseLostError.new(task.id, operation) unless acknowledge_cancel(task)
1922
+ when nil then arbiter.submit(neutral_outcome)
1849
1923
  when :lease_expired
1850
- arbiter.submit(:lease_expired)
1851
- raise LeaseLostError.new(task.id, operation)
1924
+ # A lease that lapsed under an unchanged fence can still settle a due boundary.
1925
+ reconcile_refusal(task, ownership, arbiter, operation)
1926
+ return nil
1927
+ when :cancelled
1928
+ unless acknowledge_cancel(task)
1929
+ # The worker settled nothing, so the attempt reports the lost lease.
1930
+ arbiter.submit(:lease_expired)
1931
+ arbiter.refuse_cancellation
1932
+ raise LeaseLostError.new(task.id, operation)
1933
+ end
1934
+ arbiter.submit(outcome)
1935
+ else arbiter.submit(outcome)
1852
1936
  end
1853
- arbiter.submit(outcome)
1937
+ status
1854
1938
  end
1855
1939
 
1856
1940
  # Stops renewing and reports whether the arbiter already decided the attempt. Raises the
@@ -1900,8 +1984,9 @@ module Stablemates
1900
1984
  end
1901
1985
 
1902
1986
  # Records a handler failure. PostgreSQL retries or fails the task by its retry policy.
1903
- # Returns the state PostgreSQL chose, for the handler span.
1904
- def settle_failure(task, error, arbiter)
1987
+ # Returns the state PostgreSQL chose, for the handler span, or nil after a reconciled stale
1988
+ # answer.
1989
+ def settle_failure(task, error, arbiter, ownership)
1905
1990
  envelope = error_envelope(error, task.redact_error_details)
1906
1991
  state = Telemetry.span("workhorse.retry", Telemetry.task_span_attributes(task)) do |span|
1907
1992
  delay = durable_replay_conflict(error) ? -1 : retry_delay_override(task)
@@ -1917,10 +2002,9 @@ module Stablemates
1917
2002
  Telemetry.task_span_attributes(task).merge("workhorse.attempt.outcome" => state))
1918
2003
  if retried
1919
2004
  arbiter.submit(:retry)
1920
- else
1921
- settle_status(task, state, arbiter, "fail", %w[failed], :failed)
2005
+ return state
1922
2006
  end
1923
- state
2007
+ settle_status(task, state, arbiter, "fail", %w[failed], :failed, ownership)
1924
2008
  end
1925
2009
 
1926
2010
  # Whether +error+, or an error it wraps, is a +:shutdown+ CancelledError. With the token's own
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: stablemates-workhorse
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.10.0
4
+ version: 0.11.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Stablemates