stablemates-workhorse 0.10.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +207 -0
- data/README.md +3 -2
- data/lib/stablemates/workhorse/context.rb +6 -0
- data/lib/stablemates/workhorse/dashboard/bundle.json +1 -1
- data/lib/stablemates/workhorse/dashboard/read-surface-1.tar.gz +0 -0
- data/lib/stablemates/workhorse/dashboard.rb +1 -1
- data/lib/stablemates/workhorse/errors.rb +3 -3
- data/lib/stablemates/workhorse/queue.rb +63 -29
- data/lib/stablemates/workhorse/sql_catalogue_generated.rb +1 -1
- data/lib/stablemates/workhorse/telemetry.rb +5 -5
- data/lib/stablemates/workhorse/version.rb +1 -1
- data/lib/stablemates/workhorse/worker/heartbeat.rb +14 -78
- data/lib/stablemates/workhorse/worker.rb +180 -96
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1686623ff5e225f372245830a169d9b248db6bd195ed3f446679e01098bc9610
|
|
4
|
+
data.tar.gz: bde5f7e6517d2a7a3621b437237db776673156693b22fc1a46f42aa28305b563
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: eef44a6c8715cc1d6c0f7983fb42b3662332ad1302fd6d705e60eca9f2d4ed099d32fd53d1303a2ee1a0cdfa9ee08c3624c57516c967c22cb62f0366180bdf4b
|
|
7
|
+
data.tar.gz: e9137fa4c8b8f6a5f20f800cb4abc482e6c750721758fb5432f407c229816b4a4186700e882eeb8859a8d21f8bcfa2f2be6be1ae734ef574a003227a8db82a94
|
data/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,213 @@
|
|
|
3
3
|
`stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
|
|
4
4
|
other SDKs carry, because every tag names one release of all of them.
|
|
5
5
|
|
|
6
|
+
## 0.11.0 — 2026-10-28
|
|
7
|
+
|
|
8
|
+
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
|
|
9
|
+
packages release from one source commit.
|
|
10
|
+
|
|
11
|
+
Requires **schema v76**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
|
|
12
|
+
|
|
13
|
+
### Upgrade steps
|
|
14
|
+
|
|
15
|
+
**Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
|
|
16
|
+
process of this release starts, producers too. Migrations 0088, 0089, and 0090 move the schema
|
|
17
|
+
from version 86 to version **89**. They replace function bodies and change no stored rows. The SDK
|
|
18
|
+
compatibility floor stays at **76**. A 0.10 process keeps working on version 89, because a
|
|
19
|
+
migration inside a major line only adds.
|
|
20
|
+
|
|
21
|
+
**Give each worker a pool with a known size of at least 3.** `Worker.new` no longer accepts
|
|
22
|
+
`shared_heartbeats:`. Remove the option. If a worker pool reports a size below 3, or reports no
|
|
23
|
+
size, set its size to 3 or more. Each worker process needs 3 connections: the listener, the
|
|
24
|
+
heartbeat connection, and one statement connection. A `polling_only: true` worker opens no
|
|
25
|
+
listener, but it still needs a pool of 3 (SM-1901).
|
|
26
|
+
|
|
27
|
+
### Breaking changes
|
|
28
|
+
|
|
29
|
+
- **`shared_heartbeats:` is removed.** Every worker now reserves a dedicated heartbeat connection
|
|
30
|
+
on its pool (ADR 0071). Before, `shared_heartbeats: true` let heartbeat rounds borrow ordinary
|
|
31
|
+
pool connections and let a pool below 3 run. Then a heartbeat could wait behind claims and
|
|
32
|
+
settlements, so a lease could expire and a task could run twice. Now `Worker.new` raises
|
|
33
|
+
`ArgumentError` when it gets `shared_heartbeats:`, and the message names the removal. A worker
|
|
34
|
+
on a pool with a size below 3, or with no reported size, raises `ArgumentError` at construction.
|
|
35
|
+
The worker no longer logs `workhorse.worker.polling_only`, because its pool always has a
|
|
36
|
+
connection for the listener (SM-1901).
|
|
37
|
+
|
|
38
|
+
### Behavior changes
|
|
39
|
+
|
|
40
|
+
These changes can make code that worked on 0.10 act differently:
|
|
41
|
+
|
|
42
|
+
1. An idempotency TTL of exactly 0 still selects the default. The queue now raises
|
|
43
|
+
`ArgumentError` for a TTL that is not 0 and rounds to 0 ms (SM-1857). See "An idempotency
|
|
44
|
+
TTL that rounds to 0 ms is refused".
|
|
45
|
+
2. The TypeScript `Queue`, `Admin`, and `Worker.runOnce()` now check the schema before their first
|
|
46
|
+
statement (SM-1864). This SDK already did, so it does not change.
|
|
47
|
+
3. After a lease lapses while a handler runs, a stale failure or release now settles a due
|
|
48
|
+
deadline or attempt timeout at once. The worker reports `deadline_exceeded` or `timeout`, not
|
|
49
|
+
`lease_lost` (SM-1884). See "A stale failure or release after a lapsed lease settles its
|
|
50
|
+
cause".
|
|
51
|
+
4. Every SDK removes the heartbeat opt-out (SM-1899, SM-1901, SM-1902). In this SDK,
|
|
52
|
+
`Worker.new` raises `ArgumentError` for `shared_heartbeats:`, and for a pool with a size below
|
|
53
|
+
3 or with no reported size. See "Breaking changes".
|
|
54
|
+
|
|
55
|
+
### A step of the wall clock no longer moves the expiry request
|
|
56
|
+
|
|
57
|
+
A worker asks PostgreSQL to end an attempt when its deadline or its attempt timeout passes. Before
|
|
58
|
+
this change, the worker read the wall clock again after each wait on the lease window (SM-1869). If
|
|
59
|
+
the host clock stepped back 10 minutes, the first expiry request came 10 minutes late. If the clock
|
|
60
|
+
stepped forward, the request came early and got `not_due`. Each retry then locked the task row until
|
|
61
|
+
the real due time.
|
|
62
|
+
|
|
63
|
+
Now the worker converts the expiry to a monotonic due time once, when its expiry
|
|
64
|
+
watch starts. After that, it reads only the monotonic clock.
|
|
65
|
+
|
|
66
|
+
### Row retention lag skips a member of an unfinished task group
|
|
67
|
+
|
|
68
|
+
Terminal cleanup keeps each member of a task group that has not finished, because the group needs
|
|
69
|
+
its member outcomes when it finishes. Before this change, queue health counted such a member as row
|
|
70
|
+
retention lag after it passed its windows (SM-1851). An open or sealed group can stay unfinished for
|
|
71
|
+
any length of time. Thus health reported "Retention cleanup is late for task records, finished
|
|
72
|
+
results", and the lag grew without end. The public demo keeps one open and one sealed group on
|
|
73
|
+
purpose, so its Health page showed the report every day.
|
|
74
|
+
|
|
75
|
+
Migration 0088 (`0088-count-only-prunable-rows-as-row-retention-lag.sql`) moves the schema to
|
|
76
|
+
version 87. Now `queue_health_v1` skips a member of an unfinished group when it measures task
|
|
77
|
+
identity and terminal outcome lag. For a group member, lag counts from the latest of window
|
|
78
|
+
expiry, history release, and group finish. The health document keeps its keys, and the schema floor
|
|
79
|
+
does not change.
|
|
80
|
+
Every process of every SDK version gets the new result when the migration commits.
|
|
81
|
+
|
|
82
|
+
### A stale failure or release after a lapsed lease settles its cause
|
|
83
|
+
|
|
84
|
+
A lease can lapse while a handler runs, with no change to the fence. Then `fail_v1` and
|
|
85
|
+
`release_owned_v1` answer `stale`, but `expire_owned_v1` can still settle a due deadline or attempt
|
|
86
|
+
timeout under that fence. Before this change, the worker settled nothing, and lease recovery
|
|
87
|
+
settled the task later (SM-1884).
|
|
88
|
+
|
|
89
|
+
Now the worker calls `expire_owned_v1` at once and settles the due boundary, as the Go and Rust
|
|
90
|
+
workers do:
|
|
91
|
+
|
|
92
|
+
- After a lapsed lease with a due deadline, the handler outcome was `lease_lost`. Now it is
|
|
93
|
+
`deadline_exceeded`, and the task fails at once.
|
|
94
|
+
- After a lapsed lease with a due attempt timeout, the handler outcome was `lease_lost`. Now it is
|
|
95
|
+
`timeout`, and PostgreSQL applies the retry policy at once.
|
|
96
|
+
- After a lapsed lease with no due boundary, the outcome stays `lease_lost`.
|
|
97
|
+
|
|
98
|
+
The handler span records `deadline_exceeded` or `timeout_exceeded`. Before this change, it
|
|
99
|
+
recorded `stale`.
|
|
100
|
+
|
|
101
|
+
If `fail_v1` or `release_owned_v1` answers `cancel_requested` and `acknowledge_cancel_v1` refuses,
|
|
102
|
+
the worker settles nothing. After a failed cancellation acknowledgement, the handler outcome was
|
|
103
|
+
`unknown`. Now it is `lease_lost`. The handler span records `stale`. Before this change, it recorded
|
|
104
|
+
`unknown`. A late heartbeat round can also deliver the cancellation request, after a failure, a
|
|
105
|
+
release, or a refused completion. If the acknowledgement then refuses, the handler outcome was
|
|
106
|
+
`canceled`. Now it is `lease_lost`.
|
|
107
|
+
|
|
108
|
+
### The dashboard bundle uses Mantine 9.7.4
|
|
109
|
+
|
|
110
|
+
The Rack dashboard backend serves the shared dashboard browser bundle. After 0.10.0, a dependency
|
|
111
|
+
update moved the dashboard to Mantine 9.7.4 and rebuilt that bundle (SM-1839). The gem now carries
|
|
112
|
+
the rebuilt `read-surface-1.tar.gz`. Its `bundle.json` records the sha256
|
|
113
|
+
`a7d61ed1d0433e38a8d227cbcc8f98a0bf668f2d3bb1d47391b1882fd69bf795`.
|
|
114
|
+
|
|
115
|
+
The `dashboard/v1` procedures and the read surface version do not change (SM-1858).
|
|
116
|
+
|
|
117
|
+
### A worker no longer keeps the tier entries of a task that the executor refuses
|
|
118
|
+
|
|
119
|
+
The worker records each claimed task in its dispatch state. A fast-tier claim records the task ID
|
|
120
|
+
in `@fast_task_ids`. A full-tier claim records the task ID and a deferred tier read in
|
|
121
|
+
`@tier_reads`. Only the end of a handler run removed these entries.
|
|
122
|
+
|
|
123
|
+
Before this change, the handler executor could refuse a task, for example after its shutdown. The
|
|
124
|
+
task then never ran, and the worker kept its tier entry for the life of the `Worker` object
|
|
125
|
+
(SM-1853). Each refusal kept the entry of at most one task.
|
|
126
|
+
|
|
127
|
+
Now the worker removes the task from both collections when the executor refuses the task.
|
|
128
|
+
|
|
129
|
+
### The dashboard page tells each cache not to store it
|
|
130
|
+
|
|
131
|
+
The dashboard page holds `window.workhorseDashboard`, which contains the `auditActor` of the
|
|
132
|
+
operator. Before this change, the Rack dashboard backend sent the page with only `Content-Type`. A
|
|
133
|
+
shared cache in front of the application could store the page by heuristic. A later visitor could
|
|
134
|
+
then get the stored page and see the actor name of an earlier operator. A browser could also show
|
|
135
|
+
the page from its cache after logout (SM-1870).
|
|
136
|
+
|
|
137
|
+
Now the Rack dashboard backend sends the page with `Cache-Control: no-store`, as the TypeScript host
|
|
138
|
+
does. Each asset keeps `Cache-Control: public, max-age=31536000, immutable`.
|
|
139
|
+
|
|
140
|
+
### An idempotency TTL that rounds to 0 ms is refused
|
|
141
|
+
|
|
142
|
+
The queue rounds `Idempotency#ttl` to the nearest millisecond. Before this change, a positive TTL
|
|
143
|
+
below 0.0005 s, or a small negative TTL such as -0.0004 s, rounded to 0. The queue then kept the key
|
|
144
|
+
for the default 24 hours without a warning (SM-1857).
|
|
145
|
+
|
|
146
|
+
Now the queue raises `ArgumentError` for a TTL that is not 0 and rounds to 0 ms. An exact 0 still
|
|
147
|
+
selects the default. ADR 0115 records the rule for all seven SDKs.
|
|
148
|
+
|
|
149
|
+
### A stalled registration refresh no longer delays a stop
|
|
150
|
+
|
|
151
|
+
A worker refreshes its `worker_registry` row at each registry interval. Before this change, the
|
|
152
|
+
dispatch loop sent that refresh itself, with no time limit. If the statement stalled, for example on
|
|
153
|
+
a lock, `stop` could not start the stop path until the statement returned. Also, the draining
|
|
154
|
+
registration and the deregistration ran on two threads in no order. If the registration committed
|
|
155
|
+
last, the row of the stopped worker came back until the registry prune removed it (SM-1868).
|
|
156
|
+
|
|
157
|
+
Now each registry write runs on its own thread, after the write before it. The dispatch loop starts
|
|
158
|
+
a periodic refresh only when no registry write still runs, and it does not wait for the refresh.
|
|
159
|
+
Thus `run` returns within `shutdown_grace` plus the unwind window, and the deregistration runs
|
|
160
|
+
after every registration write of the run.
|
|
161
|
+
|
|
162
|
+
### Recovery no longer locks a timed-out attempt that waits for its cancellation
|
|
163
|
+
|
|
164
|
+
An operator cancels a running task after its execution timeout passed, and its lease is still live.
|
|
165
|
+
Before migration 0089, each maintenance tick locked that attempt in its timeout scan, but settled
|
|
166
|
+
nothing, because the cancellation wins over the timeout. While the tick ran, the heartbeat of the
|
|
167
|
+
owning worker reported the lease busy. The locked rows also used the recovery limit, so a real
|
|
168
|
+
timeout behind them waited for their leases to expire (SM-1866).
|
|
169
|
+
|
|
170
|
+
Now the full-tier timeout scan skips such an attempt while its lease is live and its deadline is
|
|
171
|
+
ahead, as the fast tier does. If the deadline passed after the timeout, recovery closes the task
|
|
172
|
+
`canceled` at once.
|
|
173
|
+
|
|
174
|
+
### Fast-tier recovery closes a canceled attempt when its deadline passes
|
|
175
|
+
|
|
176
|
+
A fast-tier task has a pending cancellation and a live lease. Its execution timeout passes, and
|
|
177
|
+
later its deadline passes. Before migration 0090, each maintenance tick locked that attempt and
|
|
178
|
+
settled nothing until the lease expired. The cancellation blocks the timeout, and recovery took the
|
|
179
|
+
deadline only when the deadline was not later than the timeout (SM-1881).
|
|
180
|
+
|
|
181
|
+
Now fast-tier recovery closes the task `canceled` when its deadline passes, as the full tier does.
|
|
182
|
+
|
|
183
|
+
### The dashboard Schedules page reads a bounded number of occurrences
|
|
184
|
+
|
|
185
|
+
Before migration 0089, `dashboard_cron_v1` counted every retained occurrence of each listed
|
|
186
|
+
schedule. A schedule that fires each second keeps about 1.2 million occurrences in the default
|
|
187
|
+
retention of 14 days. Each poll of the Schedules page read all of them (SM-1866).
|
|
188
|
+
|
|
189
|
+
Now the function reads at most 1,001 occurrences for each listed schedule. `occurrenceCount` stops
|
|
190
|
+
at 1,000, and the new optional key `occurrenceCountCapped` is `true` when more occurrences are
|
|
191
|
+
retained. The bundled Schedules page then shows `1,000+`. `lastFiredAt` is the fire time of the
|
|
192
|
+
newest occurrence. The `dashboard/v1` change is additive. A database below schema 88 returns an
|
|
193
|
+
exact count and no `occurrenceCountCapped`. The page change rebuilds the shared dashboard bundle
|
|
194
|
+
again. The sha256 in "The dashboard bundle uses Mantine 9.7.4" is the sha256 of that bundle.
|
|
195
|
+
|
|
196
|
+
### The worker registry prune finds expired kept pauses through their index
|
|
197
|
+
|
|
198
|
+
Before migration 0089, `prune_worker_registry_v1` deleted expired kept pauses with a condition that
|
|
199
|
+
the planner could not match to `worker_pause_tombstone_expires_idx`. Each maintenance pass read the
|
|
200
|
+
whole table. Now the delete reads only the expired rows through the index (SM-1866).
|
|
201
|
+
|
|
202
|
+
### A child set and a task group check every request before their first statement
|
|
203
|
+
|
|
204
|
+
Before this change, `HandlerContext#run_children` and `#run_children_all` checked the options of
|
|
205
|
+
each child while they serialized it. Thus the first child could read its payload contract before
|
|
206
|
+
the SDK refused the second child. `Queue#create_task_group` and `Queue#task_group` checked the
|
|
207
|
+
callbacks after the compatibility query and the member contract reads. A member add through
|
|
208
|
+
`HandlerContext#task_group` read the membership of its task first (SM-1883).
|
|
209
|
+
|
|
210
|
+
Now these calls check the options of every request before their first statement, also before the
|
|
211
|
+
deferred tier read of a child set. The errors do not change. Only their time changes.
|
|
212
|
+
|
|
6
213
|
## 0.10.0 — 2026-10-26
|
|
7
214
|
|
|
8
215
|
The npm packages, Python distribution, Go module, Rust crate, Ruby gem, JVM artifacts, and .NET
|
data/README.md
CHANGED
|
@@ -30,7 +30,7 @@ bundle add connection_pool
|
|
|
30
30
|
Install the schema once, as a deployment step. The application never installs or migrates it.
|
|
31
31
|
|
|
32
32
|
```bash
|
|
33
|
-
npx --package @stablemates/workhorse@0.
|
|
33
|
+
npx --package @stablemates/workhorse@0.11.0 workhorse schema install
|
|
34
34
|
```
|
|
35
35
|
|
|
36
36
|
The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
|
|
@@ -108,7 +108,8 @@ The gem is `stablemates-workhorse`, and its namespace is `Stablemates::Workhorse
|
|
|
108
108
|
that was `:non_replaceable` also carries a `reason`: `:incompatible_key_mode`, `:not_pending`, or
|
|
109
109
|
`:window_elapsed_pending`.
|
|
110
110
|
- `Worker` takes a pool, registers handlers by task type, and runs them under a lease with a shared
|
|
111
|
-
heartbeat connection.
|
|
111
|
+
heartbeat connection. The pool must report a maximum size of at least 3 connections, or the
|
|
112
|
+
constructor raises `ArgumentError`. `run_worker_process` drains the worker on `TERM` or `INT`.
|
|
112
113
|
- `HandlerContext` offers checkpoints, durable sleeps, signal and human waits, child tasks, and
|
|
113
114
|
progress. PostgreSQL owns every durable decision, and an unresolved wait suspends the task.
|
|
114
115
|
- `Admin` lists, inspects, and repairs tasks, dead letters, waits, workers, and queues.
|
|
@@ -63,6 +63,10 @@ module Stablemates
|
|
|
63
63
|
|
|
64
64
|
def submit(outcome) = @outcome.compare_and_set(nil, outcome)
|
|
65
65
|
|
|
66
|
+
# Replaces a cancellation that PostgreSQL refused to acknowledge with a lost lease. A
|
|
67
|
+
# heartbeat round can submit the cancellation request first, but the worker settled nothing.
|
|
68
|
+
def refuse_cancellation = @outcome.compare_and_set(:cancelled, :lease_expired)
|
|
69
|
+
|
|
66
70
|
# Records that a fenced durable write released the task for the suspension +outcome+. The
|
|
67
71
|
# attempt then suspends whichever outcome won, because the heartbeat reports that release as a
|
|
68
72
|
# lost lease.
|
|
@@ -353,6 +357,8 @@ module Stablemates
|
|
|
353
357
|
children.each { |child| child_name(child.name) }
|
|
354
358
|
raise ArgumentError, "child names must be unique" unless children.map(&:name).uniq.length == children.length
|
|
355
359
|
|
|
360
|
+
# Every child is checked before the deferred tier read and the first contract read.
|
|
361
|
+
children.each { |child| @queue.check_child_request(@task, child.task_type, child.payload, child.options) }
|
|
356
362
|
durable!("child tasks")
|
|
357
363
|
|
|
358
364
|
build = lambda do |versions = {}|
|
|
Binary file
|
|
@@ -322,7 +322,7 @@ module Stablemates
|
|
|
322
322
|
"window.workhorseDashboard = #{JSON.generate(config).gsub(/[<>&
]/, SCRIPT_ESCAPES)}"
|
|
323
323
|
}
|
|
324
324
|
.sub("<!--__WORKHORSE_BROWSER_MODULES__-->") { modules }
|
|
325
|
-
[200, {"content-type" => "text/html; charset=utf-8"}, [html]]
|
|
325
|
+
[200, {"content-type" => "text/html; charset=utf-8", "cache-control" => "no-store"}, [html]]
|
|
326
326
|
end
|
|
327
327
|
|
|
328
328
|
def asset(name)
|
|
@@ -298,9 +298,9 @@ module Stablemates
|
|
|
298
298
|
end
|
|
299
299
|
end
|
|
300
300
|
|
|
301
|
-
# A worker
|
|
302
|
-
#
|
|
303
|
-
#
|
|
301
|
+
# A worker could not reserve its dedicated heartbeat connection, or could not borrow a
|
|
302
|
+
# connection for its readiness check, within its startup budget. The message names the pool and
|
|
303
|
+
# the budget. A pool error is the +cause+.
|
|
304
304
|
class WorkerStartupError < Error; end
|
|
305
305
|
|
|
306
306
|
# Any other PG::Error. The original is the +cause+, and +sqlstate+ is its SQLSTATE or nil.
|
|
@@ -120,6 +120,8 @@ module Stablemates
|
|
|
120
120
|
# SDK.
|
|
121
121
|
def add_to_task_group_as_member(task, worker_id, requests) # :nodoc:
|
|
122
122
|
TaskGroups.validate_add(requests)
|
|
123
|
+
# Every request is checked before the membership read.
|
|
124
|
+
request_inputs(requests)
|
|
123
125
|
link = find_task_group(task.id)
|
|
124
126
|
raise ArgumentError, "Task #{task.id} is not a member of a task group" unless link&.role == :member
|
|
125
127
|
|
|
@@ -338,6 +340,32 @@ module Stablemates
|
|
|
338
340
|
# queue of +parent+. Internal to the SDK.
|
|
339
341
|
def serialize_child_request(parent, task_type, payload, options, contracts = {}, version: :current, # :nodoc:
|
|
340
342
|
stored_queue: nil)
|
|
343
|
+
request, input = child_input(parent, task_type, payload, options, stored_queue)
|
|
344
|
+
key = (version == :current) ? request.task_type : [request.task_type, version]
|
|
345
|
+
contract = contracts.fetch(key) do
|
|
346
|
+
contracts[key] = if version != :current
|
|
347
|
+
version && Contracts.load(@executor, request.task_type, version)
|
|
348
|
+
else
|
|
349
|
+
Contracts.load(@executor, request.task_type).tap do |loaded|
|
|
350
|
+
@lock.synchronize { @contracts[request.task_type] = loaded }
|
|
351
|
+
end
|
|
352
|
+
end
|
|
353
|
+
end
|
|
354
|
+
stamp_contract(input, request, contract)
|
|
355
|
+
input
|
|
356
|
+
end
|
|
357
|
+
|
|
358
|
+
# Checks a child request with no round trip. A child set checks every child with this before
|
|
359
|
+
# its first contract read. Internal to the SDK.
|
|
360
|
+
def check_child_request(parent, task_type, payload, options) # :nodoc:
|
|
361
|
+
child_input(parent, task_type, payload, options, nil)
|
|
362
|
+
nil
|
|
363
|
+
end
|
|
364
|
+
|
|
365
|
+
private
|
|
366
|
+
|
|
367
|
+
# The request and the input of a child, without its contract stamp.
|
|
368
|
+
def child_input(parent, task_type, payload, options, stored_queue)
|
|
341
369
|
request = EnqueueRequest.new(task_type: task_type, payload: payload, **options)
|
|
342
370
|
if request.idempotency || request.debounce || request.throttle || request.dependencies
|
|
343
371
|
raise ArgumentError, "Child tasks cannot use coalescing or dependency enqueue options"
|
|
@@ -354,22 +382,9 @@ module Stablemates
|
|
|
354
382
|
input["tags"] = tags(request.tags)
|
|
355
383
|
input["traceContext"] = parent.trace_context unless parent.trace_context.nil?
|
|
356
384
|
input["runAt"] = Values.timestamp(request.run_at, "run at") if request.run_at
|
|
357
|
-
|
|
358
|
-
contract = contracts.fetch(key) do
|
|
359
|
-
contracts[key] = if version != :current
|
|
360
|
-
version && Contracts.load(@executor, request.task_type, version)
|
|
361
|
-
else
|
|
362
|
-
Contracts.load(@executor, request.task_type).tap do |loaded|
|
|
363
|
-
@lock.synchronize { @contracts[request.task_type] = loaded }
|
|
364
|
-
end
|
|
365
|
-
end
|
|
366
|
-
end
|
|
367
|
-
stamp_contract(input, request, contract)
|
|
368
|
-
input
|
|
385
|
+
[request, input]
|
|
369
386
|
end
|
|
370
387
|
|
|
371
|
-
private
|
|
372
|
-
|
|
373
388
|
# Serializes +requests+, passes their JSON document to the block, and returns the rows the block
|
|
374
389
|
# returns. On a contract mismatch, PostgreSQL enqueued nothing: the client reloads the named
|
|
375
390
|
# contracts and calls the block one more time.
|
|
@@ -388,20 +403,25 @@ module Stablemates
|
|
|
388
403
|
end
|
|
389
404
|
|
|
390
405
|
def serialize_requests(requests)
|
|
391
|
-
now = Time.now
|
|
392
406
|
trace_context = Telemetry.inject_trace_context
|
|
393
|
-
inputs = requests
|
|
394
|
-
|
|
395
|
-
input["traceContext"] = trace_context if trace_context
|
|
396
|
-
input
|
|
397
|
-
rescue ArgumentError => e
|
|
398
|
-
raise ArgumentError, "enqueue request #{index + 1}: #{e.message}"
|
|
399
|
-
end
|
|
407
|
+
inputs = request_inputs(requests)
|
|
408
|
+
inputs.each { |input| input["traceContext"] = trace_context } if trace_context
|
|
400
409
|
assert_compatible
|
|
401
410
|
inputs.zip(requests) { |input, request| apply_contract(input, request) }
|
|
402
411
|
JSON.generate(inputs)
|
|
403
412
|
end
|
|
404
413
|
|
|
414
|
+
# The inputs of +requests+, without a round trip, so a batch refuses an invalid option of any
|
|
415
|
+
# request before its first statement.
|
|
416
|
+
def request_inputs(requests)
|
|
417
|
+
now = Time.now
|
|
418
|
+
requests.each_with_index.map do |request, index|
|
|
419
|
+
serialize_request(request, now)
|
|
420
|
+
rescue ArgumentError => e
|
|
421
|
+
raise ArgumentError, "enqueue request #{index + 1}: #{e.message}"
|
|
422
|
+
end
|
|
423
|
+
end
|
|
424
|
+
|
|
405
425
|
# Creates a group with +requests+ as its member tasks, and seals it when +seal+ is true. A
|
|
406
426
|
# contract mismatch of a member or a callback creates nothing, and the retry serializes the
|
|
407
427
|
# callbacks again under the contracts the mismatch reloaded.
|
|
@@ -413,6 +433,11 @@ module Stablemates
|
|
|
413
433
|
raise ArgumentError, "task group callbacks cannot use enqueue idempotency, debounce, or throttle"
|
|
414
434
|
end
|
|
415
435
|
end
|
|
436
|
+
# The callbacks are serialized after the member contracts are read, so their options are
|
|
437
|
+
# checked here, before the first statement.
|
|
438
|
+
now = Time.now
|
|
439
|
+
callbacks.each { |kind, request| callback_input(kind, request, now) unless request.nil? }
|
|
440
|
+
request_inputs(requests)
|
|
416
441
|
group_id = nil
|
|
417
442
|
rows = submit_requests(requests) do |document|
|
|
418
443
|
created = @executor.rows(SqlCatalogue::CREATE_TASK_GROUP_V1,
|
|
@@ -435,11 +460,7 @@ module Stablemates
|
|
|
435
460
|
document = callbacks.each_with_object({}) do |(kind, request), entries|
|
|
436
461
|
next if request.nil?
|
|
437
462
|
|
|
438
|
-
input =
|
|
439
|
-
serialize_request(request, now)
|
|
440
|
-
rescue ArgumentError => e
|
|
441
|
-
raise ArgumentError, "#{kind} callback: #{e.message}"
|
|
442
|
-
end
|
|
463
|
+
input = callback_input(kind, request, now)
|
|
443
464
|
input.delete("runAt") if request.run_at.nil?
|
|
444
465
|
input["traceContext"] = trace_context if trace_context
|
|
445
466
|
apply_contract(input, request)
|
|
@@ -448,6 +469,12 @@ module Stablemates
|
|
|
448
469
|
JSON.generate(document)
|
|
449
470
|
end
|
|
450
471
|
|
|
472
|
+
def callback_input(kind, request, now)
|
|
473
|
+
serialize_request(request, now)
|
|
474
|
+
rescue ArgumentError => e
|
|
475
|
+
raise ArgumentError, "#{kind} callback: #{e.message}"
|
|
476
|
+
end
|
|
477
|
+
|
|
451
478
|
# +member+ is the task ID, worker ID, and fence token of a running member task, or three nils.
|
|
452
479
|
def add_task_group_members(group_id, requests, member)
|
|
453
480
|
TaskGroups.validate_add(requests)
|
|
@@ -545,14 +572,21 @@ module Stablemates
|
|
|
545
572
|
def idempotency_document(idempotency)
|
|
546
573
|
raise ArgumentError, "idempotency must be an Idempotency" unless idempotency.is_a?(Idempotency)
|
|
547
574
|
|
|
548
|
-
ttl = Values.milliseconds(idempotency.ttl, "idempotency TTL", 0..MAX_DURATION_MS)
|
|
549
575
|
{
|
|
550
576
|
"key" => required_bytes(idempotency.key, MAX_KEY_BYTES, "idempotency key"),
|
|
551
577
|
"scope" => scope(idempotency.scope),
|
|
552
|
-
"ttlMs" =>
|
|
578
|
+
"ttlMs" => idempotency_ttl(idempotency.ttl)
|
|
553
579
|
}
|
|
554
580
|
end
|
|
555
581
|
|
|
582
|
+
# The protocol value of an idempotency TTL. An exact zero takes the default. A nonzero value
|
|
583
|
+
# that rounds to 0 ms is refused, so a unit mistake cannot select the default.
|
|
584
|
+
def idempotency_ttl(value)
|
|
585
|
+
return DEFAULT_IDEMPOTENCY_TTL_MS if value.is_a?(Numeric) && value.zero?
|
|
586
|
+
|
|
587
|
+
Values.milliseconds(value, "idempotency TTL", 1..MAX_DURATION_MS)
|
|
588
|
+
end
|
|
589
|
+
|
|
556
590
|
def debounce_document(debounce)
|
|
557
591
|
raise ArgumentError, "debounce must be a Debounce" unless debounce.is_a?(Debounce)
|
|
558
592
|
|
|
@@ -11,7 +11,7 @@ module Stablemates
|
|
|
11
11
|
MINIMUM_PROTOCOL_VERSION = 5
|
|
12
12
|
MAXIMUM_PROTOCOL_VERSION = 5
|
|
13
13
|
MINIMUM_SCHEMA_VERSION = 76
|
|
14
|
-
MAXIMUM_SCHEMA_VERSION =
|
|
14
|
+
MAXIMUM_SCHEMA_VERSION = 89
|
|
15
15
|
# PostgreSQL's atomic enqueue batch limit.
|
|
16
16
|
MAX_ENQUEUE_BATCH_SIZE = 1000
|
|
17
17
|
DEFAULT_TASK_VALUE_MAX_BYTES = 1048576
|
|
@@ -163,11 +163,11 @@ module Stablemates
|
|
|
163
163
|
# errors of a socket), or SocketError (DNS); or a SQLSTATE that starts with 08, or 57P01,
|
|
164
164
|
# 57P02, or 57P03 (a server shutdown). IO::TimeoutError is an IOError, so a socket timeout
|
|
165
165
|
# is a connection error too.
|
|
166
|
-
# - "connection" also covers a pool error that CONNECTION_ERROR_NAMES names
|
|
167
|
-
#
|
|
168
|
-
#
|
|
169
|
-
#
|
|
170
|
-
#
|
|
166
|
+
# - "connection" also covers a pool error that CONNECTION_ERROR_NAMES names: a wait that ends at
|
|
167
|
+
# the pool's own limit, ConnectionPool::TimeoutError, Sequel::PoolTimeout, or
|
|
168
|
+
# ActiveRecord::ConnectionTimeoutError (an ActiveRecord::ConnectionNotEstablished). A round
|
|
169
|
+
# runs on the dedicated heartbeat connection and borrows none from the pool. The classifier
|
|
170
|
+
# keeps these names, so a pool error in a round still counts as "connection".
|
|
171
171
|
# - "database": any other SQLSTATE.
|
|
172
172
|
# - "other": everything else. A Timeout::Error that is not one of these, for example from
|
|
173
173
|
# Timeout.timeout, is "other": only the round deadline is "timeout".
|
|
@@ -11,9 +11,7 @@ module Stablemates
|
|
|
11
11
|
# never delay a lease round. A checkout thread borrows that connection (+Acquisition+), and the
|
|
12
12
|
# heartbeat thread waits for it at most one round bound. The heartbeat replaces a lost
|
|
13
13
|
# connection at once, with a backoff capped at the interval. A connection that arrives late
|
|
14
|
-
# has one owner, which hands it over or returns it to the pool.
|
|
15
|
-
# +shared_heartbeats: true+ a worker has its own heartbeat, which borrows a connection from
|
|
16
|
-
# the executor for each round instead.
|
|
14
|
+
# has one owner, which hands it over or returns it to the pool.
|
|
17
15
|
class Heartbeat # :nodoc:
|
|
18
16
|
# One running attempt. +renew+ takes the monotonic send time of an accepted round;
|
|
19
17
|
# +settle+ takes a status other than +accepted+ and +busy+; +fail+ takes an error +settle+
|
|
@@ -175,14 +173,11 @@ module Stablemates
|
|
|
175
173
|
# statement, plus a margin for the payload, the wait for the connection, and BEGIN.
|
|
176
174
|
BUSY_RETRY_MARGIN = 0.012
|
|
177
175
|
BUSY_RETRY_MINIMUM = MINIMUM_REMAINING + BUSY_RETRY_MARGIN
|
|
178
|
-
# The wait after a shared round raised an unexpected error, in seconds, so that an error that
|
|
179
|
-
# repeats cannot keep the thread busy.
|
|
180
|
-
FAILED_ROUND_DELAY = 0.1
|
|
181
176
|
# heartbeat_many_v2 raises an error for more leases than this in one call, so a batch of
|
|
182
177
|
# heartbeat_pool_v1 holds at most this many leases too.
|
|
183
178
|
STATEMENT_LEASE_LIMIT = 100
|
|
184
179
|
private_constant :STRINGS, :ROUND_TIMEOUTS, :MINIMUM_REMAINING, :REACQUIRE_BASE, :PROBE_INTERVAL, :RELEASE_WAIT,
|
|
185
|
-
:BUSY_RETRY_BASE, :BUSY_RETRY_MARGIN, :BUSY_RETRY_MINIMUM, :
|
|
180
|
+
:BUSY_RETRY_BASE, :BUSY_RETRY_MARGIN, :BUSY_RETRY_MINIMUM, :STATEMENT_LEASE_LIMIT
|
|
186
181
|
|
|
187
182
|
# The deadline of one round, in seconds, for an +interval+ in seconds. It is three quarters
|
|
188
183
|
# of the interval, so that one timed-out round leaves a renewal margin when 2 × interval +
|
|
@@ -213,12 +208,11 @@ module Stablemates
|
|
|
213
208
|
|
|
214
209
|
# The heartbeat every worker on +pool+ shares.
|
|
215
210
|
def self.dedicated(pool)
|
|
216
|
-
@dedicated_lock.synchronize { @dedicated[pool] ||= new(pool
|
|
211
|
+
@dedicated_lock.synchronize { @dedicated[pool] ||= new(pool) }
|
|
217
212
|
end
|
|
218
213
|
|
|
219
|
-
def initialize(source
|
|
214
|
+
def initialize(source)
|
|
220
215
|
@source = source
|
|
221
|
-
@dedicated = dedicated
|
|
222
216
|
@lock = Mutex.new
|
|
223
217
|
@members = {}
|
|
224
218
|
@users = {}.compare_by_identity
|
|
@@ -344,7 +338,7 @@ module Stablemates
|
|
|
344
338
|
# failed round does not start a new schedule.
|
|
345
339
|
@previous = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
346
340
|
@busy = nil
|
|
347
|
-
|
|
341
|
+
run_dedicated
|
|
348
342
|
ensure
|
|
349
343
|
lost
|
|
350
344
|
end
|
|
@@ -369,21 +363,6 @@ module Stablemates
|
|
|
369
363
|
probe
|
|
370
364
|
end
|
|
371
365
|
|
|
372
|
-
# Runs the rounds of a borrowing heartbeat. An unexpected error does not end the thread while
|
|
373
|
-
# a member remains: the thread ends only through +stop_if_idle+, so a register that races
|
|
374
|
-
# the end starts a new thread.
|
|
375
|
-
def run_shared
|
|
376
|
-
loop do
|
|
377
|
-
return if rounds { |sql, params, _timeout| Executor.for(@source).rows(sql, params) }
|
|
378
|
-
rescue
|
|
379
|
-
@busy = nil
|
|
380
|
-
return if stop_if_idle
|
|
381
|
-
|
|
382
|
-
@wake.wait(FAILED_ROUND_DELAY)
|
|
383
|
-
@wake.reset
|
|
384
|
-
end
|
|
385
|
-
end
|
|
386
|
-
|
|
387
366
|
# Holds one connection across rounds, also while no member remains and a worker still
|
|
388
367
|
# runs. Each attempt waits at most one round bound for its checkout (+acquire+). A round
|
|
389
368
|
# that fails or overruns discards the connection, and so does a probe that finds it unusable
|
|
@@ -597,7 +576,7 @@ module Stablemates
|
|
|
597
576
|
# member interval after the previous regular round started; a wake recomputes that from the
|
|
598
577
|
# current members. A round that ends after its next slot starts one overdue round at once,
|
|
599
578
|
# and the interval after it counts from that start, so missed slots are never replayed. A
|
|
600
|
-
# failed round raises
|
|
579
|
+
# failed round raises, so the caller can replace the connection.
|
|
601
580
|
#
|
|
602
581
|
# Between two regular rounds, a busy retry (+take_busy_retry+) sends only the members that
|
|
603
582
|
# the latest round reported busy. It runs on the same thread, so it is a round for the
|
|
@@ -642,7 +621,7 @@ module Stablemates
|
|
|
642
621
|
deadline = retry_deadline(selection, interval, regular_at, started)
|
|
643
622
|
# A bounded retry with too little time left would fail before its heartbeat statement.
|
|
644
623
|
# It sends nothing, and the next regular round renews every lease, the busy ones too.
|
|
645
|
-
next if
|
|
624
|
+
next if deadline - started < BUSY_RETRY_MINIMUM
|
|
646
625
|
|
|
647
626
|
send_round(selection, deadline, interval, &)
|
|
648
627
|
next
|
|
@@ -665,9 +644,7 @@ module Stablemates
|
|
|
665
644
|
#
|
|
666
645
|
# On the dedicated connection, one heartbeat_pool_v1 statement in one transaction renews
|
|
667
646
|
# every batch of every worker, by the one +deadline+ (+send_pooled+). The round is
|
|
668
|
-
# all-or-nothing: if the statement or COMMIT fails, no lease of the round counts.
|
|
669
|
-
# borrowing heartbeat has no deadline, and sends one heartbeat_many_v2 statement for each
|
|
670
|
-
# batch (+send_each+).
|
|
647
|
+
# all-or-nothing: if the statement or COMMIT fails, no lease of the round counts.
|
|
671
648
|
#
|
|
672
649
|
# A round replaces the pending busy retry. If one or more members are busy, it schedules one
|
|
673
650
|
# retry for them. A round that raises or expires schedules none, and the next regular round
|
|
@@ -675,7 +652,7 @@ module Stablemates
|
|
|
675
652
|
def send_round(members, deadline, interval, &)
|
|
676
653
|
@busy = nil
|
|
677
654
|
batches = batches(members)
|
|
678
|
-
busy =
|
|
655
|
+
busy = send_pooled(batches, deadline, &)
|
|
679
656
|
return if busy.empty?
|
|
680
657
|
|
|
681
658
|
@busy = busy
|
|
@@ -720,29 +697,7 @@ module Stablemates
|
|
|
720
697
|
batches.each_with_index.flat_map { |(_worker_id, group), index| settle(group, statuses[index], sent_at) }
|
|
721
698
|
end
|
|
722
699
|
|
|
723
|
-
#
|
|
724
|
-
# busy members. A batch whose statement fails gets nothing, and the other batches still go.
|
|
725
|
-
# A round with one or more failed statements counts once in
|
|
726
|
-
# +workhorse.worker.heartbeat.round_failure+, with the reason of its first failure. The count
|
|
727
|
-
# occurs when that failure occurs, so a later batch that stalls does not delay it.
|
|
728
|
-
def send_each(batches, deadline)
|
|
729
|
-
failed = false
|
|
730
|
-
batches.flat_map do |worker_id, group|
|
|
731
|
-
sent_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
732
|
-
params = [worker_id, JSON.generate(leases(group))]
|
|
733
|
-
rows = begin
|
|
734
|
-
yield SqlCatalogue::HEARTBEAT_MANY_V2, params, deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
735
|
-
rescue => e
|
|
736
|
-
# A skipped round sends no log record: the lease watchdog handles rounds that keep failing.
|
|
737
|
-
Telemetry.record_heartbeat_round_failure(e) unless failed
|
|
738
|
-
failed = true
|
|
739
|
-
next []
|
|
740
|
-
end
|
|
741
|
-
settle(group, rows.to_h { |row| [row["task_id"], row["status"]] }, sent_at)
|
|
742
|
-
end
|
|
743
|
-
end
|
|
744
|
-
|
|
745
|
-
# The lease objects of heartbeat_many_v2 for the members of one batch.
|
|
700
|
+
# The lease objects of one batch of heartbeat_pool_v1.
|
|
746
701
|
def leases(group)
|
|
747
702
|
group.map do |member|
|
|
748
703
|
{"taskId" => member.task.id, "fenceToken" => member.task.fence_token.to_s, "leaseMs" => member.lease_ms}
|
|
@@ -768,8 +723,7 @@ module Stablemates
|
|
|
768
723
|
|
|
769
724
|
# A busy retry ends by its own round bound, by the next regular slot, and by the end of the
|
|
770
725
|
# earliest local lease window in it, so it never renews a lease that the worker has already
|
|
771
|
-
# given up.
|
|
772
|
-
# the monotonic start of the retry.
|
|
726
|
+
# given up. +started+ is the monotonic start of the retry.
|
|
773
727
|
def retry_deadline(selection, interval, regular_at, started)
|
|
774
728
|
ends = selection.map { |member| member.lease_ends_at.call || started }
|
|
775
729
|
[started + round_bound(interval), regular_at, *ends].min
|
|
@@ -826,43 +780,25 @@ module Stablemates
|
|
|
826
780
|
@lock.synchronize { @members[[member.worker_id, member.task.id]].equal?(member) }
|
|
827
781
|
end
|
|
828
782
|
|
|
829
|
-
# Ends a borrowing heartbeat thread when no member and no worker remains, and returns true
|
|
830
|
-
# then. A register that races this starts a new one.
|
|
831
|
-
def stop_if_idle
|
|
832
|
-
@lock.synchronize do
|
|
833
|
-
unused = @members.empty? && @users.empty?
|
|
834
|
-
if unused
|
|
835
|
-
@thread = nil
|
|
836
|
-
clear_mark
|
|
837
|
-
end
|
|
838
|
-
unused
|
|
839
|
-
end
|
|
840
|
-
end
|
|
841
|
-
|
|
842
783
|
# The shortest member interval; an Idle with the shortest worker interval when only workers
|
|
843
|
-
# remain; or nil when nothing remains.
|
|
844
|
-
#
|
|
845
|
-
# to end. A dedicated thread ends later, in +interval_or_stop+, after the pool has its
|
|
846
|
-
# connection back.
|
|
784
|
+
# remain; or nil when nothing remains. The thread does not end here: it ends later, in
|
|
785
|
+
# +interval_or_stop+, after the pool has its connection back.
|
|
847
786
|
def shortest_interval_or_stop
|
|
848
787
|
@lock.synchronize do
|
|
849
788
|
next @members.each_value.map(&:interval).min unless @members.empty?
|
|
850
789
|
next Idle.new(@users.each_key.map(&:interval).min) unless @users.empty?
|
|
851
790
|
|
|
852
|
-
@thread = nil unless @dedicated
|
|
853
791
|
nil
|
|
854
792
|
end
|
|
855
793
|
end
|
|
856
794
|
|
|
857
795
|
# The current members; an empty Array when only workers remain; or nil when nothing
|
|
858
|
-
# remains.
|
|
859
|
-
# (+shortest_interval_or_stop+).
|
|
796
|
+
# remains (+shortest_interval_or_stop+).
|
|
860
797
|
def members_or_stop
|
|
861
798
|
@lock.synchronize do
|
|
862
799
|
next @members.values unless @members.empty?
|
|
863
800
|
next [] unless @users.empty?
|
|
864
801
|
|
|
865
|
-
@thread = nil unless @dedicated
|
|
866
802
|
nil
|
|
867
803
|
end
|
|
868
804
|
end
|
|
@@ -13,7 +13,8 @@ module Stablemates
|
|
|
13
13
|
# smallest supported +heartbeat+ is 0.1. The worker accepts a shorter value, but on a loaded
|
|
14
14
|
# host it can renew no lease.
|
|
15
15
|
#
|
|
16
|
-
#
|
|
16
|
+
# The pool must report a maximum +size+ of at least 3: the heartbeat connection, the listener,
|
|
17
|
+
# and one connection for claims. Each run first reserves the heartbeat connection that the
|
|
17
18
|
# workers on the pool share, and then borrows one other connection for the compatibility query.
|
|
18
19
|
# Both steps must end within max(+heartbeat+, 5 s), or the run raises WorkerStartupError. The
|
|
19
20
|
# worker claims nothing while the pool has no usable heartbeat connection.
|
|
@@ -207,20 +208,13 @@ module Stablemates
|
|
|
207
208
|
:gate) # :nodoc:
|
|
208
209
|
private_constant :PendingCompletion
|
|
209
210
|
|
|
210
|
-
#
|
|
211
|
-
# dedicated heartbeat connection uses its Heartbeat::Reservation as its gate.
|
|
212
|
-
OPEN_GATE = Object.new.tap do |gate|
|
|
213
|
-
gate.define_singleton_method(:ready?) { true }
|
|
214
|
-
gate.define_singleton_method(:enter) { true }
|
|
215
|
-
gate.define_singleton_method(:leave) { nil }
|
|
216
|
-
end.freeze
|
|
217
|
-
# The gate of a fused completion that no running dispatch admitted.
|
|
211
|
+
# A run uses its Heartbeat::Reservation as its claim gate. The gate of a fused completion that no running dispatch admitted.
|
|
218
212
|
CLOSED_GATE = Object.new.tap do |gate|
|
|
219
213
|
gate.define_singleton_method(:ready?) { false }
|
|
220
214
|
gate.define_singleton_method(:enter) { false }
|
|
221
215
|
gate.define_singleton_method(:leave) { nil }
|
|
222
216
|
end.freeze
|
|
223
|
-
private_constant :
|
|
217
|
+
private_constant :CLOSED_GATE
|
|
224
218
|
|
|
225
219
|
# Raised before a claim statement when the claim gate is closed at send time. The claim sends
|
|
226
220
|
# nothing.
|
|
@@ -233,6 +227,14 @@ module Stablemates
|
|
|
233
227
|
end
|
|
234
228
|
private_constant :Borrowed
|
|
235
229
|
|
|
230
|
+
POOL_SIZE_REFUSAL = "Worker needs a pool with a known maximum size of at least #{MIN_POOL_SIZE} connections " \
|
|
231
|
+
"(found %s). Give the worker a pool with a maximum size of at least #{MIN_POOL_SIZE}."
|
|
232
|
+
# Every worker reserves a dedicated heartbeat connection since 0.11.0 (ADR 0071, SM-1901).
|
|
233
|
+
SHARED_HEARTBEATS_REMOVED = "shared_heartbeats was removed in 0.11.0. Every worker now reserves a dedicated " \
|
|
234
|
+
"heartbeat connection. Remove the option, and give the worker a pool with a known maximum size of at least " \
|
|
235
|
+
"#{MIN_POOL_SIZE}."
|
|
236
|
+
private_constant :POOL_SIZE_REFUSAL, :SHARED_HEARTBEATS_REMOVED
|
|
237
|
+
|
|
236
238
|
RETRY_DELAY_RANGE = (0..2_147_483_647)
|
|
237
239
|
RETRY_DELAY_REFUSAL = "retry_delay must be a finite Numeric count of seconds between 0 and 2147483.647"
|
|
238
240
|
private_constant :RETRY_DELAY_RANGE, :RETRY_DELAY_REFUSAL
|
|
@@ -247,9 +249,10 @@ module Stablemates
|
|
|
247
249
|
def initialize(pool, queues: ["default"], worker_id: nil, concurrency: 1, lease: 30, heartbeat: nil,
|
|
248
250
|
poll_interval: nil, polling_only: false, maintenance_interval: 1, maintenance_routine_interval: 60,
|
|
249
251
|
registry_interval: 5, disable_registry: false, schedule_namespaces: [], schedule_catchup_limit: 100,
|
|
250
|
-
shutdown_grace: 25,
|
|
251
|
-
|
|
252
|
-
|
|
252
|
+
shutdown_grace: 25, retry_delay: nil, on_registration_error: nil, on_notification_error: nil,
|
|
253
|
+
cohorts: nil, logger: nil, **removed)
|
|
254
|
+
refuse_removed_options(removed)
|
|
255
|
+
@executor = worker_executor(pool)
|
|
253
256
|
@pool = pool
|
|
254
257
|
@queues = unique_names(queues, "queues")
|
|
255
258
|
raise ArgumentError, "queues must contain at least one non-empty queue name" if @queues.empty?
|
|
@@ -258,7 +261,7 @@ module Stablemates
|
|
|
258
261
|
concurrency.is_a?(Integer) && concurrency.between?(1, 100)
|
|
259
262
|
|
|
260
263
|
@concurrency = concurrency
|
|
261
|
-
@cohorts = dispatch_cohorts(cohorts, pool,
|
|
264
|
+
@cohorts = dispatch_cohorts(cohorts, pool, polling_only)
|
|
262
265
|
@worker_id = worker_id || "#{Socket.gethostname}-#{Process.pid}-#{SecureRandom.hex(4)}"
|
|
263
266
|
raise ArgumentError, "worker_id must be a non-empty String" unless @worker_id.is_a?(String) && !@worker_id.empty?
|
|
264
267
|
|
|
@@ -292,8 +295,7 @@ module Stablemates
|
|
|
292
295
|
@reporter_failures_lock = Mutex.new
|
|
293
296
|
@logger = logger
|
|
294
297
|
@queue = Queue.new(@executor, default_queue: @queues.first)
|
|
295
|
-
@heartbeat =
|
|
296
|
-
@shared_heartbeats = shared_heartbeats ? true : false
|
|
298
|
+
@heartbeat = Heartbeat.dedicated(pool)
|
|
297
299
|
@handlers = Concurrent::Map.new
|
|
298
300
|
@batch_coordinators = Concurrent::Map.new
|
|
299
301
|
@contracts = Concurrent::Map.new
|
|
@@ -310,6 +312,11 @@ module Stablemates
|
|
|
310
312
|
@last_routine_at = -Float::INFINITY
|
|
311
313
|
@last_registry_at = -Float::INFINITY
|
|
312
314
|
@registered = Concurrent::AtomicBoolean.new(false)
|
|
315
|
+
# The newest registry write thread (+registry_step+). Each write waits for the one before it.
|
|
316
|
+
@registry_lock = Mutex.new
|
|
317
|
+
@registry_write = nil
|
|
318
|
+
# The first error of the run that ended the body of a registry write (+registry_step+).
|
|
319
|
+
@registry_failure = Concurrent::AtomicReference.new
|
|
313
320
|
@instance_id = nil
|
|
314
321
|
@draining = false
|
|
315
322
|
# Guards the dispatch slots, the queue tiers, and the claim order that handler threads share
|
|
@@ -386,14 +393,25 @@ module Stablemates
|
|
|
386
393
|
|
|
387
394
|
private
|
|
388
395
|
|
|
389
|
-
|
|
396
|
+
# Raises the ArgumentError of a keyword that +initialize+ does not take. Any keyword other
|
|
397
|
+
# than the removed +shared_heartbeats+ gets the message Ruby gives for an unknown keyword, and
|
|
398
|
+
# it comes first, so the removal message never hides a typo. The removed option alone gets a
|
|
399
|
+
# message that names its replacement.
|
|
400
|
+
def refuse_removed_options(removed)
|
|
401
|
+
unknown = removed.keys - [:shared_heartbeats]
|
|
402
|
+
unless unknown.empty?
|
|
403
|
+
raise ArgumentError, "unknown keyword#{"s" if unknown.size > 1}: #{unknown.map(&:inspect).join(", ")}"
|
|
404
|
+
end
|
|
405
|
+
raise ArgumentError, SHARED_HEARTBEATS_REMOVED unless removed.empty?
|
|
406
|
+
end
|
|
407
|
+
|
|
408
|
+
def worker_executor(pool)
|
|
390
409
|
raise ArgumentError, "Worker pool must be a ConnectionPool, not a PG::Connection" if pool.is_a?(PG::Connection)
|
|
391
410
|
raise ArgumentError, "Worker pool must respond to with" unless pool.respond_to?(:with)
|
|
392
411
|
|
|
393
412
|
capacity = pool.respond_to?(:size) ? pool.size : nil
|
|
394
|
-
|
|
395
|
-
raise ArgumentError,
|
|
396
|
-
"(found #{capacity.nil? ? "unknown" : capacity}); set shared_heartbeats: true to opt out"
|
|
413
|
+
unless capacity.is_a?(Integer) && capacity >= MIN_POOL_SIZE
|
|
414
|
+
raise ArgumentError, format(POOL_SIZE_REFUSAL, capacity.is_a?(Integer) ? capacity : "unknown")
|
|
397
415
|
end
|
|
398
416
|
Executor.for(pool)
|
|
399
417
|
end
|
|
@@ -402,7 +420,7 @@ module Stablemates
|
|
|
402
420
|
# pooled connection per cohort after the listener and the heartbeat connection, which every
|
|
403
421
|
# worker on the pool shares. A polling-only worker opens no listener, so it keeps that
|
|
404
422
|
# connection for its cohorts.
|
|
405
|
-
def dispatch_cohorts(cohorts, pool,
|
|
423
|
+
def dispatch_cohorts(cohorts, pool, polling_only)
|
|
406
424
|
unless cohorts.nil?
|
|
407
425
|
raise ArgumentError, "cohorts must be an integer between 1 and concurrency" unless
|
|
408
426
|
cohorts.is_a?(Integer) && cohorts.between?(1, @concurrency)
|
|
@@ -413,7 +431,7 @@ module Stablemates
|
|
|
413
431
|
capacity = pool.respond_to?(:size) ? pool.size : nil
|
|
414
432
|
return default unless capacity.is_a?(Integer)
|
|
415
433
|
|
|
416
|
-
spare = capacity - (polling_only ? 0 : 1) -
|
|
434
|
+
spare = capacity - (polling_only ? 0 : 1) - 1
|
|
417
435
|
spare.clamp(1, default)
|
|
418
436
|
end
|
|
419
437
|
|
|
@@ -507,13 +525,13 @@ module Stablemates
|
|
|
507
525
|
return false if reservation == :stopped
|
|
508
526
|
|
|
509
527
|
begin
|
|
510
|
-
run_reserved(continuous, version, reservation
|
|
528
|
+
run_reserved(continuous, version, reservation)
|
|
511
529
|
ensure
|
|
512
530
|
# The run's members renew until the drain ends, so the reservation outlives it. Its
|
|
513
531
|
# release waits only within the stop deadline of the run.
|
|
514
532
|
final = @final_deadline || monotonic
|
|
515
533
|
@final_deadline = nil
|
|
516
|
-
reservation
|
|
534
|
+
reservation.close([final - monotonic, 0].max)
|
|
517
535
|
end
|
|
518
536
|
end
|
|
519
537
|
|
|
@@ -521,22 +539,47 @@ module Stablemates
|
|
|
521
539
|
# at most the shutdown grace plus the unwind window in total.
|
|
522
540
|
def remaining(deadline) = [deadline - monotonic, 0].max
|
|
523
541
|
|
|
524
|
-
#
|
|
525
|
-
#
|
|
526
|
-
#
|
|
527
|
-
#
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
Thread.current.name = name
|
|
531
|
-
block.call
|
|
532
|
-
rescue
|
|
533
|
-
nil
|
|
534
|
-
end
|
|
542
|
+
# Waits for the registry write +thread+ (+registry_step+) until +deadline+. A write that is
|
|
543
|
+
# still running then finishes on its own thread, which keeps its connection until its statement
|
|
544
|
+
# ends. The stop path sends its registration statements this way. The next run of this worker
|
|
545
|
+
# waits for these threads (+await_previous_stop+). Each write waits for the write before it,
|
|
546
|
+
# so these threads also outlast every periodic refresh of the run.
|
|
547
|
+
def within(deadline, thread)
|
|
535
548
|
@stop_statements << thread
|
|
536
549
|
thread.join(remaining(deadline))
|
|
537
550
|
nil
|
|
538
551
|
end
|
|
539
552
|
|
|
553
|
+
# Starts a registry write on its own thread named +name+ and returns the thread. The write
|
|
554
|
+
# first waits for the write before it, so the registrations and the deregistration of a run
|
|
555
|
+
# reach PostgreSQL in that order. The thread takes the earlier write out of its holder before
|
|
556
|
+
# it waits. Thus no write keeps each earlier write reachable (SM-1846, SM-1847).
|
|
557
|
+
#
|
|
558
|
+
# The thread never ends with an error. Otherwise the wait of the next write, and +within+,
|
|
559
|
+
# would raise it, and the chain and the stop path would end there. The write records the
|
|
560
|
+
# first error of the run in @registry_failure instead, and the run raises it
|
|
561
|
+
# (+raise_registry_failure+). Only an error that is not a StandardError gets there: the
|
|
562
|
+
# registration logs and reports a StandardError, and the deregistration ignores one.
|
|
563
|
+
def registry_step(name, &body)
|
|
564
|
+
@registry_lock.synchronize do
|
|
565
|
+
before = [@registry_write]
|
|
566
|
+
@registry_write = Thread.new do
|
|
567
|
+
Thread.current.name = name
|
|
568
|
+
before.pop&.join
|
|
569
|
+
body.call
|
|
570
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
|
571
|
+
@registry_failure.compare_and_set(nil, e)
|
|
572
|
+
nil
|
|
573
|
+
end
|
|
574
|
+
end
|
|
575
|
+
end
|
|
576
|
+
|
|
577
|
+
# Raises the first error that ended the body of a registry write of this run, if one did.
|
|
578
|
+
def raise_registry_failure
|
|
579
|
+
failure = @registry_failure.get
|
|
580
|
+
raise failure if failure
|
|
581
|
+
end
|
|
582
|
+
|
|
540
583
|
# Waits until every stop statement of the previous run of this worker has ended, at most
|
|
541
584
|
# until +deadline+. deregister_worker_v2 deletes only the row of its own run, but a late
|
|
542
585
|
# registration write of the previous run would still replace the registry row of this run, and
|
|
@@ -566,6 +609,9 @@ module Stablemates
|
|
|
566
609
|
def run_reserved(continuous, version, gate)
|
|
567
610
|
@instance_id = SecureRandom.uuid
|
|
568
611
|
@registered.make_false
|
|
612
|
+
# Every registry write of the previous run has ended (+await_previous_stop+), so a failure
|
|
613
|
+
# that one of them recorded after its run ended does not reach this run.
|
|
614
|
+
@registry_failure.set(nil)
|
|
569
615
|
@draining = false
|
|
570
616
|
run_errors = Concurrent::Array.new
|
|
571
617
|
maintenance_errors = Concurrent::Array.new
|
|
@@ -574,11 +620,12 @@ module Stablemates
|
|
|
574
620
|
startup = nil
|
|
575
621
|
claimed_any = false
|
|
576
622
|
begin
|
|
577
|
-
if continuous && !@polling_only
|
|
623
|
+
if continuous && !@polling_only
|
|
578
624
|
listener = Listener.shared(@pool).subscribe(@queues, on_wake: method(:notify),
|
|
579
625
|
on_error: ->(error) { call_reporter(:on_notification_error, @on_notification_error, error) })
|
|
580
626
|
end
|
|
581
|
-
refresh_registration(force: true)
|
|
627
|
+
registry_step("workhorse-registration") { refresh_registration(force: true) }.join
|
|
628
|
+
raise_registry_failure
|
|
582
629
|
log(:info, "workhorse.worker.started", "Worker started",
|
|
583
630
|
"workhorse.worker.concurrency" => @concurrency, "workhorse.worker.queues" => @queues)
|
|
584
631
|
if continuous && !stop_requested?(version)
|
|
@@ -604,15 +651,18 @@ module Stablemates
|
|
|
604
651
|
startup&.join(remaining(stop_deadline))
|
|
605
652
|
listener&.close(remaining(stop_deadline))
|
|
606
653
|
@draining = true
|
|
607
|
-
within(stop_deadline, "workhorse-stop-registration") { refresh_registration(force: true) }
|
|
654
|
+
within(stop_deadline, registry_step("workhorse-stop-registration") { refresh_registration(force: true) })
|
|
608
655
|
abandoned = drain(handlers, stop_deadline, final_deadline)
|
|
609
656
|
# A handler's completion may admit tasks until the drain ends, so the slots outlive it.
|
|
610
657
|
@state_lock.synchronize { @dispatch_slots = nil }
|
|
611
658
|
instance_id = @instance_id
|
|
612
|
-
within(final_deadline, "workhorse-deregistration") { deregister(instance_id) }
|
|
659
|
+
within(final_deadline, registry_step("workhorse-deregistration") { deregister(instance_id) })
|
|
613
660
|
log(:info, "workhorse.worker.stopped", "Worker stopped",
|
|
614
661
|
"workhorse.worker.active_slots" => abandoned, "workhorse.worker.queues" => @queues)
|
|
615
662
|
end
|
|
663
|
+
# A registry write that failed after the last dispatch pass raises only now, so the drain and
|
|
664
|
+
# the deregistration still ran. A write that has not ended by the final deadline is not known.
|
|
665
|
+
raise_registry_failure
|
|
616
666
|
raise ShutdownIncompleteError.new(abandoned) if abandoned.positive?
|
|
617
667
|
raise maintenance_errors.first unless maintenance_errors.empty?
|
|
618
668
|
raise run_errors.first unless run_errors.empty?
|
|
@@ -622,15 +672,10 @@ module Stablemates
|
|
|
622
672
|
|
|
623
673
|
# Reserves the heartbeat connection of the pool, then borrows one other connection for the
|
|
624
674
|
# compatibility query and returns it at once, both within the startup budget. Returns the
|
|
625
|
-
# Heartbeat::Reservation
|
|
626
|
-
#
|
|
627
|
-
#
|
|
675
|
+
# Heartbeat::Reservation. A failure releases the reservation and raises WorkerStartupError, or
|
|
676
|
+
# CompatibilityError for an incompatible schema. A stop issued after +version+ ends the wait
|
|
677
|
+
# for the reservation, closes it, and returns :stopped.
|
|
628
678
|
def start_heartbeat(version, deadline, budget)
|
|
629
|
-
if @shared_heartbeats
|
|
630
|
-
@queue.assert_compatible
|
|
631
|
-
return nil
|
|
632
|
-
end
|
|
633
|
-
|
|
634
679
|
reservation = @heartbeat.reserve(@heartbeat_ms / 1000.0, method(:wake_dispatcher))
|
|
635
680
|
begin
|
|
636
681
|
loop do
|
|
@@ -673,25 +718,12 @@ module Stablemates
|
|
|
673
718
|
raise startup_error("could not borrow a connection for its readiness check", budget), cause: probe.error
|
|
674
719
|
end
|
|
675
720
|
|
|
676
|
-
# Whether the pool can lend the shared listener a connection (SM-1585). The listener holds one
|
|
677
|
-
# pooled connection while it listens. With the opt-out, a pool smaller than MIN_POOL_SIZE is
|
|
678
|
-
# allowed, so the listener starts only on a pool that reports a size of at least 2. Otherwise
|
|
679
|
-
# it could hold the only connection, and the worker polls instead. The run logs that once.
|
|
680
|
-
def pool_lends_listener?
|
|
681
|
-
size = @pool.respond_to?(:size) ? @pool.size : nil
|
|
682
|
-
return true if size.is_a?(Integer) && size >= 2
|
|
683
|
-
|
|
684
|
-
log(:warn, "workhorse.worker.polling_only", "Worker has no notification listener and dispatches by polling",
|
|
685
|
-
"workhorse.worker.queues" => @queues, "workhorse.worker.pool_max_size" => size.is_a?(Integer) ? size : "unknown")
|
|
686
|
-
false
|
|
687
|
-
end
|
|
688
|
-
|
|
689
721
|
def startup_error(failure, budget)
|
|
690
722
|
name = @pool.name if @pool.respond_to?(:name)
|
|
691
723
|
size = @pool.respond_to?(:size) ? @pool.size : "unknown"
|
|
692
724
|
pool = "#{@pool.class.name}#{" #{name.inspect}" if name} (size #{size})"
|
|
693
725
|
WorkerStartupError.new("Worker #{failure} from pool #{pool} within the startup budget of " \
|
|
694
|
-
"#{budget.round(3)} s;
|
|
726
|
+
"#{budget.round(3)} s; grow the pool, or check that the database accepts connections")
|
|
695
727
|
end
|
|
696
728
|
|
|
697
729
|
# Keeps the slots full without one serial claim round trip per task (ADR 0076). A claim
|
|
@@ -701,9 +733,8 @@ module Stablemates
|
|
|
701
733
|
# fast-tier completion claims too, from the same accounting. On the fast tier the slots split
|
|
702
734
|
# into cohorts, and plain claims leave one at a time, each for the roomiest cohort.
|
|
703
735
|
#
|
|
704
|
-
# +gate+ is the claim gate of this run
|
|
705
|
-
#
|
|
706
|
-
# usable heartbeat connection. A busy or rejected lease does not close it.
|
|
736
|
+
# +gate+ is the claim gate of this run, the run's Heartbeat::Reservation: open while the run
|
|
737
|
+
# holds it and the pool holds a usable heartbeat connection. A busy or rejected lease does not close it.
|
|
707
738
|
def dispatch(continuous, version, handlers, listener, startup, run_errors, maintenance_errors, gate)
|
|
708
739
|
slots = DispatchSlots.new(@concurrency, @cohorts, listener, version, run_errors, handlers, gate)
|
|
709
740
|
@state_lock.synchronize { @dispatch_slots = slots }
|
|
@@ -735,7 +766,7 @@ module Stablemates
|
|
|
735
766
|
loop do
|
|
736
767
|
@wake.reset
|
|
737
768
|
settle.call(results.pop) until results.empty?
|
|
738
|
-
|
|
769
|
+
refresh_registration_if_due
|
|
739
770
|
claim_error, pass_ended, empty, empty_wait, wake_version = @state_lock.synchronize do
|
|
740
771
|
[slots.claim_error, slots.pass_ended, slots.empty, slots.empty_wait, @wake_version]
|
|
741
772
|
end
|
|
@@ -1300,6 +1331,8 @@ module Stablemates
|
|
|
1300
1331
|
# A task the executor refused never runs, so the drain must not wait for it.
|
|
1301
1332
|
@active.delete(task.id)
|
|
1302
1333
|
@dispatch_order.delete(task.id)
|
|
1334
|
+
@fast_task_ids.delete(task.id)
|
|
1335
|
+
@tier_reads.delete(task.id)
|
|
1303
1336
|
slots.cohort_active[cohort] -= 1 if slots.thread_cohorts.delete(task.id)
|
|
1304
1337
|
raise
|
|
1305
1338
|
end
|
|
@@ -1379,6 +1412,20 @@ module Stablemates
|
|
|
1379
1412
|
|
|
1380
1413
|
# --- Registration and maintenance -------------------------------------------------------
|
|
1381
1414
|
|
|
1415
|
+
# Starts the periodic registration refresh on its own thread when its interval elapsed and no
|
|
1416
|
+
# registry write still runs. Thus at most one registry statement is in flight, and a refresh
|
|
1417
|
+
# that stalls blocks neither dispatch nor the stop path. Only the dispatch loop calls it. The
|
|
1418
|
+
# error that ended an earlier refresh raises here, so it ends the run as it did on the dispatch
|
|
1419
|
+
# thread.
|
|
1420
|
+
def refresh_registration_if_due
|
|
1421
|
+
return if @registry_ms.zero?
|
|
1422
|
+
|
|
1423
|
+
raise_registry_failure
|
|
1424
|
+
return if monotonic - @last_registry_at < @registry_ms / 1000.0 || @registry_write&.alive?
|
|
1425
|
+
|
|
1426
|
+
registry_step("workhorse-registration") { refresh_registration }
|
|
1427
|
+
end
|
|
1428
|
+
|
|
1382
1429
|
def refresh_registration(force: false)
|
|
1383
1430
|
return if @registry_ms.zero?
|
|
1384
1431
|
|
|
@@ -1582,21 +1629,20 @@ module Stablemates
|
|
|
1582
1629
|
# span.
|
|
1583
1630
|
def release(task, arbiter, attributes)
|
|
1584
1631
|
log(:warn, "workhorse.handler.missing", "No handler registered for the claimed task type", attributes)
|
|
1585
|
-
release_owned(task, arbiter, attributes)
|
|
1632
|
+
release_owned(task, arbiter, attributes, nil)
|
|
1586
1633
|
end
|
|
1587
1634
|
|
|
1588
1635
|
# Hands an owned task back to PostgreSQL, which makes it claimable again without charging the
|
|
1589
|
-
# attempt. Returns the release status.
|
|
1590
|
-
def release_owned(task, arbiter, attributes)
|
|
1636
|
+
# attempt. Returns the release status, or nil after a reconciled stale answer.
|
|
1637
|
+
def release_owned(task, arbiter, attributes, ownership)
|
|
1591
1638
|
status = fenced_row(SqlCatalogue::RELEASE_OWNED_V1, task)["status"]
|
|
1592
1639
|
log(:info, "workhorse.task.release_processed", "Owned task release processed",
|
|
1593
1640
|
attributes.merge("workhorse.release.status" => status))
|
|
1594
1641
|
if status == "released"
|
|
1595
1642
|
arbiter.submit(:released)
|
|
1596
|
-
|
|
1597
|
-
settle_status(task, status, arbiter, "release", ["not_due"], :lease_expired)
|
|
1643
|
+
return status
|
|
1598
1644
|
end
|
|
1599
|
-
status
|
|
1645
|
+
settle_status(task, status, arbiter, "release", ["not_due"], :lease_expired, ownership)
|
|
1600
1646
|
end
|
|
1601
1647
|
|
|
1602
1648
|
def run_handler(task, handler, claim_started_at, active, arbiter, state)
|
|
@@ -1621,11 +1667,11 @@ module Stablemates
|
|
|
1621
1667
|
|
|
1622
1668
|
# A handler that stopped for shutdown returns the task to the queue without an attempt.
|
|
1623
1669
|
if cancellation.reason == :shutdown && shutdown_cancellation?(e)
|
|
1624
|
-
state[:span_outcome] = release_owned(task, arbiter, Telemetry.task_span_attributes(task))
|
|
1670
|
+
state[:span_outcome] = release_owned(task, arbiter, Telemetry.task_span_attributes(task), ownership)
|
|
1625
1671
|
return
|
|
1626
1672
|
end
|
|
1627
1673
|
|
|
1628
|
-
state[:span_outcome] = settle_failure(task, e, arbiter)
|
|
1674
|
+
state[:span_outcome] = settle_failure(task, e, arbiter, ownership)
|
|
1629
1675
|
state[:errors] << (task.redact_error_details ? REDACTED_NAME : error_name(e))
|
|
1630
1676
|
return
|
|
1631
1677
|
ensure
|
|
@@ -1705,15 +1751,39 @@ module Stablemates
|
|
|
1705
1751
|
# nothing. The heartbeat thread delivers statuses inline, so the lock is not held during the
|
|
1706
1752
|
# statements here.
|
|
1707
1753
|
def reconcile_rejected_completion(task, ownership, arbiter)
|
|
1708
|
-
|
|
1709
|
-
|
|
1754
|
+
reconcile_refusal(task, ownership, arbiter, "complete")
|
|
1755
|
+
end
|
|
1710
1756
|
|
|
1711
|
-
|
|
1757
|
+
# Like reconcile_rejected_completion, for the fenced write of +operation+: the completion, or a
|
|
1758
|
+
# failure or release that PostgreSQL answered stale (SM-1884). Without +ownership+, no
|
|
1759
|
+
# heartbeat round can settle the attempt, so the reconciliation does not wait.
|
|
1760
|
+
def reconcile_refusal(task, ownership, arbiter, operation)
|
|
1761
|
+
decided = ownership&.settlement&.synchronize { !arbiter.outcome.nil? }
|
|
1762
|
+
if decided
|
|
1763
|
+
begin
|
|
1764
|
+
return if finish_lifecycle(task, arbiter)
|
|
1765
|
+
rescue LeaseLostError
|
|
1766
|
+
# A heartbeat round submitted a cancellation that PostgreSQL refused to acknowledge.
|
|
1767
|
+
arbiter.refuse_cancellation
|
|
1768
|
+
raise
|
|
1769
|
+
end
|
|
1770
|
+
end
|
|
1771
|
+
|
|
1772
|
+
settle_refusal(task, arbiter, operation)
|
|
1773
|
+
end
|
|
1774
|
+
|
|
1775
|
+
# Settles the cause of a fenced write that PostgreSQL refused, under the attempt's own fence.
|
|
1776
|
+
# expire_owned_v1 does not read expires_at, so a due deadline or attempt timeout settles also
|
|
1777
|
+
# after the lease lapsed (SM-1884). A requested cancellation settles when acknowledge_cancel_v1
|
|
1778
|
+
# accepts it. Any other answer submits a lost lease and raises LeaseLostError.
|
|
1779
|
+
def settle_refusal(task, arbiter, operation)
|
|
1780
|
+
outcome = outcome_for_status(expire_owned(task, Telemetry.current_context), ["not_due"], operation)
|
|
1712
1781
|
return arbiter.submit(:cancelled) if outcome == :cancelled && acknowledge_cancel(task)
|
|
1713
1782
|
return arbiter.submit(outcome) if %i[deadline_exceeded attempt_timeout].include?(outcome)
|
|
1714
1783
|
|
|
1715
1784
|
arbiter.submit(:lease_expired)
|
|
1716
|
-
|
|
1785
|
+
arbiter.refuse_cancellation
|
|
1786
|
+
raise LeaseLostError.new(task.id, operation)
|
|
1717
1787
|
end
|
|
1718
1788
|
|
|
1719
1789
|
# Registers the attempt with the heartbeat and starts its lease watchdog. The watchdog's
|
|
@@ -1756,17 +1826,18 @@ module Stablemates
|
|
|
1756
1826
|
# deadline or an attempt timeout when one passes. While PostgreSQL answers not_due, the first
|
|
1757
1827
|
# retry waits EXPIRATION_RETRY and each later one waits twice as long, up to
|
|
1758
1828
|
# EXPIRATION_RETRY_MAX. Each request locks the task row, so a short fixed retry makes the
|
|
1759
|
-
# heartbeat of the attempt answer busy (SM-1701).
|
|
1829
|
+
# heartbeat of the attempt answer busy (SM-1701). The expiry is a database timestamp, so the
|
|
1830
|
+
# watch converts it to a monotonic due time once. Every wait reads only the monotonic clock,
|
|
1831
|
+
# so a step of the wall clock does not move it (SM-1869).
|
|
1760
1832
|
def watch_expiration(task, ownership, arbiter, cancellation, parent, fail_attempt)
|
|
1761
1833
|
expiration_at = [task.deadline_at, task.attempt_timeout_at].compact.min
|
|
1834
|
+
# A past expiry is due now.
|
|
1835
|
+
expiration_due = monotonic + [expiration_at - Time.now, 0].max if expiration_at
|
|
1762
1836
|
retry_at = nil
|
|
1763
1837
|
retry_delay = EXPIRATION_RETRY
|
|
1764
1838
|
loop do
|
|
1765
|
-
|
|
1766
|
-
|
|
1767
|
-
elsif expiration_at
|
|
1768
|
-
expiration_at - Time.now
|
|
1769
|
-
end
|
|
1839
|
+
due = retry_at || expiration_due
|
|
1840
|
+
delay = due && (due - monotonic)
|
|
1770
1841
|
lease_delay = ownership.lease_remaining
|
|
1771
1842
|
if delay.nil? || lease_delay < delay
|
|
1772
1843
|
if lease_delay.positive?
|
|
@@ -1839,18 +1910,31 @@ module Stablemates
|
|
|
1839
1910
|
STATUS_OUTCOMES.fetch(status) { raise UnexpectedStatusError.new(operation, status) }
|
|
1840
1911
|
end
|
|
1841
1912
|
|
|
1842
|
-
# Submits the outcome for a status the worker's own write returned
|
|
1843
|
-
#
|
|
1844
|
-
|
|
1913
|
+
# Submits the outcome for a status the worker's own write returned, and returns the status
|
|
1914
|
+
# for the handler span. A cancel request is acknowledged. A stale answer settles its cause, as
|
|
1915
|
+
# settle_refusal describes, and returns nil, so the span takes its outcome from that cause. A
|
|
1916
|
+
# lost lease submits :lease_expired and raises LeaseLostError. A neutral status submits
|
|
1917
|
+
# +neutral_outcome+. The reconciliation waits for a heartbeat round of +ownership+, as
|
|
1918
|
+
# reconcile_refusal describes.
|
|
1919
|
+
def settle_status(task, status, arbiter, operation, neutral, neutral_outcome, ownership)
|
|
1845
1920
|
outcome = outcome_for_status(status, neutral, operation)
|
|
1846
1921
|
case outcome
|
|
1847
|
-
when nil then
|
|
1848
|
-
when :cancelled then raise LeaseLostError.new(task.id, operation) unless acknowledge_cancel(task)
|
|
1922
|
+
when nil then arbiter.submit(neutral_outcome)
|
|
1849
1923
|
when :lease_expired
|
|
1850
|
-
|
|
1851
|
-
|
|
1924
|
+
# A lease that lapsed under an unchanged fence can still settle a due boundary.
|
|
1925
|
+
reconcile_refusal(task, ownership, arbiter, operation)
|
|
1926
|
+
return nil
|
|
1927
|
+
when :cancelled
|
|
1928
|
+
unless acknowledge_cancel(task)
|
|
1929
|
+
# The worker settled nothing, so the attempt reports the lost lease.
|
|
1930
|
+
arbiter.submit(:lease_expired)
|
|
1931
|
+
arbiter.refuse_cancellation
|
|
1932
|
+
raise LeaseLostError.new(task.id, operation)
|
|
1933
|
+
end
|
|
1934
|
+
arbiter.submit(outcome)
|
|
1935
|
+
else arbiter.submit(outcome)
|
|
1852
1936
|
end
|
|
1853
|
-
|
|
1937
|
+
status
|
|
1854
1938
|
end
|
|
1855
1939
|
|
|
1856
1940
|
# Stops renewing and reports whether the arbiter already decided the attempt. Raises the
|
|
@@ -1900,8 +1984,9 @@ module Stablemates
|
|
|
1900
1984
|
end
|
|
1901
1985
|
|
|
1902
1986
|
# Records a handler failure. PostgreSQL retries or fails the task by its retry policy.
|
|
1903
|
-
# Returns the state PostgreSQL chose, for the handler span
|
|
1904
|
-
|
|
1987
|
+
# Returns the state PostgreSQL chose, for the handler span, or nil after a reconciled stale
|
|
1988
|
+
# answer.
|
|
1989
|
+
def settle_failure(task, error, arbiter, ownership)
|
|
1905
1990
|
envelope = error_envelope(error, task.redact_error_details)
|
|
1906
1991
|
state = Telemetry.span("workhorse.retry", Telemetry.task_span_attributes(task)) do |span|
|
|
1907
1992
|
delay = durable_replay_conflict(error) ? -1 : retry_delay_override(task)
|
|
@@ -1917,10 +2002,9 @@ module Stablemates
|
|
|
1917
2002
|
Telemetry.task_span_attributes(task).merge("workhorse.attempt.outcome" => state))
|
|
1918
2003
|
if retried
|
|
1919
2004
|
arbiter.submit(:retry)
|
|
1920
|
-
|
|
1921
|
-
settle_status(task, state, arbiter, "fail", %w[failed], :failed)
|
|
2005
|
+
return state
|
|
1922
2006
|
end
|
|
1923
|
-
state
|
|
2007
|
+
settle_status(task, state, arbiter, "fail", %w[failed], :failed, ownership)
|
|
1924
2008
|
end
|
|
1925
2009
|
|
|
1926
2010
|
# Whether +error+, or an error it wraps, is a +:shutdown+ CancelledError. With the token's own
|