stablemates-workhorse 0.8.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 807d43f9b16d79430d292a3e6c70acdc55bb8cc3f2010b7e03f0356fb0731c90
-  data.tar.gz: f84ab68d2d6d03450983e74dc337880546815472ece79d8cdf1d650679d3912a
+  metadata.gz: b030c71db8e2fa0f3b2e14e4ac34442270e1643ea1105efe904bc965ab6e4823
+  data.tar.gz: 90256bbd761dfa37265e9e0fce4f5f49534e378361a625bbc6bc27cf24b7634b
 SHA512:
-  metadata.gz: 706d9cfd755c1bc971116245009433dba6d4880dc7b7dec081f486d95f5395395a69b55527527b33252f4cf4bbc5e1a1b94c74ad91b267fe6893884c3bdb09ef
-  data.tar.gz: dd7e13f971e790cc10c4f07843a4b7ec9f086f32365c665f681206bf1376d1d7d03bf11e6dfa78aacc75ea8cb2ce292da3246b9471431d7fa6b1630c58c46a17
+  metadata.gz: 9d580a9360fe1772b51567673be3d787ea287ba7df8693d93d8ea4b153620f0a4c0b4c5ad492593830127ccd2c78a1063b5ac5f9174bbbea6aae3f1ca2880b70
+  data.tar.gz: d353d8e0ce3d2b03b869212dee916bcead4c41034de9403c83ab1f3e3a74eaaadfa36bf0c0c0c03f22728efdee770ef437c3de369f51d1f36765f5861948679f

data/CHANGELOG.md

@@ -3,6 +3,126 @@
 `stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
 other SDKs carry, because every tag names one release of all of them.
 
+## 0.9.0 — 2026-10-20
+
+The npm packages, Python distribution, Go module, Rust crate, Ruby gem, and JVM artifacts release from one source commit.
+
+Requires **schema v73**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
+
+### Upgrade steps
+
+**Migrate the schema before starting updated processes.** Run `workhorse schema migrate` before any
+process of this release starts, producers too. Migrations 0073 and 0074 move the schema to version
+**73**. The SDK compatibility floor rises from 69 to **73**, because every SDK statement catalogue
+now names `heartbeat_many_v2` and `heartbeat_v2`. An updated process refuses a schema below version
+73 at startup. A 0.8 process keeps working on version 73, because a migration inside a major line
+only adds.
+
+### Heartbeats do not wait for a locked task row
+
+Before this release, a heartbeat waited for a task row that another transaction locked. If a
+settlement held the row, the heartbeat then kept the row locked after the settlement committed, and
+a claim skipped the ready task (SM-1434). The worker now renews leases through `heartbeat_many_v2`
+(SM-1443, ADR 0105). It skips a locked row and reports that lease as `busy`: no renewal and no
+ownership verdict. The worker keeps a busy lease, retries only the busy leases after a short delay
+with jitter, and does not move the regular heartbeat schedule. A busy answer never extends the local
+lease watchdog, so a lease that stays busy until the watchdog ends is lost, as before. Old and new
+workers can run together: an old worker keeps calling `heartbeat_many_v1`, which does not change.
+
+**One-lease heartbeat without a row lock wait.** Migration 0074 adds `heartbeat_v2` (SM-1461). It
+renews one lease by the rule of `heartbeat_many_v2`, so it reports a locked row as `busy`. This SDK
+names it in its statement catalogue but does not call it. `heartbeat_v1` does not change.
+
+### Heartbeat rounds keep a renewal margin and end on the server
+
+Before this release, the dedicated heartbeat connection bounded each heartbeat statement by one full
+interval. After a failed round, the heartbeat thread started a new schedule on its new connection.
+With the defaults, one timed-out round thus left about 0 s before the lease watchdog expired
+(SM-1444). Also, when the worker stopped waiting for a round, PostgreSQL continued to run the
+heartbeat statement, and that statement kept its row locks (SM-1442). These changes apply to each
+round on the dedicated heartbeat connection:
+
+- **Rounds start on a fixed schedule.** A round falls due one interval after the previous round
+  started, not after it ended. The schedule uses a monotonic clock. If a round ends after its next
+  slot, the worker starts one overdue round at once and sends no round for the other missed slots.
+- **The round bound is shorter.** A round ends by
+  `max(1 ms, interval × 3 / 4, min(50 ms, interval))`. For an interval of 200 ms or more, this is
+  three quarters of the interval. With the defaults (lease 30 s, interval 10 s), the bound is 7.5 s,
+  so one timed-out round leaves a margin of 2.5 s. An attempt keeps its lease after one timed-out
+  round if `2 × interval + round bound < lease`. The worker does not reject an interval that breaks
+  this rule. Before this release, each statement of a round had its own bound of one interval. Now
+  all statements of one round share one deadline, and a statement that would start after it waits
+  for the next round. Each round starts with the next statement in turn, so a statement that is
+  always slow cannot keep a later statement from its turn.
+- **Each round sets server timeouts.** The round sends `BEGIN`, then one `set_config` statement for
+  `statement_timeout` and `idle_in_transaction_session_timeout`, then `heartbeat_many_v2`, then
+  `COMMIT`. Both settings are transaction-local. The heartbeat statement gets one half of the time
+  that remains, and each idle gap one eighth. If less than 8 ms remains at the settings statement,
+  the round fails before the heartbeat. The worker counts a renewal only after a `COMMIT` that ends
+  before the deadline. No setting bounds `COMMIT` on PostgreSQL 15 and later.
+- **A failed round discards its connection only when a statement may still run** (SM-1492). A
+  round that expires before it sends the heartbeat statement rolls back and keeps its connection.
+  Under load, PostgreSQL can log
+  `canceling statement due to statement timeout` for a heartbeat statement. It can also end the
+  session with `terminating connection due to idle-in-transaction timeout`. Each message means that
+  one round failed. Every task continues, and the next round tries again. A deadlock (`40P01`) also
+  fails the round.
+- **A busy retry with less than 20 ms left sends nothing** (SM-1492). A bounded busy retry runs
+  only if its deadline is at least 20 ms after its start: the 8 ms that a round needs at its
+  settings statement, plus a 12 ms margin. Otherwise the next regular round renews the busy leases.
+- **Replacement connections.** The heartbeat thread keeps the start of the latest round when it
+  replaces a failed connection.
+- **100 ms is the smallest supported heartbeat interval** (SM-1471). This applies to `heartbeat:` of
+  `Worker`, in seconds: the smallest supported value is 0.1. At 100 ms, the round bound is 75 ms.
+  The worker still accepts a shorter interval, and it does not warn. On a loaded host, a shorter
+  interval can make each round miss its deadline, and attempts can then lose their leases.
+
+Rounds under `shared_heartbeats: true` have no client deadline and set no server timeouts, as
+before. They follow the new schedule. The [heartbeat round
+deadline](../docs/architecture/lifecycle.md#heartbeat-round-deadline) and the [heartbeat
+schedule](../docs/architecture/lifecycle.md#heartbeat-schedule) state each rule.
+
+### A payload contract compares the decimals of numbers
+
+Before this release, `ContractSchema` compared an Integer with the binary value of a Float in the
+bound keywords, `const`, `enum`, and `uniqueItems` (SM-1539). Thus `1152921504606846980` failed
+`{"maximum": 1.152921504606847e18}`, although PostgreSQL stores the constant as
+`1152921504606847000`. A contract now compares each Float through the decimal of `Float#to_s`
+(ADR 0039). Ruby still reads a stored schema constant that a double cannot hold as a Float
+(SM-1541).
+
+### A bad worker interval raises ArgumentError
+
+Before this release, a bad value for `heartbeat`, `poll_interval`, `maintenance_interval`,
+`maintenance_routine_interval`, or `registry_interval` raised `NoMethodError`, because these options
+have no upper limit (SM-1511). For example, `heartbeat: 0` did this. Now `Worker.new` raises
+`ArgumentError`, and the message names the option and its lower limit.
+
+### The `workhorse.tasks.retried` description matches `api/telemetry.txt`
+
+The Ruby worker now describes `workhorse.tasks.retried` as "Failed or expired tasks returned to live
+work", as the contract does (SM-1514). The counter already counted expired tasks. Only the
+description changes.
+
+### A wrapped durable replay conflict is terminal
+
+Before this release, the Ruby worker checked only the outermost error for a durable replay conflict
+(ADR 0098). If a handler wrapped a conflict in its own error, the worker recorded a retryable
+failure, and the conflict repeated until `max_attempts` (SM-1512). The worker now follows the cause
+chain of the error, as the Go and Rust workers do. It reads at most 16 errors, so a cyclic chain
+cannot stop it. It reads `Exception#cause`, so an error raised in a `rescue` block also counts. A
+wrapped conflict fails the task at once. The failure envelope records the name of the conflict. Its
+fields `message` and `stack` stay those of the outer error.
+
+### A heartbeat round sends at most 100 leases in each statement
+
+`heartbeat_many_v2` accepts at most 100 leases. A fused completion claim registers its tasks before
+the completing attempts unregister, so a round can hold more leases than the concurrency limit. That
+round raised an error, and no lease in it renewed (SM-1495). The worker now splits a round into
+statements of at most 100 leases. Each statement counts on its own. A failed statement renews none
+of its leases, and the statements before it still count, as before for the statements of each
+worker.
+
 ## 0.8.0 — 2026-10-13
 
 The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.

data/README.md

@@ -3,6 +3,13 @@
 The Ruby queue client, operator client, and worker runtime for the Workhorse durable task queue for
 PostgreSQL.
 
+<picture>
+  <source media="(max-width: 600px) and (prefers-color-scheme: dark)" srcset="https://workhorse.run/diagrams/readme-overview-narrow-dark.png">
+  <source media="(max-width: 600px)" srcset="https://workhorse.run/diagrams/readme-overview-narrow-light.png">
+  <source media="(prefers-color-scheme: dark)" srcset="https://workhorse.run/diagrams/readme-overview-dark.png">
+  <img src="https://workhorse.run/diagrams/readme-overview-light.png" width="960" alt="One protocol for your workers and your dashboard">
+</picture>
+
 > **Public beta:** Workhorse is usable for evaluation and early production adoption. A 0.x minor
 > release may change behaviour, so read the
 > [changelog](https://github.com/stablemates/workhorse/blob/main/ruby/CHANGELOG.md) before you
@@ -23,7 +30,7 @@ bundle add connection_pool
 Install the schema once, as a deployment step. The application never installs or migrates it.
 
 ```bash
-npx --package @stablemates/workhorse@0.8.0 workhorse schema install
+npx --package @stablemates/workhorse@0.9.0 workhorse schema install
 ```
 
 The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs

data/lib/stablemates/workhorse/contracts.rb

@@ -144,7 +144,7 @@ module Stablemates
 
       private
 
-      # A reference names the root schema or one root definition. The libraries behind the five SDKs
+      # A reference names the root schema or one root definition. The validators behind the six SDKs
       # resolve those two forms alike, and both name a schema position the profile check visited. The
       # check runs before +valid?+, so +valid?+ cannot fail on the schema.
       def check_references(schema, path)
@@ -213,11 +213,11 @@ module Stablemates
 
       def number_valid?(keyword, value, instance)
         case keyword
-        when "minimum" then instance >= value
-        when "maximum" then instance <= value
-        when "exclusiveMinimum" then instance > value
-        when "exclusiveMaximum" then instance < value
-        when "multipleOf" then (Rational(instance.to_s) / Rational(value.to_s)).denominator == 1
+        when "minimum" then exact(instance) >= exact(value)
+        when "maximum" then exact(instance) <= exact(value)
+        when "exclusiveMinimum" then exact(instance) > exact(value)
+        when "exclusiveMaximum" then exact(instance) < exact(value)
+        when "multipleOf" then finite?(instance) && (exact(instance) / exact(value)).denominator == 1
         else true
         end
       end
@@ -287,6 +287,13 @@ module Stablemates
 
       def number?(instance) = instance.is_a?(Integer) || instance.is_a?(Float)
 
+      # The decimal that JSON writes to PostgreSQL for a number. Float#to_s gives the shortest
+      # decimal that reads back as the same Float, so 2**60 as a Float compares as
+      # 1152921504606847000, not as its binary value. A non-finite Float stays a Float.
+      def exact(number) = (number.is_a?(Float) && number.finite?) ? Rational(number.to_s) : number
+
+      def finite?(number) = !number.is_a?(Float) || number.finite?
+
       # JSON equality: 1 equals 1.0, and true equals neither 1 nor 1.0.
       def equal?(left, right)
         case left
@@ -295,7 +302,7 @@ module Stablemates
         end
         when Array then right.is_a?(Array) && left.size == right.size && left.zip(right).all? { |l, r| equal?(l, r) }
         when true, false, nil then left.equal?(right)
-        when Integer, Float then number?(right) && left == right
+        when Integer, Float then number?(right) && exact(left) == exact(right)
         else left == right
         end
       end

data/lib/stablemates/workhorse/dashboard/bundle.json

@@ -2,5 +2,5 @@
   "formatVersion": 1,
   "readSurfaceVersion": 1,
   "archive": "read-surface-1.tar.gz",
-  "sha256": "b65d68d2a7dd68e1cb7e58512f4d6467e9f26dea7f405515503da5f78d974234"
+  "sha256": "698c094e8ef4a0ccee589f425b03243abb97e7c817b271b920ddbb394cf99454"
 }

data/lib/stablemates/workhorse/sql_catalogue_generated.rb

@@ -10,8 +10,8 @@ module Stablemates
       CLIENT_PROTOCOL_VERSION = 5
       MINIMUM_PROTOCOL_VERSION = 5
       MAXIMUM_PROTOCOL_VERSION = 5
-      MINIMUM_SCHEMA_VERSION = 69
-      MAXIMUM_SCHEMA_VERSION = 71
+      MINIMUM_SCHEMA_VERSION = 73
+      MAXIMUM_SCHEMA_VERSION = 73
       # PostgreSQL's atomic enqueue batch limit.
       MAX_ENQUEUE_BATCH_SIZE = 1000
       DEFAULT_TASK_VALUE_MAX_BYTES = 1048576
@@ -132,12 +132,15 @@ module Stablemates
       # `get_wait` (admin)
       GET_WAIT = 'SELECT task_id::text task_id,wait_name,mode,duration_ms::text duration_ms,requested_wake_at,wake_at,attempt,fence_token::text fence_token,worker_id,created_at FROM workhorse.task_wait WHERE task_id=$1::uuid AND wait_name=$2::text'
 
-      # `heartbeat_many_v1` (protocol)
-      HEARTBEAT_MANY_V1 = 'SELECT task_id::text AS task_id, status FROM workhorse.heartbeat_many_v1($1::text, $2::jsonb) ORDER BY ordinal'
+      # `heartbeat_many_v2` (protocol)
+      HEARTBEAT_MANY_V2 = 'SELECT task_id::text AS task_id, status FROM workhorse.heartbeat_many_v2($1::text, $2::jsonb) ORDER BY ordinal'
 
       # `heartbeat_v1` (protocol)
       HEARTBEAT_V1 = 'SELECT workhorse.heartbeat_v1($1::uuid, $2::text, $3::bigint, $4::integer) AS status'
 
+      # `heartbeat_v2` (protocol)
+      HEARTBEAT_V2 = 'SELECT workhorse.heartbeat_v2($1::uuid, $2::text, $3::bigint, $4::integer) AS status'
+
       # `list_checkpoint` (internal)
       LIST_CHECKPOINT = 'SELECT checkpoint_value FROM workhorse.task_checkpoint WHERE task_id = $1::uuid AND checkpoint_name = $2::text'
 

data/lib/stablemates/workhorse/telemetry.rb

@@ -16,7 +16,7 @@ module Stablemates
         "workhorse.tasks.claimed" => [:counter, "Tasks claimed for handler execution", "{task}"],
         "workhorse.tasks.completed" => [:counter, "Tasks completed under a valid lease", "{task}"],
         "workhorse.tasks.failed" => [:counter, "Handler failures submitted to PostgreSQL", "{task}"],
-        "workhorse.tasks.retried" => [:counter, "Failed tasks returned to live work", "{task}"],
+        "workhorse.tasks.retried" => [:counter, "Failed or expired tasks returned to live work", "{task}"],
         "workhorse.leases.expired" => [:counter, "Expired leases recovered by maintenance", "{lease}"],
         "workhorse.handler.runtime" => [:counter, "Cumulative handler execution time", "ms"],
         "workhorse.handler.executions" => [:counter, "Worker handler activations by outcome", "{execution}"],

data/lib/stablemates/workhorse/types.rb

@@ -216,13 +216,15 @@ module Stablemates
       end
 
       # Whole milliseconds from a finite Numeric count of seconds, within +range+ milliseconds.
-      # An ActiveSupport::Duration is Numeric and converts the same way.
+      # An ActiveSupport::Duration is Numeric and converts the same way. An endless +range+ has
+      # only a lower limit.
       def milliseconds(value, label, range)
         milliseconds = whole_milliseconds(value)
         return milliseconds if milliseconds && range.cover?(milliseconds)
 
-        raise ArgumentError, "#{label} must be a finite Numeric count of seconds between " \
-          "#{seconds(range.begin)} and #{seconds(range.end)}"
+        limits = range.end.nil? ? "of at least #{seconds(range.begin)}" :
+          "between #{seconds(range.begin)} and #{seconds(range.end)}"
+        raise ArgumentError, "#{label} must be a finite Numeric count of seconds #{limits}"
       end
 
       def whole_milliseconds(value)

data/lib/stablemates/workhorse/version.rb

@@ -2,6 +2,6 @@
 
 module Stablemates
   module Workhorse
-    VERSION = "0.8.0"
+    VERSION = "0.9.0"
   end
 end

data/lib/stablemates/workhorse/worker/heartbeat.rb

@@ -12,14 +12,58 @@ module Stablemates
       # the executor for each round instead.
       class Heartbeat # :nodoc:
         # One running attempt. +renew+ takes the monotonic send time of an accepted round;
-        # +settle+ takes a status other than +accepted+; +fail+ takes an error +settle+ raised.
-        Member = Data.define(:worker_id, :task, :lease_ms, :interval, :renew, :settle, :fail, :logger)
+        # +settle+ takes a status other than +accepted+ and +busy+; +fail+ takes an error +settle+
+        # raised. +lease_ends_at+ returns the monotonic end of the local lease window, or nil once
+        # the window is closed. A busy retry never runs past it.
+        Member = Data.define(:worker_id, :task, :lease_ms, :interval, :renew, :settle, :fail, :logger,
+          :lease_ends_at)
 
-        # A round that did not finish within its interval.
+        # A round that did not finish by its deadline.
         class RoundTimeout < StandardError; end
 
+        # A round that stopped before it sent its heartbeat statement, and left its connection clean:
+        # it sent nothing, or its transaction rolled back. The worker keeps the connection.
+        class RoundExpired < StandardError; end
+
         STRINGS = PG::TypeMapAllStrings.new
-        private_constant :STRINGS
+        # Sets both server timeouts for the current transaction only. SET LOCAL cannot take a bind
+        # parameter, and set_config(..., true) has the same scope.
+        ROUND_TIMEOUTS = "SELECT set_config('statement_timeout', $1, true), " \
+          "set_config('idle_in_transaction_session_timeout', $2, true)"
+        # A round with less time than this left, in seconds, fails before it sends its heartbeat. At
+        # this size every budget of +round_timeouts+ is at least 1 ms, and the budgets still fit.
+        MINIMUM_REMAINING = 0.008
+        # The longest base delay before a busy retry, in seconds (Heartbeat.busy_retry_delay).
+        BUSY_RETRY_BASE = 0.1
+        # A bounded busy retry with less time than BUSY_RETRY_MINIMUM left does not run, and the next
+        # regular round renews its leases. The minimum is the time a round needs at the settings
+        # statement, plus a margin for the payload, the wait for the connection, and BEGIN.
+        BUSY_RETRY_MARGIN = 0.012
+        BUSY_RETRY_MINIMUM = MINIMUM_REMAINING + BUSY_RETRY_MARGIN
+        # The wait after a shared round raised an unexpected error, in seconds, so that an error that
+        # repeats cannot keep the thread busy.
+        FAILED_ROUND_DELAY = 0.1
+        # heartbeat_many_v2 raises an error for more leases than this in one statement.
+        STATEMENT_LEASE_LIMIT = 100
+        private_constant :STRINGS, :ROUND_TIMEOUTS, :MINIMUM_REMAINING, :BUSY_RETRY_BASE, :BUSY_RETRY_MARGIN,
+          :BUSY_RETRY_MINIMUM, :FAILED_ROUND_DELAY, :STATEMENT_LEASE_LIMIT
+
+        # The deadline of one round, in seconds, for an +interval+ in seconds. It is three quarters
+        # of the interval, so that one timed-out round leaves a renewal margin when 2 × interval +
+        # bound < lease. Below an interval of 200 ms the bound is at least min(50 ms, interval). Up
+        # to 50 ms the bound thus equals the interval, as before SM-1444, and a timed-out round
+        # leaves about no margin. It is at least 1 ms. Every SDK uses the same formula.
+        def self.round_bound(interval)
+          milliseconds = (interval * 1000).round
+          [1, milliseconds * 3 / 4, [50, milliseconds].min].max / 1000.0
+        end
+
+        # The delay before a busy retry, in seconds, for the shortest member +interval+ in seconds:
+        # base × (1 + +random+), with base = min(100 ms, interval / 8). The jitter keeps two workers
+        # from meeting the same lock holder in step. Every SDK uses the same formula.
+        def self.busy_retry_delay(interval, random = rand)
+          [BUSY_RETRY_BASE, interval / 8.0].min * (1 + random)
+        end
 
         @dedicated = {}.compare_by_identity
         @dedicated_lock = Mutex.new
@@ -36,6 +80,13 @@ module Stablemates
           @members = {}
           @wake = Concurrent::Event.new
           @thread = nil
+          # The worker that the next dedicated round sends first. Only the heartbeat thread uses it.
+          @rotation = 0
+          # The busy members of the latest round, which a retry renews before the next regular
+          # round, and the monotonic time when that retry falls due. A round of either kind
+          # replaces both. Only the heartbeat thread uses them.
+          @busy = nil
+          @busy_at = 0
         end
 
         # Adds +member+ and returns a callable that removes it. Removing twice is harmless. A member
@@ -64,15 +115,31 @@ module Stablemates
         private
 
         def run
-          if @dedicated
-            run_dedicated
-          else
-            rounds { |sql, params, _timeout| Executor.for(@source).rows(sql, params) }
+          # The start of the latest round, on the monotonic clock. A new connection keeps it, so a
+          # failed round does not start a new schedule.
+          @previous = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          @busy = nil
+          @dedicated ? run_dedicated : run_shared
+        end
+
+        # Runs the rounds of a borrowing heartbeat. An unexpected error does not end the thread while
+        # a member remains: the thread ends only through +stop_if_idle+, so a register that races
+        # the end starts a new thread.
+        def run_shared
+          loop do
+            return if rounds { |sql, params, _timeout| Executor.for(@source).rows(sql, params) }
+          rescue
+            @busy = nil
+            return if stop_if_idle
+
+            @wake.wait(FAILED_ROUND_DELAY)
+            @wake.reset
           end
         end
 
         # Holds one connection across rounds. A round that fails or overruns discards the
-        # connection, and the next round checks out a new one.
+        # connection, and the next round checks out a new one. A round that expires before its
+        # heartbeat statement keeps the connection (+bounded_rows+).
         def run_dedicated
           loop do
             finished = false
@@ -84,62 +151,181 @@ module Stablemates
               end
             rescue
               # The pool refused a connection. The lease watchdog covers rounds that keep failing.
-              finished = idle?
+              finished = stop_if_idle
             end
             return if finished
           end
         end
 
         # Runs rounds until no member remains, and returns true then. A round falls due one shortest
-        # member interval after the previous one; a wake recomputes that from the current members.
-        # A failed round raises only from a dedicated connection, so the caller can replace it.
-        def rounds
-          previous = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        # member interval after the previous regular round started; a wake recomputes that from the
+        # current members. A round that ends after its next slot starts one overdue round at once,
+        # and the interval after it counts from that start, so missed slots are never replayed. A
+        # failed round raises only from a dedicated connection, so the caller can replace it.
+        #
+        # Between two regular rounds, a busy retry (+take_busy_retry+) sends only the members that
+        # the latest round reported busy. It runs on the same thread, so it is a round for the
+        # one-round-in-flight rule, and it never moves the regular slot.
+        def rounds(&)
           loop do
-            interval = @lock.synchronize { @members.each_value.map(&:interval).min }
-            return true if interval.nil? && stop_if_idle
+            interval = shortest_interval_or_stop
+            if interval.nil?
+              @busy = nil
+              return true
+            end
 
-            remaining = previous + (interval || 0) - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            regular_at = @previous + interval
+            retrying = !@busy.nil? && @busy_at < regular_at
+            remaining = (retrying ? @busy_at : regular_at) - Process.clock_gettime(Process::CLOCK_MONOTONIC)
             if remaining.positive?
               @wake.wait(remaining)
               @wake.reset
               next
             end
-            previous = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-            members = @lock.synchronize { @members.values }
-            return true if members.empty? && stop_if_idle
-
-            members.group_by(&:worker_id).each do |worker_id, group|
-              sent_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-              leases = group.map do |member|
-                {"taskId" => member.task.id, "fenceToken" => member.task.fence_token.to_s, "leaseMs" => member.lease_ms}
-              end
-              rows = begin
-                yield SqlCatalogue::HEARTBEAT_MANY_V1, [worker_id, JSON.generate(leases)], interval
-              rescue RoundTimeout, PG::Error
-                raise if @dedicated
+            if retrying
+              selection = take_busy_retry(regular_at)
+              # With nothing left to retry, the next regular round waits for its slot.
+              next if selection.nil?
 
-                next
-              rescue
-                # A skipped round is silent: the lease watchdog handles rounds that keep failing.
-                next
-              end
-              settle(group, rows.to_h { |row| [row["task_id"], row["status"]] }, sent_at)
+              started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+              deadline = retry_deadline(selection, interval, regular_at, started)
+              # A bounded retry with too little time left would fail before its heartbeat statement.
+              # It sends nothing, and the next regular round renews every lease, the busy ones too.
+              next if @dedicated && deadline - started < BUSY_RETRY_MINIMUM
+
+              send_round(selection, deadline, interval, rotate: false, &)
+              next
             end
+            @previous = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            # The deadline starts before the payloads are built, so serialization counts against it.
+            deadline = @previous + round_bound(interval)
+            members = members_or_stop
+            return true if members.nil?
+
+            send_round(members, deadline, interval, rotate: true, &)
+          end
+        end
+
+        # Sends one round for +members+. A round sends one statement for each worker, and more for a
+        # worker with more than +STATEMENT_LEASE_LIMIT+ members (+statements+). On the
+        # dedicated connection, all of them share one +deadline+. A statement that would start after
+        # it waits for the next round, so a slow statement of one worker cannot delay the renewal of
+        # another past it. Each regular round starts with the next worker (+ordered_groups+), so a
+        # worker whose statement is always slow cannot keep another from its turn. A borrowing
+        # heartbeat has no deadline, and sends every statement.
+        #
+        # A round replaces the pending busy retry. If one or more members are busy, it schedules one
+        # retry for them. A round that raises schedules none, and the next regular round tries again.
+        #
+        # A statement that expires before its heartbeat (+RoundExpired+) stops the round on its
+        # connection. The statements before it still count, and the next round sends the rest. A
+        # busy retry that stops this way schedules no other retry.
+        def send_round(members, deadline, interval, rotate:)
+          @busy = nil
+          busy = []
+          expired = false
+          statements(members, rotate:).each do |worker_id, group|
+            sent_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            leases = group.map do |member|
+              {"taskId" => member.task.id, "fenceToken" => member.task.fence_token.to_s, "leaseMs" => member.lease_ms}
+            end
+            params = [worker_id, JSON.generate(leases)]
+            timeout = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            break if @dedicated && !timeout.positive?
+
+            rows = begin
+              yield SqlCatalogue::HEARTBEAT_MANY_V2, params, timeout
+            rescue RoundExpired
+              expired = true
+              break
+            rescue RoundTimeout, PG::Error
+              raise if @dedicated
+
+              next
+            rescue
+              # A skipped round is silent: the lease watchdog handles rounds that keep failing.
+              next
+            end
+            busy.concat(settle(group, rows.to_h { |row| [row["task_id"], row["status"]] }, sent_at))
+          end
+          return if busy.empty? || (expired && !rotate)
+
+          @busy = busy
+          @busy_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) + Heartbeat.busy_retry_delay(interval)
+        end
+
+        # Takes the pending busy retry, and returns the members it can still renew, or nil. Each
+        # member must still be registered as the same object, so a member that completed, left, or
+        # was replaced is gone, and its local lease window must not have ended. At or after
+        # +regular_at+ the retry is dropped, and the regular round takes every member.
+        def take_busy_retry(regular_at)
+          busy = @busy
+          @busy = nil
+          now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          return nil if busy.nil? || now >= regular_at
+
+          selection = busy.select do |member|
+            ends_at = member.lease_ends_at.call
+            ends_at && ends_at > now && registered?(member)
           end
+          selection.empty? ? nil : selection
         end
 
+        # A busy retry ends by its own round bound, by the next regular slot, and by the end of the
+        # earliest local lease window in it, so it never renews a lease that the worker has already
+        # given up. A borrowing heartbeat ignores the deadline, as in its regular rounds. +started+ is
+        # the monotonic start of the retry.
+        def retry_deadline(selection, interval, regular_at, started)
+          ends = selection.map { |member| member.lease_ends_at.call || started }
+          [started + round_bound(interval), regular_at, *ends].min
+        end
+
+        # The statements of one round, as [worker_id, members] pairs: the +ordered_groups+, each
+        # split into slices of at most +STATEMENT_LEASE_LIMIT+ members. A fused completion claim
+        # registers its tasks before the completing attempts unregister, so a worker can hold more
+        # members than its concurrency.
+        def statements(members, rotate:)
+          ordered_groups(members, rotate:).flat_map do |worker_id, group|
+            group.each_slice(STATEMENT_LEASE_LIMIT).map { |slice| [worker_id, slice] }
+          end
+        end
+
+        # The members of one round, one group for each worker, as [worker_id, members] pairs. On the
+        # dedicated connection, each regular round starts one worker later than the round before
+        # it. A busy retry (+rotate+ false) keeps the order and does not move the rotation.
+        def ordered_groups(members, rotate: true)
+          groups = members.group_by(&:worker_id).to_a
+          return groups unless rotate && @dedicated && groups.size > 1
+
+          first = @rotation % groups.size
+          @rotation = first + 1
+          groups.rotate(first)
+        end
+
+        # The deadline of one round, in seconds, for the shortest member +interval+
+        # (Heartbeat.round_bound).
+        def round_bound(interval) = Heartbeat.round_bound(interval)
+
+        # Delivers each status to its member, and returns the busy members. A result counts only
+        # for the member object that the round sent, so a member that left or was replaced while
+        # the round ran gets nothing from it. A busy member got no renewal and no ownership verdict:
+        # its lease watchdog keeps its window, and the member stays registered.
         def settle(group, statuses, sent_at)
-          group.each do |member|
+          group.filter_map do |member|
             next unless registered?(member)
 
             status = statuses.fetch(member.task.id, "stale")
             begin
               report(member, status)
-              (status == "accepted") ? member.renew.call(sent_at) : member.settle.call(status)
+              case status
+              when "accepted" then member.renew.call(sent_at)
+              when "busy" then next member
+              else member.settle.call(status)
+              end
             rescue => e
               member.fail.call(e)
             end
+            nil
           end
         end
 
@@ -159,34 +345,108 @@ module Stablemates
           @lock.synchronize { @members[[member.worker_id, member.task.id]].equal?(member) }
         end
 
-        def idle? = @lock.synchronize { @members.empty? }
-
-        # Ends the thread when no member remains. A register that races this starts a new one.
+        # Ends the thread when no member remains, and returns true then. A register that races this
+        # starts a new one.
         def stop_if_idle
           @lock.synchronize do
-            return false unless @members.empty?
+            @thread = nil if @members.empty?
+            @members.empty?
+          end
+        end
 
-            @thread = nil
-            true
+        # The shortest member interval, or nil when no member remains. Nil also ends the thread. The
+        # read and the end take one lock, so a register between them cannot find a thread that is
+        # about to end.
+        def shortest_interval_or_stop
+          @lock.synchronize do
+            @thread = nil if @members.empty?
+            @members.each_value.map(&:interval).min
           end
         end
 
-        # Sends one round without blocking past +timeout+ seconds.
+        # The current members, or nil when none remains. Nil also ends the thread, under the same
+        # lock as the read (+shortest_interval_or_stop+).
+        def members_or_stop
+          @lock.synchronize do
+            @thread = nil if @members.empty?
+            @members.values unless @members.empty?
+          end
+        end
+
+        # Sends one round in its own transaction, without blocking past +timeout+ seconds.
+        #
+        # Closing a connection does not stop its statement: PostgreSQL notices a closed client only
+        # when it next writes to it. So the server must end a round that the client stopped waiting
+        # for. The timeouts (+round_timeouts+) end the heartbeat, and a transaction whose client
+        # stopped before COMMIT, by the deadline. A COMMIT that PostgreSQL has already started can
+        # still finish later. Both settings end with the transaction, so the connection keeps no
+        # session state.
+        #
+        # A round that expires before its heartbeat statement, while no statement is in flight,
+        # raises RoundExpired: with less than MINIMUM_REMAINING left before BEGIN it sends nothing,
+        # and after BEGIN it rolls back first (+expire+). Any other failure raises RoundTimeout or
+        # PG::Error, and the caller discards the connection, which rolls back.
         def bounded_rows(connection, sql, params, timeout)
-          connection.send_query_params(sql, params, 0, STRINGS)
-          raise RoundTimeout unless connection.block(timeout)
+          deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
+          raise RoundExpired if timeout < MINIMUM_REMAINING
+
+          bounded_result(connection, "BEGIN", [], deadline).clear
+          set_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          remaining = deadline - set_at
+          expire(connection, deadline) if remaining < MINIMUM_REMAINING
 
-          result = connection.get_last_result
-          begin
+          bounded_result(connection, ROUND_TIMEOUTS, round_timeouts(remaining), deadline).clear
+          # The budgets fit only if the settings statement used at most the quarter kept for COMMIT.
+          expire(connection, deadline) if Process.clock_gettime(Process::CLOCK_MONOTONIC) - set_at > remaining / 4
+
+          result = bounded_result(connection, sql, params, deadline)
+          rows = begin
             result.type_map = STRINGS
             result.to_a
           ensure
             result.clear
           end
+          bounded_result(connection, "COMMIT", [], deadline).clear
+          # A COMMIT that ends after the deadline does not count as a renewal.
+          raise RoundTimeout unless deadline > Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+          rows
+        end
+
+        # Rolls back a round that expired before its heartbeat statement, and raises RoundExpired. The
+        # ROLLBACK has the round's deadline, so it is not sent after it. If the ROLLBACK cannot be
+        # sent, fails, or does not end by the deadline, RoundTimeout or PG::Error goes to the
+        # caller, which discards the connection.
+        def expire(connection, deadline)
+          bounded_result(connection, "ROLLBACK", [], deadline).clear
+          raise RoundExpired
+        end
+
+        # The server budgets for the rest of a round with +remaining+ seconds left, in whole
+        # milliseconds. PostgreSQL measures +statement_timeout+ for each statement and
+        # +idle_in_transaction_session_timeout+ for each idle gap, and restarts both every time.
+        # While the round can hold a row lock, it has one idle gap before the heartbeat, the
+        # heartbeat, and one idle gap before COMMIT. So the heartbeat gets one half, each gap one
+        # eighth, and the last quarter is kept for COMMIT. No setting bounds COMMIT itself.
+        def round_timeouts(remaining)
+          milliseconds = (remaining * 1000).floor
+          [[1, milliseconds / 2].max.to_s, [1, milliseconds / 8].max.to_s]
+        end
+
+        # Sends one statement and waits for its result until +deadline+, a monotonic time. Nothing
+        # is sent after the deadline.
+        def bounded_result(connection, sql, params, deadline)
+          raise RoundTimeout unless deadline > Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+          connection.send_query_params(sql, params, 0, STRINGS)
+          remaining = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          raise RoundTimeout unless remaining.positive? && connection.block(remaining)
+
+          connection.get_last_result
         end
 
-        # A connection that overran a round still runs its statement, so it cannot go back to the
-        # pool. A pool without +discard_current_connection+ gets the connection back closed, and
+        # A connection that overran a round can still hold its transaction until the server
+        # timeouts end it, so it cannot go back to the pool. A pool without +discard_current_connection+ gets the connection back closed, and
         # its owner reconnects it before reuse.
         def discard(connection)
           if @source.respond_to?(:discard_current_connection)

data/lib/stablemates/workhorse/worker.rb

@@ -9,11 +9,15 @@ module Stablemates
     # tasks run at once. Every write an attempt makes carries its fence token, so an attempt that
     # lost its lease cannot change the task.
     #
-    # Durations are Numeric seconds. +run+ blocks until +stop+; +run_once+ claims and runs one
-    # batch. A stopping worker claims nothing new, waits up to +shutdown_grace+ for running
-    # handlers, cancels the rest with +:shutdown+, and raises ShutdownIncompleteError when any
-    # handler still runs after one short unwind window. A handler that raises the CancelledError of
-    # that cancellation, as +check!+ does, returns its task to the queue without charging the attempt.
+    # Durations are Numeric seconds. +heartbeat+ defaults to a third of +lease+, at least 0.1. The
+    # smallest supported +heartbeat+ is 0.1. The worker accepts a shorter value, but on a loaded
+    # host it can renew no lease.
+    #
+    # +run+ blocks until +stop+; +run_once+ claims and runs one batch. A stopping worker claims
+    # nothing new, waits up to +shutdown_grace+ for running handlers, cancels the rest with
+    # +:shutdown+, and raises ShutdownIncompleteError when any handler still runs after one short
+    # unwind window. A handler that raises the CancelledError of that cancellation, as +check!+
+    # does, returns its task to the queue without charging the attempt.
     class Worker
       MAX_EMPTY_POLL_MS = 5_000
       NOTIFICATION_POLL_MS = 5_000
@@ -72,10 +76,26 @@ module Stablemates
           @watchdog = nil
         end
 
-        def renew(sent_at) = @renewal.synchronize { @renewed_at = [@renewed_at, sent_at].max }
+        # Moves the local lease window to start at +sent_at+, and returns true. If the window has
+        # already ended, it stays closed and this returns false: a late accepted heartbeat cannot
+        # reopen it. The check and the move hold the lock that +lease_remaining+ holds, so they
+        # cannot race the watchdog.
+        def renew(sent_at)
+          @renewal.synchronize do
+            return false unless Process.clock_gettime(Process::CLOCK_MONOTONIC) < @renewed_at + @lease_seconds
+
+            @renewed_at = [@renewed_at, sent_at].max
+            true
+          end
+        end
 
         def lease_deadline = @renewal.synchronize { @renewed_at + @lease_seconds }
 
+        # The seconds until the local lease window ends, zero or less once it has ended.
+        def lease_remaining
+          @renewal.synchronize { @renewed_at + @lease_seconds - Process.clock_gettime(Process::CLOCK_MONOTONIC) }
+        end
+
         def start(unregister, watchdog)
           @unregister = unregister
           @watchdog = watchdog
@@ -1324,7 +1344,9 @@ module Stablemates
         end
         member = Heartbeat::Member.new(
           worker_id: @worker_id, task: task, lease_ms: @lease_ms, interval: @heartbeat_ms / 1000.0,
-          renew: ->(sent_at) { ownership.renew(sent_at) },
+          # An accepted heartbeat that arrives after the window ended ends the attempt, as the
+          # watchdog does.
+          renew: ->(sent_at) { ownership.renew(sent_at) || lose_lease(ownership, arbiter, cancellation) },
           settle: lambda do |status|
             ownership.settlement.synchronize do
               if %w[deadline_exceeded timeout_exceeded].include?(status)
@@ -1335,7 +1357,9 @@ module Stablemates
             end
           end,
           fail: fail_attempt,
-          logger: @logger
+          logger: @logger,
+          # The local lease window closes when the watchdog stops the attempt.
+          lease_ends_at: -> { ownership.lease_deadline unless ownership.stop.set? }
         )
         unregister = @heartbeat.register(member)
         watchdog = Thread.new { watch_expiration(task, ownership, arbiter, cancellation, parent, fail_attempt) }
@@ -1354,16 +1378,14 @@ module Stablemates
           elsif expiration_at
             expiration_at - Time.now
           end
-          lease_delay = ownership.lease_deadline - monotonic
+          lease_delay = ownership.lease_remaining
           if delay.nil? || lease_delay < delay
             if lease_delay.positive?
               return if ownership.stop.wait(lease_delay)
 
               next
             end
-            arbiter.submit(:lease_expired)
-            ownership.abandon
-            cancellation.cancel(:lease_lost)
+            lose_lease(ownership, arbiter, cancellation)
             return
           end
           return if ownership.stop.wait([0, delay].max)
@@ -1377,6 +1399,14 @@ module Stablemates
         fail_attempt.call(e)
       end
 
+      # Ends an attempt whose local lease window ended: the lease counts as lost, the heartbeat stops,
+      # and the handler is cancelled. The watchdog and a late accepted heartbeat both end it here.
+      def lose_lease(ownership, arbiter, cancellation)
+        arbiter.submit(:lease_expired)
+        ownership.abandon
+        cancellation.cancel(:lease_lost)
+      end
+
       def settle_expiration(task, arbiter, cancellation, parent)
         row = fenced_row(SqlCatalogue::EXPIRE_OWNED_TELEMETRY_V1, task)
         status = row["status"]
@@ -1477,7 +1507,7 @@ module Stablemates
       def settle_failure(task, error, arbiter)
         envelope = error_envelope(error, task.redact_error_details)
         state = Telemetry.span("workhorse.retry", Telemetry.task_span_attributes(task)) do |span|
-          delay = error.is_a?(ConflictError) ? -1 : retry_delay_override(task)
+          delay = durable_replay_conflict(error) ? -1 : retry_delay_override(task)
           state = fenced_row(SqlCatalogue::FAIL_V1, task, JSON.generate(envelope), delay&.to_s)["state"]
           Telemetry.set_attribute(span, "workhorse.retry.outcome", state.to_s)
           state
@@ -1509,6 +1539,20 @@ module Stablemates
         false
       end
 
+      # The durable replay conflict (ADR 0098) that +error+ is or wraps, or nil. A handler can wrap a
+      # conflict in its own error. The conflict is terminal also then, as in Go and Rust, and the
+      # failure envelope records the name of the conflict. The walk stays bounded against a cyclic
+      # cause chain.
+      def durable_replay_conflict(error)
+        16.times do
+          return nil if error.nil?
+          return error if error.is_a?(ConflictError)
+
+          error = error.cause
+        end
+        nil
+      end
+
       def retry_delay_override(task)
         delay = @retry_delay.respond_to?(:call) ? @retry_delay.call(task.attempt, task) : @retry_delay
         return nil if delay.nil?
@@ -1519,7 +1563,8 @@ module Stablemates
       def error_envelope(error, redact)
         return {"name" => REDACTED_NAME, "message" => REDACTED_MESSAGE} if redact
 
-        {"name" => error.class.name, "message" => error.message.to_s,
+        # A wrapped durable replay conflict keeps its name. The message and stack stay the outer ones.
+        {"name" => (durable_replay_conflict(error) || error).class.name, "message" => error.message.to_s,
          "stack" => error.full_message(highlight: false, order: :top)}
       end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: stablemates-workhorse
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 0.9.0
 platform: ruby
 authors:
 - Stablemates