stablemates-workhorse 0.6.0 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 98c5582b19c14aaf69dd9b317684c9a5e49a5508caf5759b79894fbe78c54146
-  data.tar.gz: 6f9a14b8dff61da2cadacc25f792c44a8b25281877b12bb47530a296b5a1e9e0
+  metadata.gz: e72b7d14542014971d81d818e073acaadbaef569c2516ba66427583e607fd0f3
+  data.tar.gz: 0e1d31974f65086473c750137249edd45722d75aa3b630e6c0a4c6c66bf7eb2e
 SHA512:
-  metadata.gz: 98805ebb4ddc387026073a530960c1494ea7fbfac0a294829b1152b0f300dbb551d1ae4c944a87fdf698e6a0debd96ddf825e55444c8dffb9b584beb74053e05
-  data.tar.gz: 81fef1e8bf1c9e5850b031aa5ac697a93b85ecbda2ae63687b60dfd6c2d821f08304b45e816dafadeea6105ef12820734e81086f3543bbfb5bb9bd7f857c9061
+  metadata.gz: 3de1cc0e31810df55903ec87b96e553edc96bfa2e7a699e6b926b669a995d6231ae03d5bd1ccfa62c8908d1ed6095e3a775998d092ac27b28fd92c8c9288cd20
+  data.tar.gz: 4683b2cebb636655c2c5c608d56059cedb5e6d837c0eb96735443aa78d2abad27371c3231737ae5be3d8cdfa504bf179a468f70cd15a038cf4516f0388e02459

data/CHANGELOG.md

@@ -3,6 +3,187 @@
 `stablemates-workhorse` gem versions and release notes live here. The gem carries the version the
 other SDKs carry, because every tag names one release of all of them.
 
+## 0.7.1 — 2026-10-09
+
+The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
+
+Requires **schema v54**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
+
+The gem has no changes in 0.7.1. It releases at 0.7.1 to keep one version across the
+registries ([ADR 0050](../docs/decisions/0050-release-0-1-0-without-a-prerelease-suffix.md)).
+The release exists because the npm publish of 0.7.0 stopped partway.
+The gem never published at 0.7.0, so 0.7.1 is the first release with the 0.7.0 changes below.
+
+**A 0.7.0 database needs no migration.** 0.7.1 adds no migration, so the final schema version is
+**64** and the SDK compatibility floor stays at schema version **54**. An installation on 0.6
+follows the [0.7.0 upgrade steps](https://github.com/stablemates/workhorse/blob/main/CHANGELOG.md#070--2026-10-09).
+
+## 0.7.0 — 2026-10-09
+
+The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
+
+Requires **schema v54**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
+
+### Upgrade steps
+
+**Migrate the schema before starting updated processes.** The upgrade is a rolling deployment: run `workhorse schema migrate`, then roll out the 0.7
+processes. The final schema version is **64**. The SDK compatibility floor is schema version
+**54**, which migration 0055 reaches. A 0.7 process refuses a schema below version 54 at startup,
+and a 0.6 process keeps working on version 64. Each migration commits on its own, and each SQL
+change takes effect when its migration commits.
+
+**Migration 0059 clears incomplete rate settings.** A per-key rate limit or a budget rate with one
+or two of its three fields null used to pass its CHECK constraint. The migration clears such a
+setting, and deletes a budget that the clearing leaves without a limit. After the migration,
+PostgreSQL rejects a synchronization that writes an incomplete rate setting. To keep a limit the
+migration cleared, complete it in your synchronized definitions and synchronize after the
+migration.
+
+**Accept the new `terminal-cleanup-backlog` health reason.** Queue health can now report it as
+degraded, so code that switches on reason codes should accept it. Its budget,
+`terminal_cleanup_backlog_ms`, starts from your `row_retention_lag_ms` setting, so existing alerts
+evaluate as before. After migration 0065 the two budgets are independent.
+
+**Update `dashboard/v1` clients that read `humanWait.context` from a task listing.**
+`dashboard.tasks` and `dashboard.tasksCursor` rows carry `humanWait.quickAction` instead, which is
+`{ label }` or `null`. `dashboard.taskDetail` still returns the full context (SM-1171).
+
+**Handler authors: a durable replay conflict now fails its task on the first occurrence.** The
+behaviour changes below describe it. A 0.6 worker still retries a conflict under the task's retry
+policy until the 0.6 processes are gone.
+
+### Breaking changes
+
+- **`dashboard/v1`:** task listing rows carry `humanWait.quickAction` instead of
+  `humanWait.context`, as the upgrade steps describe. The break is taken in place under the ADR 0064
+  exception in [`docs/compatibility.md`](../docs/compatibility.md) (SM-1171).
+
+### Behaviour changes
+
+Durable execution and workers:
+
+- Durable checkpoint, timer, child, child-set, and human-decision replay conflicts fail the task on
+  their first occurrence. The worker preserves the current attempt and records the conflict class.
+  Before, a conflict retried under the task's retry policy, and every retry met the same conflict.
+  Redaction still hides error details. Transient failures, lease loss, child-limit errors, and
+  already-waiting signal errors keep their existing behaviour. Conflict settlement bypasses the
+  worker's retry-delay callback (SM-1107, migration 0055).
+- A renamed individual child on replay raises `ConflictError` with the stored and requested names,
+  and `ConflictError.new` takes a `stored_name:` keyword. Before, it raised a child-limit error. A
+  second child after joining the retained child in the same handler run still exceeds the child
+  limit (SM-1106, migration 0054).
+- A handler that raises the `CancelledError` of its `:shutdown` cancellation no longer charges an
+  attempt. The worker hands its task to `release_owned_v1`, as it does for a task without a handler.
+  Before, `CancelledError(:shutdown)` reached `fail_v1`, so a task on its last attempt failed
+  because its worker stopped (SM-1164).
+- The `workhorse.handler` span descends only from the task's stored trace context. A task without
+  one starts a new trace instead of joining a span that a context-propagating executor carries into
+  the handler thread (SM-1179).
+
+Dashboard host:
+
+- `Dashboard.new(audit_actor: ...)` no longer replaces the actor of a `Principal` that `authorize`
+  returns. The dashboard records the principal's actor, as the `dashboard/v1` protocol requires.
+  `audit_actor` still names the actor when `authorize` returns `true`, and defaults to `dashboard`
+  there (SM-1152).
+- `Dashboard` refuses an RPC body over 2 MiB, or a malformed declared length, with `413` before
+  parsing. An unexpected schema-compatibility failure, such as a connection error, answers a generic
+  `503` and is written to `rack.errors` instead of escaping the Rack app (SM-1168).
+
+Maintenance, health, and administration, from the SQL changes:
+
+- Terminal cleanup repeats its batch while each one fills, for up to one second per pass. A pass
+  that still ends with a full batch makes its follow-up due five seconds later. Full-tier and
+  fast-tier tasks share every batch (SM-1160, migration 0058).
+- `tick_v1` runs the expired-lease scan only when no tick ran it within half the shortest
+  maintenance interval of the live registered workers. A worker that opts out of the registry with
+  `disable_registry: true` does not shorten that spacing. Recovery of an expired lease can then take
+  longer (SM-1167, SM-1192, migration 0061).
+- Row retention lag counts from the scheduled history pass that released the row, so health no
+  longer reports task records and finished results as late after every daily pass (SM-1134,
+  migration 0056).
+- Queue health raises `terminal-cleanup-backlog` once `terminal_cleanup_backlog_since` is older than
+  its budget (SM-1178, SM-1197, migrations 0063 and 0065).
+- `run_task_now_v1` reports `not_scheduled` for a blocked task instead of raising an error. A rate
+  synchronization refills each changed bucket at the old rate up to one clock reading. An update
+  can no longer change a dependency edge's endpoints or outcome policies (SM-1165, migration 0059).
+- A full-tier task that a worker without a handler released can be canceled, and deadline recovery
+  terminalizes it. Both close it like never-started work (SM-1158, migration 0057).
+
+Dashboard, from the shared browser bundle:
+
+- Pausing a schedule asks for confirmation, as resuming one already did (SM-1147).
+- The Events pager stops at the last page the server accepts and offers older events through a
+  custom range. The activity chart shows loading, error, and stale states with a Retry button
+  (SM-1171).
+- The Schedules page says how many schedules past the first 50 it does not show, and the System
+  page lists the terminal cleanup backlog check (SM-1171, SM-1178).
+
+### Migrations
+
+Migration 0054 is the first since 0.6.1. The
+[root changelog](https://github.com/stablemates/workhorse/blob/main/CHANGELOG.md) describes each in
+full.
+
+- **0054** adds versioned child functions and the nullable `task_child.last_seen_fence_token`
+  fence marker. Older clients keep their v1 functions (SM-1106).
+- **0055** reserves a retry delay of -1 in `fail_v1` and `fast_fail_v1` as the terminal failure
+  override. It is the SDK compatibility floor, schema version 54 (SM-1107).
+- **0056** changes `queue_health_v1` to count row retention lag from the releasing history pass
+  (SM-1134).
+- **0057** changes `cancel_v1` and `terminalize_deadline_v1` to close a released task like
+  never-started work (SM-1158).
+- **0058** lets terminal cleanup keep pace and share its batches across tiers, and adds two
+  `maintenance_state` columns (SM-1160).
+- **0059** closes five SQL integrity gaps. It adds `rate_limit_policy_per_key_complete_check`,
+  `budget_rate_complete_check`, and a trigger that rejects a dependency edge update (SM-1165).
+- **0060** judges `fast_complete_many_v1` and `fast_acknowledge_cancel_v1` at the time they act
+  (SM-1163).
+- **0061** adds `fast_task_outcome_failed_finished_idx`, bounds `aggregate_stats_v1`, and spaces the
+  expired-lease scan in `tick_v1` with the new `maintenance_state.lease_recovery_started_at` column
+  (SM-1167).
+- **0062** adds `dashboard_human_wait_quick_action_v1` and bounds four dashboard read functions
+  (SM-1171).
+- **0063** raises the `terminal-cleanup-backlog` health reason and adds
+  `terminal_cleanup_follow_up_delay_ms_v1` (SM-1178).
+- **0064** disables JIT for `aggregate_stats_v1` (SM-1193).
+- **0065** gives `terminal-cleanup-backlog` its own budget in `queue_health_policy`, adds
+  `sync_queue_health_policy_v2`, and keeps `sync_queue_health_policy_v1` (SM-1197).
+
+### Added
+
+- `Queue#health` reports the backlog budget as `budgets.terminalCleanupBacklogMs` with schema 64,
+  and `status.reasons` can carry `terminal-cleanup-backlog` (SM-1178, SM-1197).
+
+### Dependencies
+
+- The gemspec declares `logger` at least 1.6 and below 2. Ruby 4.0 no longer ships `logger` as a
+  default gem. On Ruby 4.0, 0.6.1 raises `LoadError` at `require "stablemates/workhorse"` unless
+  the application already installs `logger` (SM-1138).
+- The dashboard requires `cgi/escape` instead of `cgi`, which Ruby 4.0 reduces to a stub that warns
+  (SM-1138).
+
+### Recipes
+
+- A new tested recipe enqueues through Sequel on its native `pg` connection (SM-1114).
+- The quick start and the agent playbook print their results as JSON (SM-1139), and a batch handler
+  example shows how batch-handler tasks are enqueued (SM-1135).
+
+## 0.6.1 — 2026-10-02
+
+The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.
+
+Requires **schema v43**, Ruby **3.3** or newer, and PostgreSQL **15** or newer.
+
+The gem has no changes in 0.6.1. It releases at 0.6.1 to keep one version across the
+registries ([ADR 0050](../docs/decisions/0050-release-0-1-0-without-a-prerelease-suffix.md)).
+The release exists for the Python distribution, whose 0.6.0 files on PyPI have no PEP 740
+attestations.
+
+**A 0.6.0 database needs no migration.** 0.6.1 adds no migration, so the final schema version is
+**52** and the SDK compatibility floor stays at schema version **43**. An installation on 0.5
+follows the [0.6.0 upgrade steps](../CHANGELOG.md#060--2026-10-02).
+
 ## 0.6.0 — 2026-10-02
 
 The npm packages, Python distribution, Go module, Rust crate, and Ruby gem release from one source commit.

data/README.md

@@ -7,10 +7,7 @@ PostgreSQL.
 > release may change behaviour, so read the
 > [changelog](https://github.com/stablemates/workhorse/blob/main/ruby/CHANGELOG.md) before you
 > upgrade. It will not ask you to recreate your database: migrations are ordered, and inside a major
-> line a migration only adds, so a running deployment upgrades in place. The one exception is
-> migration 0025: a database from before 0.5.0 crosses it offline, with the
-> [0.5.0 upgrade steps](https://github.com/stablemates/workhorse/blob/main/CHANGELOG.md#050--2026-09-28).
-> The upgrade from 0.5 to 0.6 only adds.
+> line a migration only adds, so a running deployment upgrades in place.
 
 An AI agent should read [the Workhorse documentation index](https://workhorse.run/llms.txt) first.
 
@@ -26,15 +23,17 @@ bundle add connection_pool
 Install the schema once, as a deployment step. The application never installs or migrates it.
 
 ```bash
-npx --package @stablemates/workhorse@0.6.0 workhorse schema install
+npx --package @stablemates/workhorse@0.7.1 workhorse schema install
 ```
 
 The machine that runs that deployment step needs Node.js 22 or newer. The application itself needs
 no Node.js.
 
-Pin that version to the gem version the application depends on. The two come from one commit of
-this repository, so the numbers match. A schema tool older than the application leaves a schema the
-application refuses to start against.
+Pin that version to the gem version the application depends on. The two come from one commit of this
+repository, so the numbers match. The application refuses a schema below its schema floor, the
+oldest schema version it runs against. A schema at or above the floor is compatible while it still
+serves the application's protocol. A schema tool from an older release can leave a schema below that
+floor.
 
 Runtime processes verify compatibility instead of changing the schema. Call
 `Queue#assert_compatible` or `Admin#assert_compatible` at startup. A refusal is
@@ -46,6 +45,7 @@ Requires Ruby 3.3 or newer and PostgreSQL 15 through 18.
 
 ```ruby
 require "connection_pool"
+require "json"
 require "pg"
 require "stablemates/workhorse"
 
@@ -62,7 +62,7 @@ end
 worker.run_once # production uses Stablemates::Workhorse.run_worker_process(worker)
 
 task = Stablemates::Workhorse::Admin.new(pool).get_task(enqueued.task_id)
-puts "#{task.state} #{task.result}" if task
+puts "#{task.state} #{JSON.generate(task.result)}" if task
 ```
 
 Handlers receive at-least-once delivery. Use stable provider idempotency keys around external

data/lib/stablemates/workhorse/context.rb

@@ -215,12 +215,12 @@ module Stablemates
         encoded = initial(build)
         once(:run_child, name, encoded) do
           live!(:run_child)
-          row = write(SqlCatalogue::CREATE_CHILD_V1, name, encoded)
-          row = replayed(row, encoded, build) { |accepted| write(SqlCatalogue::CREATE_CHILD_V1, name, accepted) }
+          row = write(SqlCatalogue::CREATE_CHILD_V2, name, encoded)
+          row = replayed(row, encoded, build) { |accepted| write(SqlCatalogue::CREATE_CHILD_V2, name, accepted) }
           status = row.fetch("status")
           case status
           when "stale" then raise LeaseLostError.new(@task.id, :run_child)
-          when "conflict" then raise ConflictError.new(:run_child, name)
+          when "conflict" then raise ConflictError.new(:run_child, name, stored_name: row["stored_child_name"])
           when "limit_exceeded" then raise LimitExceededError.new(:run_child, name)
           when "created", "completed"
             processed(status == "created" && :suspended_for_child) do

data/lib/stablemates/workhorse/dashboard/bundle.json

@@ -2,5 +2,5 @@
   "formatVersion": 1,
   "readSurfaceVersion": 1,
   "archive": "read-surface-1.tar.gz",
-  "sha256": "67459299efcee8926d357badc947113529399453d08d59d4f240bf06f022f539"
+  "sha256": "38ca62438266d41ab61f9ea0c84b1264b8a2397896a1db75b22ca90c0b199fc0"
 }

data/lib/stablemates/workhorse/dashboard.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require "cgi"
+require "cgi/escape"
 require "json"
 require "rubygems/package"
 require "stringio"
@@ -62,6 +62,10 @@ module Stablemates
         ".woff2" => "font/woff2"
       }.freeze
       JSON_TYPE = "application/json; charset=utf-8"
+      # The largest RPC request body the dashboard reads, the same bound the Go and Rust hosts apply.
+      MAX_REQUEST_BYTES = 2 << 20
+      UNVERIFIED_COMPATIBILITY =
+        "Unable to verify Workhorse schema compatibility because the database query failed."
       DEFAULT_PORTS = {"http" => ":80", "https" => ":443"}.freeze
       # Characters escaped in the runtime configuration so it cannot close its inline script.
       SCRIPT_ESCAPES = {"<" => "\\u003c", ">" => "\\u003e", "&" => "\\u0026",
@@ -74,8 +78,8 @@ module Stablemates
         "issues" => [{"code" => "custom", "path" => ["feature"],
                       "message" => "The feature demo kind requires a feature family"}]
       }.freeze
-      private_constant :MUTATIONS, :OPTIONAL_MUTATIONS, :CONTENT_TYPES, :JSON_TYPE, :DEFAULT_PORTS,
-        :SCRIPT_ESCAPES, :TOO_SMALL_PAGE, :MISSING_FEATURE
+      private_constant :MUTATIONS, :OPTIONAL_MUTATIONS, :CONTENT_TYPES, :JSON_TYPE, :MAX_REQUEST_BYTES,
+        :UNVERIFIED_COMPATIBILITY, :DEFAULT_PORTS, :SCRIPT_ESCAPES, :TOO_SMALL_PAGE, :MISSING_FEATURE
 
       attr_reader :base_path
 
@@ -122,12 +126,16 @@ module Stablemates
         return authorization if authorization.is_a?(Array)
         return json(401, {"error" => "Unauthorized"}) if authorization == false
 
-        principal = (authorization == true) ? Principal.new(@audit_actor || "dashboard") : authorization
-        actor = @audit_actor || principal.actor
+        # A verified principal names the actor; the audit actor stands in only for a bare true.
+        actor = (authorization == true) ? (@audit_actor || "dashboard") : authorization.actor
         begin
           assert_compatible
-        rescue Error => e
+        rescue CompatibilityError => e
           return json(503, {"error" => e.message})
+        rescue => e
+          # A driver error can name the database host, user, or path, so it stays in the log.
+          env["rack.errors"]&.puts("workhorse dashboard: schema compatibility check failed: #{e.class}: #{e.message}")
+          return json(503, {"error" => UNVERIFIED_COMPATIBILITY})
         end
 
         # A mounting stack may hand over the mount point itself as SCRIPT_NAME plus a PATH_INFO of "/".
@@ -201,9 +209,11 @@ module Stablemates
           return rpc_error(404, "NOT_FOUND", "Procedure not found")
         end
 
+        body = read_body(env)
+        return rpc_error(413, "PAYLOAD_TOO_LARGE", "Payload Too Large") if body.nil?
+
         input = nil
         begin
-          body = env["rack.input"]&.read.to_s
           envelope = JSON.parse(body.empty? ? "{}" : body)
           raise InputValidationError, "request envelope must be an object" unless envelope.is_a?(Hash)
 
@@ -231,6 +241,17 @@ module Stablemates
         json(200, result.nil? ? {} : {"json" => result})
       end
 
+      # At most MAX_REQUEST_BYTES of the request body, or nil for a body that is larger.
+      def read_body(env)
+        declared = env["CONTENT_LENGTH"].to_s
+        unless declared.empty?
+          return nil unless declared.match?(/\A\d+\z/) && declared.to_i <= MAX_REQUEST_BYTES
+        end
+
+        body = env["rack.input"]&.read(MAX_REQUEST_BYTES + 1).to_s
+        (body.bytesize > MAX_REQUEST_BYTES) ? nil : body
+      end
+
       def same_origin?(env)
         origin = env["HTTP_ORIGIN"]
         return false unless origin.is_a?(String)

data/lib/stablemates/workhorse/errors.rb

@@ -41,7 +41,14 @@ module Stablemates
 
     # A named primitive received a different request under its retained identity.
     class ConflictError < NamedOperationError
-      def initialize(operation, name) = super(operation, name, "conflicts with a different retained request")
+      def initialize(operation, name, stored_name: nil)
+        message = if stored_name && stored_name != name
+          "conflicts: stored child #{stored_name.inspect}, requested child #{name.inspect}"
+        else
+          "conflicts with a different retained request"
+        end
+        super(operation, name, message)
+      end
     end
 
     # A named primitive exceeded its limit.

data/lib/stablemates/workhorse/sql_catalogue_generated.rb

@@ -10,8 +10,8 @@ module Stablemates
       CLIENT_PROTOCOL_VERSION = 5
       MINIMUM_PROTOCOL_VERSION = 5
       MAXIMUM_PROTOCOL_VERSION = 5
-      MINIMUM_SCHEMA_VERSION = 43
-      MAXIMUM_SCHEMA_VERSION = 52
+      MINIMUM_SCHEMA_VERSION = 54
+      MAXIMUM_SCHEMA_VERSION = 64
       # PostgreSQL's atomic enqueue batch limit.
       MAX_ENQUEUE_BATCH_SIZE = 1000
       DEFAULT_TASK_VALUE_MAX_BYTES = 1048576
@@ -40,6 +40,9 @@ module Stablemates
       # `create_child_v1` (protocol)
       CREATE_CHILD_V1 = 'SELECT status, child_task_id, child_type, created_at, joined_at, result FROM workhorse.create_child_v1($1::uuid, $2::text, $3::bigint, $4::text, $5::jsonb)'
 
+      # `create_child_v2` (protocol)
+      CREATE_CHILD_V2 = 'SELECT status, child_task_id, child_type, created_at, joined_at, result, stored_child_name FROM workhorse.create_child_v2($1::uuid, $2::text, $3::bigint, $4::text, $5::jsonb)'
+
       # `create_children_v1` (protocol)
       CREATE_CHILDREN_V1 = 'SELECT status, children, results, result_bytes, result_limit_bytes FROM workhorse.create_children_v1($1::uuid, $2::text, $3::bigint, $4::jsonb, $5::text)'
 
@@ -867,6 +870,8 @@ module Stablemates
         WITH queue_names AS (
           SELECT queue_name FROM workhorse.task_runtime
           UNION
+          SELECT queue_name FROM workhorse.fast_task_runtime
+          UNION
           SELECT queue_name FROM workhorse.queue_control
         ), depth AS (
           SELECT names.queue_name,
@@ -886,7 +891,17 @@ module Stablemates
        WHERE runtime.state = \'active\' AND runtime.attempt_timeout_at <= clock_timestamp()
      )::text AS overdue_execution_timeouts
       FROM queue_names names
-      LEFT JOIN workhorse.task_runtime runtime
+      LEFT JOIN (
+        SELECT task_id, queue_name, state, ready_at, expires_at, deadline_at, attempt_timeout_at
+          FROM workhorse.task_runtime
+        UNION ALL
+        SELECT task_id, queue_name,
+               CASE WHEN state = \'ready\' AND run_at > clock_timestamp() THEN \'scheduled\'
+                    ELSE state END,
+               CASE WHEN state = \'ready\' THEN run_at END, expires_at, deadline_at,
+               attempt_timeout_at
+          FROM workhorse.fast_task_runtime
+      ) runtime
         ON runtime.queue_name = names.queue_name
      GROUP BY names.queue_name
         )
@@ -956,6 +971,7 @@ module Stablemates
       QUEUE_METRIC_SNAPSHOT = 'WITH queue_names AS (
          SELECT $1::text AS queue_name
          UNION SELECT queue_name FROM workhorse.task_runtime
+         UNION SELECT queue_name FROM workhorse.fast_task_runtime
          UNION SELECT queue_name FROM workhorse.queue_control
          UNION SELECT queue_name FROM workhorse.concurrency_policy
          UNION SELECT queue_name FROM workhorse.rate_limit_policy
@@ -977,7 +993,16 @@ module Stablemates
        WHERE runtime.state = \'ready\'
      )) * 1000 AS oldest_ready_age_ms
       FROM queue_names names
-      LEFT JOIN workhorse.task_runtime runtime
+      LEFT JOIN (
+        SELECT task_id, queue_name, state, ready_at, expires_at
+          FROM workhorse.task_runtime
+        UNION ALL
+        SELECT task_id, queue_name,
+               CASE WHEN state = \'ready\' AND run_at > clock_timestamp() THEN \'scheduled\'
+                    ELSE state END,
+               CASE WHEN state = \'ready\' THEN run_at END, expires_at
+          FROM workhorse.fast_task_runtime
+      ) runtime
         ON runtime.queue_name = names.queue_name
      GROUP BY names.queue_name
        )

data/lib/stablemates/workhorse/telemetry.rb

@@ -63,9 +63,14 @@ module Stablemates
 
       # Yields a span, or nil without OpenTelemetry. A +trace_context+ carrier takes precedence
       # over +parent+. The span records no exception by itself; +record_error+ marks a failure.
+      #
+      # A consumer span runs a claimed task, which descends only from the trace its enqueue stored.
+      # A span current where the worker runs is unrelated to the task, so a consumer span starts
+      # from an empty context, and a task without a stored context starts a new trace.
       def span(name, attributes, trace_context: nil, parent: nil, consumer: false)
         return yield(nil) unless tracing?
 
+        parent = ::OpenTelemetry::Context.empty if consumer
         if trace_context.is_a?(Hash)
           carrier = trace_context.each_with_object({}) do |(key, value), result|
             result[key.downcase] = value if key.is_a?(String) && value.is_a?(String)

data/lib/stablemates/workhorse/version.rb

@@ -2,6 +2,6 @@
 
 module Stablemates
   module Workhorse
-    VERSION = "0.6.0"
+    VERSION = "0.7.1"
   end
 end

data/lib/stablemates/workhorse/worker.rb

@@ -12,7 +12,8 @@ module Stablemates
     # Durations are Numeric seconds. +run+ blocks until +stop+; +run_once+ claims and runs one
     # batch. A stopping worker claims nothing new, waits up to +shutdown_grace+ for running
     # handlers, cancels the rest with +:shutdown+, and raises ShutdownIncompleteError when any
-    # handler still runs after one short unwind window.
+    # handler still runs after one short unwind window. A handler that raises the CancelledError of
+    # that cancellation, as +check!+ does, returns its task to the queue without charging the attempt.
     class Worker
       MAX_EMPTY_POLL_MS = 5_000
       NOTIFICATION_POLL_MS = 5_000
@@ -1182,10 +1183,16 @@ module Stablemates
           attributes.merge("workhorse.handler.outcome" => outcome))
       end
 
-      # Hands a task with no handler back to PostgreSQL, which makes it claimable again without
-      # charging the attempt. Returns the release status for the handler span.
+      # Hands a task with no handler back to PostgreSQL. Returns the release status for the handler
+      # span.
       def release(task, arbiter, attributes)
         log(:warn, "workhorse.handler.missing", "No handler registered for the claimed task type", attributes)
+        release_owned(task, arbiter, attributes)
+      end
+
+      # Hands an owned task back to PostgreSQL, which makes it claimable again without charging the
+      # attempt. Returns the release status.
+      def release_owned(task, arbiter, attributes)
         status = fenced_row(SqlCatalogue::RELEASE_OWNED_V1, task)["status"]
         log(:info, "workhorse.task.release_processed", "Owned task release processed",
           attributes.merge("workhorse.release.status" => status))
@@ -1217,6 +1224,12 @@ module Stablemates
         rescue => e
           return if finish_suspended(task, ownership, arbiter)
 
+          # A handler that stopped for shutdown returns the task to the queue without an attempt.
+          if cancellation.reason == :shutdown && shutdown_cancellation?(e)
+            state[:span_outcome] = release_owned(task, arbiter, Telemetry.task_span_attributes(task))
+            return
+          end
+
           state[:span_outcome] = settle_failure(task, e, arbiter)
           state[:errors] << (task.redact_error_details ? REDACTED_NAME : e.class.name)
           return
@@ -1453,7 +1466,7 @@ module Stablemates
       def settle_failure(task, error, arbiter)
         envelope = error_envelope(error, task.redact_error_details)
         state = Telemetry.span("workhorse.retry", Telemetry.task_span_attributes(task)) do |span|
-          delay = retry_delay_override(task)
+          delay = error.is_a?(ConflictError) ? -1 : retry_delay_override(task)
           state = fenced_row(SqlCatalogue::FAIL_V1, task, JSON.generate(envelope), delay&.to_s)["state"]
           Telemetry.set_attribute(span, "workhorse.retry.outcome", state.to_s)
           state
@@ -1472,6 +1485,19 @@ module Stablemates
         state
       end
 
+      # Whether +error+, or an error it wraps, is a +:shutdown+ CancelledError. With the token's own
+      # reason, it proves the handler saw the stop, so a failure of its own that races the grace
+      # still charges its attempt. The walk stays bounded against a cyclic cause chain.
+      def shutdown_cancellation?(error)
+        16.times do
+          return false if error.nil?
+          return true if error.is_a?(CancelledError) && error.reason == :shutdown
+
+          error = error.cause
+        end
+        false
+      end
+
       def retry_delay_override(task)
         delay = @retry_delay.respond_to?(:call) ? @retry_delay.call(task.attempt, task) : @retry_delay
         return nil if delay.nil?

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: stablemates-workhorse
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.1
 platform: ruby
 authors:
 - Stablemates
@@ -49,6 +49,26 @@ dependencies:
     - - "<"
       - !ruby/object:Gem::Version
         version: '4'
+- !ruby/object:Gem::Dependency
+  name: logger
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.6'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.6'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
 - !ruby/object:Gem::Dependency
   name: pg
   requirement: !ruby/object:Gem::Requirement