ace-herdr 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 11805892b3ae5bdf74d4bdd555f8e508e7fe3693476d07fa537a41b9f891857e
-  data.tar.gz: 2aa907e23b109838153215ac004f344045220d2c77cf0f295a99c6ff55a75cb2
+  metadata.gz: a3c3149a40374844ef590a17e1ff5908daf08153ebf64f3d82f776efb22985b8
+  data.tar.gz: 5c8ba6b8316a3c5f3808a138e7b4e543006107286edefa6d9b5fc12403a975d8
 SHA512:
-  metadata.gz: 53e2cbd3f3c97db17b5e14609e54deb210e0d7474886dc4f6fe7d78a33154df1cb7fe680a565aa500d534c0ce06235dd20445f9ae6bb867d5b989732c5a5fa70
-  data.tar.gz: 9397dfe3b7fb8ac0d9b42f2a43ad7ac56204c7b70197e5f1a6095a00f62f38ecb8d7a0967ca0ab5d68c91864656c86db25a9cc9bc0274fa65c63d74216771e09
+  metadata.gz: ffd4b11b53a480492c625a3f43bab4710a732ef0dda2dae9d84bf5caa9a15aba156c4f9484abf47ac0f36c2caed0c8f36947c3788999b8e21a4070c8c3790e21
+  data.tar.gz: '03794e613cbce4ea3edcfdc06e8e04bff70d0e75afa8e3cc0755b85eceddbed6f79a2a2f1f245b6171dc140569b63353b9646746d1fdec78df2922d08558ccc5'

data/.ace-defaults/herdr/config.yml

@@ -19,6 +19,10 @@ timeouts:
 # Delivery record root (ADR-029 short-name layout), relative to project root
 deliveries_dir: .ace-local/herdr/deliveries
 
+# Absolute path to the supervisor-managed receipt verification key. Configure
+# before enqueue; each event pins this key's fingerprint for reconciliation.
+inbox_receipt_public_key:
+
 # Tidy (spec 8wq.t.1w0): archive delivered records strictly older than this
 # many days (age = record updated_at); failed/retryable/pending are never
 # touched

data/CHANGELOG.md

@@ -5,6 +5,34 @@ All notable changes to `ace-herdr` will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.3.0] - 2026-10-02
+
+### Added
+- Durable `inbox enqueue`, `status`, and `deliver` commands for attempt-linked messages. Enqueue binds the live pane, terminal, agent, and native session; per-event locked claims and saved submission intent prevent duplicate dispatch after a crash. Codex and Pi use exact-session native queues, and idle agents receive a bounded, payload-free Herdr wake after accepted submission.
+- Inbox status reports claim generation, target binding, accepted submission receipt, and uncertainty. `reconcile` accepts an operator or supervisor native-outcome receipt file only when its trusted detached signature and event, attempt, generation, digest, and full bound identity match; missing or mismatched proof returns a machine-readable refusal and keeps the event uncertain.
+- Installed acceptance covers Codex delivery, signed uncertainty reconciliation, and idle Pi delivery; bounded wake tests cover subprocess stalls and structured errors.
+
+### Fixed
+- Post-launch subprocess I/O failures terminate the child process group before surfacing, so a failed run can never leave an orphaned native client behind past the deadline.
+- Post-launch subprocess I/O failures are classified separately from spawn failures, so a native client that already ran keeps its event uncertain instead of being re-queued for a duplicate submission; and a pane that changed agent is classified as target drift before agent-specific validation can mask it.
+- Unreadable or malformed reconciliation receipt files surface as the documented machine-readable refusal while persistence failures still exit as command errors.
+- Orphaned claims recover by proof: a claim saved before any submission intent requeues as retryable, while only intent-carrying claims stay uncertain; every spawn-time failure is classified at the process boundary as proven pre-launch (retryable) across both executors; and the inbox CLI reports persistence failures as command failures instead of receipt refusals.
+- Target identity drift is classified before agent-specific event id rules, so a bound event whose pane changed agent lands in reconcilable uncertainty instead of looping on a pre-send validation error.
+- Native queue bindings require an immutable session id for both Codex and Pi, so an agent restarting under the same name in a reused pane can never inherit an old event; signed replacement targets are validated against the replacement agent's own event-id and payload rules before a supersession re-binds the event.
+- Pane probe responses are shape-checked and exit-checked before use, so malformed or failed `herdr pane get` output becomes a classified retryable error instead of stranding a claimed event or delivering against a failed observation; the Pi identity probe rejects non-object JSON the same way.
+- The pane identity probe runs under a bounded child deadline (process-group kill), so a stalled herdr pane get can no longer hold the per-event inbox lock; a probe timeout classifies as a retryable pre-submission failure. The idle-wake decision rides the first delivered transition (a busy target can never be prompted by a crash-recovery retry), and enqueue rejects payloads that are not valid UTF-8 text with an actionable validation error.
+- Pane observation launch failures are normalized to executor unavailability so a delivery probe can never strand an event in `claimed`; idle-wake retry outcomes persist across restarts; reverse-address files that are not JSON objects fail enqueue with an actionable validation error instead of crashing.
+- Enqueue applies the target agent's native payload limit (Codex argv bound, Pi body bound) before persisting an event, validates recognized agent statuses before submission (undetermined statuses stay retryable pre-send), and shares one Pi event id validator between enqueue and delivery, so events that could never be delivered are rejected up front instead of queuing forever.
+- Idle-agent wakes are persisted as pending before they are attempted and retried by later deliver calls (re-verifying the live target; identity drift demotes the event to uncertain) without ever resubmitting the native payload, so a crash or transient wake failure can no longer leave an accepted message undelivered to an idle agent. Pi targets now enforce the `inb-`/`wnk-` event id constraint at enqueue instead of queuing an event that can never be delivered.
+- Native queue submission and the Pi identity probe enforce their deadline at the child boundary through a shared bounded-process molecule (process-group kill, bounded pipe draining, nonblocking partial stdin writes). A stalled or non-reading native client can no longer hold the per-event inbox lock past the configured timeout the way a cleanup-blocking `Timeout.timeout` around `Open3.capture3` allowed. Codex payloads are size-bounded before launch and a spawn-time `E2BIG` is classified as a proven pre-submission rejection, keeping the event retryable instead of stranding it uncertain.
+
+## [0.2.0] - 2026-10-02
+
+### Added
+- `Ace::Herdr::Organisms::RuntimeAdapter` and the lazy `ace/runtime/adapters/herdr` entrypoint implement the shared ace-runtime intent contract. It preserves native agent prompt and wait semantics, polls all four lifecycle observations, and maps Herdr failures to contract errors. Native tab and pane probes support context, focus, and retained pane preparation. The packaged shared adapter contract suite runs against a Herdr-shaped fake executor.
+
 ## [0.1.0] - 2026-09-27
 
 ### Added

data/README.md

@@ -26,12 +26,15 @@
 4. Retryable failures back off on a fixed deterministic schedule; terminal failures are reported and persisted in the delivery record.
 5. `ace-herdr dispatch`, `wait`, and `close` give agents one-command subagent lifecycle with sensible defaults (same workspace as the caller, label = task id, prompt from file or stdin).
 
+The `inbox` command records one attempt-linked agent message before native submission. It binds the live terminal, agent session, and configured receipt verification key at enqueue, serializes claims by event ID, and exposes uncertain outcomes through `inbox status`. An uncertain event stays uncertain until `inbox reconcile --receipt FILE` receives a matching, signed operator or supervisor observation of consumption or supersession; missing or mismatched proof returns a JSON refusal without resending. See the [usage guide](docs/usage.md#durable-agent-inbox) for states, key configuration, and receipt shape.
+
 ## Use Cases
 
 - Push an HITL answer to the pane that asked, even if its agent was restarted since.
 - Dispatch a subagent in one command with deterministic defaults and no LLM decisions.
 - Wait for agent readiness or completion without scraping pane noise.
 - Close out finished agent panes (rename/close) and return their result.
+- Use Herdr through the shared `ace-runtime` Ruby intent API, with workspace/session and tab/window mapping handled by the adapter.
 
 ## Installation
 

data/docs/usage.md

@@ -3,8 +3,8 @@ doc-type: user
 title: ace-herdr Usage
 purpose: Full CLI and configuration reference for ace-herdr: push delivery, agent bootstrap, the terminal-control surface (list, send, capture, wait, presets), and tidy cleanup.
 ace-docs:
-  last-updated: 2026-09-28
-  last-checked: 2026-09-28
+  last-updated: 2026-10-02
+  last-checked: 2026-10-02
 ---
 
 # Usage
@@ -14,6 +14,7 @@ ace-docs:
 ## Command Surface
 
 - `ace-herdr deliver [OPTIONS]`
+- `ace-herdr inbox enqueue|status|deliver|reconcile [OPTIONS]`
 - `ace-herdr dispatch [OPTIONS]`
 - `ace-herdr list [--panes|--tabs|--workspaces] [--workspace ID] [--quiet]`
 - `ace-herdr send [--cmd TEXT] [--msg TEXT...] [--key NAME...] --pane ID [--quiet]`
@@ -31,6 +32,60 @@ Control commands print exactly one deterministic JSON line on stdout; `--quiet`
 
 herdr *sessions* (server persistence) are intentionally not exposed; the tmux session analogue is the herdr **workspace**.
 
+## ace-runtime adapter
+
+Install `ace-herdr` alongside `ace-runtime`, then resolve the adapter with `Ace::Runtime.resolve("herdr")`. The contract's `session` is a Herdr **workspace**, `window` is a **tab**, and `pane` is an opaque Herdr pane ID. The adapter does not construct pane IDs. It locates the caller through `HERDR_SESSION` and `HERDR_PANE` and reads the explicit pane with `pane get`; the pane's native workspace is authoritative and a conflicting `HERDR_WORKSPACE_ID` hint is rejected instead of honored. Outside Herdr, `context` reports `in_runtime: false`; operations requiring a caller workspace raise `Ace::Runtime::RuntimeUnavailableError`.
+
+The adapter implements `context`, `ensure_window`, `prepare_pane`, `focus`, ordered `send` and its convenience methods, `capture`, `wait_output`, `wait_agent`, `wait_lifecycle`, `close_window`, `list_windows`, and `list_panes`. `ensure_window` scopes the sanitized tab label to the caller workspace. It records the tab's root, preset, and prepared pane under `~/.ace/local/herdr/runtime-tabs/` — keyed by workspace and label, so adapter instances from any working directory or requested root share one record, create through the same lock, and serialize prepared-pane creation through it. Later instances verify an idempotent request and reject a conflicting one; a tab whose ownership cannot be proven is never closed or replaced (a failed create rolls back only tabs the same attempt created). `prepare_pane` splits a retained shell target and returns the native pane ID after verifying it exists and has a shell process.
+
+Sends probe for a live agent. Plain panes receive raw text and keys in order; agent panes receive one self-submitting `agent prompt`, with one trailing Enter dropped and reported. Agent waits use native `agent wait` states (`idle`, `working`, `blocked`, `done`). Contract timeout values are seconds and are converted to Herdr milliseconds. Lifecycle waits observe `tab get`, focused tab state, `pane get`, and `pane process-info`; a missing pane keeps `pane-exists` waiting but immediately satisfies `pane-exited`. A pane with only its retained shell also satisfies `pane-exited` because no submitted foreground command remains. This observation does not prove assignment success or authorize cleanup.
+
+Native `agent_blocked` becomes `SendRejectedError`; `agent_prompt_stalled` becomes `SendStalledError` and must not be automatically resent. Native wait timeouts become `WaitTimeoutError`, missing targets become `TargetNotFoundError`, and an unavailable binary or socket becomes `RuntimeUnavailableError`.
+
+## Durable agent inbox
+
+Use a stable event ID for one message and the assignment attempt that owns it. The reverse-address JSON contains `session` and `pane`, as with `deliver`.
+
+```bash
+ace-herdr inbox enqueue --event inb-12345678 --attempt ATTEMPT --ref ref.json --file prompt.txt
+ace-herdr inbox status --event inb-12345678 --format json
+ace-herdr inbox deliver --event inb-12345678
+```
+
+`enqueue` checks the live pane, durable terminal, agent kind, and native thread identity. Repeating the same event, attempt, target, and payload returns its existing record; a changed value is an error. `deliver` claims the record once. Idle and busy Codex or Pi agents receive an exact-session native queue submission. After an accepted idle submission, Herdr sends a generic wake prompt without the inbox payload. A pane replacement can receive at most that generic wake, never the payload. The JSON result includes `state`, `claim_generation`, `binding`, `last_error`, and any submission receipt.
+
+`queued` can be retried after a proven pre-submission rejection. `delivered` means native submission was accepted; it does not mean the agent consumed the message. A crash after claim, changed target identity, or a native result without a verified acceptance receipt is `uncertain`; another `deliver` call does not resend it. A `completed` record has a positively verified consumption receipt. A superseded uncertain record can return to `queued` only after positive nonconsumption proof.
+
+```bash
+ace-herdr inbox reconcile --event inb-12345678 --receipt proof.json
+```
+
+The receipt is a file supplied from an operator or supervisor observation of the native outcome. Codex and Pi expose submission but no queryable consumption or eviction proof, so `reconcile` never polls the native queue or infers an outcome from age. Before enqueue, the supervisor sets `inbox_receipt_public_key` in `.ace/herdr/config.yml` to the absolute path of its trusted RSA public key. Each event pins that key's fingerprint; a later process cannot substitute a different key. Protect this configuration from delivery requesters. The receipt file must have a detached SHA-256 RSA signature at `FILE.sig`. The private key stays with the operator or supervisor. Its JSON must include the recorded event, attempt, claim generation, payload digest, and complete binding, plus an outcome (`consumed` or `superseded`), an identified observer, and a native observation reference:
+
+```json
+{
+  "event_id": "inb-12345678",
+  "attempt_id": "ATTEMPT",
+  "claim_generation": 1,
+  "payload_sha256": "<digest from inbox status>",
+  "binding": {"session": "<exact binding from inbox status>"},
+  "outcome": "consumed",
+  "observer": {"role": "operator", "id": "operator-id"},
+  "evidence": {"kind": "consumed_acknowledged", "native_reference": "session-log:42", "observation": "message consumed and acknowledged"}
+}
+```
+
+Copy the **entire** `binding` object from `inbox status`; the shortened object above only illustrates the field. `consumed` requires `evidence.kind: consumed_acknowledged`. `superseded` requires `queue_evicted`, `queue_expired`, or `thread_replaced`, with an observation that the old queue entry cannot be consumed. A matching receipt changes `uncertain` to `completed` for consumption, or to `queued` for proven supersession. The original target binding stays pinned. To authorize a replacement native session, include `replacement_target` in the signed receipt with the complete target identity from a fresh Herdr pane observation; reconciliation verifies its session, pane, terminal, agent, and thread against that live replacement address before saving it. Without that field, a new claim can only use the original target. A missing or mismatched receipt returns JSON with unchanged `state: uncertain` and `reconciliation_refusal`; it does not resend. Keep the observation record with the receipt. Signature validation authenticates the configured key; the operator or supervisor remains responsible for checking the cited native outcome before signing.
+
+After inspecting the native outcome and writing `proof.json`, the trusted operator signs the exact bytes:
+
+```bash
+openssl dgst -sha256 -sign operator-private.pem -out proof.json.sig proof.json
+ace-herdr inbox reconcile --event inb-12345678 --receipt proof.json
+```
+
+The command verifies the detached signature against the configured public key before any transition. Missing, malformed, or self-signed receipts leave the event uncertain and return a machine-readable refusal. Keep the signed receipt and source observation for audit.
+
 ## tmux-intent ↔ herdr-command parity
 
 Every common terminal-control intent available through `ace-tmux` is available through `ace-herdr` (herdr 0.9.1). Parity is of INTENT and FLAG VOCABULARY, not byte-format: ace-herdr keeps one-line JSON where ace-tmux renders human tables.

data/exe/ace-herdr

@@ -1,7 +1,6 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
-require "bundler/setup"
 require "ace/herdr"
 
 # No args -> show help

data/lib/ace/herdr/atoms/json_find.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module Ace
+  module Herdr
+    module Atoms
+      # Shared recursive extraction from herdr's native JSON responses
+      # (ids nest under result.*; native shapes vary slightly between
+      # commands). Pure functions — no side effects.
+      module JsonFind
+        module_function
+
+        # Walk a path of hash keys; nil unless every hop is a Hash. A hop
+        # resolves only to a non-empty String.
+        def dig_value(json, path)
+          current = json
+          path.each do |key|
+            return nil unless current.is_a?(Hash)
+
+            current = current[key]
+          end
+          current.is_a?(String) && !current.empty? ? current : nil
+        end
+
+        # Depth-first search for the first non-empty String under any of
+        # the given keys (tolerant extraction of ids from native shapes).
+        def find_value(json, keys)
+          case json
+          when Hash
+            keys.each { |key| return json[key] if json[key].is_a?(String) && !json[key].empty? }
+            json.each_value do |value|
+              found = find_value(value, keys)
+              return found if found
+            end
+            nil
+          when Array
+            json.each do |value|
+              found = find_value(value, keys)
+              return found if found
+            end
+            nil
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ace/herdr/cli/commands/inbox.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+require "json"
+require "openssl"
+require "ace/support/cli"
+require_relative "support"
+
+module Ace
+  module Herdr
+    module CLI
+      module Commands
+        class Inbox < Ace::Support::Cli::Command
+          include Ace::Support::Cli::Base
+          include Runtime
+
+          desc "Enqueue, inspect, deliver, or reconcile a durable agent inbox event"
+          argument :operation, desc: "enqueue, status, deliver, or reconcile"
+          option :event, type: :string, desc: "Stable event ID"
+          option :attempt, type: :string, desc: "Assignment attempt ID"
+          option :ref, type: :string, desc: "Herdr reverse-address JSON file"
+          option :file, type: :string, desc: "Payload file"
+          option :receipt, type: :string, desc: "Native proof JSON file"
+          option :format, type: :string, desc: "Output format (json)"
+
+          def initialize(executor: nil, native: nil)
+            @executor = executor
+            @native = native
+          end
+
+          def call(operation: nil, **options)
+            inbox = nil
+            translate_errors do
+              cli_error("--event is required") if options[:event].to_s.empty?
+              cli_error("only --format json is supported") if options[:format] && options[:format] != "json"
+              inbox = Organisms::Inbox.new(executor: executor,
+                native: @native || Molecules::NativeQueueExecutor.new,
+                deliveries_dir: deliveries_dir,
+                receipt_public_key: %w[enqueue reconcile].include?(operation) ? receipt_public_key : nil)
+              result = case operation
+              when "enqueue"
+                %i[attempt ref file].each { |key| cli_error("--#{key} is required") if options[key].to_s.empty? }
+                inbox.enqueue(event: options[:event], attempt: options[:attempt],
+                  ref: options[:ref], payload: File.binread(options[:file]))
+              when "status"
+                inbox.status(event: options[:event])
+              when "deliver"
+                inbox.deliver(event: options[:event])
+              when "reconcile"
+                path = options[:receipt]
+                begin
+                  bytes = path.to_s.empty? ? nil : File.binread(path)
+                  receipt = bytes && JSON.parse(bytes)
+                  signature = bytes && File.binread("#{path}.sig")
+                rescue SystemCallError, JSON::ParserError => e
+                  # Receipt input is caller-owned: unreadable or malformed
+                  # proof files surface as the documented refusal, never as
+                  # command failures.
+                  puts JSON.generate(inbox.status(event: options[:event]).merge(
+                    "reconciliation_refusal" => "invalid receipt: #{e.message}"))
+                  return
+                end
+                inbox.reconcile(event: options[:event], receipt: receipt,
+                  signed_bytes: bytes, signature: signature)
+              else
+                cli_error("operation must be enqueue, status, deliver, or reconcile")
+              end
+              puts JSON.generate(result)
+            end
+          rescue Errno::ENOENT, JSON::ParserError => e
+            raise Ace::Support::Cli::Error, e.message
+          end
+
+          private
+
+          def receipt_public_key
+            path = config["inbox_receipt_public_key"]
+            return nil unless path.is_a?(String) && path.start_with?("/")
+
+            key = OpenSSL::PKey.read(File.read(path))
+            key if key.is_a?(OpenSSL::PKey::RSA) && !key.private?
+          rescue SystemCallError, OpenSSL::PKey::PKeyError
+            nil
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ace/herdr/cli.rb

@@ -4,6 +4,7 @@ require "ace/support/cli"
 require "ace/core"
 require_relative "../herdr"
 require_relative "cli/commands/deliver"
+require_relative "cli/commands/inbox"
 require_relative "cli/commands/dispatch"
 require_relative "cli/commands/wait"
 require_relative "cli/commands/close"
@@ -26,6 +27,7 @@ module Ace
       # Application commands with descriptions (for help output)
       REGISTERED_COMMANDS = [
         ["deliver", "Push an answer to an agent pane (ace-hitl delivery contract)"],
+        ["inbox", "Durable agent message queue and reconciliation"],
         ["dispatch", "Start an agent in one command: tab + agent + prompt"],
         ["list", "List live panes, tabs, or workspaces as one JSON line"],
         ["send", "Send a command, raw text, or named keys to a pane"],
@@ -40,6 +42,7 @@ module Ace
 
       HELP_EXAMPLES = [
         "ace-herdr deliver --session ws-1 --pane p5 --event-id evt-1 --answer-file answer.md",
+        "ace-herdr inbox enqueue --event evt-1 --attempt att-1 --ref ref.json --file prompt.txt",
         "echo 'the answer' | ace-herdr deliver --pane p5",
         "ace-herdr dispatch --label 8wm.t.vs0 --kind pi --prompt-file prompt.md",
         "ace-herdr list --workspace w1",
@@ -64,6 +67,7 @@ module Ace
 
       # Register commands
       register "deliver", CLI::Commands::Deliver.new
+      register "inbox", CLI::Commands::Inbox.new
       register "dispatch", CLI::Commands::Dispatch.new
       register "list", CLI::Commands::List.new
       register "send", CLI::Commands::Send.new

data/lib/ace/herdr/errors.rb

@@ -22,6 +22,9 @@ module Ace
       end
     end
 
+    # Native timeout is distinct from a prompt accepted but not observed.
+    class ExecutorTimeoutError < AgentNotReadyError; end
+
     # Target pane does not exist (pane_not_found)
     class PaneNotFoundError < ExecutorError; end
 

data/lib/ace/herdr/models/delivery_record.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "json"
+require "digest"
 
 module Ace
   module Herdr
@@ -17,14 +18,23 @@ module Ace
       # before the prompt never loses the content; it can be re-delivered
       # via resume. Record files are 0600 (answers may be sensitive).
       class DeliveryRecord
-        STATES = %w[pending delivered retryable failed].freeze
+        STATES = %w[pending delivered retryable failed queued claimed uncertain completed].freeze
 
         attr_reader :event_id, :session, :pane, :answer_digest, :answer,
-          :state, :attempts, :history, :created_at, :updated_at
+          :state, :attempts, :history, :created_at, :updated_at, :inbox
 
         def initialize(event_id:, session:, pane:, answer_digest:, answer: nil,
-          state: "pending", attempts: 0, history: [], created_at: nil, updated_at: nil)
+          state: "pending", attempts: 0, history: [], created_at: nil, updated_at: nil, inbox: nil)
           raise ArgumentError, "unknown state: #{state}" unless STATES.include?(state)
+          if inbox
+            unless inbox.is_a?(Hash) && inbox["attempt_id"].is_a?(String) &&
+                inbox["claim_generation"].is_a?(Integer) && inbox["claim_generation"] >= 0 &&
+                answer.is_a?(String) && Digest::SHA256.hexdigest(answer) == answer_digest
+              raise ArgumentError, "invalid inbox record"
+            end
+          elsif %w[queued claimed uncertain completed].include?(state)
+            raise ArgumentError, "inbox state requires inbox metadata"
+          end
 
           @event_id = event_id
           @session = session
@@ -36,6 +46,7 @@ module Ace
           @history = history.dup.freeze
           @created_at = created_at
           @updated_at = updated_at
+          @inbox = inbox && deep_freeze(JSON.parse(JSON.generate(inbox)))
           freeze
         end
 
@@ -45,7 +56,7 @@ module Ace
             answer_digest: hash["answer_digest"], answer: hash["answer"],
             state: hash["state"] || "pending",
             attempts: hash["attempts"] || 0, history: hash["history"] || [],
-            created_at: hash["created_at"], updated_at: hash["updated_at"]
+            created_at: hash["created_at"], updated_at: hash["updated_at"], inbox: hash["inbox"]
           )
         end
 
@@ -59,7 +70,8 @@ module Ace
             "answer_digest" => answer_digest, "answer" => answer,
             "state" => state,
             "attempts" => attempts, "history" => history,
-            "created_at" => created_at, "updated_at" => updated_at
+            "created_at" => created_at, "updated_at" => updated_at,
+            "inbox" => inbox
           }
         end
 
@@ -84,7 +96,7 @@ module Ace
             state: state || self.state,
             attempts: attempts,
             history: history + [detail.merge("at" => timestamp)],
-            created_at: created_at || timestamp, updated_at: timestamp
+            created_at: created_at || timestamp, updated_at: timestamp, inbox: inbox
           )
         end
 
@@ -100,9 +112,32 @@ module Ace
             state: state,
             attempts: attempts + 1,
             history: appended.history,
-            created_at: appended.created_at, updated_at: appended.updated_at
+            created_at: appended.created_at, updated_at: appended.updated_at, inbox: inbox
           )
         end
+
+        def advance_inbox(state:, inbox:, detail:, timestamp:)
+          self.class.new(
+            event_id: event_id, session: session, pane: pane,
+            answer_digest: answer_digest, answer: answer,
+            state: state, attempts: attempts + (detail["action"] == "claim" ? 1 : 0),
+            history: history + [detail.merge("at" => timestamp)],
+            created_at: created_at || timestamp, updated_at: timestamp,
+            inbox: inbox
+          )
+        end
+
+        private
+
+        def deep_freeze(value)
+          case value
+          when Hash
+            value.each { |key, item| deep_freeze(key); deep_freeze(item) }
+          when Array
+            value.each { |item| deep_freeze(item) }
+          end
+          value.freeze
+        end
       end
     end
   end

data/lib/ace/herdr/molecules/bounded_process.rb

@@ -0,0 +1,118 @@
+# frozen_string_literal: true
+
+require "open3"
+require "timeout"
+
+module Ace
+  module Herdr
+    module Molecules
+      # Runs a child process in its own process group with a real deadline.
+      # Pipes drain with bounded memory and a stalled child is killed at the
+      # deadline, so the caller never blocks in cleanup past the timeout the
+      # way an Open3.capture3 + Timeout.timeout combination can.
+      module BoundedProcess
+        # Immutable outcome of one bounded child run.
+        Result = Struct.new(:stdout, :stderr, :status, :oversized)
+
+        # A SystemCallError raised while the child was already running (pipe
+        # I/O after launch). Distinct from spawn failures: the child exists,
+        # so the caller cannot classify the run as pre-launch.
+        class PostLaunchError < StandardError
+          def initialize(message)
+            super("post-launch process failure: #{message}")
+          end
+        end
+
+        module_function
+
+        # @param argv [Array<String>] child argv (no shell)
+        # @param stdin_data [String] payload written to the child's stdin
+        # @param timeout_s [Numeric] wall-clock deadline for the whole run
+        # @param output_limit [Integer] per-stream retained byte cap
+        # @return [Result] stdout/stderr are capped; oversized reports truncation
+        # @raise [Timeout::Error] when the child outlives the deadline (killed)
+        # @raise [SystemCallError] spawn failures only (child never launched)
+        # @raise [PostLaunchError] SystemCallError while the child was live
+        def call(argv, stdin_data: "", timeout_s:, output_limit: 65_536)
+          Open3.popen3(*argv, pgroup: true) do |stdin, stdout, stderr, waiter|
+            begin
+              run_loop(stdin, stdout, stderr, waiter,
+                stdin_data: stdin_data, timeout_s: timeout_s, output_limit: output_limit)
+            rescue SystemCallError => e
+              # The child was already spawned: an I/O failure now cannot be
+              # rewound into a pre-launch classification, and the child must
+              # not outlive the failure inside popen3's cleanup wait.
+              kill_group(waiter)
+              waiter.join
+              raise PostLaunchError, e.message
+            end
+          end
+        end
+
+        def run_loop(stdin, stdout, stderr, waiter, stdin_data:, timeout_s:, output_limit:)
+          payload = stdin_data.to_s
+          sent = 0
+          stdin_closed = payload.empty?
+          stdin.close if stdin_closed
+          buffers = {stdout => +"", stderr => +""}
+          streams = [stdout, stderr]
+          oversized = false
+          deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout_s
+            buffers = {stdout => +"", stderr => +""}
+            streams = [stdout, stderr]
+            oversized = false
+            deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout_s
+
+            until streams.empty? && stdin_closed && waiter.join(0)
+              remaining = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+              if remaining <= 0
+                kill_group(waiter)
+                raise Timeout::Error, "process did not finish within #{timeout_s}s"
+              end
+
+              ready = IO.select(streams, stdin_closed ? [] : [stdin], nil, [remaining, 0.02].min)
+              next unless ready
+
+              ready.first.each do |io|
+                chunk = io.read_nonblock(4096, exception: false)
+                if chunk.nil?
+                  io.close
+                  streams.delete(io)
+                elsif chunk != :wait_readable
+                  buffer = buffers.fetch(io)
+                  available = output_limit - buffer.bytesize
+                  oversized = true if chunk.bytesize > available
+                  buffer << chunk.byteslice(0, available) if available.positive?
+                end
+              end
+
+              next if stdin_closed || !ready[1].to_a.include?(stdin)
+
+              begin
+                # Nonblocking with partial writes: a child that stops reading
+                # must never trap us inside one large stdin.write; the loop
+                # re-checks the deadline between write attempts.
+                written = stdin.write_nonblock(payload.byteslice(sent, payload.bytesize - sent),
+                  exception: false)
+                sent += written if written != :wait_writable
+              rescue Errno::EPIPE, IOError
+                sent = payload.bytesize
+              end
+              if sent >= payload.bytesize
+                stdin.close
+                stdin_closed = true
+              end
+            end
+
+          Result.new(buffers.fetch(stdout), buffers.fetch(stderr), waiter.value, oversized)
+        end
+
+        def kill_group(waiter)
+          Process.kill("KILL", -waiter.pid)
+        rescue Errno::ESRCH
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/ace/herdr/molecules/delivery_record_store.rb

@@ -46,8 +46,11 @@ module Ace
           tmp = "#{path}.tmp.#{Process.pid}"
           File.open(tmp, IO::CREAT | IO::TRUNC | IO::WRONLY, 0o600) do |file|
             file.write(JSON.generate(record.to_h))
+            file.flush
+            file.fsync
           end
           File.rename(tmp, path)
+          File.open(deliveries_dir) { |dir| dir.fsync }
           path
         end