kairos-chain 3.83.0 → 3.84.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e601238b25375836907f7eec155beb0297e940cd5fd419ea855b7d48a3335993
-  data.tar.gz: 1c720eaa88d3204eedc436e38cdabaa3e79df3b9b5aca6260c925ec4173c6e62
+  metadata.gz: 1ebfc6fe1ff6c901c620512c56c698f6dca7bd058143e611547864a0a5460e29
+  data.tar.gz: caa3229fcb55a9f057a88be84a7a7694e514aff6e9b31ec5a8796d5e1ca93138
 SHA512:
-  metadata.gz: d13dbb392110ebc2f5f913bc38e370f7c4c36ee8067d69d060ab4379f1a34cc9db98a37ee09b845d633a4c76821194bb5aa8eaeef6db6c77e43784f7831150f7
-  data.tar.gz: 34eca89babce68b01e805b6510967f0d1e2dca0faa0a856d1ec7a8881fd7e4a00cf90932c69ed7685cefe89e482144566ca4733d1efc433c2bf0b0038d9080af
+  metadata.gz: 91588bb7de5f9f211409c77e90ba688432415dae81a142341481201ea412d2769d597ed1d10382a726b7ef301fbfdda77439bfa7f27e870102a296f520957656
+  data.tar.gz: a7a9bd001776163b148e690508f6aa6f44639f01974f00da552f5264676f37f4a25a286c9acb6e502edd6c77f07f9c5de2afd963a6053a89a8d0999721bf55a4

data/CHANGELOG.md

@@ -4,6 +4,54 @@ All notable changes to the `kairos-chain` gem will be documented in this file.
 
 This project follows [Semantic Versioning](https://semver.org/).
 
+## [3.84.0] - 2026-09-03
+
+### Fixed — agent: second field-defect bundle (D5-b, D6, exit-record housekeeping), real-process worker tests
+
+`agent` SkillSet templates only; the at-most-once advance gate (`lib/agent/advance_gate.rb`) is
+byte-identical to 3.83.0 and the normal path writes the same records with the same content.
+
+**D5-b — the review-verdict parser lost to nested objects and braces inside strings.** The old
+candidate scan walked back to the nearest `{` before `"overall_verdict"` and counted braces with no
+notion of strings, so `{"summary": {"a": 1}, "overall_verdict": …}` yielded the sibling `{"a": 1}`
+and a `}` inside a string value corrupted the depth count. Candidates now come from one string-aware
+forward pass over the reply; for each key occurrence they are emitted by enclosing depth ascending
+(so a panel's top-level verdict beats a per-persona one), with a size tie-break within a depth; an
+occurrence no span encloses (an odd quote on the same line) gets a line-anchored string-aware scan,
+and the old nearest-brace walk-back only as the candidate of last resort. Fence and crude fallbacks
+are unchanged.
+
+**D6 — a TERM/INT/HUP during the gated call left no record.** The shutdown flag was read once before
+the call; a signal landing during a minutes-long call was swallowed and, under the usual TERM-then-KILL
+sequence, the worker vanished as `no_record`. A recorder thread now writes an interim `trapped_signal`
+record (`phase: during_gated_call`) the moment a signal lands while the call is active; the final
+`normal` / `uncaught` / watchdog record supersedes it and carries `signals_during_call`. The call is
+still left to finish (aborting it would turn a sound advance into a crash-path recovery); no result
+file is written on any signal path. The flag is re-checked after tool-registry bootstrap, so a signal
+in that window exits 130 with a `before_gated_call` record; the recorder is stopped before every
+final write so it cannot overwrite a final record; and the exit record (and its atomic-write temp
+file) no longer counts as session activity — before this fix a TERM to a hung worker pushed the
+stall bound out by a full stall window (measured +4.0 s → +7.1 s with stall=3).
+
+**Housekeeping.** A fresh delegation handle removes the previous worker's `worker_exit.json`
+(readers already filtered by token; the file lingered). New env knob
+`KAIROS_WORKER_WATCHDOG_TICK_SECONDS` (default 30, floor 1) sets how often the watchdog re-evaluates
+its two bounds — the default is unchanged; the knob exists so real-process tests can drive both
+bounds in seconds.
+
+**Not changed, by evidence.** D3 (read-side queries blocked during a delegated run) is MCP stdio
+serialisation in the server's request loop, not the SkillSet; mitigation today is a short
+`max_wait_seconds` on `agent_wait`. D7 (a GVL-holding call starving the watchdog) did not reproduce:
+every blocking primitive on the worker's path releases the GVL (probe table in the implementation log).
+
+**Tests.** New `test_agent_worker_process.rb` starts the shipped worker script as a real child
+process and drives 7 of its 8 exit classes (normal, uncaught, superseded, signal before the call,
+bootstrap failure, stall bound, hard cap) plus the D6 interim record, a TERM to a hung worker, a TERM
+inside registry bootstrap, and signals carried on a watchdog record. Agent suite 96 runs / 209
+assertions → 131 / 345; every fix has a test shown red on the pre-fix code (one exception recorded:
+the recorder-serialisation window is sub-millisecond and has no deterministic test). Reviewed in two
+implementation rounds (7 seats each); records in the instance's L2 store.
+
 ## [3.83.0] - 2026-09-01
 
 ### Added — readable_gate measures sentence length

data/lib/kairos_mcp/version.rb

@@ -1,4 +1,4 @@
 module KairosMcp
-  VERSION = "3.83.0"
+  VERSION = "3.84.0"
   CHANGELOG_URL = "https://github.com/masaomi/KairosChain_2026/blob/main/CHANGELOG.md"
 end

data/templates/skillsets/agent/bin/agent_step_worker.rb

@@ -19,7 +19,12 @@
 #       KAIROS_DATA_DIR     (the server's effective data dir; makes the
 #                            worker resolve the SAME .kairos)
 #
-# Exit codes: 0 success; 1 exception; 125 setsid failed; 130 signal.
+# Exit codes: 0 success (also: superseded); 1 exception / bootstrap failure;
+#             124 watchdog (stall bound or hard cap); 125 setsid failed;
+#             130 signal received BEFORE the gated call. A signal received
+#             DURING the call is recorded but the call is left to finish (D6).
+#
+# Every exit path above leaves a worker_exit.json record (field defect D4).
 #
 # NB: bootstrap failures (LoadError/ScriptError from require) are caught too,
 # so the driver always sees a result rather than a silently hung handle.
@@ -79,9 +84,15 @@ end
 delegation = KairosMcp::SkillSets::Agent::StepDelegation.new(session_dir)
 my_token = boot_identity['step_token']
 
-shutdown = { requested: false }
+# Signals are trapped, not honoured mid-flight: a TERM during a multi-minute
+# gated call must not turn a sound advance into a crash-path recovery. The
+# trap only records; who reads the record is decided below (field defect D6).
+shutdown = { requested: false, signals: [] }
 %w[TERM INT HUP].each do |sig|
-  Signal.trap(sig) { shutdown[:requested] = true }
+  Signal.trap(sig) do
+    shutdown[:requested] = true
+    shutdown[:signals] << sig
+  end
 end
 
 begin
@@ -118,25 +129,82 @@ end
 # worker_exit record BEFORE exit!(124). That record is not a result, so the
 # collector still takes the crash path and a committed advance is still
 # recovered from the gate log, never masked.
+#
+# Impl review R1 (2026-09-02, I5): a watchdog record is a FINAL record, so it
+# carries the signals that landed during the call — otherwise the operator's
+# TERM vanished from the crash report the moment the watchdog superseded the
+# D6 interim record.
+#
+# Impl review R2 (2026-09-03, J1): clearing the phase only NARROWED the window
+# in which the recorder, already past its phase check, could rename an interim
+# record over the final one. Every final write now first stops the recorder
+# (kill, then join — join is safe after kill) so the two writers are
+# serialised, not raced. A recorder killed mid-write may leave a
+# worker_exit.json.tmp.* file behind; readers ignore it (they open the
+# renamed file only) and so does the activity clock (last_activity_time).
+call_phase = { active: false }
+signal_recorder = nil # assigned below; the watchdog closes over it
+stop_recorder = lambda do
+  signal_recorder&.kill
+  signal_recorder&.join
+end
 worker_started = Time.now
 stall_s = KairosMcp::SkillSets::Agent::StepDelegation.worker_stall_seconds
 cap_s   = KairosMcp::SkillSets::Agent::StepDelegation.worker_hard_cap_seconds
+tick_s  = KairosMcp::SkillSets::Agent::StepDelegation.worker_watchdog_tick_seconds
 watchdog = Thread.new do
   loop do
-    sleep 30
+    sleep tick_s
     elapsed = (Time.now - worker_started).round
     silent  = (Time.now - delegation.last_activity_time).round
-    if elapsed > cap_s
-      delegation.write_worker_exit(boot_identity, 'self_timeout_hard_cap',
-                                   'elapsed_seconds' => elapsed,
-                                   'silent_seconds' => silent)
-      exit!(124)
-    elsif silent > stall_s
-      delegation.write_worker_exit(boot_identity, 'self_timeout_stalled',
-                                   'elapsed_seconds' => elapsed,
-                                   'silent_seconds' => silent)
-      exit!(124)
-    end
+    exit_class = if elapsed > cap_s
+                   'self_timeout_hard_cap'
+                 elsif silent > stall_s
+                   'self_timeout_stalled'
+                 end
+    next unless exit_class
+
+    call_phase[:active] = false
+    stop_recorder.call
+    detail = { 'elapsed_seconds' => elapsed, 'silent_seconds' => silent }
+    detail['signals_during_call'] = shutdown[:signals].dup unless shutdown[:signals].empty?
+    delegation.write_worker_exit(boot_identity, exit_class, detail)
+    exit!(124)
+  end
+end
+
+# Field defect D6 (2026-08-27): the shutdown flag used to be read exactly once,
+# just before the gated call, so a TERM/INT/HUP that landed DURING the call —
+# which can run for minutes — was trapped, set a flag nobody read again, and
+# left no record; the worker then finished and reported 'normal'. The usual
+# shutdown sequence is TERM, then KILL after a grace period, so that worker
+# vanished with 'no_record'. The call is still left to finish (see the trap
+# comment above), but the signal is now recorded the moment it lands: an
+# interim 'trapped_signal' exit record with phase 'during_gated_call'. If the
+# worker survives to its own exit, the final record supersedes it and carries
+# the signal list; if it is killed first, the interim record is what the
+# collector's crash report finds. Recording from a thread rather than inside
+# the trap keeps file I/O out of trap context.
+#
+# Impl review R1 (I2, I3): `seen` advances ONLY while the phase is active. A
+# signal that lands before the call starts (e.g. inside ToolRegistry.new) is
+# not consumed here — it is either caught by the main thread's re-check below
+# or recorded on the recorder's first active tick. The phase is cleared by
+# the main thread around the call (ensure) and before each final write; R2 J1
+# then stops this thread (stop_recorder) before every final record, which is
+# what actually prevents it overwriting an `uncaught` / watchdog / normal one.
+signal_recorder = Thread.new do
+  seen = 0
+  loop do
+    sleep 0.25
+    next unless call_phase[:active]
+    sigs = shutdown[:signals]
+    next unless sigs.size > seen
+    seen = sigs.size
+    delegation.write_worker_exit(boot_identity, 'trapped_signal',
+                                 'phase' => 'during_gated_call',
+                                 'signals' => sigs.dup,
+                                 'note' => 'gated call left to finish; the final exit record supersedes this one')
   end
 end
 
@@ -176,13 +244,26 @@ begin
   args = (pending['arguments'] || {}).merge('session_id' => session_id)
   args.delete('execution') # never recurse into another delegation
 
-  if shutdown[:requested]
-    delegation.write_worker_exit(boot_identity, 'trapped_signal')
+  exit_before_call = lambda do
+    delegation.write_worker_exit(boot_identity, 'trapped_signal',
+                                 'phase' => 'before_gated_call',
+                                 'signals' => shutdown[:signals].dup)
     exit 130
   end
+  exit_before_call.call if shutdown[:requested]
 
   registry = KairosMcp::ToolRegistry.new
-  raw = registry.call_tool('agent_step', args)
+  # Re-check (I3): registry bootstrap takes long enough for a signal to land
+  # in it, and the recorder deliberately does not consume signals while the
+  # phase is inactive — so this check is what turns that signal into a record.
+  exit_before_call.call if shutdown[:requested]
+
+  call_phase[:active] = true
+  begin
+    raw = registry.call_tool('agent_step', args)
+  ensure
+    call_phase[:active] = false # the exception path clears it too (I2)
+  end
 
   # Normalize the MCP content shape to the response hash the inline call
   # would have returned.
@@ -201,7 +282,10 @@ begin
   # races a concurrently-opened fresh delegation by clearing state it may no
   # longer own or by mislabeling its result as a newer delegation's.
   delegation.write_result(response, identity: identity)
-  delegation.write_worker_exit(identity, 'normal')
+  call_phase[:active] = false
+  stop_recorder.call
+  signal_detail = shutdown[:signals].empty? ? {} : { 'signals_during_call' => shutdown[:signals].dup }
+  delegation.write_worker_exit(identity, 'normal', signal_detail)
   exit 0
 rescue SystemExit, SignalException
   # A deliberate exit (including our own `exit 0`) or a signal is not a
@@ -221,12 +305,17 @@ rescue Exception => e # rubocop:disable Lint/RescueException
     # best effort
   end
   begin
-    delegation.write_worker_exit(boot_identity, 'uncaught', 'error' => "#{e.class}: #{e.message}")
+    call_phase[:active] = false
+    stop_recorder.call
+    detail = { 'error' => "#{e.class}: #{e.message}" }
+    detail['signals_during_call'] = shutdown[:signals].dup unless shutdown[:signals].empty?
+    delegation.write_worker_exit(boot_identity, 'uncaught', detail)
   rescue StandardError
     # best effort
   end
   exit 1
 ensure
   watchdog&.kill
+  signal_recorder&.kill
   heartbeat_thread&.kill
 end

data/templates/skillsets/agent/lib/agent/step_delegation.rb

@@ -38,6 +38,9 @@ module KairosMcp
       #   delegation.heartbeat  — touched by the live worker every 2s
       #   delegation_result.json— the worker's final response, tagged with the
       #                           issue_anchor it belongs to
+      #   worker_exit.json      — the worker's exit record (why it left), one
+      #                           file overwritten per exit; readers match on
+      #                           step_token, a fresh open_handle removes it
       class StepDelegation
         LOCK_FILE      = 'delegation.lock'
         PENDING_FILE   = 'delegation.json'
@@ -61,9 +64,18 @@ module KairosMcp
         # record their firing in WORKER_EXIT_FILE before exiting.
         WORKER_STALL_SECONDS_DEFAULT      = 2700
         WORKER_HARD_CAP_SECONDS_DEFAULT   = 10800
+        # How often the watchdog re-evaluates the two bounds. Env-overridable
+        # (floor 1 s) so a real-process test can drive both firings in seconds
+        # instead of minutes; the default is unchanged from the D4 fix.
+        WORKER_WATCHDOG_TICK_SECONDS_DEFAULT = 30
 
         WORKER_SCRIPT = File.expand_path('../../bin/agent_step_worker.rb', __dir__)
 
+        # atomic_write's temp file is "<path>.tmp.<pid>.<thread>"; the infix
+        # is shared with last_activity_time's exclusion (and the test that
+        # pins it) so the three cannot drift apart.
+        ATOMIC_TMP_INFIX = '.tmp.'
+
         def initialize(session_dir)
           @dir = session_dir
           FileUtils.mkdir_p(@dir)
@@ -154,11 +166,16 @@ module KairosMcp
               # (crashed) — fall through to open a fresh handle below.
             end
 
-            # Fresh delegation: clear any stale result and ALL prior per-token
+            # Fresh delegation: clear any stale result, ALL prior per-token
             # heartbeats (a superseded worker's heartbeat must not count toward
-            # this handle's liveness), then write the new handle. The
+            # this handle's liveness) and the previous worker's exit record
+            # (it belongs to a token that is stale by construction once a new
+            # token is minted; crash_detail already filters by token, so this
+            # is housekeeping — the record would otherwise linger until the
+            # session dir is removed), then write the new handle. The
             # issue_anchor is injected into the worker args.
             FileUtils.rm_f(result_path)
+            FileUtils.rm_f(worker_exit_path)
             clear_all_heartbeats
             token = SecureRandom.uuid
             worker_args = recorded_args.merge('anchor' => issue_anchor)
@@ -249,13 +266,18 @@ module KairosMcp
 
         # Most recent write anywhere in the session dir EXCLUDING liveness and
         # locking artifacts: heartbeats tick every 2 s even when the main
-        # thread hangs, so counting them would blind the stall bound.
+        # thread hangs, so counting them would blind the stall bound. The
+        # worker's own exit record (and its atomic-write temp file) is excluded
+        # too: the D6 interim record is written by a TERM to a possibly HUNG
+        # worker, and counting it pushed the stall bound out by a whole stall
+        # window for exactly the worker the operator had asked to stop.
         def last_activity_time
           newest = nil
           Dir.glob(File.join(@dir, '*')).each do |f|
             base = File.basename(f)
             next if base.start_with?(HEARTBEAT_FILE)
             next if base == LOCK_FILE || base.end_with?('.lock')
+            next if base == WORKER_EXIT_FILE || base.start_with?("#{WORKER_EXIT_FILE}#{ATOMIC_TMP_INFIX}")
             t = begin
               File.mtime(f)
             rescue StandardError
@@ -353,6 +375,10 @@ module KairosMcp
           (ENV['KAIROS_WORKER_TIMEOUT_SECONDS'] || WORKER_HARD_CAP_SECONDS_DEFAULT).to_i
         end
 
+        def self.worker_watchdog_tick_seconds
+          [(ENV['KAIROS_WORKER_WATCHDOG_TICK_SECONDS'] || WORKER_WATCHDOG_TICK_SECONDS_DEFAULT).to_i, 1].max
+        end
+
         private
 
         # A pending handle whose worker is not (yet) declared crashed.
@@ -399,11 +425,15 @@ module KairosMcp
         end
 
         def atomic_write(path, content)
-          tmp = "#{path}.tmp.#{Process.pid}.#{Thread.current.object_id}"
+          tmp = atomic_tmp_path(path)
           File.write(tmp, content)
           File.rename(tmp, path)
         end
 
+        def atomic_tmp_path(path)
+          "#{path}#{ATOMIC_TMP_INFIX}#{Process.pid}.#{Thread.current.object_id}"
+        end
+
         def clear_all_heartbeats
           Dir.glob(File.join(@dir, "#{HEARTBEAT_FILE}.*")).each { |f| FileUtils.rm_f(f) }
           FileUtils.rm_f(File.join(@dir, HEARTBEAT_FILE)) # legacy unscoped, if any