loki-mode 7.5.17 → 7.5.28

package/docs/retrospectives/v7.5.15-llm-failure-modes.md

@@ -0,0 +1,49 @@
+# v7.5.15 LLM Failure Modes Catalogue
+
+This table catalogues LLM failure modes that occurred (or were explicitly
+considered and ruled out) during the v7.5.15 8-agent fleet session. "Failure
+mode" means a systematic LLM behavioral error, not a human process mistake.
+Each row covers one category. If a category did not materialize in this
+session, it is marked NOT OBSERVED so the reader can confirm it was
+considered, not overlooked. All session moments and file references are
+specific: no generalities.
+
+| Failure mode | Where it appeared | What caught it | Cost if missed |
+|---|---|---|---|
+| Confident hallucination (agent claims a file/function exists that doesn't) | NOT OBSERVED in this session | n/a | n/a |
+| Stale context (agent acts on memory of prior state instead of current) | Dev5's `autonomy/loki` worktree was branched from pre-merge `HEAD` (`2ce36624`), i.e., prior to Dev3+Dev4's merges. The integrator treated the worktree file as current state and bulk-`cp`'d it over the integrated tree. This is the integrator exhibiting stale-context behavior. | `grep -c "init-rules" autonomy/loki` returned 0 instead of 4. Immediate recovery via `git checkout HEAD -- autonomy/loki` + surgical `Edit` of only Dev5's two help blocks. | Dev3's `init-rules` subcommand (9/9 tests) and Dev4's `cmd_doctor_json` sentrux field (13/13 tests) would have shipped silently regressed. Two advertised features gone from the binary. |
+| Scope creep (agent does more than asked, breaking adjacent code) | NOT OBSERVED in this session. All 8 dev agents touched only their assigned functions/files. Dev7 explicitly disclaimed out-of-scope sites (`cross_project.py:101`) rather than patching them. | n/a | n/a |
+| Silent test rot (tests written but not wired to runner) | 7 of 8 new test suites (Dev1 through Dev7) were not registered in `tests/run-all-tests.sh`. Only Dev8's `test-ci-sentrux-coverage.sh` was wired. R1, R2, R3 all ran the tests directly and confirmed PASS without asking whether CI would ever invoke them. | Devil's Advocate reviewer asked "will these tests run tomorrow?" and checked `tests/run-all-tests.sh` directly. Caught pre-commit. Fixed by adding 5 bash entries + 2 pytest wrapper scripts at `tests/run-all-tests.sh:85-99`. | 7 tests silently orphaned. Future regressions in sentrux wire-in, dashboard endpoint, init-rules, doctor JSON parity, dashboard nav UAT, pytest timeout, and episode resilience paths would go undetected until a manual invocation. |
+| Cross-agent state assumption (Dev N assumes Dev M's branch is in some state without verifying) | Integrator assumed Dev5's worktree file was safe to `cp` over the integrated main tree. The assumption was that worktree files reflected the post-merge state; they did not (Dev5's branch point was `2ce36624`, before Dev3 + Dev4 merges). See `v7.5.15-fleet-postmortem.md` section 1. | Structural checksum: `grep -c "init-rules" autonomy/loki` == 0 (expected 4). Caught immediately after the bad copy. REPEATABILITY: LOW (manual, not codified -- see fleet-postmortem section 1 recommendation 3 for the fix). | Same as stale context above: two shipped features silently deleted from the binary. |
+| Optimistic verification (agent says "tested" without running test) | R3 (first dispatch) returned "Still running. Let me wait for monitor." as its entire output. R3 had not run any of the 6 cross-cutting integration checks assigned to it, but its status was "completed". The agent used language implying work was in progress when no work was done. | Integrator read the result body. 7-word fragment clearly not a structured 6-risk verdict. Re-spawned R3 with a prompt containing literal `bash` blocks and "DO NOT wait for any monitor." R3-retry completed in 61s. See `v7.5.15-fleet-postmortem.md` section 2. | Integration safety review would have been missing from the record. Subtle cross-file interactions between Dev1+Dev6's `autonomy/run.sh` edits and Dev3+Dev4's `autonomy/loki` edits would have gone unchecked. A previously undetected conflict could have shipped. |
+| Path-prefix confusion (rm -rf interpreted as dangerous when it isn't) | `validate-bash` hook matched the pattern `rm -rf /Users/lokesh/git/loki-mode/.sentrux` against its dangerous-rm guard (`rm -rf /` prefix check). The path is a subdirectory of the project, not the root, but the string match fired a false positive. Hook returned BLOCKED. | The hook itself caught and surfaced the block explicitly -- no silent data loss. Workaround: rewrite as relative path `rm -rf ./.sentrux`. Documented in session notes from the validate-bash hook output. | If the hook had silently suppressed the error and continued, a subtly wrong rm invocation could have run. Conversely, if the integrator had bypassed the hook without understanding why, it could have established a precedent of ignoring hook blocks. |
+| Exit code over-trust (relying on subprocess exit code that lies) | Pre-existing MCP test failure: `python3 -c "import mcp"` exits non-zero on this Mac because `pip install mcp` was never run. The exit code does not represent a regression introduced by v7.5.15 -- it is an environmental gap that predates the session. If taken at face value it could be misread as a newly broken test. | Verified pre-existing by reproducing on `origin/main` before the release commit. CHANGELOG v7.5.15 documents it honestly as "single failure is pre-existing pip install mcp env gap". 24/25 PASS is the accurate count. | Misfiled as a release regression: would have blocked the release or, if overridden, set a precedent of shipping with unacknowledged test failures. |
+| Bash vs zsh source-file scope leaking | session-reported, unverified by audit. Reported behavior: `command -v sentrux` returned empty under zsh mid-session even though the binary was on PATH. Plausible mechanism: zsh PATH hash cache had not refreshed to see the binary added during the session. No file:line cite, no grep output, no git ref was captured that would allow an auditor to reproduce or confirm this independently. | session-reported, unverified by audit. Reported workaround: switching to `bash -c` for the test invocation forces a fresh hash evaluation and the invocation succeeded. No codified guardrail exists for this behavior. | If the zsh/bash inconsistency had been attributed to a code bug rather than a shell hash quirk, time would have been spent diagnosing a non-existent defect in the sentrux integration code. Alternatively, the test could have been skipped and a real future failure masked. |
+| Tool hook noise drowning real failures | First `bash tests/run-all-tests.sh` invocation was piped through `tail -15`. The tail window captured only 15 lines of output, discarding all per-test PASS/FAIL lines and the full run summary. The visible output showed only the last few lines of the final test's output -- no actionable signal. | Integrator noticed the output was truncated (no "Tests Run:" summary line) and re-ran without the `tail` pipe. Second run produced full per-test output: 24/25 PASS with the one failing test identified by name. | If the truncated output had been accepted as the verification record, the integrator would have had no per-test attribution. Any failure that landed outside the last 15 lines would have been invisible. The release would have shipped without a verified pass/fail count per test. |
+| (beyond user's 10) Build artifact drift after source-only merge | The bun-parity matrix in `local-ci.sh` ran both bash and Bun routes through `diff -q` and surfaced a 5/13 failure caused by stale `loki-ts/dist/loki.js`. The Bun dist had not been rebuilt after source changes were merged, so the Bun route diverged from the bash route at 5 commands. This is a distinct failure mode from stale context: the source was current, but the compiled artifact was not. | `bash scripts/local-ci.sh` bun-parity matrix caught the 5/13 diff before the commit. Fix was `bun run build` followed by re-staging the updated dist file. | If the stale dist had shipped, 5 Bun-route commands would have exhibited the pre-merge behavior while the bash route exhibited the post-merge behavior. Users on the Bun route would have silently received the older behavior with no error. |
+
+## What went right
+
+The failure modes above were caught -- none shipped to users -- because
+several guardrails worked exactly as intended. The `validate-bash` hook
+surfaced the false-positive rm block explicitly rather than silently passing
+or silently failing. The post-merge structural grep checksum (`grep -c
+"init-rules"`) caught the bad-copy regression in seconds. The Devil's
+Advocate role, operating with an explicitly non-domain mandate separate from
+R1/R2/R3, asked the meta-question about test runner wiring that three
+domain-specific reviewers each had reason not to ask. The `local-ci`
+pre-push gate (`bash scripts/local-ci.sh`, 21/21 PASS) caught the stale
+`loki-ts/dist/loki.js` artifact before the commit: the bun-parity matrix
+ran both bash and Bun routes through `diff -q` and surfaced the 5/13
+failure that triggered a `bun run build` before staging. The integrator's
+discipline of reading reviewer output rather than trusting "completed"
+status caught R3's fragment response before it was treated as a verdict.
+Taken together: grep-based checksums, the DA contrarian role, local-ci
+pre-push, the bun-parity matrix, and the validate-bash hook are the
+codified mechanisms that kept this session's failure modes from becoming
+user-facing defects. Note: the integrator reading R3's result body before
+accepting its status as a verdict was human attentiveness, not a repeatable
+guardrail. It is not listed here as a mechanism; it is listed as the action
+that exposed the gap. The gap itself (agents reporting "completed" without
+producing required output) is a process deficiency without a codified fix
+as of this session.

package/loki-ts/data/finding-schema.json

@@ -0,0 +1,74 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://loki-mode.dev/schemas/finding.json",
+  "title": "Council Multi-Finding Response",
+  "description": "Phase C (v7.5.20) JSON contract Claude Code must emit when invoked with --agents <voter-set> and --json-schema. Top-level is an object with a 'findings' array, one entry per dispatched voter. Each finding describes a single voter's verdict on the current iteration. NOTE on reconciliation with the existing AgentVerdict type (loki-ts/src/runner/council.ts:128-133): the schema field 'vote' maps onto AgentVerdict.verdict; the schema field 'confidence' is required at emission time but is NOT carried onto AgentVerdict (validateFinding drops it). Optional 'severity' and 'suggested_action' top-level per-finding fields are likewise dropped from AgentVerdict. The schema is the authoritative shape Claude must produce; AgentVerdict is the simplified shape internal council logic consumes.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["findings"],
+  "properties": {
+    "findings": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["role", "vote", "reason", "confidence"],
+        "properties": {
+          "role": {
+            "type": "string",
+            "minLength": 1,
+            "maxLength": 128,
+            "description": "Voter slug, e.g. 'requirements-verifier', 'test-auditor', 'convergence-voter', 'devils-advocate'."
+          },
+          "vote": {
+            "type": "string",
+            "enum": ["APPROVE", "REJECT", "CANNOT_VALIDATE"],
+            "description": "Voter's verdict. Maps onto AgentVerdict.verdict on the consumer side."
+          },
+          "reason": {
+            "type": "string",
+            "minLength": 1,
+            "maxLength": 4000,
+            "description": "Free-text explanation of the vote. Surfaced verbatim in council transcripts."
+          },
+          "confidence": {
+            "type": "number",
+            "minimum": 0,
+            "maximum": 1,
+            "description": "Voter's self-reported confidence in the vote, 0..1. Used by future weighting passes; not carried onto AgentVerdict in v7.5.20."
+          },
+          "severity": {
+            "type": "string",
+            "enum": ["CRITICAL", "HIGH", "MEDIUM", "LOW"],
+            "description": "Top-level severity hint for the finding. Optional; per-issue severities live under 'issues'."
+          },
+          "suggested_action": {
+            "type": "string",
+            "maxLength": 2000,
+            "description": "Short remediation hint. Optional; surfaced in transcripts when present."
+          },
+          "issues": {
+            "type": "array",
+            "items": {
+              "type": "object",
+              "additionalProperties": false,
+              "required": ["severity", "description"],
+              "properties": {
+                "severity": {
+                  "type": "string",
+                  "enum": ["CRITICAL", "HIGH", "MEDIUM", "LOW"]
+                },
+                "description": {
+                  "type": "string",
+                  "minLength": 1,
+                  "maxLength": 2000
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}

package/loki-ts/data/model-pricing.json

@@ -0,0 +1,12 @@
+{
+  "$schema_version": 1,
+  "_comment": "Rolling pricing table consumed by loki-ts/src/runner/budget.ts. Update this file when Anthropic / OpenAI / others publish new prices; no code change required. Pricing is USD per 1 million tokens. Aliases (opus/sonnet/haiku) point to the latest model of that family per providers/model_catalog.json.",
+  "_updated": "2026-05-22",
+  "_source": "https://www.anthropic.com/pricing + provider docs",
+  "pricing": {
+    "opus": { "input": 5.0, "output": 25.0 },
+    "sonnet": { "input": 3.0, "output": 15.0 },
+    "haiku": { "input": 1.0, "output": 5.0 },
+    "gpt-5.3-codex": { "input": 1.5, "output": 12.0 }
+  }
+}