@codebam/dsh-thinking-auditor 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sean Behan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,179 @@
+# @codebam/dsh-thinking-auditor
+
+A **claim/evidence auditor** for live [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) reasoning traces.
+
+It observes the model's `reasoning-delta` stream without changing the request, checks what the trace claims against what the session actually saw, asks the **same provider/model route** for a structured second opinion in a session-less call the monitored agent never sees, and lets deterministic code decide whether to force verification, gate an irreversible action, or (only if explicitly configured) cancel.
+
+It is **not a lie detector**. It does not read intent and does not label a model dishonest. Chain-of-thought is an unfaithful generated artifact, not a window into hidden state, so the plugin reports only *claim-to-evidence relations*: `supported`, `unsupported`, `contradicted`, or `speculative`.
+
+## Why this shape
+
+- **Covert capture.** A Cordis listener on `agent/assistant-stream` copies reasoning deltas into a bounded per-session ring window and passes every chunk through untouched. Neither the prompt nor the request route changes, so the monitored model has no new input to react to.
+- **Mechanical evidence first.** The session log is the harness's model-visible source of truth. The plugin maintains a bounded ledger of `tool/call`, `tool/result`, and user messages from committed `session/event`s. Claims that require a tool result but have no antecedent are settled in code, not by a model. This is the highest-precision signal the design offers.
+- **Same-model, isolated auditor.** The auditor call uses the session's latest logged `request/header` route (so it follows model changes) but omits `sessionId` and `purpose`. It attaches to no session, enters no persistence log, and emits no `agent/assistant-stream` frame. The monitored model never sees the auditor's request or response.
+- **Deterministic gates.** The auditor's structured JSON enriches a verdict; it cannot clear a mechanical contradiction or a missing tool-call antecedent. A small pure policy module owns escalation.
+- **Separate audit store.** Verdicts live in a bounded in-memory store and, by default, a private `0600` JSONL file beside the harness home. They are deliberately *not* appended to the session log: an out-of-tree session event without the envelope's `ignorable` marker can make a log unreadable on resume, and `Session.append()` cannot set that marker.
+
+## How a verdict is produced
+
+1. **Capture.** Every `reasoning-delta` is appended to the session's pending window. At `windowChars`, after a completed tool result, at an attempt end, or before a turn closes, the window is taken and cleared exactly once.
+2. **Deterministic pass.** `src/claims.mjs` extracts claims and anchors (quoted excerpts, paths, URLs, versions, quantities, action verbs), then joins them against the ledger:
+   - an asserted tool result with no matching tool call in the turn → `unsupported` / high / `noMatchingToolCall`;
+   - a success claim contradicted by the paired result's failure markers → `contradicted` / high;
+   - a quoted file/output excerpt absent from every evidence source → `unsupported` / high (or `contradicted` when the matching read result exists without it);
+   - hedging, plans, and predictions → `speculative`, never a fabrication.
+3. **Auditor pass.** The same route receives the user-request excerpt, the bounded evidence digest, the mechanical findings, and the trace window. Trace and evidence are XML-escaped and declared untrusted data; the system prompt explicitly refuses to follow instructions found inside them.
+4. **Gate.** Mechanical findings stay authoritative. Trailing-window hysteresis, the configured `maxTier`, and the reversibility of the next action decide the actuator response.
+
+## Install
+
+Add a row to a profile patch (`~/.dsh/profiles/<profile>/cordis.patch.yml`):
+
+```yaml
+- insert:
+    - id: thinking-auditor
+      name: '@codebam/dsh-thinking-auditor' # or an absolute path to index.mjs
+      config:
+        enabled: true
+        gates:
+          maxTier: verify # observe | verify | block | cancel
+        store:
+          persist: true
+```
+
+The stock `dsh` base bundle already mounts the `llm`, `sessions`, and `tools` services this plugin injects. Node resolves the plugin's `@deepseek-ai/*` peers from the profile's `node_modules`; when loading by absolute file path, make that resolution available next to `index.mjs` (the NixOS route below symlinks it).
+
+A live settings section with the namespace `thinking-audit` overlays the row when `dsh-settings-file` is mounted; use it for thresholds and gates you want to change without restarting dsh. Unknown or invalid values fail the write, never silently drift.
+
+### Configuration
+
+Top level:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `enabled` | `true` | Master switch; `false` makes `apply()` a no-op. |
+| `auditOnToolResult` | `true` | Audit the pending window after a completed tool result. |
+| `auditOnTurnStop` | `true` | Audit/await a verdict before a turn closes (T1/T3 decision point). |
+| `auditOnAttemptEnd` | `true` | Audit a short final slice when one model attempt ends. |
+| `windowChars` | `1800` | Reasoning characters that force a background audit window. |
+| `minWindowChars` | `500` | Minimum pending reasoning for an opportunistic trigger. |
+| `maxTraceChars` | `12000` | Hard cap on the trace excerpt sent to one audit (tail preferred). |
+| `cooldownMs` | `1500` | Minimum spacing between background audits per session. |
+| `requestTimeoutMs` | `20000` | Deadline for a background session-less auditor call. |
+| `turnStopTimeoutMs` | `5000` | Maximum time `agent/turn-stopping` waits for a verdict. |
+| `maxAuditsPerTurn` | `8` | Maximum background audits one turn may start. |
+
+`auditor`:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `provider`, `model` | `null` | Explicit route override; `null` follows the session's latest logged `request/header`, then the agent's creation options. |
+| `maxTokens` | `1600` | Auditor output-token cap. |
+| `temperature` | `0` | Greedy decoding. |
+| `followReasoningEffort` | `true` | Reuse the session's reasoning-effort selection when the route advertises one. |
+| `maxResponseChars` | `50000` | Cap on the decoded response before JSON parsing; over-cap fails open. |
+
+`evidence`:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `maxToolCalls` | `120` | Tool calls retained per session. |
+| `maxResultChars` | `6000` | Per-result text retained in the ledger. |
+| `maxDigestChars` | `16000` | Overall evidence digest cap sent to the auditor. |
+| `maxUserChars` | `4000` | Per-user-message cap in the digest. |
+
+`gates`:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `maxTier` | `verify` | Highest actuator tier permitted: `observe`, `verify`, `block`, `cancel`. |
+| `minimumSeverity` | `medium` | Findings below this severity remain audited but never drive a gate. |
+| `verifyOn` | `unsupported` | Lowest deterministic status that makes T1 force an extra step. |
+| `blockOn` | `confirmed` | What T2 treats as sufficient: `never`, `contradicted`, or `confirmed`. |
+| `cancelOn` | `never` | T3 evidence requirement: `never` or `confirmed`. |
+| `consecutive` | `2` | Windows at `verify`+ for T1; block needs this many; cancel needs at least two. |
+| `t2Mode` | `ask` | `ask` routes through the approval seam; `deny` blocks outright. |
+| `askOnTimeoutForIrreversible` | `true` | If an irreversible action races an in-flight audit, ask instead of allowing. |
+
+`store`:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `persist` | `true` | Append verdict/action records to `audit.jsonl` in the audit dir. |
+| `dir` | `null` | Explicit audit directory; `null` derives from `dshHomePath('thinking-auditor')`, `$DSH_HOME/thinking-auditor`, or `~/.dsh/thinking-auditor`. |
+| `maxRecordsPerSession` | `200` | Verdict/action records retained per session. |
+| `maxSessions` | `200` | Sessions retained in memory. |
+
+## Actuator tiers
+
+- **T0 — observe (always on).** The verdict is stored and exposed through `ctx.thinkingAudit`; nothing acts.
+- **T1 — force verification (default maximum).** At `agent/turn-stopping`, if the trailing history reaches `verify`, the plugin steers one neutral message asking the model to re-check each factual claim against a tool result or file read from the session. It never says why and it fires at most once per turn. This is the reference design's false-positive killer: most unbacked claims resolve here.
+- **T2 — block the action, not the thought (opt-in).** At `tools/pre-execute`, if the verdict reaches `block`, the next action is `irreversible` (push/publish/network write/recursive removal/activation/…) and hysteresis is satisfied, the plugin returns `ask` or `deny`. Reversible and read-only actions continue.
+- **T3 — cancel (opt-in, both switches).** Requires `maxTier: cancel` and `cancelOn: confirmed`, a mechanically confirmed fabrication, and at least two trailing block windows. It calls `agent.cancel({ kind: 'hook', reason })`; irreversible actions already executed are not rolled back, so pair T3 with T2 rather than relying on cancellation alone.
+
+Defaults are deliberately capped at T1. A single model opinion is never enough to stop an interaction, and the auditor route is not independent: same-model monitors share blind spots and self-preference leniency.
+
+## Model experience
+
+T1 is the only default behavior the model can notice. The injected message is neutral:
+
+> Before finalizing: verify each factual claim against an actual tool result or file read from this session, cite the evidence for anything you assert, and remove or explicitly mark as unverified any claim you cannot support from this session's evidence. Do not state a result you have not verified here.
+
+T2's ask/denial reason is equally neutral and carries no monitoring details. The plugin never blocks a thought, never rewrites a request, and never injects context into the model's context unless a gate fires.
+
+## Privacy and safety
+
+- The audit store contains claim quotes, brief bases, and route metadata. It is written only locally, `0600`, under the harness home; it is not part of the canonical session log and is not uploaded by the official DeepSeek route. Delete the file or set `store.persist: false` to keep records memory-only.
+- The auditor call still sends the trace window and evidence excerpts to the same provider the session already used. It adds a second same-model request whose content may contain session material; treat its cost and exposure as you treat the session itself.
+- Trace and evidence are treated as hostile: escaped, tagged, and bounded before prompt assembly. The auditor is instructed never to follow instructions found inside them. The response parser accepts only a balanced JSON object and normalizes every field.
+- A trace that infers it is being monitored can change its behavior. Covertness buys measurement fidelity now, not a guarantee.
+- The auditor model never calls tools; only deterministic code can act, and only through documented dsh seams.
+
+## Known limitations
+
+- **Reasoning routes only.** A model/route that emits no `reasoning-delta` chunks produces no audits. The plugin is a no-op there.
+- **Claim extraction is heuristic.** It is tuned to be conservative; the model auditor and the LLM pass cover richer paraphrase, but neither is proof. `unsupported` means "not in this session's evidence", not "false".
+- **Same-model correlation.** The auditor shares architecture, tokenizer, and blind spots with the subject; it is not an independent judge. Strategic deception modelling a same-capability monitor can defeat it.
+- **CoT unfaithfulness.** A clean trace is not evidence of a clean computation; a suspicious trace is not evidence of intent.
+- **Audit latency.** `agent/turn-stopping` may wait up to `turnStopTimeoutMs` for a verdict. Lower it, or set `auditOnTurnStop: false`, if the deployment values latency over the final gate.
+- **Boundary latency is real.** T1 adds one model step; T2 can ask for approval. That is the point of the design, not a bug.
+- **No session-log events.** Verdicts are available through `ctx.thinkingAudit` and the JSONL store, not through the session surface except for synthetic T1/T2 messages.
+- **Windows are form-aware, not semantic.** A window can contain multiple attempts or span a step boundary; the auditor is told to use only the ledger.
+- **Gates are per-agent.** A parent agent's block verdict does not gate a subagent's irreversible action; each session is audited and gated independently.
+- **Every audit is a model call.** On a reasoning-heavy session, window triggers can add a second same-route request per window. Raise `windowChars`/`cooldownMs`, lower `maxAuditsPerTurn`, or set `auditOnToolResult`/`auditOnAttemptEnd` to `false` to trade coverage for cost.
+
+## Public service
+
+When mounted, the plugin provides `ctx.thinkingAudit`:
+
+- `status()` — enabled state, tier, session/verdict counts, in-flight audits, audit-store path/error.
+- `latest(sessionId)` — latest detached verdict record.
+- `verdicts(sessionId, limit?)` / `actions(sessionId, limit?)` — bounded detached history.
+- `onRecord(callback)` — subscribe to `{ kind: 'verdict' | 'action', record }`; observer failures are contained.
+
+## Development
+
+```sh
+npm run peers       # link node_modules/@deepseek-ai to the installed dsh closure
+npm test            # node --test (pure suites plus the mock-context integration suite)
+npm run test:local  # peers + tests in one shot
+npm run check       # syntax-check every ESM source
+npm run pack:check  # npm pack --dry-run
+```
+
+Tests run against the real `@deepseek-ai/*` packages from the installed dsh closure for schema and message construction, and use a mock Cordis context for lifecycle/event orchestration. No model API calls are made.
+
+Verified against dsh `0.1.6-alpha.1` on Node 26; the plugin itself targets Node >= 20.
+
+## Deployment (NixOS profile)
+
+This repository carries a staged module for the NixOS dsh profile:
+
+- [`packaging/nixos/dsh-thinking-auditor.nix`](packaging/nixos/dsh-thinking-auditor.nix) — the `fetchFromGitHub` derivation.
+- [`packaging/nixos/default.nix.snippet`](packaging/nixos/default.nix.snippet) — the `pkgs/default.nix` callPackage line.
+- [`packaging/nixos/agents.nix.snippet`](packaging/nixos/agents.nix.snippet) — the `home/agents.nix` copy block and profile-patch row.
+- [`packaging/README.md`](packaging/README.md) — placement and build steps.
+
+## License
+
+MIT. See [`LICENSE`](LICENSE).