loki-mode 9.17.0 → 9.18.2

package/docs/STALE-STATE-AUDIT.md

@@ -0,0 +1,174 @@
+# Stale-State Audit
+
+Audit for siblings of the `loki.pgid` session-killer (fixed in 4792b521).
+
+**The shape being hunted:** a file recording a PID, PGID, port, lock, or session
+id, written by one run, NOT removed on abnormal exit, later TRUSTED by another
+run after the OS recycled the identifier.
+
+**Method:** grep for the write, find its `rm -f`, check whether a trap covers
+INT/TERM/crash, then check whether the reader proves the record is still
+current. Every candidate was measured on this host rather than judged by
+reading, because the pgid finding was credible on evidence (155h/202h orphans),
+not on the code looking wrong.
+
+Scope: read-only across the repo; fix applied to one file (`autonomy/run.sh`).
+
+---
+
+## Measured state of this host
+
+Run from the repo root on 2026-08-08:
+
+| File | Contents | Live? | Age |
+|---|---|---|---|
+| `~/.loki/dashboard/dashboard.pid` | absent | n/a | n/a |
+| `.loki/dashboard/dashboard.pid` | `87992` | **DEAD** | **mtime Jul 31 (8 days)** |
+| `$TMPDIR/loki-local-ci.lock` | `61876` | LIVE (real local-ci) | current |
+| `.loki/app-runner/app.pid` | absent | n/a | n/a |
+
+The dashboard pid file is the same measured shape as the pgid orphans: a dead
+identifier, days old, still on disk, still trusted by a code path that kills.
+
+---
+
+## Findings, ranked by blast radius
+
+### FINDING 1 (HIGH -- fixed here): shared dashboard pid is killed unverified
+
+**Evidence.** `autonomy/run.sh:1597-1600` reads a pid from
+`~/.loki/dashboard/dashboard.pid` and sends it `kill` then `kill -9` with no
+liveness and no identity check.
+
+Removal of that file happens ONLY on explicit stop paths -- `run.sh:16608`,
+`:16658`, `:17399`, and `autonomy/loki:7078-7079`. **No trap covers a crash or
+Ctrl+C.** That is precisely why the measured copy in this checkout is 8 days
+stale with a dead pid.
+
+**Reachable:** yes. Called from `cleanup` on both stop paths (`run.sh:25311`,
+`:25384`).
+
+**Blast radius: the worst class.** The file lives under `~/.loki`, so it is
+machine-global -- the victim need not belong to this project. A recycled pid
+names an unrelated live process and receives `kill -9`.
+
+**Why `kill -0` would not have fixed it:** a recycled pid IS alive, so a
+liveness check passes. This is the identical insufficiency the `loki.pgid`
+self-check had. The guard has to check IDENTITY.
+
+**Fix applied.** `_loki_pid_looks_like_dashboard()` at `autonomy/run.sh:1526`,
+gating the kill at `:1597`. It mirrors `_app_runner_pid_is_ours`
+(`app-runner.sh:242`) and **fails OPEN**: when `ps` reports nothing we signal
+exactly as before, so a legitimate dashboard is never left running by this
+check. The only behavior change is refusing to kill a process positively
+identified as not a dashboard.
+
+Deliberately NOT done: adding a trap to remove the pid file on crash. That is a
+larger change across the dashboard lifecycle, and the identity guard already
+makes a stale file harmless at the point where it does damage. The stale file
+still being present is untidy, not dangerous, once the killer verifies.
+
+**Test:** `tests/test-stale-dashboard-pid.sh`, 6/6, mutation-verified below.
+
+### FINDING 2 (MEDIUM -- reported, not fixed): `app_runner_stop` group-kills unverified
+
+**Evidence.** `app-runner.sh:1407` falls back to reading `app.pid` from disk,
+then `:1456` sends `kill -TERM "-$_APP_RUNNER_PID"` -- a **process-group**
+signal -- without an identity check.
+
+The repo already has the right guard: `_app_runner_pid_is_ours`
+(`app-runner.sh:242`). It is called at `:1657` and `:1829` but **not** on the
+stop path. `app-runner.sh` has **no trap at all** (`grep -n "trap " ` returns
+nothing), so `app.pid` survives a crash exactly like the pgid file did.
+
+**Blast radius:** higher per-hit than Finding 1 (a group signal reaches a whole
+tree) but narrower reach: the file is project-local (`.loki/app-runner/`), not
+machine-global, and was absent on this host. Not fixed because it is a third
+file and the lead scoped this task to one; it needs its own change adding
+`_app_runner_pid_is_ours` to the stop path.
+
+### FINDING 3 (LOW -- reported, not fixed): local-ci lock guard is a no-op under `/bin/bash`
+
+**Evidence.** `scripts/local-ci.sh:104`:
+
+```
+trap '[ "$$" = "$_lci_owner" ] && rm -f "$_lci_lock" 2>/dev/null || true' EXIT
+```
+
+The scar comment above it (`:99-103`) correctly diagnoses that a bare EXIT trap
+fires in every subshell and that a finishing child deleted the parent's lock.
+**But `$$` does not change in a bash subshell, so this guard does not
+discriminate.** Verified:
+
+```
+$ bash -c 'p=$$; ( [ "$$" = "$p" ] && echo same )'
+same
+```
+
+The correct discriminators are `BASHPID` (bash 4+) or `BASH_SUBSHELL` (works in
+3.2). Measured on this host: the script is `#!/usr/bin/env bash` which resolves
+to **bash 5.3**, where `BASHPID` is available. But `/bin/bash` here is **3.2**,
+where `BASHPID` is unset and `BASH_SUBSHELL` is the version-safe choice.
+
+**Blast radius: benign** by the lead's own criterion. Worst case is two
+concurrent local-ci runs starving each other into phantom failures -- costly in
+time and trust, but it kills nothing. Left unfixed as out of scope; the one-line
+change is `[ "${BASH_SUBSHELL:-0}" = 0 ]`.
+
+Note on the sibling fix already shipped: `_loki_remove_pgid_file`
+(`run.sh:25189`) uses `${BASHPID:-$$}`. Under bash 3.2 that collapses to `$$`
+and the guard degrades to the no-op. **The degradation direction is safe** -- a
+subshell deletes the pgid file early, the reap then finds no file and skips, so
+an orphan survives and nobody gets killed. `run.sh` is also `#!/usr/bin/env
+bash` (5.3 here), so this is a portability note, not a live defect. Worth
+knowing: the mutation test for that guard asserts *source text* contains
+`BASHPID`, so it proves the code is present, not that it behaves correctly on a
+3.2 host.
+
+---
+
+## Already guarded -- do not re-audit
+
+Checked and found correct. Listed explicitly so this ground is not covered
+twice.
+
+- **`autonomy/lib/lock.sh:56-67`** -- `_loki_lock_is_stale` requires the
+  sentinel PID to be dead **AND** mtime > 30s. Both conditions, not either.
+  Correct.
+- **`cleanup_orphan_pids` (`run.sh:2300+`)** -- reaps only on liveness AND
+  parent death AND (for wrappers) idle-past-budget AND no live engine child.
+  Also self-skips `$$`. Correct, and notably stricter than the pgid reap was.
+- **`_app_runner_pid_is_ours` (`app-runner.sh:242`)** -- a real identity token
+  captured post-exec, failing open on a missing token so a live app is never
+  falsely killed. Correct where it is called; see Finding 2 for where it is not.
+- **`_app_runner_collect_descendants` (`app-runner.sh:289`)** -- refuses pid
+  0/1 and walks parent-child links from our own pid only, so it structurally
+  cannot signal outside our subtree. Correct.
+- **`status.ts:266-270` and the bash status reader (`loki:5092`)** -- both do
+  `os.kill(pid, 0)` before reporting. A wrong answer only mislabels a URL in
+  status output. Benign even when stale.
+- **The `CLEAR`/`KEEP` registry check (`run.sh:1545-1560`)** -- correctly
+  refuses to tear the shared dashboard down while any other project holds a
+  live pid. This gates Finding 1's call site; the defect was the missing check
+  on the pid itself, not this decision.
+
+---
+
+## Mutation verification (Finding 1's fix)
+
+Each guard was reverted individually; the test must go red, and only on its own
+assertion.
+
+| Mutation | Before | After | Assertion killed |
+|---|---|---|---|
+| Identity check downgraded to bare `kill -0` | 6 pass / 0 fail | 5 / 1 | "a live non-dashboard process is refused" |
+| Call site ungated (guard present but unused) | 6 / 0 | 5 / 1 | "the kill is gated on the identity guard" |
+| `pid > 1` check removed | 6 / 0 | 5 / 1 | "pid 0/1, empty and malformed inputs refused" |
+
+Restored after each. Final: `bash -n autonomy/run.sh` clean,
+`tests/test-stale-dashboard-pid.sh` 6/6, `tests/test-pgid-stale-reap.sh` 8/8
+(unaffected).
+
+The first mutation is the important one: it replaces the identity check with
+exactly the insufficient guard (`kill -0`) that a reviewer would most likely
+propose, and the test catches it.

package/docs/VERIFICATION-COST.md

@@ -25,6 +25,8 @@ different answer on your hardware.
 | FULL local gate | 23 to 26 minutes | `LOCAL_CI_TIER=full bash scripts/local-ci.sh`, 166 checks, five runs on an M-series Mac |
 | FAST local gate | about 1 minute | `bash scripts/local-ci.sh`, the documented pre-push tier |
 | Shell suite, sharded | 352 seconds | 323 suites, `LOCAL_CI_SHARDS=4`; 1440 seconds serial, so 4.1x |
+| `loki verify` | 77 seconds | `loki verify HEAD~1` on this repository, 5 changed files; dominated by the project's own test suite, not by our gates |
+| of which, shipped-vs-dev CVE split | 418 ms | one extra `npm audit --omit=dev`; 0.5% of the run, and the reason the audit finding can say whether a CVE reaches users |
 | `loki outcomes` | under 1 second per receipt | `git blame` and one `git log` per changed file |
 | `loki intent status` | milliseconds | hash comparison against `.loki/spec/spec.lock` |
 | Agent readiness | milliseconds | filesystem checks only, no network |
@@ -50,6 +52,24 @@ non-forgeability requires the signed path
 (`LOKI_PROOF_GPG_KEY`, see [SIGNED-RECEIPTS.md](SIGNED-RECEIPTS.md)). We removed
 our own "non-forgeable" claim in v7.111.0 after finding it false on that path.
 
+**The tests gate depends on correctly identifying YOUR test runner, and it has
+been wrong before.** Until 2026-08-07 the runner was chosen by grepping
+`package.json` for `"jest"` / `"vitest"` / `"mocha"`, which matches a
+**devDependency**. This repository is the case that exposed it: jest is a
+devDependency with no jest config while `scripts.test` runs `bash -n` plus
+`node --test`, so verify ran jest, jest globbed 895 files that are not jest
+tests, and `loki verify` returned BLOCKED on a clean tree -- permanently, for a
+defect that did not exist. It now reads `scripts.test` with a JSON parser and
+runs what the project declares.
+
+We record this rather than quietly fixing it because a false BLOCK is the more
+damaging direction of that error: a gate that cries wolf on every run trains
+you to ignore the verdict, which costs more than the gate ever earned. If the
+tests gate reports a runner you do not use, that is a bug in our detection, not
+a finding about your code -- the runner is named in
+`.loki/verify/evidence.json` under `deterministic_gates[].runner` so you can
+check which one it picked.
+
 **Only four of the eight quality gates are agent-independent.** Static analysis,
 mock-integrity, test-mutation and documentation coverage do not ask a model
 anything. The other four involve model judgment and are labelled ASSESSMENTS
@@ -95,179 +115,39 @@ above less trustworthy:
 LOCAL_CI_TIER=full bash scripts/local-ci.sh   # the full gate, timed
 loki outcomes --json                          # post-merge outcomes, or UNKNOWN with reasons
 loki intent status --json                     # spec-vs-intent drift
-loki proof verify <id>                        # re-hash a receipt, exit 1 on tamper
+loki proof verify <id>                        # re-hash a receipt (see below)
 bash tests/test-competitor-verify-surface.sh  # the competitor CLI measurement
 ```
 
-If a number here does not reproduce on your machine, that is a defect and we
-want the report.
-
-## How we compare, and what we cannot measure
-
-A goal was set to be "2-10x better than factory.ai, cognition devin, 8090.ai
-and replit". This section reports what is measurable and refuses the rest.
-
-### What is NOT benchmarked, and why
-
-None of those four products has a runnable local arm. Checked on this machine:
-
-```
-droid    NOT installed        devin    NOT installed
-replit   NOT installed
-claude   on PATH              aider    on PATH
-codex    on PATH              loki     on PATH
-```
-
-Devin and 8090 are hosted services with no CLI. Factory's droid and Replit run
-cloud-side. `benchmarks/bench/adapters/` can drive a competing arm as a
-subprocess (`claude_code.py` invokes `claude -p` live), but it cannot drive a
-product that has no local binary.
-
-**So there is no "2-10x vs Factory/Devin/8090/Replit" number here, and any such
-figure elsewhere should be treated as unearned.** Publishing one would be the
-same error as sourcing a colour token from a frontend that does not ship: a
-number that looks authoritative and measures something else.
+### Prove the tamper detection yourself, in three commands
 
-### What IS measured: the verification surface
+The claim is narrow and worth stating exactly: editing a receipt's recorded
+facts is DETECTED. Run this against any receipt in `.loki/proofs/`:
 
-`tests/test-competitor-verify-surface.sh`, run on this machine:
-
-> **5 installed competitor CLIs. 0 expose an output-verification command.**
-
-That is not a multiplier, it is a category. The comparison is not "our
-verification is faster" but "there is nothing on the other side to compare
-against". Re-run it yourself; it names each CLI it checked and SKIPs the ones
-it could not find rather than counting them as absent.
-
-### What the competitors say about verification, in their own words
-
-Verbatim, with the file each came from, so every line is checkable against the
-scraped corpora:
-
-| Source | Their words |
-|---|---|
-| `factory_ai/docs.factory.ai_missions_overview.md` | "**How do you maximize correctness?** Long-running plans accumulate errors." -- published as an OPEN QUESTION |
-| `factory_ai/docs.factory.ai_missions_overview.md` | "Without it, the mission **cannot reliably verify its own work**" (Missions require repo readiness Level 4+) |
-| `devin_cognition_ai/docs.devin.ai_admin_security.md.md` | "it can still experience **hallucinations, introduce bugs into code**, or suggest insecure code" |
-| `8090_ai/www.8090.ai_terms-of-service.md` | "**HUMAN REVIEW AND VERIFICATION OF ALL OUTPUT**" required, while the same ToS caps liability at "FIFTY US DOLLARS" and disclaims "ACCURACY" |
-
-Factory's is the most honest of the four: they name verification as an open
-research question rather than a solved feature, and they state that
-self-verification is a property of the ENVIRONMENT, not of the agent. We agree,
-which is why `loki readiness` measures the repo and not the model.
-
-### The claim we will actually defend
-
-Not a multiplier. A receipt you can recompute:
-
-```
-loki proof verify <id>     # re-hash the receipt; exit 1 on tamper
-loki outcomes --json       # anchored, or UNKNOWN with a named reason
+```bash
+ID=$(ls .loki/proofs | head -1)
+V() { loki proof verify "$ID" --json | python3 -c 'import json,sys;print(json.load(sys.stdin)["hash_ok"])'; }
+V                                                    # True
+python3 -c "import json;p='.loki/proofs/$ID/proof.json';d=json.load(open(p));d['files_changed']={'count':999999};json.dump(d,open(p,'w'))"
+V                                                    # False
 ```
 
-Against "human review and verification of all output" and "cannot reliably
-verify its own work", an anchored receipt is a categorical difference. It is
-also falsifiable: if `loki outcomes` reports UNKNOWN, we say UNKNOWN. On this
-repo it reported ANCHORED 0 of 9 for weeks, and the fix
-(`facts.git.base_sha` was empty) is in the history.
+Measured on this repository: `True` -> `False` -> `True` after restoring.
 
-### Honest limits on this section
+**Read `hash_ok`, not `ok`.** They answer different questions and conflating
+them produces a false alarm. `hash_ok` is integrity: do the recorded facts still
+hash to the recorded digest. `ok` also folds in `tree_drift`, which is true
+whenever the working tree has moved since the receipt was written -- so an
+untampered receipt from last month correctly reports `ok: false` with
+`hash_ok: true`. An earlier version of this document said "exit 1 on tamper",
+which is wrong in exactly that way: a drifted-but-intact receipt also exits 1.
 
-- The verify-surface count is a check for a COMMAND, not for internal
-  verification a product may do without exposing it. A hosted product could
-  verify server-side and expose nothing to a CLI.
-- It measures what is installed HERE. A CLI absent from this machine is
-  reported as SKIP, never as a competitor that lacks the feature.
-- Nothing here measures build quality, speed, or cost against those four.
-  Those comparisons are not available to us and are not claimed.
+**What this does NOT establish.** Integrity is not provenance. On the unsigned
+path a party who rewrites the facts AND recomputes the digest passes this check
+-- see the forgeability limit above. Provenance requires the signed path
+(`LOKI_PROOF_GPG_KEY`, [SIGNED-RECEIPTS.md](SIGNED-RECEIPTS.md)), and the remote
+client reports the two separately for that reason: VERIFIED, UNSIGNED,
+UNCHECKED and TAMPERED are four distinct verdicts, never collapsed.
 
-### Measured: what the harness is worth, model held constant
-
-The one comparison we CAN run. `benchmarks/bench/matrix.sh` defines two
-configs against the same task and the same model:
-
-- `baseline` -- raw model, minimal orchestration (in-repo comment: "the
-  Replit/Cursor mode")
-- `full` -- the harness: council, code review, self-heal, auto-tune
-
-Paired on identical tasks, both arms on `haiku`, from
-`benchmarks/bench/results/`:
-
-| Task | harness (`full`) | raw model (`baseline`) | verdict |
-|---|---|---|---|
-| `hard-2-ledger` | **4/4**, $0.89 | **0/4**, $0.19 | harness wins outright |
-| `hard-1-order-api` | 3 runs, 1.00, $0.58 | 1 run, 1.00, $0.20 | harness cost 2.8x for nothing |
-| `multifail-1-two-modules` | 2 runs, 1.00, $0.27 | 1 run, 1.00, $0.14 | harness cost 1.9x for nothing |
-
-**Two of the three paired tasks are UNFAVOURABLE to the harness.** That is the
-honest headline, and it is narrower than "the harness is worth more than the
-model":
-
-**The harness matters on hard tasks and is pure overhead on easy ones.**
-
-On `hard-2-ledger` the raw model never finished the task across 4 trials and
-the harness finished it every time. That is not a percentage improvement; the
-baseline success rate is zero. It cost 4.7x more per run and produced a working
-result instead of nothing.
-
-On the other two, both arms succeed and the harness simply costs 1.9x-2.8x
-more. We publish those rows because the direction is unfavourable to us, and a
-benchmark table that only survives its favourable rows is an advertisement.
-
-**Two further baseline cells were attempted and produced NO data.**
-`tokenheavy-1-crm` (2 trials) and a wider `hard-1-order-api` (3 trials) both hit
-the 1200s cell timeout, which caps all trials of a cell together. The harness
-wrote no result file and the runner said so:
-
-> WARNING: cell haiku-baseline / tokenheavy-1-crm wrote NO new result (likely
-> timed out at 1200s). This cell is MISSING from the report.
-
-They are absent from the table rather than counted as failures. A timeout is
-not evidence the raw model cannot do the task; it is evidence we did not
-measure it. Reporting them as baseline losses would have made the harness look
-better on data that does not exist.
-
-An earlier version of this section reported only the first two tasks and read
-as a stronger claim than the data supported. The `multifail` baseline arm was
-run afterwards specifically to test whether the claim would survive more data.
-It did not survive intact, and the table was corrected rather than the
-measurement dropped.
-
-Aggregate across all recorded cells:
-
-| Cell | n | success (median) | cost (median) |
-|---|---|---|---|
-| `haiku` + harness | 11 | 1.00 | $0.54 |
-| `opus` + baseline | 5 | 1.00 | $0.83 |
-| `haiku` + baseline | 8 | 0.50 | $0.19 |
-
-The cheap model WITH the harness matches the expensive model without it, at
-35% lower cost. Treat that as directional, not as a headline: the task sets
-differ between those three cells, which is exactly why the paired table above
-is the one that carries the argument.
-
-**Limits of this measurement, stated so it cannot be over-read:**
-
-- Two paired tasks. `hard-1-order-api`'s baseline arm is n=1.
-- It measures OUR harness against OUR baseline config. `baseline` is a
-  documented stand-in for "raw model, minimal orchestration", NOT a
-  measurement of Replit, Cursor, or any other product.
-- Success is a held-out acceptance exit code, not a judgement of code quality.
-- Reproduce it: `LOKI_BENCH_SPEND_APPROVED=1 bash benchmarks/bench/matrix.sh pilot`.
-  The spend interlock is default-deny on purpose; a benchmark that starts a
-  paid tool without an explicit opt-in is how a surprise bill happens.
-
-**The exclusion rule, observed on a live run rather than asserted.** A fresh
-`haiku-full` trial on `hard-1-order-api` (2026-08-07) hit the 1200s adapter
-timeout. The held-out grader inspected the workdir and returned
-`success: true` -- there WAS a passing artifact. The harness still recorded
-`measured: false`, `unmeasured_reasons: ["adapter exit_status=timeout"]`, and
-excluded the trial from k/N, with this note in the result file:
-
-> "the held-out grader's own verdict on whatever was in the workdir. Real
-> evidence about the ARTIFACT; NOT evidence that a run happened."
-
-A naive harness scores that 1/1. Ours scores it 0/0 and says why, so the
-`hard-1-order-api` row above stayed at n=3 instead of being inflated to n=4 by
-a run that never finished. The number in this document is smaller because of
-that rule, which is the point of having it.
+If a number here does not reproduce on your machine, that is a defect and we
+want the report.