loki-mode 9.78.0 → 9.81.0

package/docs/v10/GUARDS.md

@@ -450,3 +450,84 @@ previously mismarked two merged guards (S-16, S-74) as PENDING.
   (S-139) asserts the VIOLATION fires for a `released@` row stamped after
   npm's newest publish; T39b asserts it stays silent for a row stamped
   before it.
+
+## 14. Eval harness orphans survived the arm's process-group KILL (EV-10)
+
+- **Incident:** 00:08Z, pulse ORPHAN_WORKTREE flagged 6 orphaned processes
+  left running after `eval/loki10/harness.py` runs against the legacy arm;
+  all 6 were stopped by PID once found. Each was a detached
+  `/tmp/loki-run-*.sh` loop legacy loki spawns via `setsid`, which had
+  reparented to launchd (ppid 1) once the arm process it came from exited.
+- **Root cause with evidence:** `kill_orphans()` only ever did
+  `os.killpg(timeout_pid, SIGKILL)` -- correct for anything still in the
+  `timeout` process group the arm ran under, but a `setsid`'d child
+  deliberately leaves that group and gets its own pgid, so the group KILL
+  can never reach it once the escape has completed. Reproduced locally: a
+  script that setsid-spawns a sleeper, confirms (by polling its pgid) that
+  the escape finished, then exits still leaves the sleeper running after
+  `capped_run()` returns and `kill_orphans()` has already run.
+- **The guard:** `kill_orphans()` now also reaps by two signals that are
+  never a process name or pattern: any PID recorded in a `*.pid` file under
+  the run's own clone (`<work>/.loki/**/*.pid`, one per line, never through
+  a symlinked `.loki/` or a non-regular/oversized file), and any process
+  anywhere on the box whose cwd (`/proc/<pid>/cwd` where present, else one
+  system-wide `lsof -a -d cwd -Fpn` call, resolved with `os.path.realpath`)
+  is that same clone directory or a path inside it. Both are scoped to
+  `cwd`, the exact per-run clone `capped_run()` was given. Every candidate
+  PID is vetted by `_pid_is_ours()` before it ever reaches `os.kill`: never
+  `<= 1` (kill(2) treats -1/0 as "every process the caller may signal," not
+  a single-PID cleanup), never this process or its parent, and never older
+  than the run's own start (a stale or forged `.pid` entry naming an
+  unrelated, longer-lived process is left alone).
+- **The test that proves it fires:** `eval/loki10/test-harness.sh`, EV-10.
+  Leg Rj: the `setsidorphan` stub-arm mode setsid-spawns a sleeper in its
+  own clone, polls until the sleeper's pgid equals its own pid and only
+  then records an "escaped" marker (ruling out a vacuous pass from a spawn
+  that never actually left the group), then exits; the leg asserts the
+  marker is present and the sleeper is dead once the harness run returns.
+  Leg Rk: the same escape, chdir'd out of the clone first so the cwd path
+  cannot see it, reaped instead from its own PID in a `.loki/*.pid` file; a
+  decoy PID that predates the run, planted in the same file (plus a literal
+  `-1`), is asserted to survive. Leg Rl exercises `_clone_orphan_pids`
+  directly to assert `-1`/`0`/`1` and a pre-run PID are filtered before any
+  kill is attempted, without ever risking a live `kill(-1, SIGKILL)`.
+  Mutation-tested: disabling the cwd path fails only Rj, disabling the
+  pidfile path fails only Rk, and bypassing `_pid_is_ours` (checked offline
+  against a throwaway copy of the module, never run against a live process)
+  lets `-1` through, which Rl's assertion catches.
+## 15. Two releases pushed with unbaselined secret-shaped fixtures never reached npm (E-60)
+
+- **Incident:** Security Audit (the `secret-scan` job in
+  `.github/workflows/security-audit.yml`, a required-ci gate) failed on the
+  release SHAs for v9.79.0 (`9dc0b1d3`, 2026-09-27) and v9.80.0 (`085306ae`,
+  same day). Both pushes landed on main, but `publish-npm` never ran because
+  the required Security Audit conclusion was red, so npm latest stayed at
+  9.78.0 through both releases -- 00:45Z is when the fix (`5190756f`) landed
+  and unblocked v9.80.1.
+- **Root cause with evidence:** `5190756f`'s own commit message names the
+  findings exactly: "a planted AWS-shaped string for the deep-verify
+  secret-scan test and two CANARY/WITHHELD token canaries." Three new Loki 10
+  test fixtures were secret-shaped (matched gitleaks' built-in rules) and
+  landed on main across the two releases without a matching
+  `.gitleaksignore` entry, so gitleaks' full-history scan in CI reported
+  three new, unreviewed findings and blocked exactly as designed -- the gate
+  worked; nothing local had run the same scan before push to catch it there
+  instead of at CI, so two releases went out DOA on the fastest-available
+  feedback channel (CI, not a local gate).
+- **The guard:** E-60 adds a fast-tier gitleaks step to `scripts/local-ci.sh`
+  (`_FAST_KEEP` entry `"gitleaks (secrets, origin/main..HEAD)"`): when a
+  `gitleaks` binary is on PATH, every pre-push run scans just the commits the
+  branch adds on top of `origin/main` against the same reviewed
+  `.gitleaksignore` baseline CI uses, so an unbaselined secret-shaped fixture
+  is caught before push, not after a release SHA is already tagged. Fails
+  closed the other way too: an absent binary prints an explicit SKIP line
+  (never a silent PASS), matching this file's shellcheck-absent posture.
+- **The test that proves it fires:** `tests/test-local-ci-gitleaks.sh`
+  (registered in `tests/run-all-tests.sh` and `tests/shard-durations.tsv`).
+  Static assertions confirm the skip-not-pass path, the `.gitleaksignore`
+  baseline, the `origin/main..HEAD` scope, and `_FAST_KEEP` membership; when
+  a gitleaks binary is present, two live temp-repo scenarios drive the exact
+  command shape `scripts/local-ci.sh` runs: a commit adding a literal
+  AKIA-shaped token fails the step, and a commit adding the identical
+  characters via source-level concatenation (never contiguous in the
+  committed bytes) passes it.

package/docs/v10/GUIDE.md

@@ -0,0 +1,154 @@
+# Loki 10 engine guide
+
+Opt-in: set LOKI_ENGINE=v10 <!-- loki10-default -->
+
+Loki 10 is the rewritten engine (docs/v10/ENGINE.md). It is not the default
+yet. Every command below only runs when LOKI_ENGINE=v10 is set; leave it
+unset and `loki` keeps using the current (legacy) engine exactly as before.
+
+Some pieces named in this guide are still being built. Each one below says
+so plainly instead of describing a finished feature.
+
+## Quickstart
+
+Put the task first, as a quoted multi-word string, so bin/loki's router
+sends it to the v10 engine instead of the legacy one. A one-word argument
+that is not an issue ref, or any first argument starting with `-`, stays
+on the legacy engine.
+
+```
+LOKI_ENGINE=v10 loki "fix the login redirect loop" --no-pr
+```
+
+Issue refs work the same way:
+
+```
+LOKI_ENGINE=v10 loki owner/repo#123
+LOKI_ENGINE=v10 loki https://github.com/owner/repo/issues/123
+```
+
+bin/loki routes GitHub, GitLab (`.../-/issues/N`) and Jira
+(`.../browse/KEY`) issue URLs to the v10 engine, and the issue-fetch step
+(autonomy/issue-providers.sh) fetches title/body/labels from all three.
+Only the extra already-done check (an issue's `state` and whether it was
+closed by a merged PR) is GitHub-only; a GitLab or Jira issue still runs,
+just without that extra signal.
+
+Flags, from the engine's own `--help`:
+
+- `--no-pr`: build and verify, but skip opening a pull request.
+- `--deep`: request the deep verify pass; the implement time budget goes
+  from 480s to 1800s (`loki-ts/src/engine10/types.ts`).
+- `--provider <name>`: pick the coding provider for this run. See the
+  provider table below.
+- `--resume <run-id>`: continue a run that was interrupted.
+
+`--no-pr`, `--deep` and `--provider` are parsed by the supervisor
+(`loki-ts/src/engine10/supervisor.ts`) and take effect on a real run.
+`--resume` is not wired yet: passing it prints `engine10: --resume is not
+wired yet` and exits 2. Its shape is documented here because it is already
+fixed in `--help`.
+
+`loki status`, `loki verify` and `loki dashboard` are all built on main
+(status.ts, verify_cmd.ts, dashboard/server.ts) and route through cli.ts's
+table like any other subcommand.
+
+The free-text and issue-ref run itself (the supervisor that plans, builds
+and verifies) is wired into the CLI on main: `loki "<task>"` and `loki
+<issue-ref>` run the full worker flow (intake, plan, wall, implement,
+verify, seal) and then, in the supervisor, the pr stage, ending in the
+5-line summary below.
+
+Slack is not part of the v10 engine surface (no command, no notification
+hook) and is in progress, tracked with the other wave 3 items.
+
+## The 5-line summary
+
+A finished run prints five lines, one label each:
+
+```
+PR:         https://github.com/owner/repo/pull/1
+Verdict:    VERIFIED
+NOT PROVEN: full suite, app boot, council, security scan
+Cost:       $0.42 (claude, 212k tokens)
+Time:       4m12s (intake 11s, plan 20s, implement 3m10s, verify 31s)
+```
+
+- **PR**: the opened pull request URL, or `none` when the run used
+  `--no-pr` or never got that far. A draft PR adds the draft reason in
+  parentheses.
+- **Verdict**: one of VERIFIED, PARTIAL, ALREADY_SATISFIED, SPEC_CONFLICT or
+  FAILED.
+- **NOT PROVEN**: everything the run did not check, comma-joined. Never
+  empty by omission: a check that was skipped or deferred is always listed
+  here, not silently dropped. A flaky test found during verify appears as a
+  separate `; flaky ...` clause on the same line.
+- **Cost**: the provider-reported dollar figure and token count. When the
+  provider does not report cost, this line reads `not measured` instead of
+  `$0.00`; a missing cost is never rendered as free.
+- **Time**: total wall time, then each stage's own duration in parentheses.
+
+This formatter (`loki-ts/src/engine10/output.ts`, `formatSummary`) is
+called by the supervisor at the end of every run, so a real `loki "<task>"`
+or `loki <issue-ref>` run prints this block on completion. The block above
+is that function's real output, not a mockup.
+
+## NOT PROVEN
+
+NOT PROVEN lists what a run did not verify, so a passing run is never read
+as more thorough than it was. One entry is unconditional, written by Seal
+every time:
+
+- **The deep checks**: `full suite, app boot, council, security scan`.
+  They are deferred to a separate deep-verify pass, so Seal always lists
+  all four at seal time.
+
+Seal also adds an entry whenever it applies, for example: `cost not
+measured (no iteration ids recorded)`, `kill blocking not enforced` for
+any provider other than claude, `worker killed by the supervisor backstop
+(cap plus grace)`, `event log modified outside the engine`, a flaky test,
+a reverted test edit, a model override a non-claude provider could not
+honor, a task source the intake step did not record, an uncomputable
+diff, an individual check that did not run, or a failed receipt signature.
+The Cost line above separately reads `not measured` (never `$0.00`)
+whenever the provider reported no dollar figure.
+
+## Receipts
+
+A run writes `.loki/runs/<run-id>/receipt.json` and a matching
+`receipt.md`. The JSON receipt's `receipt_sha256` hashes the canonical JSON
+without the `verification` block, so verifying a receipt never has to trust
+its own signature field to compute the hash it is checking.
+
+`loki verify [run-id]` re-hashes the receipt and checks its signature. A
+receipt with no signing key configured (or a key that failed to load) reads
+`UNSIGNED` and is never presented as attested; a signed receipt reads
+`SIGNED (kid ...)`.
+
+## Provider table
+
+| Provider | Cost measured | Kill blocking enforced |
+|----------|----------------|--------------------------|
+| claude   | yes            | yes                      |
+| codex    | no             | no                       |
+| cline    | no             | no                       |
+| aider    | no             | no                       |
+| opencode | not supported  | n/a                      |
+
+For codex, cline and aider, a run's Cost line reads `not measured` and the
+NOT PROVEN line adds the literal entry `kill blocking not enforced`
+(`loki-ts/src/engine10/stages/seal.ts`). opencode is not one of engine10's
+provider names (`claude`, `codex`, `cline`, `aider`); it is not supported.
+
+## After the flip
+
+Once v10 becomes the default (docs/v10/DECISIONS.md, D29), `loki`, `loki
+status`, `loki verify` and `loki dashboard` with no `LOKI_ENGINE` set will
+mean the v10 commands described above. The previous engine stays fully
+reachable: `loki legacy <args>` and `LOKI_ENGINE=legacy` both route to it,
+unchanged. Nothing is removed at the flip.
+
+That flip has not happened on main as of this guide. The opt-in line near
+the top of this file and of README.md's section is the one place that
+records which state we are in; it changes in the same commit as the flip
+itself.

package/docs/v10/METRICS.md

@@ -217,3 +217,29 @@ remains the only cap on this step. Guarded by
 ## Over-budget stops
 
 - 2026-09-27T21:33Z: batch 8 workflow wf_ff8d537c-40e stopped. Its last two reviewers (S-191, S-195) ran about 55 minutes against the 30-minute reviewer budget with no verdict (pulse AGENT_OVER_BUDGET at 21:31Z: "S-191 building MEDIUM (55.9 min, budget 30 min)"). Re-dispatched as two standalone reviewers with 20 and 25 minute scopes. Load average was 15 to 17 during the stall.
+
+## Loki 10 eval: first real arm numbers (2026-09-27, smoke, not the gate)
+
+Raw `claude -p` arm (EV-4), model claude-opus-5-5, 900s cap, 3 public tasks, harness from the EV-1 branch with EV-3 isolation (auth via keychain access token; no credential in any result file):
+
+| task | completed | time to PR | cost (provider-reported) |
+|---|---|---|---|
+| pub-more-itertools-1192 | yes | 20s | $0.1266 |
+| pub-click-2877 | yes | 34s | $0.1861 |
+| pub-humanize-152 | yes | 45s | $0.2704 |
+
+`summarize --markdown`: 3/3 completed, p50 34s, p90 45s, $0.1944 per completed task, 0 capped. This is a 3-task smoke, not the 29-task gate. It sets the bar the v10 fast lane must meet on completion and cost per completed task (D29), which is far tighter than the 5-minute p50 target.
+
+## Loki 10 eval: raw claude -p arm, full 29 tasks (2026-09-27 23:05-23:16Z)
+
+Model claude-opus-5-5, harness 84c22568, 900s cap, --parallel 3, EV-3 isolation. Results: ~/loki-ci-logs/eval-raw-claude-20260927T231653Z/.
+
+| Arm | Completed | Rate | p50 time to PR | p90 time to PR | Cost per completed | Cost measured | Capped |
+|---|---|---|---|---|---|---|---|
+| raw-claude | 27/29 | 93.1% | 39s | 68s | $0.2363 | 29/29 | 0 |
+
+Misses: pub-click-3059 (hidden tests failed: wrong metavar bracketing), pub-humanize-174 (hidden tests failed: rounding boundary cases). Total provider-reported spend $6.38.
+
+## Loki 10 engine: first real run (E-14 smoke, not the gate)
+
+pub-more-itertools-1192 through the E-14 glue entry, real claude, --no-pr: 30s wall (intake 0s, plan 16s, implement 12s, verify 3s, commit and seal under 1s), hidden test passes, $0.4486 over 2 sessions (plan $0.2264, implement $0.2223), verdict PARTIAL only because ruff is not installed. Raw claude on the same task: 15s, $0.1290. The planner session costs as much as the implement session.

package/docs/v10/PROGRESS.md

@@ -1248,3 +1248,44 @@ sample.
 - Eval numbers so far: none (no arm has run yet).
 - ETA for v10.0.0: thin path end to end targeted for about 01:00 UTC; gate decision at 03:00 UTC.
 - Top blocker: E-01 (shared types) gates phase B; augmentiq has only 3 issues, so the augmentiq share of the eval is 1 task, not 5.
+
+## 2026-09-27T22:57Z: Loki 10 engine progress
+- On main (`git ls-remote origin refs/heads/main` = 694866fc): engine wave 1 E-01..E-13 except E-14, plus E-11 (both halves), E-16, E-20; the eval harness EV-1 with EV-3 isolation and the 29 EV-2 tasks (EV-7 nonce fixes). tests/engine10 plus spawn guard 185 pass, 0 fail; eval/loki10/test-harness.sh 77 passed.
+- Eval baseline: a no-op arm over all 29 tasks graded every task not completed, none task_invalid (108s). No real arm numbers yet: EV-4 (raw claude -p, 3 tasks) and EV-5 (legacy, 1 task) are running.
+- In flight: E-14 end to end plus the first real v10 run on pub-more-itertools-1192; reworks E-15, E-17, E-18, E-21, E-22; E-19, E-23, E-TG.
+- Incidents fixed forward: two orphan loki test runs over 24 hours (stopped by PID; E-00 guard merged); main CI red on Coverage from engine spawns without env (72423684, 19128f6b).
+- ETA for v10.0.0: first real v10 task within the next hour; gate decision at 03:00 UTC.
+- Top blocker: E-14 integration (first time the modules run together).
+
+## 2026-09-27T23:03Z: first real eval numbers
+- Raw claude -p arm, 3 public tasks: 3/3 completed, p50 34s, p90 45s, $0.1944 per completed task (docs/v10/METRICS.md). Legacy arm and v10 arm not yet measured.
+- Implication for the gate: v10 must match raw completion and cost per completed task. Any extra model call (planner, Wall author, fix round) adds cost against a $0.19 baseline, so the fast lane must skip stages a small task does not need.
+
+## 2026-09-27T23:27Z: Loki 10 progress
+- Raw claude -p full arm: 27/29 (93.1%), p50 39s, p90 68s, $0.2363 per completed task (docs/v10/METRICS.md).
+- v10 first real task (E-14 glue entry, commit 9d9ff7d2): fixed in 30s, hidden test result "1 passed, exit 0", $0.45 (plan $0.23 plus implement $0.22; `cost-summary.py --json` fully_measured true) against raw $0.13 on the same task.
+- On main (`git ls-remote origin refs/heads/main` = 44731e0f): every engine slice except E-14's glue, plus E-41 local-origin push; v9.79.0 tagged 23:12:55.
+- In flight: E-42 (real entry end to end, Rule of Two for the pr stage, harness v10 arm on one task), E-43 cost knobs, E-44 output fixes, E-33..E-35, EV-5 legacy full arm.
+- ETA: first harness-scored v10 task by about 00:15Z; full v10 arm by about 01:30Z; gate decision 03:00Z.
+- Top blocker: cost. Plan plus implement is about 2x raw per task on this sample; the gate requires v10 cost per completed task at or below raw ($0.2363).
+
+## 2026-09-28T00:11Z: Loki 10 progress
+- On main (`git ls-remote origin refs/heads/main` = addda547): E-42 real entry end to end (run.ts glue deleted), E-44 output fixes, agent SDK 0.3.283 (9db2a4ed), E-51 eval PR path regression, and train E12 (23b20e6f): E-33 gate report, EV-6 publish script, E-38 legacy arm pin, E-39 interrupt/resume, E-40 live PR smoke. Train checks: engine10 349 pass 0 fail, tsc exit 0, test-harness 79/0, test-gate-report 34/0.
+- SDK route: claude-opus-5-5 now runs through the SDK (probe: `success OK`); 0.3.267 rejected it. The same probe cost $0.33 for a one-word reply, so fixed per-call overhead, not task work, drives v10 cost.
+- Legacy arm (EV-5): 3 results so far (2 hidden pass), each at the 900s cap; runner alive at parallel 3, ETA about 02:15Z. Incident: 6 then 2 legacy watchdog loops escaped the harness timeout group (ppid 1, 30 min); stopped by PID; guard slice EV-10.
+- In flight (wave E13, 12 builders): E-45 cost path rework, E-36 preflight (plus E-37), E-32 rebase, E-34 docs refresh, E-52..E-56, E-58 (makes E-47 green), E-59 (makes E-50 green), EV-10.
+- ETA: E-45 merge about 00:45Z, EV-8 5-task cost check right after, EV-9 full v10 arm about 01:00Z to 01:45Z, gate decision 03:00Z.
+- Top blocker: cost per completed task (raw $0.2363); E-45 plus the per-call overhead finding decide it.
+
+## 2026-09-28T01:51Z: Loki 10 progress (gate numbers, EV-8, MODERNIZE.md)
+- Gate numbers, small tier, claude-opus-5-5 (D30 targets: completion 96.5% or higher, cost per completed task $0.118 or lower, time to a correct result at most raw):
+  - raw claude -p, 29 tasks: 27/29 (93.1%), p50 39s, p90 68s, $0.2363 per completed task (~/loki-ci-logs/eval-raw-claude-20260927T231653Z/results.jsonl).
+  - legacy (EV-5, global loki v9.78.0, 29 tasks, finished): completed 15/29 (51.7%), hidden tests pass 26/29, p50 190s, p90 441s; cost not measured on any run (0/29 provider-sourced) (~/loki-ci-logs/ev5-legacy/results.jsonl). Legacy commits locally and opens no PR, which is why hidden-pass exceeds completed.
+  - v10: full arm not run yet. EV-8 (5 tasks, results below) is the only v10 measurement.
+- EV-8 status: first run invalid (harness ran the global loki v9.78.0, and the main checkout had agent SDK 0.3.267, which rejects claude-opus-5-5; every session exited in 1s with 0 tokens). Rerun from the repo's bin/loki after `bun install` (node_modules 0.3.283), same 5 tasks (raw on them: 5/5, $0.2161 per completed, p50 41s):
+  - A, full design (Wall on sonnet): 4/5, $0.5954 per completed (one run's cost unrecorded, so a floor), p50 116.5s, p90 322s.
+  - B, Wall off: 5/5, $0.3976 per completed, p50 65s, p90 115s (~/loki-ci-logs/ev8r-{A,B}/results.jsonl).
+  - Reading: v10 is 1.8x raw cost at best and 1.6x raw time; the D30 target is 0.5x cost. The Wall session alone was $0.119 in one run. Next lever: the cascade (sonnet first, opus only on a Wall or test failure), then per-call-type cost records.
+- MODERNIZE.md: CTO and Architect dispatched 01:52Z; ETA 02:45Z for the design doc, Part 2 slices cut by the PO right after.
+- CI: main red at d00c5ede and 7c3dea3d from train E13 (providers.test.ts dies where claude is absent because preflight exits in-process; test-engine10-dispatch.sh expected 2 cli.ts lines, E-32 made it 3). Both fixes are in train E14 (404 pass, 0 fail locally, plus a no-claude PATH run); push after the remaining no-claude check.
+- Top blocker: cost. v10 at $0.40 per completed task vs the $0.118 target.