loki-mode 9.12.6 → 9.16.0

package/docs/LOOP-CANDIDATE-PROPOSAL-v1.md

@@ -0,0 +1,167 @@
+# loop-candidate-v1: proposal only
+
+**GATE VERDICT: FALSE.** Nothing here is implemented, and nothing may be until
+the gate closes. See the last section for the single gate-closing action.
+
+| Gate | Verdict | Evidence |
+|---|---|---|
+| Local full tier | **DO NOT PUSH** | 161 passed, 3 failed, 0 deferred |
+| Remote exact-SHA | **failure** | `Tests@dda8beec` completed/failure, Shell shard 2 |
+
+## 0. Why this proposal is small
+
+The six cheaper surfaces come first by directive, and the honest finding is
+that **the one evidenced product defect this session surfaced cannot be
+addressed by any of them**: `loki onboard --stdout` emits 0 bytes and exits 0
+on a 4100-file repository while working on a small one. It is a bash command
+that never calls a model, so memory, retrieval, skills, prompts, tool
+descriptions, compression and routing are all inapplicable by construction.
+
+That leaves routing as the only surface with both an evidenced question and
+existing instrumentation. This proposal is therefore scoped to **one routing
+candidate**, and it is deliberately the smallest thing that could clear the
+preregistered bar.
+
+## 1. Provenance
+
+Every field pinned, no "current" or "latest":
+
+| Field | Value |
+|---|---|
+| baseline runtime | `dda8beec` (origin/main), loki-mode 9.12.5 |
+| candidate runtime | **none** -- this proposal changes no runtime code |
+| harness | `tools/loop-harness-report.py` @ `2910a95d` (read-only) |
+| task corpus | 1 spec, `wordcount` (pure helper + test), verbatim in both arms |
+| grader | the produced test suite, executed; plus receipt `iterations.succeeded` |
+| prompt version | unchanged; main-loop prompt is **NOT ATTRIBUTABLE** (assembled in memory, `run.sh:8987`) |
+| skill version | unchanged |
+| routing arm A | `LOKI_SESSION_MODEL=sonnet` |
+| routing arm B | `LOKI_SESSION_MODEL=opus` |
+| verifier set | unchanged; **records carry no cost/latency**, see §5 |
+
+**The prompt row is a known provenance hole.** The main-loop prompt cannot be
+versioned today, so any candidate that claims a prompt effect is unfalsifiable.
+This proposal therefore claims none.
+
+## 2. Success and noise criteria, preregistered
+
+- **Primary**: cost_usd per successful run, matched on the identical spec.
+- **Secondary**: wall_clock_sec, and `progress duration_ms` reported separately
+  -- these are NOT the same quantity and are never summed. Wall clock includes
+  orchestration; progress duration is measured work.
+- **Success requires**: `iterations.succeeded >= 1` AND the produced test suite
+  passes when executed. A receipt is not evidence the code works.
+- **Noise rule**: a difference below **25%** on the primary is declared noise
+  and not reported as an effect. Justification: the two prior UNMATCHED runs
+  differed by 2.0x on the same nominal models, which bounds run-to-run
+  variance well above any effect a single pair could detect.
+- **Minimum n**: 5 matched pairs. Below that, report "insufficient", never a
+  direction.
+
+## 3. Caps
+
+| Cap | Value |
+|---|---|
+| retries | 0 -- a failed arm is recorded as failed, never retried |
+| per-run timeout | 900s (matches the runs already executed) |
+| iterations per run | `LOKI_MAX_ITERATIONS=2` |
+| total spend ceiling | **$25**, hard stop; at ~$0.70/run that is ~35 runs |
+| canary population | **none** -- offline only; no production traffic |
+| canary window | n/a until offline clears |
+
+## 4. Evals
+
+**Deterministic regression**: the existing suites, unchanged and frozen at the
+baseline SHA. Any new failure disqualifies the candidate outright, regardless
+of primary-metric movement.
+
+**Ambitious artifact-level outcome**: one real `loki start` on a spec requiring
+a multi-file artifact (module + test + usage doc), graded by (a) the produced
+tests executing green, and (b) the receipt verifying via
+`api_evidence.receipts_report`. Flat-or-better is required; a cost win with a
+degraded artifact is a rejection, not a trade.
+
+## 5. Verifier lift versus p95 latency and cost -- BLOCKED, stated as such
+
+This element **cannot be satisfied today** and the proposal does not pretend
+otherwise.
+
+Verifier records carry no cost, latency, criterion, or terminal-outcome effect:
+`code_review_complete` emits exactly `review_id`, `source`, `iteration`; the
+three gate functions (`_evidence_`, `_invariant_`, `_semantic_gate_and_surface`)
+emit nothing structured at all. So the denominator for "lift vs p95 latency and
+cost" does not exist, and no matched on/off cohort can be computed.
+
+Closing this needs runtime instrumentation, which the directive excludes
+without an evidenced deterministic requirement. **This proposal therefore makes
+no verifier claim and proposes no verifier change.**
+
+## 6. One reversible candidate at a time
+
+Exactly one variable moves: `LOKI_SESSION_MODEL`. It is an environment
+variable, so reversal is unsetting it -- no code, no migration, no state.
+
+Structured-trace learning is **read-only**: `loop-harness-report.py` reports
+what traces contain and marks every underivable field UNKNOWN. It proposes
+nothing automatically.
+
+## 7. Replay, rollback, retention
+
+- **Frozen replay**: each run's receipt pins `base_sha`, `head_sha`,
+  `diff_sha256`, model, provider and cost. The spec is stored verbatim.
+- **Automatic rollback**: none needed -- no runtime change to roll back. If the
+  candidate loses, the env var is simply not set.
+- **Retained prior version**: baseline is `dda8beec` on origin/main, immutable.
+
+## 8. Trigger-to-receipt path
+
+Already present and verified by execution, not by grep:
+
+| Property | State |
+|---|---|
+| authentication | present |
+| idempotency | present |
+| dedupe | present -- `seen_delivery()`, lock-guarded bounded OrderedDict |
+| backpressure | present -- bounded queue, 503 shed |
+| bounded retry | present |
+| timeout | present |
+| dead-letter | **failures logged, not queryable** |
+
+The one gap is a queryable failure record. It is narrow and **not proposed for
+change** on this evidence.
+
+## 9. Human approval
+
+Required for: promotion to default, any spend beyond the $25 ceiling,
+destructive or security-boundary changes, and any external action (publish,
+tag, post). Not required for: running the offline arms within the ceiling.
+
+## What would falsify this candidate
+
+- any new deterministic regression -> reject
+- artifact outcome degraded -> reject even if cheaper
+- primary difference < 25% -> declare noise, no promotion
+- fewer than 5 matched pairs -> report insufficient, no direction claimed
+
+## THE SINGLE GATE-CLOSING ACTION
+
+The gate is red because of **three failures, none introduced by the held
+commits**:
+
+| Failure | Class | Fixable here? |
+|---|---|---|
+| `test-onboard-command` (6) | pre-existing; identical at `1c80c85ff~1` | needs a runtime fix to `autonomy/loki` |
+| `test-model-override` (1) | pre-existing; 65/66 identical at `1c80c85ff~1` | unknown, undiagnosed |
+| `bun run typecheck` | environmental; `tsc` not installed locally | one install |
+
+**The single next action: install the TypeScript toolchain so
+`bun run typecheck` can execute locally.** It is the only one of the three that
+is a local environment gap rather than a code defect, it is non-destructive and
+reversible, and it removes the one failure that is not telling us anything
+about the repository.
+
+That alone does not turn the gate green -- the two pre-existing suite failures
+remain, and deciding whether to fix them, accept the remote gate as the
+documented equivalent, or waive them is a founder call, not mine.
+
+**Implementation stops here.**

package/docs/LOOP-HARNESS-AUDIT.md

@@ -629,3 +629,56 @@ Three procedural constraints were learned by getting them wrong first:
 2. run arms sequentially, or contention confounds the latency column
 3. verify the produced artifact by executing it -- a receipt records what a run
    claimed to do, not whether the code works
+
+## Read-only gap check: one evidenced product defect
+
+The directive asks whether memory, skills, prompts, tool descriptions,
+compression, or routing can address any evidenced gap. Running the full tier
+surfaced a defect that none of those six can touch, because it is not a model
+problem at all.
+
+### `loki onboard --stdout` produces nothing on a large repository
+
+Reproduced deterministically, not inferred:
+
+```
+small repo (1 source file), explicit path   394 bytes
+small repo, cwd form                        394 bytes
+this repo (4100 tracked files), any form      0 bytes
+this repo, --depth 1                          0 bytes
+```
+
+It is not the argument form: both spellings work on a small repo. It is not
+scan depth: depth 1 fails identically. The command logs three INFO lines --
+"Analyzing project at", "Depth: 2 | Format: markdown", "Scanning source
+files..." -- then stops, writes no file, emits nothing to stdout, and **exits
+0**.
+
+Exit 0 with no output is the worst available shape. A caller that checks the
+exit code sees success; a caller that reads stdout gets an empty string and
+cannot tell "this project has no structure" from "the command gave up". It is
+the same class as the dashboard rendering an unmeasured cost as `$0.00`, which
+this release line already fixed twice.
+
+This is what `tests/test-onboard-command.sh` has been failing on: six
+assertions, all downstream of empty output. The suite fails identically at
+`1c80c85ff~1`, so it is pre-existing, not introduced this session.
+
+### Why none of the six named levers apply
+
+Memory, retrieval, skills, prompts, tool descriptions, compression and routing
+all shape what a MODEL is asked or told. This defect is in a bash analysis
+command that never calls a model: it walks the filesystem and formats markdown.
+No prompt change makes it emit output, and no routing decision is involved.
+
+That is itself the useful finding. The directive's ordering -- try the cheap
+model-facing levers before touching runtime -- is correct as a default and
+does not apply here, and saying so is more honest than proposing a prompt
+change that could not work.
+
+### Not fixed here
+
+Fixing it means changing `autonomy/loki`, which is a runtime change this
+directive excludes, and the failure mode (silent give-up at scale) needs its
+own diagnosis before a patch. Recorded with a reproduction so it is actionable
+rather than rediscovered.

package/docs/VERIFICATION-COST.md

@@ -0,0 +1,103 @@
+# What our verification costs, and what it does not prove
+
+This page exists because the most credible thing 8090 AI published was not a
+capability claim. It was a cost:
+
+> "the most common failure mode of vendor evaluation frameworks is to promise
+> that no operational burden falls on the customer team. That promise is
+> incompatible with a measurement signal that survives audit. The eval that
+> costs nothing to run is, in our experience, the eval that cannot be defended
+> in a regulatory inspection."
+
+They name a seven-minute-per-document human cost, a 50-document golden dataset,
+and several days per quarter of recalibration, and they refuse to automate the
+one human signal away. That refusal is what makes the rest of their numbers
+believable.
+
+So here are ours. Every figure below was measured on this repository, and the
+command that produces it is given so you can measure it yourself and get a
+different answer on your hardware.
+
+## Measured cost
+
+| Check | Cost | How it was measured |
+|---|---|---|
+| FULL local gate | 23 to 26 minutes | `LOCAL_CI_TIER=full bash scripts/local-ci.sh`, 166 checks, five runs on an M-series Mac |
+| FAST local gate | about 1 minute | `bash scripts/local-ci.sh`, the documented pre-push tier |
+| Shell suite, sharded | 352 seconds | 323 suites, `LOCAL_CI_SHARDS=4`; 1440 seconds serial, so 4.1x |
+| `loki outcomes` | under 1 second per receipt | `git blame` and one `git log` per changed file |
+| `loki intent status` | milliseconds | hash comparison against `.loki/spec/spec.lock` |
+| Agent readiness | milliseconds | filesystem checks only, no network |
+| Pre-edit snapshot | one `git diff` per run | write-once, at agent stop |
+
+The FULL gate is the honest headline: **a full verification run costs about 25
+minutes of wall clock**. We do not offer a mode that makes that free, because
+the checks that take the time are the ones doing the work.
+
+## What our verification does NOT prove
+
+Stated as plainly as we can, because a limits section that reads like marketing
+is worse than none.
+
+**A receipt is not proof the code is correct.** It proves a specific diff was
+subjected to specific checks and records what each returned. A change can pass
+every gate and still be wrong.
+
+**The unsigned receipt path is forgeable.** Someone who rewrites both the facts
+and the headline into a mutually consistent lie and recomputes the hash will
+pass verification. That is defense in depth, not non-forgeability. Neutral
+non-forgeability requires the signed path
+(`LOKI_PROOF_GPG_KEY`, see [SIGNED-RECEIPTS.md](SIGNED-RECEIPTS.md)). We removed
+our own "non-forgeable" claim in v7.111.0 after finding it false on that path.
+
+**Only four of the eight quality gates are agent-independent.** Static analysis,
+mock-integrity, test-mutation and documentation coverage do not ask a model
+anything. The other four involve model judgment and are labelled ASSESSMENTS
+rather than FACTS in every receipt.
+
+**Verification cannot prove the spec was right.** This is the sharpest limit and
+it is structural. Our gates prove code matches spec; if the spec diverged from
+what you actually wanted, a passing gate is a correct answer to the wrong
+question. `loki intent` measures that divergence where an intent has been
+recorded, and reports UNKNOWN where it has not, which is most runs today.
+
+**`loki outcomes` reports UNKNOWN on most existing receipts.** Measured on this
+repository: 0 of 9 receipts are anchored, because 8 carry no recorded baseline
+and 1 uses the empty-tree sha. A receipt is measured only when sha algebra
+proves `base..head` is that change. We would rather print UNKNOWN than a
+change-failure rate of 0.0 that no data supports.
+
+**Generation is not air-gapped.** The verification path is local and offline;
+generating code calls a model provider.
+
+**We have no independent benchmark placement and no enterprise case studies.**
+Neither exists yet. When they do they will be linked here, and until then their
+absence is not evidence of anything except their absence.
+
+## What we refuse to build
+
+Each of these would look good in a comparison table and would make the numbers
+above less trustworthy:
+
+- **A semantic fidelity score.** Asking a model whether a spec expresses an
+  intent and printing a percentage is a judgment wearing the costume of a
+  measurement.
+- **A composite trust score.** Averaging a revert count, a hash comparison, a
+  path match and a model id yields a number whose movement nobody can explain.
+- **A readiness percentage.** "There is no test command" tells you what to do.
+  "Readiness 62%" does not.
+- **Any gate that punishes an agent for needing human edits.** It would train
+  the agent toward diffs nobody edits, which is not the same as good diffs.
+
+## Check any of this yourself
+
+```bash
+LOCAL_CI_TIER=full bash scripts/local-ci.sh   # the full gate, timed
+loki outcomes --json                          # post-merge outcomes, or UNKNOWN with reasons
+loki intent status --json                     # spec-vs-intent drift
+loki proof verify <id>                        # re-hash a receipt, exit 1 on tamper
+bash tests/test-competitor-verify-surface.sh  # the competitor CLI measurement
+```
+
+If a number here does not reproduce on your machine, that is a defect and we
+want the report.

package/docs/WANG-PRINCIPLES-PLAN.md

@@ -53,7 +53,7 @@ Measured on this machine, not asserted.
 | **1 Systems Thinking** | STRONG | 8 quality gates, RARV-C loop, council, Evidence Receipt, dual-route parity enforced by test |
 | **2 Speed** | MEASURED, UNOPTIMISED | agent call = **980s = 96%** of iteration; all gates together = 44s |
 | **3 Reliability** | STRONG, newly so | 73 mutation-proven trust invariants; four gates were shipping broken until v8.38.0 |
-| **4 Extensibility** | STRONG | 4 providers, 41 agent types, MCP (34 tools), plugin marketplace |
+| **4 Extensibility** | STRONG | 4 providers, 41 agent types, MCP (36 tools), plugin marketplace |
 | **5 Feedback Loops / Evals** | **BROKEN** | `cost_usd == 0` on **3 of 5** efficiency records; no eval score for our own harness |
 
 **Principle 5 is the gap, and it is the one Wang weights highest.** He states the