loki-mode 9.16.0 → 9.17.0

package/docs/QUEUE-OPERATIONS.md

@@ -0,0 +1,107 @@
+# Queue operations: what happens when a worker dies
+
+Written for the person on call when builds stop coming out.
+
+## The short version
+
+A build that vanished is recoverable. Run:
+
+```bash
+bash autonomy/queue-consumer.sh --reap
+```
+
+It requeues in-flight items whose worker died. It never touches a build that is
+still running, and it never deletes anything.
+
+## What the queue guarantees, and what it does not
+
+| | redis backend | file backend |
+|---|---|---|
+| Delivery | at-least-once (default) | at-least-once |
+| Crash leaves the item | in `<key>:processing` | in `processing/` |
+| Automatic recovery | `--reap` | see below |
+| Claim time recorded | `<key>:claims` hash | file mtime |
+
+**A crashed worker does not lose the build.** It used to: the redis backend was
+at-most-once (`LPOP`-then-run), so a worker killed between the pop and the
+finish took the item with it and nothing anywhere recorded that it existed.
+Proven against a real redis 8.6.3, then fixed with `LMOVE`, which pops and
+records in-flight in one atomic step.
+
+`LOKI_QUEUE_ACK=0` restores the old lossy behaviour if you are running a broker
+that already guarantees delivery and do not want a second in-flight record.
+
+## Recovering a stalled fleet
+
+Symptom: items are queued, workers are up, and nothing completes.
+
+```bash
+# 1. What is stuck?
+redis-cli -u "$LOKI_QUEUE_URL" LLEN loki-builds             # waiting
+redis-cli -u "$LOKI_QUEUE_URL" LLEN loki-builds:processing  # claimed, in flight
+
+# 2. Requeue anything whose worker is gone.
+bash autonomy/queue-consumer.sh --reap
+# -> reap: requeued 2, left 1 in flight (timeout 7200s)
+```
+
+The reaper reports what it did. "left 1 in flight" means one item's claim is
+still inside the visibility timeout, so it is treated as a running build and not
+touched.
+
+### The timeout is the safety knob
+
+`LOKI_QUEUE_VISIBILITY_SEC` (default 7200) is how long an item may be in flight
+before the reaper assumes its worker died.
+
+**It must exceed your longest legitimate build.** Set it too low and the reaper
+requeues work that is still running, and the user gets the same build twice.
+That is the only way this tool can hurt you, so it errs long by default.
+
+### An item with no claim is reaped immediately
+
+If a worker dies between claiming the item and stamping the claim time, the
+item sits in `processing` with no entry in `<key>:claims`. The reaper treats
+that as infinitely old, not infinitely young -- it is exactly the case that must
+be recoverable, and erring the other way would strand the items this exists to
+rescue.
+
+## Running the reaper unattended
+
+A CronJob is the intended shape. It is a separate mode rather than part of the
+consume loop on purpose: a consumer that reaped on every poll would race its own
+peers on a busy queue.
+
+```yaml
+schedule: "*/15 * * * *"
+command: ["bash", "autonomy/queue-consumer.sh", "--reap"]
+```
+
+## What is NOT claimed
+
+- **The file backend has no automatic reaper.** `--reap` is redis-only today; a
+  crashed worker on the file backend leaves its item in `processing/` for a
+  human to move back to `pending/`. The item is not lost, but nothing recovers
+  it on its own.
+- **No dead-letter queue.** An item that fails repeatedly is requeued
+  repeatedly. There is no automatic give-up-and-park.
+- **`LREM` clears every equal copy.** Two identical specs queued twice both
+  clear on the first ack. Accepted deliberately: the alternative needs a
+  per-item token from the producer, and the failure mode here is one duplicate
+  re-drive rather than a lost build.
+- **Only redis and file ship.** SQS, Pub/Sub, RabbitMQ and Kafka are
+  bring-your-own: override `queue.command` with your own consumer.
+
+## Check any of this yourself
+
+```bash
+bash tests/test-queue-at-least-once.sh    # 16 assertions; live half needs a broker
+redis-server --port 6399 --save '' --appendonly no &
+LOKI_TEST_REDIS_PORT=6399 bash tests/test-queue-at-least-once.sh
+```
+
+The live checks SKIP with a named reason when no broker is present rather than
+passing on a mock. Both bugs found building this were invisible to a fake:
+`redis-cli --no-raw` escapes the inner quotes of a JSON payload, which mangled
+the item and made the ack a silent no-op. A green mock would have certified both
+as working.

package/docs/VERIFICATION-COST.md

@@ -101,3 +101,173 @@ bash tests/test-competitor-verify-surface.sh  # the competitor CLI measurement
 
 If a number here does not reproduce on your machine, that is a defect and we
 want the report.
+
+## How we compare, and what we cannot measure
+
+A goal was set to be "2-10x better than factory.ai, cognition devin, 8090.ai
+and replit". This section reports what is measurable and refuses the rest.
+
+### What is NOT benchmarked, and why
+
+None of those four products has a runnable local arm. Checked on this machine:
+
+```
+droid    NOT installed        devin    NOT installed
+replit   NOT installed
+claude   on PATH              aider    on PATH
+codex    on PATH              loki     on PATH
+```
+
+Devin and 8090 are hosted services with no CLI. Factory's droid and Replit run
+cloud-side. `benchmarks/bench/adapters/` can drive a competing arm as a
+subprocess (`claude_code.py` invokes `claude -p` live), but it cannot drive a
+product that has no local binary.
+
+**So there is no "2-10x vs Factory/Devin/8090/Replit" number here, and any such
+figure elsewhere should be treated as unearned.** Publishing one would be the
+same error as sourcing a colour token from a frontend that does not ship: a
+number that looks authoritative and measures something else.
+
+### What IS measured: the verification surface
+
+`tests/test-competitor-verify-surface.sh`, run on this machine:
+
+> **5 installed competitor CLIs. 0 expose an output-verification command.**
+
+That is not a multiplier, it is a category. The comparison is not "our
+verification is faster" but "there is nothing on the other side to compare
+against". Re-run it yourself; it names each CLI it checked and SKIPs the ones
+it could not find rather than counting them as absent.
+
+### What the competitors say about verification, in their own words
+
+Verbatim, with the file each came from, so every line is checkable against the
+scraped corpora:
+
+| Source | Their words |
+|---|---|
+| `factory_ai/docs.factory.ai_missions_overview.md` | "**How do you maximize correctness?** Long-running plans accumulate errors." -- published as an OPEN QUESTION |
+| `factory_ai/docs.factory.ai_missions_overview.md` | "Without it, the mission **cannot reliably verify its own work**" (Missions require repo readiness Level 4+) |
+| `devin_cognition_ai/docs.devin.ai_admin_security.md.md` | "it can still experience **hallucinations, introduce bugs into code**, or suggest insecure code" |
+| `8090_ai/www.8090.ai_terms-of-service.md` | "**HUMAN REVIEW AND VERIFICATION OF ALL OUTPUT**" required, while the same ToS caps liability at "FIFTY US DOLLARS" and disclaims "ACCURACY" |
+
+Factory's is the most honest of the four: they name verification as an open
+research question rather than a solved feature, and they state that
+self-verification is a property of the ENVIRONMENT, not of the agent. We agree,
+which is why `loki readiness` measures the repo and not the model.
+
+### The claim we will actually defend
+
+Not a multiplier. A receipt you can recompute:
+
+```
+loki proof verify <id>     # re-hash the receipt; exit 1 on tamper
+loki outcomes --json       # anchored, or UNKNOWN with a named reason
+```
+
+Against "human review and verification of all output" and "cannot reliably
+verify its own work", an anchored receipt is a categorical difference. It is
+also falsifiable: if `loki outcomes` reports UNKNOWN, we say UNKNOWN. On this
+repo it reported ANCHORED 0 of 9 for weeks, and the fix
+(`facts.git.base_sha` was empty) is in the history.
+
+### Honest limits on this section
+
+- The verify-surface count is a check for a COMMAND, not for internal
+  verification a product may do without exposing it. A hosted product could
+  verify server-side and expose nothing to a CLI.
+- It measures what is installed HERE. A CLI absent from this machine is
+  reported as SKIP, never as a competitor that lacks the feature.
+- Nothing here measures build quality, speed, or cost against those four.
+  Those comparisons are not available to us and are not claimed.
+
+### Measured: what the harness is worth, model held constant
+
+The one comparison we CAN run. `benchmarks/bench/matrix.sh` defines two
+configs against the same task and the same model:
+
+- `baseline` -- raw model, minimal orchestration (in-repo comment: "the
+  Replit/Cursor mode")
+- `full` -- the harness: council, code review, self-heal, auto-tune
+
+Paired on identical tasks, both arms on `haiku`, from
+`benchmarks/bench/results/`:
+
+| Task | harness (`full`) | raw model (`baseline`) | verdict |
+|---|---|---|---|
+| `hard-2-ledger` | **4/4**, $0.89 | **0/4**, $0.19 | harness wins outright |
+| `hard-1-order-api` | 3 runs, 1.00, $0.58 | 1 run, 1.00, $0.20 | harness cost 2.8x for nothing |
+| `multifail-1-two-modules` | 2 runs, 1.00, $0.27 | 1 run, 1.00, $0.14 | harness cost 1.9x for nothing |
+
+**Two of the three paired tasks are UNFAVOURABLE to the harness.** That is the
+honest headline, and it is narrower than "the harness is worth more than the
+model":
+
+**The harness matters on hard tasks and is pure overhead on easy ones.**
+
+On `hard-2-ledger` the raw model never finished the task across 4 trials and
+the harness finished it every time. That is not a percentage improvement; the
+baseline success rate is zero. It cost 4.7x more per run and produced a working
+result instead of nothing.
+
+On the other two, both arms succeed and the harness simply costs 1.9x-2.8x
+more. We publish those rows because the direction is unfavourable to us, and a
+benchmark table that only survives its favourable rows is an advertisement.
+
+**Two further baseline cells were attempted and produced NO data.**
+`tokenheavy-1-crm` (2 trials) and a wider `hard-1-order-api` (3 trials) both hit
+the 1200s cell timeout, which caps all trials of a cell together. The harness
+wrote no result file and the runner said so:
+
+> WARNING: cell haiku-baseline / tokenheavy-1-crm wrote NO new result (likely
+> timed out at 1200s). This cell is MISSING from the report.
+
+They are absent from the table rather than counted as failures. A timeout is
+not evidence the raw model cannot do the task; it is evidence we did not
+measure it. Reporting them as baseline losses would have made the harness look
+better on data that does not exist.
+
+An earlier version of this section reported only the first two tasks and read
+as a stronger claim than the data supported. The `multifail` baseline arm was
+run afterwards specifically to test whether the claim would survive more data.
+It did not survive intact, and the table was corrected rather than the
+measurement dropped.
+
+Aggregate across all recorded cells:
+
+| Cell | n | success (median) | cost (median) |
+|---|---|---|---|
+| `haiku` + harness | 11 | 1.00 | $0.54 |
+| `opus` + baseline | 5 | 1.00 | $0.83 |
+| `haiku` + baseline | 8 | 0.50 | $0.19 |
+
+The cheap model WITH the harness matches the expensive model without it, at
+35% lower cost. Treat that as directional, not as a headline: the task sets
+differ between those three cells, which is exactly why the paired table above
+is the one that carries the argument.
+
+**Limits of this measurement, stated so it cannot be over-read:**
+
+- Two paired tasks. `hard-1-order-api`'s baseline arm is n=1.
+- It measures OUR harness against OUR baseline config. `baseline` is a
+  documented stand-in for "raw model, minimal orchestration", NOT a
+  measurement of Replit, Cursor, or any other product.
+- Success is a held-out acceptance exit code, not a judgement of code quality.
+- Reproduce it: `LOKI_BENCH_SPEND_APPROVED=1 bash benchmarks/bench/matrix.sh pilot`.
+  The spend interlock is default-deny on purpose; a benchmark that starts a
+  paid tool without an explicit opt-in is how a surprise bill happens.
+
+**The exclusion rule, observed on a live run rather than asserted.** A fresh
+`haiku-full` trial on `hard-1-order-api` (2026-08-07) hit the 1200s adapter
+timeout. The held-out grader inspected the workdir and returned
+`success: true` -- there WAS a passing artifact. The harness still recorded
+`measured: false`, `unmeasured_reasons: ["adapter exit_status=timeout"]`, and
+excluded the trial from k/N, with this note in the result file:
+
+> "the held-out grader's own verdict on whatever was in the workdir. Real
+> evidence about the ARTIFACT; NOT evidence that a run happened."
+
+A naive harness scores that 1/1. Ours scores it 0/0 and says why, so the
+`hard-1-order-api` row above stayed at n=3 instead of being inflated to n=4 by
+a run that never finished. The number in this document is smaller because of
+that rule, which is the point of having it.