loki-mode 7.11.0 → 7.13.0

package/docs/INSTALLATION.md

@@ -2,7 +2,7 @@
 
 The flagship product of [Autonomi](https://www.autonomi.dev/). Complete installation instructions for all platforms and use cases.
 
-**Version:** v7.11.0
+**Version:** v7.13.0
 
 ---
 

package/docs/R5-AUTO-WIKI-DESIGN.md

@@ -0,0 +1,137 @@
+# R5: Auto-wiki + Cited Codebase Q&A (Loki's DeepWiki) -- Design Note
+
+Status: implemented in worktree (do not commit to main). Target release: R5 of the
+competitive-stickiness arc. NO version bump in this worktree.
+
+## Goal
+
+A persistent, queryable per-project wiki generated from the codebase, surfaced in
+the dashboard, with cited answers. Loki's answer to Devin DeepWiki. Sections cite
+the real source files they were built from; `loki wiki ask` returns a grounded,
+cited answer (citations = `file:line`). Generation is incremental: it skips when
+the codebase has not changed (reuses the R1 codebase-signature idea).
+
+## What already exists (verified against source, 2026-06-03)
+
+| Asset | File | Reused? |
+|---|---|---|
+| `loki docs generate` (LLM-written README/ARCHITECTURE/...) | `autonomy/loki:20577` (`cmd_docs`) | Patterns reused: `_docs_scan_project`, `_docs_build_context`, `_docs_invoke_provider`, `_docs_write_manifest`. Not the command itself -- docs has no citations and no Q&A. |
+| Proof-of-run generator (Python core, thin CLI readers) | `autonomy/lib/proof-generator.py`, bash `cmd_proof`, `loki-ts/src/commands/proof.ts` | Architecture precedent reused exactly: Python core does the heavy work; bash + Bun are thin readers; dashboard exposes read APIs. |
+| PII redaction | `autonomy/lib/proof_redact.py` (`redact_tree`, `_redact_paths`) | Reused: wiki output is normalized to repo-relative paths and passed through the redactor so no `/Users/<name>/...` leaks. |
+| Org knowledge graph | `memory/knowledge_graph.py` (`OrganizationKnowledgeGraph`) | Token-overlap scoring idea reused (`_tokenize` / `query_patterns`). NOT a codebase index -- it aggregates `.loki/memory/semantic/*.json` patterns across projects, keyed on `~/.loki/knowledge`. See "Honest reuse" below. |
+| Memory retrieval | `memory/retrieval.py` (`MemoryRetrieval`) | Inspected. It retrieves memory entries (episodic/semantic/procedural), NOT source code. Not a code indexer. Not reused for code retrieval. |
+| Dashboard read-API + traversal-safety | `dashboard/server.py:7191` (`_safe_proof_run_dir`) | Pattern reused for the wiki section/path param (`_safe_wiki_section`). |
+| Dashboard web components | `dashboard-ui/components/*.js` (Web Components) | New `loki-wiki-browser.js` follows the same `LokiElement` convention; registered in `index.js`. |
+
+### Honest reuse statement
+
+The task says "reuse memory/knowledge_graph.py and memory/retrieval.py." Both were
+read. Neither is a *codebase* index: `knowledge_graph.py` aggregates cross-project
+*memory patterns* (`.loki/memory/semantic`), and `retrieval.py` retrieves *memory
+entries*, not source files. There was no existing per-file code index to query for
+grounded citations. So R5 adds a new lightweight, dependency-free code index
+(`autonomy/lib/wiki_index.py`) and reuses the parts that genuinely fit: the
+token-overlap retrieval scoring (ported from `knowledge_graph._tokenize` /
+`query_patterns`), the docs scanner, the proof manifest/signature idea, and the
+redactor. Reuse is stated where real; not fabricated to satisfy a constraint.
+
+ChromaDB (`tools/index-codebase.py`, MEMORY.md) is an OPTIONAL future backend. The
+core deliberately does NOT depend on it: it needs Docker + python3.12 and is not
+CI-safe. Default retrieval is deterministic and dependency-light.
+
+## The grounding contract (the part that must be right)
+
+Fabricated citations are made structurally impossible, not merely prompt-discouraged:
+
+1. **Index**: `wiki_index.py` scans source files (git-tracked when available, else a
+   filtered `find`), splits each into line-anchored chunks
+   `{file, start_line, end_line, text}` where `file` is REPO-RELATIVE.
+2. **Retrieve** (`ask`): deterministic token-overlap scoring (no LLM, no network)
+   selects the top-K chunks for the question. Each is a record we own.
+3. **Prompt**: the LLM sees NUMBERED chunks `[1]..[K]` and is told to cite by chunk
+   index only (`[1]`, `[2]`). It never emits raw paths.
+4. **Map + validate**: indices in the answer are mapped back to `{file, start_line}`
+   from the retrieval records. Every citation is then validated against the
+   filesystem (file exists AND start_line <= file length). Non-resolving citations
+   are DROPPED. The LLM can only reference chunks we supplied, and only ones that
+   resolve on disk -- so a fabricated citation cannot survive.
+5. **generate**: per-section "sources" are CODE-DERIVED (the files the scanner read,
+   the real def/class line numbers from a grep parse), not LLM-emitted. The LLM
+   writes prose; the citation list comes from the index.
+
+If the LLM is unavailable (CI, no provider), `ask` returns an EXTRACTIVE answer
+(the top chunk snippets with their real citations) and `generate` writes a
+template-based wiki whose citations are still the real scanned files. No fabrication
+in any path.
+
+## Mocking the LLM in CI
+
+The Python core reads `LOKI_WIKI_LLM_STUB`:
+- unset -> call the real provider via the same path `_docs_invoke_provider` uses
+  (`claude -p` etc.), OR fall back to extractive/template if no provider on PATH.
+- set to a file path -> read the stubbed completion from that file.
+- set to any other value -> use it literally as the completion.
+
+Tests set `LOKI_WIKI_LLM_STUB` so CI makes ZERO paid calls. This mirrors how the
+proof tests fake `gh`/`open` via PATH and env.
+
+## Storage layout (per project, generated)
+
+```
+.loki/wiki/
+  wiki.json              # structured: sections[], each with title, body, citations[]
+  index.md               # human-readable rendered wiki (overview + module list)
+  architecture.md        # rendered architecture section
+  modules.md             # rendered key-modules section
+  data-flow.md           # rendered data-flow section
+  wiki-manifest.json     # signature (git sha + per-file content hash), generated_at
+  code-index.json        # the chunk index (file, start_line, end_line, tokens)
+```
+
+NOTE: `.loki/wiki/` (this deliverable, per-project, generated, gitignored) is a
+DIFFERENT namespace from the repo-root `wiki/` (the GitHub wiki in the release
+workflow). R5 never touches the latter.
+
+## Incremental regeneration (R1 signature idea)
+
+`wiki-manifest.json` stores a `signature` = sha256 over (git HEAD sha + sorted list
+of `path:content-hash` for every scanned source file). `loki wiki generate` computes
+the current signature; if it equals the stored one, it SKIPS regeneration and prints
+"up to date" (unless `--force`). This is the same cheap-incremental idea as the docs
+manifest and the R1 codebase signature.
+
+## Command surface
+
+```
+loki wiki generate [path] [--force]   # build/refresh .loki/wiki/ (incremental)
+loki wiki show [section]              # print rendered wiki (or one section)
+loki wiki ask "<question>"           # grounded, cited answer (file:line)
+```
+
+## Build surface (mirrors the proof precedent)
+
+- `autonomy/lib/wiki_index.py` -- scan + chunk + token-overlap retrieve + signature
+  (importable module; underscore name).
+- `autonomy/lib/wiki-generator.py` -- generate wiki.json + rendered md (LLM or
+  template), citations code-derived; subprocess-invoked (hyphen in name, like
+  proof-generator.py).
+- `autonomy/lib/wiki-ask.py` -- retrieve K chunks, prompt (stub-aware), map + validate
+  citations, print grounded answer. Subprocess-invoked.
+- bash `cmd_wiki` in `autonomy/loki` (generate|show|ask) -- thin dispatcher to Python.
+- Bun `loki-ts/src/commands/wiki.ts` -- native `show` (reads `.loki/wiki/`); `generate`
+  and `ask` delegate to the bash/Python core (heavy work, provider). Added to the
+  `bin/loki` allowlist and `cli.ts` switch.
+- Dashboard: `GET /api/wiki` (list sections + manifest), `GET /api/wiki/{section}`,
+  `POST /api/wiki/ask` -- all traversal-safe; web component `loki-wiki-browser.js`.
+
+## Tests
+
+- `tests/test_wiki_index.py` -- chunking is line-accurate; retrieval is deterministic;
+  signature stable + changes on edit; repo-relative paths only.
+- `tests/test_wiki_generator.py` -- generate on a fixture repo; citations point to REAL
+  files; incremental skip when unchanged; LLM stubbed; no absolute paths (no PII).
+- `tests/test_wiki_ask.py` -- `ask` returns grounded answer; every citation resolves on
+  disk; a stub that emits a bogus `[99]` index is dropped (anti-fabrication).
+- `tests/cli/test-wiki-command.sh` -- bash route generate/show/ask on a fixture, stubbed.
+- `loki-ts/tests/commands/wiki.test.ts` -- Bun `show` parity with the rendered md.
+- `tests/dashboard/test_wiki_routes.py` -- API list/get/ask + traversal rejection.

package/docs/R7-ZERO-CONFIG-FIRST-RUN-PLAN.md

@@ -0,0 +1,137 @@
+# R7: Zero-config killer first run (time-to-first-value)
+
+Design note for the R7 release in the competitive-stickiness arc. Worktree
+deliverable for the integrator to cherry-pick. NO version bumps here.
+
+## Goal
+
+Convert trials to habits. The #1 acquisition-to-retention gate is the first
+run. Today a blank first run is mediocre and Loki's deep RARV-C / council can
+feel heavy on run 1. R7 = a frictionless first run: a user types
+`loki start "<one line>"` (or `loki start` in an existing repo) and sees a
+VISIBLE valuable artifact in minutes, with depth opt-in later.
+
+Honest "fast": we do NOT fake progress. We actually shorten the path by running
+a lightweight execution profile first (capped iterations, completion council
+off, simple complexity tier, heavy phases off) so the first visible artifact
+plus a proof-of-run land quickly. "Go deeper" = re-run plain `loki start` for
+the full RARV-C depth.
+
+## Verified current behavior (real code, traced 2026-06-03)
+
+- `cmd_start()` (`autonomy/loki:746`) is the unified entry. It parses args,
+  calls `detect_arg_type()` (`autonomy/loki:667`), then dispatches:
+  issue -> `cmd_run`; prd -> sets `prd_file`; empty -> no-PRD path; unknown
+  -> treated as a PRD path for back-compat.
+- `cmd_start` ends in `_loki_new_session_exec "$RUN_SH" ...` (`autonomy/loki:1678`).
+  Every branch of `_loki_new_session_exec` (`autonomy/loki:167-186`) uses
+  `exec`, so NOTHING after that line in `cmd_start` runs. Any end-of-run
+  message must live in `run.sh`, not after the exec in `cmd_start`.
+- `cmd_quick()` (`autonomy/loki:8849`) already synthesizes a PRD from a
+  one-line task and sets the lightweight profile
+  (`LOKI_MAX_ITERATIONS=3`, `LOKI_COMPLEXITY=simple`,
+  `LOKI_COUNCIL_ENABLED=false`, heavy phases off), then execs `run.sh`.
+- No-PRD + generated-PRD-reuse (v7.8.1): in `run.sh` around line 11102,
+  `decide_generated_prd_action()` (`run.sh:4032`) returns reuse|update|generate
+  for the no-arg in-repo path; signature persisted by
+  `persist_prd_signature_if_present()` (`run.sh:4064`).
+- Proof-of-run (R1): `generate_proof_of_run()` (`run.sh:4101`) wraps
+  `autonomy/lib/proof-generator.py`. It runs at session end (`run.sh:13312`)
+  on both success and failure, gated only by `LOKI_PROOF` (NOT by council
+  state), writing `.loki/proofs/<run_id>/{proof.json,index.html}`. Viewable
+  via `loki proof list` / `loki proof open <id>` (Bun-routed, `bin/loki:119`).
+
+### The exact gap R7 closes (traced, not assumed)
+
+`loki start "build a todo app"` TODAY:
+1. `detect_arg_type("build a todo app")` returns `unknown` (has spaces, no
+   extension, not a file, not an issue ref).
+2. The PRD-not-found guard at `autonomy/loki:1243` and `:1268` only fires for
+   `*.md|*.json|*.txt|*.yaml|*.yml`, so a brief with spaces slips past.
+3. `prd_file="build a todo app"` is passed to `run.sh`, which fails:
+   `[ERROR] PRD file not found: build a todo app`.
+
+So the one-line-brief path is broken today. R7 makes it work. This is ADDITIVE:
+no existing valid input (`.md` PRD, issue ref, single-token name) changes
+behavior.
+
+## Design (additive, no behavior change to existing inputs)
+
+1. `detect_arg_type`: add a `brief` return ONLY for args that contain
+   whitespace and match none of the file/issue/path patterns. A single-token
+   `unknown` arg still falls back to PRD path (back-compat preserved).
+2. `--brief "<text>"` explicit flag: deterministic escape hatch for the rare
+   single-word brief (e.g. `loki start --brief "snake"`).
+3. Shared helper `synthesize_brief_prd <file> <text>`: factored so `cmd_quick`
+   and the new brief path write the same forward-looking PRD. The brief PRD is
+   written to `.loki/brief-prd-$$.md` -- DISTINCT from `.loki/generated-prd.md`
+   so it never pollutes the v7.8.1 generated-PRD-reuse signature logic
+   (generated-prd is for codebase analysis of an existing repo; brief is a
+   forward spec).
+4. `cmd_start` brief sub-path: set the lightweight TTFV profile (same env as
+   quick), synthesize the brief PRD, set `LOKI_TTFV=brief`, then continue
+   through the normal exec path. Upfront framing ("fast first pass") is printed
+   BEFORE the exec.
+5. `cmd_start` no-arg in-repo path: UNCHANGED execution (existing no-PRD +
+   reuse, full RARV-C depth), but set `LOKI_TTFV=repo` so the end-of-run
+   what-next framing appears.
+6. `run.sh` end-of-session: after proof generation, when `LOKI_TTFV` is set and
+   stdout is a TTY, call `print_ttfv_next_steps <mode> <result>`. The wording
+   BRANCHES on mode so the message always matches what actually ran:
+   - `brief`: lightweight first pass, council off; proof has diffs/cost/time
+     (NO council verdicts, because the council was disabled).
+   - `repo`: full-depth codebase analysis, council on; proof has
+     diffs/cost/time/council verdicts.
+   Both point at `loki proof list` / `loki proof open` (the visible artifact)
+   and the depth opt-in. Gated so it is silent in CI / pipes and never fires
+   for normal PRD runs. Factored into `print_ttfv_next_steps` so it is
+   unit-testable.
+
+Honesty note: the `brief` message intentionally does NOT advertise "council
+verdicts" because brief mode runs with the council off (`_collect_council` in
+proof-generator.py finds no council state, so that proof section is blank on the
+brief path). The `repo` message claims verdicts because the full-depth path runs
+the council. This keeps the end-of-run summary truthful per the no-fabrication
+rule.
+
+### Why fast is honest
+
+The brief path uses the same lightweight profile `cmd_quick` already ships:
+3 iterations max, council off, simple tier, heavy phases (perf, a11y,
+regression, UAT, web-research) off. That genuinely shortens the path to first
+visible value. We do not print fake progress or claim work that did not happen;
+the proof-of-run is generated from real `.loki/` state. Depth is opt-in: the
+end-of-run message tells the user to re-run plain `loki start` (or
+`loki start <prd.md>`) for the full council-gated build.
+
+## Parity (bash + Bun)
+
+`loki start` and `loki quick` are NOT in the Bun shim allowlist
+(`bin/loki:119`), so dispatch is bash-only by design; this change is bash-only
+for the CLI surface. The runtime pieces it reuses are already shared across
+routes: `proof-generator.py` (one implementation, both routes) and the no-PRD /
+generated-PRD-reuse path in `run.sh` (both routes source run.sh). No Bun CLI
+change is required for parity.
+
+## Files
+
+- `autonomy/loki`: `detect_arg_type` brief return; `--brief` flag;
+  `synthesize_brief_prd` helper; `cmd_quick` refactor to use it; `cmd_start`
+  brief sub-path + `LOKI_TTFV` wiring; help text.
+- `autonomy/run.sh`: end-of-session TTFV what-next block.
+- `tests/cli/test_zero_config_first_run.sh`: new test suite.
+
+## Tests (no paid runs; mock via early exit)
+
+Following `tests/cli/test_start_run_unified.sh`: extract `detect_arg_type` and
+`synthesize_brief_prd` in a subshell and assert on them; force `cmd_start` to
+exit before `run.sh` boots via `--provider nonexistent-provider`.
+
+- `detect_arg_type("build a todo app")` = `brief`; single tokens still `unknown`;
+  `.md` still `prd`; issue refs still `issue`; empty still `empty`.
+- `synthesize_brief_prd` writes a PRD containing the brief text and TTFV markers.
+- `loki start "<brief>"` enters the brief path (lightweight env, not
+  "PRD file not found").
+- `loki start --brief "<one word>"` works.
+- existing-repo no-arg path still routes to no-PRD (unchanged).
+- `loki start <prd.md>` (real PRD) still routes to PRD mode (no regression).