@vibecompass/vibecompass 0.14.0 → 0.16.0

package/CHANGELOG.md

@@ -1,5 +1,188 @@
 # Changelog
 
+## 0.16.0 - 2026-09-30
+
+The session brief ships as an opt-in experiment (D-368). In a baseline-versus-brief
+evaluation that did not fully follow its own protocol, the brief arm met the numerical
+recall targets and made none of the labeled forbidden authority claims, but lost nine
+facts that agents without a brief found. That does not support turning it on by
+default, so `start-session` and `continue-session` write a lane brief only when
+`project.yaml` sets `brief.enabled: true`. `vibecompass brief` works on request
+either way.
+
+- Read model: architecture components now carry `retrieval_guidance` (the
+  `## Retrieval guidance` section) and `retrieval_scope` (the "Retrieval
+  scope" bullet under `## Review metadata`); both are `null` when absent.
+  Existing fields are unchanged.
+- Add typed decision references and declared decision lineage
+  (`decision_lineage` on the read model, contract version 1). Architecture
+  docs `cites` decisions, session notes `references` them and `made` the ones
+  listed at the start of a top-level "Decisions made" bullet, and decision
+  entries declare `supersedes` / `amends` / `preserves` only from their own
+  wording, with extent (`full` / `partial` / `scoped` / `unqualified`) and
+  scope. Other lineage-like wording is `unknown`; mentions are `references`;
+  nothing is inferred as "governs" or "currently valid" (D-359). Every
+  relation carries source path, section, line, document and section hashes,
+  and explicit-or-inferred evidence (range interiors are inferred).
+- Lineage stays conservative: fenced examples, another decision's restated,
+  reported, or uncertain claim ("D-020 supersedes D-010", "As D-020
+  supersedes …", "Documents how D-020 supersedes …"), contrast objects
+  ("separate from D-169"), hypothetical wording (may, would, if, unless, …),
+  and negated preservation ("cannot preserve") never certify a relation;
+  prohibited change ("cannot supersede", "cannot be superseded by <this
+  entry>") is preservation, and a requirement ("must retain") still
+  certifies preservation. An
+  `unknown` is kept beside certified relations unless one shares its object
+  (coordinated verbs). Historical retention ("kept for historical reference")
+  never overrides partial or scoped wording. Decision IDs must be positive
+  safe integers, so malformed ranges cannot loop.
+- Recognize the optional structured lineage fields `**Supersedes:**`,
+  `**Partially supersedes:**`, `**Amends:**`, and `**Preserves:**` on new
+  decision entries (D-363); they win over prose for the targets they name, and
+  malformed items become diagnostics.
+- Add `getDecisionLineage(readModel, id)` (declared successors transitively,
+  incoming and outgoing lineage, citations, and the notes that made it) and
+  export the pure extractors (`extractDecisionFileRelations`,
+  `extractSessionNoteRelations`, `extractArchitectureDocCitations`,
+  `scanDecisionReferences`, `buildDecisionLineageModel`) with shared fixtures
+  in `src/tests/fixtures/decision-lineage/` for the app port.
+- Add `vibecompass brief` and the read-only session brief engine
+  (`buildSessionBrief`, `renderBrief`; plan task A3, D-359). Given a task, and
+  optionally files, feature slugs, claims, a lane, and a budget, it selects the
+  memory a session should read first — through the file-owner index (repo
+  aliases and workspace paths such as `vibecompass-app/src/…` both match),
+  lane-declared docs, feature slugs, and deterministic keyword matching on
+  titles, Description, and Retrieval guidance (no embeddings, no network, no
+  model) — and packs whole units (lane, decision-lineage, doc, note,
+  watch-out) into an estimated token budget (Unicode code points ÷ 4 over the
+  whole rendered brief; default 6,000, minimum 800). Every brief states
+  `complete`, `partial`, `incomplete` (with prioritized required reads, also
+  used for retrieval errors), or `no-match`, lists at most 12 follow-up reads
+  plus "+N more" (the full list is in `--json`), and never emits a decision
+  without every declared superseding or amending successor. Uncertified
+  lineage wording becomes a follow-up read, never a status. The command
+  writes nothing: no manifest refresh, state write, lane write, or agent-file
+  sync, so it is safe against a hand-maintained root.
+- Session brief in the lane lifecycle (plan task A4, D-359, D-364, D-368),
+  opt-in: when `project.yaml` sets `brief.enabled: true`, `start-session` and
+  `continue-session` write `sessions/active/<lane-id>/brief.md` from the lane's working-on,
+  claims, and feature slugs, after every lifecycle write succeeded and inside
+  the same lock. `continue-session` keeps a current brief and regenerates a
+  missing or stale one, and lists it under "Read next". A brief failure is a
+  warning plus, when the file can be written, an `incomplete` brief naming the
+  reason; it never fails the command, and existing lifecycle refusals fail as
+  before. Without the setting (or with `brief.enabled: false`), lifecycle
+  commands print `Brief: off (…)`, write no brief, and leave a brief already
+  in the lane untouched; `--no-brief` (start and continue) skips generation
+  for one command on a root that opted in. Every other lifecycle effect is
+  unchanged. `vibecompass brief --write` writes or regenerates the lane brief
+  on request whatever the setting; `vibecompass brief --check [--json]`
+  reports whether it is stale, without writing.
+- `brief.md` opens with a YAML header that binds its inputs: source root and
+  lane with the lane's session date and number, destination root and lane, a
+  digest of the canonical corpus it read, short hashes of the docs, decision
+  entries, and notes it drew text from, the lane scratch it read (`handoff.md`
+  bytes, the `session.yaml` fields it uses, the other active lanes), and the
+  repo alias map, plus ready-to-run check and refresh commands that keep the
+  root, lane, and source explicit. The body is exactly what
+  `vibecompass brief` prints for the lane. Staleness is computed from the
+  bindings, not stored: any hash, the session identity, or the alias map
+  differs, or the check cannot read what it needs; resume bookkeeping
+  (`resumed_at`, `resume_count`) never makes a brief stale. Memory that
+  cannot be read (invalid settings included) is a failed generation
+  (`generation: "failed"`; `brief --write` exits 1), while mandatory units
+  overflowing the budget is a successful `incomplete` brief.
+- `project.yaml` gains an optional `brief:` mapping (D-364): `enabled`
+  (default `false`; `true` opts the root into lifecycle generation, D-368),
+  and `exclude`, a list of root-relative globs (`*`, `**`, `?`; case-insensitive)
+  naming canonical documents every brief must never read. Excluded files are
+  dropped from the scan before they are opened, so they contribute nothing to
+  selection, ranking, citation propagation, bindings, or output. Malformed or
+  unknown `brief` settings, or an unreadable `project.yaml`, fail closed: the
+  brief reads no canonical document and is `incomplete`; scans warn with
+  `project-brief-invalid`. `loadProjectReadModel` and other consumers are
+  unaffected.
+- Dual-root source for dogfood roots (D-255, D-364): `continue-session`,
+  `brief --write`, and `brief --check` accept `--source-root <root>` (and
+  `--source-session <lane-id>`, which must equal the lane) to build the brief
+  from another memory root's matching lane — same id, session date, and
+  session number, or the brief is refused (`incomplete`, with the reason).
+  The source root is only read (no lock, manifest, lane, stamp, or agent-file
+  write) and is byte-identical afterwards; both roots' exclusions apply;
+  destination repo ids translate to the source's by matching remotes. The
+  option is never inferred: a brief built from another root is stale for a
+  run that does not name it, and that root is not read.
+- Generated `context.md`, the managed agent block, and the scratch listings
+  now tell agents to read the lane's `brief.md`, when present, before planning,
+  and describe lifecycle generation as opt-in. The lane
+  `wip.md` template gains an optional `## Context used` section for the docs
+  and decisions the lane actually relied on.
+
+## 0.15.0 - 2026-09-25
+
+- Add a layered local credential store for hosted sync tokens (D-355). Every
+  hosted command now resolves the token in a fixed order: the binding's env
+  var when set (per-shell override for CI, automation, and agents), then the
+  per-user credential store keyed by hosted URL + project id. The store lives
+  outside every project root (`VIBECOMPASS_CONFIG_DIR`, else
+  `$XDG_CONFIG_HOME/vibecompass`, else `~/.config/vibecompass`;
+  `%APPDATA%\vibecompass` on Windows); its `credentials.json` index is
+  written `0600` inside a `0700` directory and a world-readable file is
+  refused. Secrets go to the OS keychain when one is available (macOS
+  Keychain via `security`, Linux `secret-tool`) and otherwise into the file;
+  the index records which backend holds each entry. `project.yaml` is
+  unchanged (`credential_source: env` + `credential_env_var` stay required),
+  so older CLIs keep working against the env var.
+- `connect-hosted` captures the token after writing the binding: a hidden
+  prompt on a TTY, `--token-stdin` for scripts and agents, `--no-store` to
+  keep the env-only path, and `--credential-store <auto|keychain|file>`.
+  When the env var is already set it says so and points at
+  `sync-credential set --from-env` instead of storing silently.
+- Add `vibecompass login [--sync-target <name>] [--api-url <url>]
+  [--project-id <id>] [--label <text>] [--no-browser] [--poll-timeout <s>]`:
+  a browser device-code sign-in. The CLI prints and opens the dashboard
+  approval link with a short code, polls until the project owner approves,
+  stores the minted token, and writes the sync binding when the root is not
+  bound yet (promoting `local-only` to `local-primary` like `connect-hosted`).
+  An unknown `--sync-target` is created only with an explicit `--api-url`;
+  a mismatched `--api-url` on a bound root fails closed.
+- Add `vibecompass sync-credential <set|remove|list>`: store a dashboard
+  token (hidden prompt, `--token-stdin`, or `--from-env`), forget one, or
+  list stored entries (metadata only, never values; `--json`). `--api-url`
+  + `--project-id` address a hosted project without a bound root.
+- `status` prints `Hosted sync:` and `Sync credential:` lines (source only:
+  env var, OS keychain, credential store file, or missing) and recommends
+  `vibecompass login` when nothing holds a token. Missing-credential errors
+  from push, pull, mode transitions, and hosted docs-review now name
+  `login` first, then `sync-credential set`, then the env var override.
+- Init and connect-hosted no longer tell users to export the token in
+  every terminal or persist it in a shell profile.
+- Credential store hardening (review passes 1–3): every store mutation
+  takes a cross-process lock file next to the index that appears atomically
+  (owner record hard-linked into place) and names its owner (id, pid, host).
+  The lock is never taken from a live process on this host, however long it
+  waits on a keychain prompt; writers wait up to 10s, then fail naming the
+  owner. A lock whose owner process no longer exists is recovered through a
+  second reclaim lock so exactly one reclaimer removes it after re-reading
+  the owner; locks from another host, unreadable locks, and a reclaim lock
+  left by a crashed reclaimer are never removed automatically (the error
+  names them). The index is written through an exclusively created random
+  temp file plus atomic rename while the lock is held, so parallel logins,
+  `sync-credential set`, and `remove` never drop each other's entries;
+  target identity lowercases only scheme and host (paths keep their case);
+  keychain labels are sanitized and keys outside the keychain-safe charset
+  map to a digest account, so any project name can be stored; under the
+  default `auto` policy a failed keychain write (headless session, locked
+  keychain, missing Secret Service) falls back to the file store with a
+  printed warning instead of losing a one-time claim. `login` bounds every
+  network wait (start, polls, body reads) by `--poll-timeout`.
+- `login` is the one recommended connection path (D-356): help lists it
+  right after `init`, `init` and missing-token errors point only at it, and
+  `connect-hosted` on a terminal starts the browser sign-in when you press
+  Enter at its token prompt. Dashboard tokens, `--token-stdin`,
+  `sync-credential`, and the env-var override remain for CI, scripts, and
+  browserless machines and are labelled advanced.
+
 ## 0.14.0 - 2026-09-23
 
 - Add `continue-session [<lane-id>]` (D-353): resume an already-open lane

package/README.md

@@ -143,6 +143,17 @@ npx -y @vibecompass/vibecompass@latest close-session --root .compass --session a
 # start-session --id <existing-lane> fails closed and points at continue-session.
 npx -y @vibecompass/vibecompass@latest continue-session auth-flow --root .compass --working-on "Auth flow, phase 2"
 
+# Session brief in the lane (D-359, D-364), opt-in (D-368): with
+# project.yaml `brief: { enabled: true }`, start-session and continue-session
+# write sessions/active/<lane-id>/brief.md — the memory the lane's task should
+# start from, with a YAML header binding the inputs it read. continue-session
+# keeps a current brief and regenerates a stale one; a brief failure is a
+# warning, never a failed start or resume. Without the setting nothing is
+# written, and --no-brief skips it for one command. brief --write briefs any
+# lane on request; `brief.exclude` lists root-relative globs no brief may read.
+npx -y @vibecompass/vibecompass@latest brief --root .compass --session auth-flow --write   # write or regenerate it
+npx -y @vibecompass/vibecompass@latest brief --root .compass --session auth-flow --check   # current or stale, and why
+
 # Optional git binding for a lane (D-281): create/reuse a branch in each bound
 # repo; --worktree provisions per-repo worktrees under
 # <workspace>/worktrees/<lane-id>/<repo-id> with the lane marker in the
@@ -186,6 +197,12 @@ npx -y @vibecompass/vibecompass@latest sync-agents --root .compass
 npx -y @vibecompass/vibecompass@latest append-decision --root .compass --target cross-cutting.md --entry staged-entry.md
 npx -y @vibecompass/vibecompass@latest refresh-decision-index --root .compass --check
 
+# Session brief (read-only): the memory a task should start from, packed into
+# an estimated token budget, with a complete/partial/incomplete/no-match status
+# and follow-up reads. Decisions always appear with their declared successors.
+npx -y @vibecompass/vibecompass@latest brief --root .compass --task "Change the refresh denial message" --files app:src/lib/entitlements.ts
+npx -y @vibecompass/vibecompass@latest brief --root .compass --session auth-flow --budget 3000 --json
+
 # Targeted and comprehensive architecture docs maintenance
 npx -y @vibecompass/vibecompass@latest docs-update --root .compass --session auth-flow --changed app:src/auth/login.ts
 npx -y @vibecompass/vibecompass@latest docs-review --root .compass --guided
@@ -240,10 +257,57 @@ workflow files, updating handoffs, and applying reviewed changes. Use
 `docs update` / `vibecompass docs-update` for ordinary session-delta
 maintenance tied to the current lane.
 
+## Session Brief (opt-in)
+
+`vibecompass brief` picks the architecture docs, decisions, and session notes a
+task should start from, and packs them into a short markdown brief with a
+`complete`, `partial`, `incomplete`, or `no-match` status and a list of
+follow-up reads. It runs locally and deterministically: no network, no model,
+no account. A decision always appears together with any later decision whose
+own text explicitly declares that it replaces or amends it, and the brief never
+claims that a decision governs your task. It cannot spot a replacement that is
+only implied, so read the follow-up list too.
+
+The brief in a session lane is an experiment you turn on (D-368):
+
+```yaml
+# .compass/project.yaml
+brief:
+  enabled: true
+```
+
+With it on, `start-session` and `continue-session` write
+`sessions/active/<lane-id>/brief.md`, and your AI tool reads it before
+planning. Without it, nothing changes, and
+`vibecompass brief --session <lane-id> --write` still briefs one lane on
+request. `brief.exclude` lists files no brief may read.
+
+The brief is opt-in because of the project's own evaluation, which did not
+fully follow its protocol. Agents with a brief found about as much of the
+relevant memory as agents without one, and made none of the specific false
+authority claims the evaluation checked for. But they missed nine facts the
+others found, in one case after the brief left out the key document. Treat a
+brief as a starting point, not the whole answer.
+
 ## Hosted Sync
 
 Use hosted sync when you want browser review, team workflows, or hosted
-docs-review proposals:
+docs-review proposals. The quickest path signs you in from the terminal and
+keeps the token in your OS keychain (or a private per-user file), so it works
+in every new terminal and in agents launched from a desktop app:
+
+```bash
+npx -y @vibecompass/vibecompass@latest login --root .compass
+# opens vibecompass.dev, you confirm a short code and pick the project,
+# the token is stored locally, and the hosted binding is written to project.yaml
+
+npx -y @vibecompass/vibecompass@latest push --root .compass
+```
+
+Advanced (CI, scripts, a machine without a browser, or a hosted-only demote):
+bind the root with a dashboard-created token, pasted at the hidden prompt or
+piped in. On a terminal, pressing Enter at that prompt starts the browser
+sign-in instead:
 
 ```bash
 npx -y @vibecompass/vibecompass@latest connect-hosted \
@@ -251,8 +315,12 @@ npx -y @vibecompass/vibecompass@latest connect-hosted \
   --sync-api-url https://vibecompass.dev \
   --sync-project-id vc_proj_example \
   --sync-credential-env-var VIBECOMPASS_SYNC_TOKEN
+# "Paste the sync token from the dashboard (input hidden)"
 
-npx -y @vibecompass/vibecompass@latest push --root .compass
+# non-interactive:
+printf '%s' "$TOKEN" | npx -y @vibecompass/vibecompass@latest connect-hosted --root .compass \
+  --sync-api-url https://vibecompass.dev --sync-project-id vc_proj_example \
+  --sync-credential-env-var VIBECOMPASS_SYNC_TOKEN --token-stdin
 ```
 
 After `close-session`, run `push` again when a connected local-primary root
@@ -260,16 +328,29 @@ changed canonical project-memory files and the hosted dashboard should catch
 up. Hosted-only projects do not use a local authoritative push; update or
 refresh them through the hosted dashboard/proposal flow.
 
-The sync token is read from the environment variable you bind (for example
-`VIBECOMPASS_SYNC_TOKEN`). One-off `export`s do not survive new terminals —
-persist the token in your shell profile (`~/.zshenv` or `~/.bashrc`):
+### Where the token lives (D-355)
+
+Hosted commands look for the token in this order:
+
+1. the environment variable named in `project.yaml` (`VIBECOMPASS_SYNC_TOKEN`
+   by default) — a per-shell override for CI and scripts;
+2. the local credential store, keyed by hosted URL + project id: the OS
+   keychain when available (macOS Keychain, Linux `secret-tool`), otherwise
+   `~/.config/vibecompass/credentials.json` (`0600`; `%APPDATA%\vibecompass`
+   on Windows; `VIBECOMPASS_CONFIG_DIR` overrides the location).
+
+The token value is never written to `project.yaml`, `state/`, or any file in
+your repo. Manage stored tokens with:
 
 ```bash
-echo 'export VIBECOMPASS_SYNC_TOKEN="<your sync token>"' >> ~/.zshenv
+npx -y @vibecompass/vibecompass@latest sync-credential list --root .compass
+npx -y @vibecompass/vibecompass@latest sync-credential set --root .compass    # hidden prompt, --token-stdin, or --from-env
+npx -y @vibecompass/vibecompass@latest sync-credential remove --root .compass
 ```
 
-If the token is lost, rotate it on the hosted dashboard under Setup -> Hosted
-sync and export the new value.
+`vibecompass status` shows which source is in use (never the value). Lost a
+token? Run `login` again, or create a new one on the dashboard under Setup ->
+Hosted sync and revoke the old one there.
 
 ### Multiple environments
 
@@ -304,6 +385,9 @@ import {
   scanProjectMemory,
   loadProjectReadModel,
   getProjectContext,
+  getDecisionLineage,
+  buildSessionBrief,
+  renderBrief,
   detectChangelogShapedLines,
   ARCHITECTURE_DOC_SOFT_SIZE_LIMIT_BYTES,
 } from "@vibecompass/vibecompass";
@@ -319,6 +403,25 @@ so external validators (for example the hosted proposal parser) stay in parity.
 `ARCHITECTURE_DOC_SOFT_SIZE_LIMIT_BYTES` is the byte-valued soft size budget
 (12000) both docs-review apply warnings and the docs-update size advisory read.
 
+`loadProjectReadModel(root)` also returns `decision_lineage`: evidence-typed
+relations between documents and decisions. Architecture docs `cites`
+decisions; session notes `references` them and `made` the ones listed under
+"Decisions made". Decision entries declare `supersedes`, `amends`, or
+`preserves` only from their own wording, with an extent and scope; anything
+the parser cannot certify is `unknown`. Every relation carries its source
+path, section, line, and content hashes. `getDecisionLineage(readModel, id)`
+returns one decision's incoming and outgoing lineage and its declared
+successors, followed transitively. Nothing is ever reported as "governs" or
+"currently valid". Architecture components also carry `retrieval_guidance`
+and `retrieval_scope` when their docs define them.
+
+`buildSessionBrief({ rootDir, task, files, featureSlugs, claims, laneId, budget })`
+is the read-only engine behind `vibecompass brief`: it returns the selection
+result (status, packed units, omitted units, follow-up reads, gaps, and the
+estimated size), and `renderBrief(result)` turns it into markdown. It never
+writes to the root, and it never reads a path the root's `project.yaml`
+`brief.exclude` names.
+
 ## More Documentation
 
 - Developer guide: https://vibecompass.dev/developers

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vibecompass/vibecompass",
-  "version": "0.14.0",
+  "version": "0.16.0",
   "description": "Local-first project-memory core for VibeCompass",
   "type": "module",
   "bin": {

package/src/brief-keywords.js

@@ -0,0 +1,203 @@
+/**
+ * Deterministic keyword matching for the session brief (plan task A3; D-359):
+ * a tokenizer with light suffix stemming and a BM25F-style scorer over a
+ * small fielded index. No embeddings, no network, no model — the same inputs
+ * always produce the same terms and scores.
+ */
+
+const STOPWORDS = new Set(
+  (
+    'a about above after again against all also am an and any are as at be because been before being below between both but by ' +
+    'can cannot could did do does doing done down during each either etc even ever every few for from further had has have having ' +
+    'he her here hers him his how however i if in into is it its itself just me more most much my neither no nor not now of off on ' +
+    'once one only onto or other our ours out over own per same she should so some such than that the their theirs them then there ' +
+    'these they this those though through thus to too under until up upon us very via was we were what whatever when where whether ' +
+    'which while who whom whose why will with within without would yet you your yours'
+  ).split(' '),
+);
+
+// Instruction framing that says what to do, not what the task is about.
+// Dropped from query text only; indexed memory keeps every content word.
+const QUERY_FRAMING = new Set(
+  (
+    'add answer anything change create edit everything afterward afterwards happen implement include including instead make ' +
+    'modify need please rewrite something thing things walk want write'
+  ).split(' '),
+);
+
+const IRREGULAR = new Map([
+  ['paid', 'pay'],
+  ['made', 'make'],
+  ['built', 'build'],
+  ['ran', 'run'],
+  ['written', 'write'],
+  ['wrote', 'write'],
+  ['chosen', 'choose'],
+  ['chose', 'choose'],
+]);
+
+// Ordered suffix rules: the first rule whose suffix matches (with at least
+// `min` characters left) applies. A final `e` is dropped afterwards so
+// "translate" and "translation" meet at "translat".
+const SUFFIX_RULES = [
+  ['ational', 'ate', 3],
+  ['izations', 'ize', 3],
+  ['ization', 'ize', 3],
+  ['ations', 'ate', 3],
+  ['ation', 'ate', 3],
+  ['ements', '', 4],
+  ['ement', '', 4],
+  ['ments', '', 3],
+  ['ment', '', 3],
+  ['nesses', '', 3],
+  ['ness', '', 3],
+  ['ingly', '', 3],
+  ['ings', '', 3],
+  ['ing', '', 3],
+  ['edly', '', 3],
+  ['ies', 'y', 2],
+  ['ied', 'y', 2],
+  ['sses', 'ss', 2],
+  ['xes', 'x', 2],
+  ['ches', 'ch', 2],
+  ['shes', 'sh', 2],
+  ['ed', '', 3],
+  ['ly', '', 4],
+  ['s', '', 3],
+];
+
+export function stemWord(word) {
+  const lower = word.toLowerCase();
+  if (IRREGULAR.has(lower)) return IRREGULAR.get(lower);
+  if (lower.length <= 3 || lower.endsWith('ss') || lower.endsWith('us') || lower.endsWith('is')) {
+    return lower;
+  }
+
+  let stem = lower;
+  for (const [suffix, replacement, min] of SUFFIX_RULES) {
+    if (stem.endsWith(suffix) && stem.length - suffix.length >= min) {
+      stem = stem.slice(0, stem.length - suffix.length) + replacement;
+      break;
+    }
+  }
+
+  // "running" → "runn" → "run"; keep "ll"/"ss"/"zz" doubles ("install", "access").
+  if (stem !== lower && /([b-df-hj-kmnp-rtv-y])\1$/.test(stem) && !/(ll|ss|zz)$/.test(stem)) {
+    stem = stem.slice(0, -1);
+  }
+  if (stem.length > 4 && stem.endsWith('e')) {
+    stem = stem.slice(0, -1);
+  }
+  return stem;
+}
+
+/** Raw lowercase word tokens: letters and digits, split on everything else. */
+export function splitWords(text) {
+  return String(text ?? '')
+    .toLowerCase()
+    .split(/[^a-z0-9]+/)
+    .filter((word) => word.length >= 2 && !/^\d+$/.test(word));
+}
+
+/** Content terms for indexing: stopwords dropped, stemmed, order kept. */
+export function indexTerms(text) {
+  const terms = [];
+  for (const word of splitWords(text)) {
+    if (STOPWORDS.has(word)) continue;
+    terms.push(stemWord(word));
+  }
+  return terms;
+}
+
+/** Unique query terms: stopwords and instruction framing dropped, stemmed. */
+export function queryTerms(text) {
+  const seen = new Set();
+  const terms = [];
+  for (const word of splitWords(text)) {
+    if (STOPWORDS.has(word) || QUERY_FRAMING.has(word)) continue;
+    const stem = stemWord(word);
+    if (QUERY_FRAMING.has(stem) || seen.has(stem)) continue;
+    seen.add(stem);
+    terms.push(stem);
+  }
+  return terms;
+}
+
+/** The first raw word behind each query term, for disclosures ("japanes" → "Japanese"). */
+export function queryTermOrigins(text) {
+  const origins = new Map();
+  for (const word of String(text ?? '').split(/[^A-Za-z0-9]+/)) {
+    if (word.length < 2 || /^\d+$/.test(word)) continue;
+    const stem = stemWord(word.toLowerCase());
+    if (!origins.has(stem)) origins.set(stem, word);
+  }
+  return origins;
+}
+
+const K1 = 1.2;
+
+/**
+ * A fielded keyword index. `fields` maps a field name to `{ weight, b }`;
+ * each document supplies text per field.
+ */
+export function createKeywordIndex(fields, documents) {
+  const entries = documents.map((document) => {
+    const perField = {};
+    for (const name of Object.keys(fields)) {
+      const terms = indexTerms(document.fields[name] ?? '');
+      const counts = new Map();
+      for (const term of terms) counts.set(term, (counts.get(term) ?? 0) + 1);
+      perField[name] = { counts, length: terms.length };
+    }
+    return { id: document.id, perField };
+  });
+
+  const averageLength = {};
+  for (const name of Object.keys(fields)) {
+    const total = entries.reduce((sum, entry) => sum + entry.perField[name].length, 0);
+    averageLength[name] = entries.length > 0 ? Math.max(1, total / entries.length) : 1;
+  }
+
+  const documentFrequency = new Map();
+  for (const entry of entries) {
+    const seen = new Set();
+    for (const name of Object.keys(fields)) {
+      for (const term of entry.perField[name].counts.keys()) seen.add(term);
+    }
+    for (const term of seen) documentFrequency.set(term, (documentFrequency.get(term) ?? 0) + 1);
+  }
+
+  const count = entries.length;
+  const idf = (term) => {
+    const frequency = documentFrequency.get(term) ?? 0;
+    return Math.log(1 + (count - frequency + 0.5) / (frequency + 0.5));
+  };
+
+  return {
+    count,
+    documentFrequency: (term) => documentFrequency.get(term) ?? 0,
+    idf,
+    /** `{ id, score, matched }` per document with any matched term. */
+    score(terms) {
+      const results = [];
+      for (const entry of entries) {
+        let score = 0;
+        const matched = [];
+        for (const term of terms) {
+          let pseudo = 0;
+          for (const [name, { weight, b }] of Object.entries(fields)) {
+            const field = entry.perField[name];
+            const frequency = field.counts.get(term);
+            if (!frequency) continue;
+            pseudo += (weight * frequency) / (1 - b + (b * field.length) / averageLength[name]);
+          }
+          if (pseudo === 0) continue;
+          matched.push(term);
+          score += idf(term) * ((pseudo * (K1 + 1)) / (K1 + pseudo));
+        }
+        if (matched.length > 0) results.push({ id: entry.id, score, matched });
+      }
+      return results;
+    },
+  };
+}