@spikedpunch/mast 0.2.0 → 0.3.0

package/MAST_SPEC.md

@@ -41,13 +41,14 @@ the pre-deletion system is preserved at the git tag
 - Code generation or explanation.
 - PR review, wiki generation, story generation.
 - A persistent background daemon — freshness is handled by the startup check and
-  `mast_reindex`. *Narrow carve-out:* `mast serve --watch` (§11.4) is an opt-in
-  file watcher for **interactive, non-container** use only. It is scoped to the
-  serve process lifetime (not a daemon), the SDD pipeline never uses it, and it
-  is a discovery-freshness optimization — it lets new files and symbols become
-  searchable sooner than waiting for an explicit `mast_reindex`. JIT staleness
-  handling (§9.0) already guarantees line-coordinate and content correctness for
-  already-indexed files without it.
+  `mast_reindex`. *Narrow carve-out:* `mast serve`'s file watcher (§11.4, on by
+  default since 2026-09-03; `--no-watch` opts out, and the SDD pipeline passes
+  it). It is scoped to the serve process lifetime (not a daemon) and is a
+  discovery-freshness optimization — it lets new files and symbols become
+  searchable without waiting for an explicit `mast_reindex`. JIT staleness
+  handling (§9.0) guarantees line-coordinate and content correctness for
+  already-indexed files without it, but is structurally blind to files the
+  index has never seen, which is what the watcher covers.
 - Support for non-TypeScript/JavaScript projects in v1 (AST layer is extensible but
   v1 targets the SDD stack).
 
@@ -157,7 +158,7 @@ The claude-runner passes `--state-dir` at serve time:
   "mcpServers": {
     "mast": {
       "command": "mast",
-      "args": ["serve", "--state-dir", "/workspace/.kluster/.mast"],
+      "args": ["serve", "--no-watch", "--state-dir", "/workspace/.kluster/.mast"],
       "type": "stdio"
     }
   }
@@ -596,8 +597,15 @@ startup
           (mast_search, mast_callers verified+potential, etc.)
 ```
 
-If `--watch` was passed to `mast serve`, the file watcher (§11.4) starts
-immediately after Step 3's transport opens, independent of Step 4.
+Unless `--no-watch` was passed, the file watcher (§11.4) starts immediately
+after Step 3's transport opens, independent of Step 4.
+
+The freshness probe (`mcp/freshness-probe.ts`) behind `mast_search`'s
+`unindexed_files` signal (§9.0) is primed here too — in the same background
+task as Step 4 and strictly AFTER it, so the measurement reflects the index the
+reindex just produced rather than racing it. Under `--no-startup-reindex` it is
+primed immediately: no reindex is coming, so the drift the operator chose to
+keep is exactly what wants measuring.
 
 **`--no-startup-reindex` refusal (M6 Part A).** The empty-during-Step-4 window
 above is legitimate and by design — but `--no-startup-reindex` disables Step 4
@@ -632,13 +640,21 @@ reindex runs in the background.
 **Fast first-task latency.** With a baked seed (§13.8), Steps 1–3 typically complete
 in **2–4 seconds** on a cold container. Step 4 then catches up any files changed
 since the seed was built in the background — the agent can begin useful work as
-soon as Step 3 completes; JIT staleness handling (§9.0) guarantees any individual
-file it queries is correct even before Step 4 reaches it.
+soon as Step 3 completes.
+
+Be precise about what covers that window, because the obvious summary is wrong.
+JIT re-parse (§9.0) makes an individual *already-indexed* file correct on read,
+and only on the five re-parsing tools; `mast_search` and `mast_implementors`
+flag such a file rather than refreshing it, and a file the seed never contained
+is invisible to all seven until Step 4 reaches it. The window is therefore
+bounded by Step 4, not eliminated by §9.0.
 
 This is the **only hook required for the SDD pipeline**. The BT orchestrator needs no
 reindex calls. Files committed by the previous task are picked up by Step 4's
-filesystem scan. JIT staleness handling (§9) covers files modified mid-task before
-Step 4 has caught up to them.
+filesystem scan. JIT re-parse (§9) covers files *modified* mid-task before Step 4
+has caught up to them — on the five re-parsing tools; elsewhere they are flagged.
+Files **created** mid-task are covered by neither and need `mast_reindex` (§11.2)
+or the watcher (§11.4).
 
 ### 7.5 Mid-Task Reindex (`mast_reindex` MCP tool)
 
@@ -773,8 +789,9 @@ Start the MCP server over stdio.
 Options:
   --state-dir <dir>       State directory
   --no-startup-reindex    Skip the startup staleness check (not recommended)
+  --no-watch              Do not watch source files (batch/container use — see §11.4)
   --watch                 Watch source files and incrementally reindex on change
-                          (interactive use — see §11.4)
+                          (the default; accepted for compatibility)
 ```
 
 The server runs until the parent process (Claude CLI) closes stdin.
@@ -787,10 +804,12 @@ state dir with the startup reindex left enabled (the default) is unaffected
 and starts normally, as does `--no-startup-reindex` against an already-indexed
 state dir (including one indexed over a genuinely empty file set).
 
-`--watch` is opt-in and intended for interactive local development; the SDD
-container does not use it (§3, §11.4). The watcher is closed on stdin close,
-SIGTERM, and SIGINT; a watcher startup failure logs a warning and the server
-continues without watch.
+Watching is **on by default** as of 2026-09-03 (§11.4); `--no-watch` opts out and
+is what the SDD container passes (§3). `--watch` is still accepted and selects
+the default, so existing MCP client configurations keep working unchanged. The
+watcher is closed on stdin close, SIGTERM, and SIGINT; a watcher startup failure
+logs a warning and the server continues without watch — which is what makes the
+default safe: it cannot prevent the server from starting.
 
 ---
 
@@ -807,6 +826,7 @@ Options:
 Output:
 ```
 state_dir:      /workspace/.kluster/.mast
+project_root:   /workspace/.kluster
 schema_version: 1.3.0
 last_indexed:   2026-05-13T14:22:00Z (3 minutes ago)
 indexed_files:  142
@@ -818,10 +838,22 @@ index_fresh:    true
 freshness_cause: none
 ```
 
+`stale_files` prints its split inline when non-zero — `stale_files: 47  (changed
+0, unindexed 27, deleted 20)` — because the field names one of the three things
+it counts (§9, `mast_status`).
+
+`project_root` names the tree the report was measured against. It is reported
+because `state_dir` alone does not identify it and the two are set
+independently: a relative `--state-dir` resolves against the **path argument**,
+not the shell's working directory, so one project's index can be read while
+another project's files are asked about.
+
 `freshness_cause` carries the same semantics as the `mast_status` MCP tool (§9) —
-it prints `none` in human output when the JSON value would be `null`; the JSON
-value is `"phase1_stale"` when `stale_files > 0`. On a never-indexed project the
-state directory is not created as a side effect of running `status`.
+it prints `none` in human output when the JSON value would be `null`. When it is
+`root_mismatch` the table is followed by a plain-language block naming the tree
+that was measured and saying that reindexing will not move the numbers. On a
+never-indexed project the state directory is not created as a side effect of
+running `status`.
 
 ---
 
@@ -834,12 +866,31 @@ Invoke any MCP tool (§9) directly from the CLI — the read tools (`mast_search
 
 ```
 Options:
-  --state-dir <dir>    State directory
+  --state-dir <dir>     State directory
+  --reindex             Run an incremental index (§7.1) before dispatching
   --json                Emit the exact single-line MCP response text (machine
                         use); default pretty-prints the parsed response with
                         2-space indent for humans
 ```
 
+`--reindex` (also on `mast search`) is the CLI's freshness mechanism. The CLI
+has neither of the server's: no watcher (§11.4) and no freshness probe
+(§9.0 `unindexed_files`), because a one-shot process has no lifetime over
+which to amortise a TTL-cached measurement. Semantics:
+
+- **Incremental, never full.** `mast index --full` remains the way to rebuild.
+- **Refreshes, does not bootstrap.** It runs strictly *after* the never-indexed
+  guard below, so a missing `graph.db` still fails fast with that guard's
+  message rather than being silently created as a side effect of the flag.
+- **Reports on stderr**, never stdout — the refresh is a side effect of the
+  query, not part of its answer, and `--json` consumers must keep a parseable
+  stdout.
+- **Never fatal.** The run takes `structure.lock`, and with `mast serve`
+  watching by default (§11.4) losing that race to a concurrent writer is an
+  ordinary outcome. A failed refresh warns and the query is answered from the
+  existing index; failing the query over a refresh that did not happen would
+  be a worse answer than a slightly stale one.
+
 `json` (positional, default `'{}'`) is the tool's argument object as a JSON
 string; `path` is the project root (same resolution as every other command's
 `[path]`).
@@ -948,8 +999,9 @@ how many files a single call's results can span:
 
 JIT re-parse covers files already known to the index. It does not discover a
 brand-new file or a newly-created symbol — those become searchable via the next
-`mast_reindex` call or the background/`--watch` reindex (§7.4/§11.4) reaching
-them. The agent prompt should still recommend `mast_reindex` after writing new
+`mast_reindex` call, the startup reindex, or the watcher (§7.4/§11.4, on by
+default) reaching them. `mast_search` reports the size of that blind spot as
+`unindexed_files` when a serve process has measured it. The agent prompt should still recommend `mast_reindex` after writing new
 files or symbols — not because JIT leaves existing files stale (it doesn't), but
 because discovery of new ones requires an actual indexing pass.
 
@@ -1045,11 +1097,12 @@ across all MCP tools, and what to do with each one:
 | `file_busy_returning_stale_cache` | JIT-refresh tools' results/envelopes (`mast_signature`, `mast_exports`, `mast_callers`, `mast_dependencies`, `mast_rename_impact`) | A refresh **was attempted** (this file's JIT re-parse) and lost to genuine write contention (`populateFile`'s `BEGIN IMMEDIATE` exhausted its `busy_timeout`), so the previous, possibly-stale chunk was returned instead. | Contended, not wrong-by-design. Retry shortly — the contention is expected to clear (§7.6). |
 | `stale` | `mast_search` / `mast_implementors` per-result (F7) | This result's `file_path` stat'd newer-on-disk than its indexed mtime, or the stat failed — **no refresh was attempted by design** (stat-and-flag, not JIT re-parse; see above). | Treat this result's line coordinates as untrustworthy. A `mast_reindex` call, or any JIT-refreshing tool call against the file, heals it. |
 | `index_empty` | Every primary-result read tool's envelope (M6) | Nothing is indexed at all — the empty result set is not "no match", it is "no index (yet)". | Run `mast init`/`mast index`, or — if a startup reindex is in progress — wait and retry. |
+| `unindexed_files` | Every primary-result read tool's envelope (D054). The five whose answer claims to be an exhaustive *set* — `mast_search`, `mast_callers`, `mast_implementors`, `mast_rename_impact`, `mast_project_skeleton` — carry it whether or not they found anything, because a short list drawn from an incomplete corpus reads exactly like a complete one and is the form that gets acted on. The three that answer about one *named* thing — `mast_signature`, `mast_exports`, `mast_dependencies` — carry it only on an empty answer, where "absent" and "never indexed" are indistinguishable; a hit is correct however much else is missing. | This many files exist on disk and are **not in the index** — measured by the serve process and TTL-cached; the request never awaits a measurement, though it may schedule one (amortised, not free — `mcp/freshness-probe.ts` documents the residual). Where `index_empty` says "nothing is indexed", this says "the index is populated but behind": these results were ranked over a corpus missing N files. Advisory — it does not claim any of the N would have matched. Absent means either nothing is unindexed or no measurement has landed yet; the two are deliberately not distinguished on the wire, because both mean "no warning to give". | Do not conclude a symbol is absent. Call `mast_reindex`, then re-query. |
 | `truncated` | `TypeContextEntry` (`mast_signature`'s `type_context`) | This referenced type's declaration was clipped at the 50-line cap. | Re-read the file directly (or call `mast_exports`/a narrower `mast_signature` query) for the full declaration if the clipped portion matters. |
 | `potential_truncated` | `CallersResponse.summary` / `RenameImpactResponse.summary` (`mast_callers`, `mast_rename_impact`) | The `identifier_fts` fetch behind `potential_matches` is capped at 50 entries; this carries the real, uncapped match count when the cap is hit (F10, Stage 3). Reports RAW fetch truncation only — `potential_matches` may still be smaller than the cap even when this field is present, because verified-overlap exclusion and checker-verdict filtering run AFTER the capped fetch (already visible via `checker_classified_*`). | The potential set is incomplete — narrow the query, or run `mast index --checker` to classify candidates away. |
 | `results_truncated` / `exports_truncated` | `SignatureResponse` / `ImplementorsResponse` / `ExportsResponse` envelopes (`mast_signature`, `mast_implementors`, `mast_exports`) | The result list is capped at `limit` (default 50, the same constant `potential_matches` uses); this carries the real, uncapped total when the cap is hit (D043). Unlike `potential_truncated` there is no post-cap filtering, so the returned page is always exactly `limit` long when this field is present. | A first page, not the answer. Pass a larger `limit` (max 500), or narrow with `file_path`. Before D043 these tools were unbounded: `mast_signature{symbol:'execute'}` over a 14k-file monorepo returned 580 declarations / 331k tokens in 78 s, which over MCP exceeded the client timeout and returned nothing. |
 
-`file_busy_returning_stale_cache`, `stale`, `index_empty`,
+`file_busy_returning_stale_cache`, `stale`, `index_empty`, `unindexed_files`,
 `potential_truncated`, `results_truncated` and `exports_truncated` all follow the same **omitted-when-false /
 present-only-when-true** convention (never present-and-false) established
 above — `potential_truncated`'s "false" case is "the fetch came back under
@@ -1698,6 +1751,7 @@ Index health snapshot.
   "indexed_files": 142,
   "chunk_count": 1840,
   "stale_files": 0,
+  "stale_breakdown": { "changed": 0, "unindexed": 0, "deleted": 0 },
   "parse_errors": 0,
   "write_errors": 0,
   "index_fresh": true,
@@ -1726,14 +1780,42 @@ conflated). Non-zero in either indicates files the agent should investigate.
 `seed_commit` is present only when the state directory was bootstrapped from a
 Docker-baked seed (§13.8) and reports the git revision the seed was built from.
 
-**Freshness diagnostics.** `freshness_cause` is `"phase1_stale"` when `stale_files > 0`
-(chunk line coordinates lag disk — corrected by JIT re-parse on read, §9.0, or by
-running `mast_reindex`) and `null` when the index is fully fresh. `index_fresh` is
-`true` only when `stale_files === 0` and the index has been run at least once.
+**Freshness diagnostics.** `freshness_cause` names which of `stale_files`'
+categories the count is actually made of, and `null` when the index is fully
+fresh. `index_fresh` is `true` only when `stale_files === 0` and the index has
+been run at least once.
+
+| value | meaning |
+|---|---|
+| `"root_mismatch"` | The index disagrees with this tree in both directions more than it agrees in either: more files here are unknown to it than known (`unindexed > walked - unindexed`), **and** it lists more absent files than known ones (`deleted > walked - unindexed`). It was built for a different project root, so reindexing will not move the numbers — the path argument or `--state-dir` is wrong. Both halves are required: the first alone would flag a mass deletion, the second alone a never-indexed project. |
+| `"phase1_stale"` | Indexed files whose content changed since — chunk line coordinates lag disk, corrected by JIT re-parse on read by the five re-parsing tools (§9.0), flagged `stale` by the other two, or fixed for all of them by `mast_reindex`. |
+| `"unindexed_files"` | Files on disk this index has never seen. |
+| `"deleted_files"` | Files the index still lists that are gone from disk. |
+
+The `"unindexed_files"` **cause** here and `mast_search`'s `unindexed_files`
+**count** (§9.0) name the same population — files on disk this index has never
+seen — and are deliberately spelled the same. They differ in shape and in
+freshness: `mast_status` reports a cause label chosen from a measurement taken
+during that call, while `mast_search` reports an integer from the serve
+process's TTL-cached probe, which may be up to one TTL old. When the two
+disagree, `mast_status` is the newer measurement.
+
+`root_mismatch` is tested first and is the only compound condition; the other
+three are ordered by what the caller should do about them, `phase1_stale` first
+because it is the one JIT re-parse corrects silently on read. The exact split
+always travels beside the cause in `stale_breakdown`, so a caller never has to
+infer the composition from the chosen label.
+
+Until 2026-09-01 this field was `"phase1_stale"` for **any** non-zero count,
+including counts containing no changed file at all (`docs/defects/LEDGER.md`
+D049); it is now decided from counted categories rather than from the total.
 
 `stale_files` counts three things, not one: files whose content changed since they
 were indexed, files on disk that are **not in the index at all**, and files the index
-still lists that are gone from disk. It is computed by `indexer/freshness.ts`
+still lists that are gone from disk. `stale_breakdown` reports that split as
+`{changed, unindexed, deleted}` — the total alone was documented as a union here
+and published as a scalar by both surfaces, so a caller who had read this
+paragraph still could not act on the distinction it draws (D049). It is computed by `indexer/freshness.ts`
 `measureFreshness`, which `mast status` and `mast_status` both call — one producer,
 because they answer one question. It reads the manifest and the `files.mtime` stamps
 and takes the union: only the manifest can see a file that was never indexed, and only
@@ -2238,9 +2320,11 @@ candidates out of `potential_matches` and report honest counts in
 
 Defined in full in §7.4. Summary: a four-step ladder that brings the whole index
 (graph + FTS) online in 2–4 seconds via a Docker-baked seed index (§13.8), with all
-11 tools registered and ready to serve as soon as Step 3 completes — there is no
-reduced-capability warm-up window. Step 4 then catches up any files changed since
-the seed was built, in the background.
+11 tools registered and ready to serve as soon as Step 3 completes — no tool is
+unavailable or degraded during startup. One *signal* does warm up: `mast_search`'s
+`unindexed_files` is served from a cached probe, and until the first measurement
+lands the count is unknown and the field is omitted (ADR 016). Step 4 then catches
+up any files changed since the seed was built, in the background.
 
 This is the **only hook required for the SDD pipeline**.
 
@@ -2249,7 +2333,7 @@ This is the **only hook required for the SDD pipeline**.
 The agent calls this explicitly after writes. JIT staleness handling (§9.0) already
 keeps already-indexed files correct on read; `mast_reindex` is what makes a
 **brand-new** file or symbol discoverable by `mast_search`/`mast_callers`/etc. before
-the next scheduled or `--watch` reindex reaches it. The implement prompt instructs:
+the next scheduled reindex or the watcher (§11.4, on by default) reaches it. The implement prompt instructs:
 
 > After writing or editing files, call `mast_reindex` before any search query that
 > depends on symbols you just created. This is the only way to guarantee the index
@@ -2274,19 +2358,39 @@ mast index "$(git rev-parse --show-toplevel)" --incremental
 Not required for the automated SDD pipeline — the startup hook covers the same
 scenario (files changed since last index).
 
-### 11.4 Optional Interactive Hook — `mast serve --watch`
+### 11.4 File Watching — `mast serve` (default on, `--no-watch` to opt out)
 
 Local interactive development has no equivalent of the container's startup
 ladder: git hooks are opt-in and fire only on commit/checkout, so a long-lived
 interactive session can leave newly-created files and symbols undiscoverable
 between explicit `mast_reindex` calls, even though JIT re-parse (§9.0) keeps
-every already-indexed file correct on read. `--watch` closes that gap as an
-**opt-in** flag — it is a discovery-freshness optimization, never a correctness
+every already-indexed file correct on read.
+
+Watching closes that gap and is **on by default** as of 2026-09-03. It was
+opt-in until then, on the reasoning that JIT staleness plus the startup ladder
+kept reads correct. That reasoning holds only for files the index already
+knows: JIT re-parses a *known* file whose mtime moved, and is structurally
+blind to one that was never indexed. So in the default configuration a file
+created mid-session was invisible to every read tool for the rest of the
+session, and `mast_search` returned `{"results":[]}` for symbols that plainly
+existed on disk — the §0 failure mode, reached through a default.
+
+It remains a discovery-freshness optimization rather than a correctness
 mechanism, and it does not reopen the §3 no-daemon non-goal (it lives and dies
-with the serve process).
+with the serve process). `--no-watch` is for batch and container use, where the
+tree does not change under the server and the fd cost buys nothing.
 
 Behaviour:
 
+- **Readiness is announced.** The watcher starts *after* the MCP transport is
+  accepting calls, and chokidar's initial scan runs with `ignoreInitial: true`
+  — correct, because the startup ladder has already indexed the tree, but it
+  means a file created before that scan completes is treated as pre-existing
+  and fires **no event at all**. When the scan finishes, `mast serve` writes
+  `[mast] watch: watching for changes` to stderr, once. Before this existed
+  (D061) silence covered three different states — watching, not watching yet,
+  and watcher failed to start — because only the failure paths wrote anything.
+  A tool or operator that must not miss a change waits for this line.
 - A chokidar watcher covers `file_extensions` under the project root,
   respecting `exclude_patterns` **and the state directory itself** — watching
   the state dir would self-trigger on every index write.
@@ -2380,7 +2484,7 @@ TypeScript (Node.js LTS). Rationale:
 | Locking | `proper-lockfile` | PID-based advisory lock; set `stale: 10000` (10s) to handle abrupt container exits |
 | CLI | `commander` | Standard TS CLI |
 | File walking | `fast-glob` | Glob pattern support for `exclude_patterns` |
-| File watching | `chokidar` | Powers `mast serve --watch` (§11.4) |
+| File watching | `chokidar` | Powers `mast serve`'s watcher (§11.4, default on; `--no-watch` opts out) |
 | Token counting | `@anthropic-ai/tokenizer` | Counts `tokens_returned`/`tokens_full_file_upper_bound` for `_stats` (§14.5) |
 | Identifiers | `uuid` | Per-`mast serve`-session `session_id` for metrics attribution |
 
@@ -2417,7 +2521,7 @@ packages/mast/
 │   ├── indexer/
 │   │   ├── index.ts                 # orchestrates the single indexing pass (§7.1)
 │   │   ├── walker.ts                # file discovery, exclude pattern matching, manifest diff
-│   │   ├── watcher.ts               # chokidar-backed `mast serve --watch` (§11.4)
+│   │   ├── watcher.ts               # chokidar-backed `mast serve` watcher (§11.4)
 │   │   └── import-resolver.ts       # tsconfig paths + pnpm workspace resolution (§13.7)
 │   ├── graph/
 │   │   ├── db.ts                    # better-sqlite3 + Kysely connection, schema init
@@ -2515,6 +2619,23 @@ scope — MAST indexes implementation files. Without this rule, ESM `.js` specif
 `re_export_files` rows. See the TypeScript Modules Reference, "File extension
 substitution".
 
+**0a. Trailing-slash specifiers are directory-only**
+
+A specifier written with a trailing slash (`./routes/`) names a **directory** and
+nothing else. Node refuses it outright when only a sibling file exists —
+`require.resolve('./routes/')` throws `MODULE_NOT_FOUND` — so the resolver skips
+every file probe for such a specifier and considers only `<base>/index.<ext>`,
+returning `null` rather than inventing an edge the runtime does not have.
+
+This is a rule about the *specifier*, not the path: `path.resolve`/`join`
+normalise the slash away, so the intent must be read before the base path is
+built. Without it the extension probes ran first and a sibling file won over the
+directory the author explicitly asked for — a `Routes.ts` importing `./routes/`
+resolved to **itself**, and `routes/index.ts`, which had a live importer, was
+recorded with none (`docs/defects/LEDGER.md` D047). Note the ordering is
+deliberate only for the slash-terminated form: without a slash, `./routes` stays
+file-first, which is what tsc does.
+
 **1. tsconfig `paths` aliases** (e.g. `@api/types` → `./src/types/index.ts`)
 
 Read the nearest `tsconfig.json` at `mast init` time using `tsconfig-paths`. Build a
@@ -2570,11 +2691,15 @@ Two important properties of the seed:
 1. **The index runs fully at build time.** The seed contains a fully-populated
    `graph.db` (chunks, symbols, edges, `chunk_fts`, `identifier_fts`). The runtime
    container is ready to serve at full capability immediately (Step 3 of §7.4) — no
-   warm-up window.
+   tool is degraded while it starts. The one exception is `mast_search`'s
+   `unindexed_files` count, which is omitted until the freshness probe's first
+   measurement lands (ADR 016); it reads as unknown, never as zero.
 2. **Frozen at build commit.** The seed reflects whatever code was in the image at
    `docker build` time. Files modified since the build commit are picked up by
    §7.4 Step 4's filesystem scan (a few seconds for typical incremental staleness),
-   and uncommitted-tree edits trigger JIT re-parse on first query (§9.0).
+   and uncommitted-tree edits to files the seed already contains are re-parsed on
+   first query by the five re-parsing tools, or flagged `stale` by the other two
+   (§9.0). An uncommitted **new** file is in neither category until Step 4 runs.
 
 **Runtime copy.** The container entrypoint runs:
 
@@ -2583,7 +2708,7 @@ Two important properties of the seed:
 if [ -z "$(ls -A /workspace/.kluster/.mast 2>/dev/null)" ]; then
   cp -r /opt/mast-seed/. /workspace/.kluster/.mast/
 fi
-exec mast serve --state-dir /workspace/.kluster/.mast
+exec mast serve --no-watch --state-dir /workspace/.kluster/.mast
 ```
 
 The copy is conditional on the workspace state being empty. Subsequent container

package/README.md

@@ -33,7 +33,7 @@ MAST takes a different approach:
 - **AST-level chunking** — every function, class, interface, and type alias is its own chunk. The assistant gets the exact declaration it needs, not the file it happens to live in.
 - **Ranked search** — BM25 (FTS5) handles keyword and identifier queries; a declaration-exact ranker ("ranker D") catches exact-symbol-name queries that BM25's trigram tokenizer can rank inconsistently. Both are fused via Reciprocal Rank Fusion so a chunk that both rankers agree on outranks one that only one of them found.
 - **Structural queries** — "who calls this function?", "what implements this interface?", "what does this file import?" are answered from a pre-built symbol graph, not by grepping source. Answers are instantaneous and structurally correct.
-- **JIT staleness detection** — on every read, MAST checks whether the file on disk has changed since it was last indexed. If it has, the file is transparently re-parsed in the background before the result is returned. The index never goes stale without the assistant knowing.
+- **Staleness handling that says what it actually does** — five read tools (`mast_signature`, `mast_callers`, `mast_exports`, `mast_dependencies`, `mast_rename_impact`) re-parse a changed file inline before answering; `mast_search` and `mast_implementors` flag affected results with `stale: true` rather than re-parsing. Both only cover files the index *already knows*: a brand-new file is invisible until a reindex, which is why `mast serve` watches by default and why `mast_search` carries an `unindexed_files` warning when it does not.
 - **Token accounting** — every tool response includes `_stats` with the token count returned and the counterfactual "what would a naive full-file read have cost?", giving a concrete measure of efficiency over time.
 
 ---
@@ -240,8 +240,8 @@ absolute. The CLI and editor integrations below infer it from the working direct
 
 ### Any other MCP client
 
-Run `mast serve` over stdio from the project root. It advertises eleven read tools and
-needs no arguments beyond `serve`.
+Run `mast serve` over stdio from the project root. It advertises eleven tools — ten that
+read and `mast_reindex`, which writes — and needs no arguments beyond `serve`.
 
 ### Tell the assistant how to use it
 
@@ -289,7 +289,10 @@ tool rather than one index per package.
 
 **What is indexed.** `.ts`, `.tsx`, `.js`, `.jsx`, and `.md`, minus `node_modules`,
 `dist`, `build`, `coverage`, `.next`, `.turbo`, `.mast`, and test files. Override with
-`--extensions` and `--exclude` on `mast init`, or edit `.mast/config.json`.
+`--extensions` and `--exclude` on `mast init`, or — for a setting the whole team should
+get — `file_extensions` / `exclude_patterns` in `mast.config.json` at the project root.
+Editing `.mast/config.json` also works and is read back, but that file is gitignored and
+per-machine, so the change will not travel; `mast.config.json` outranks it.
 
 **Other languages are not indexed, and this matters.** MAST parses TypeScript and
 JavaScript only. A symbol defined in Python, Go, Java, or Rust is absent from the index,
@@ -297,7 +300,16 @@ which looks exactly like absent from the repository. Treat an empty result as "M
 not find it", never as "it does not exist" — `mast skill` says this to the model too.
 
 **Add `.mast/` to `.gitignore`.** It is derived state, it is large, and it is
-machine-specific.
+machine-specific — on a 14k-file monorepo it is around 420 MB, almost all of it `graph.db`.
+Ignore the whole directory, including `.mast/config.json`: that file is a *resolved*
+snapshot and carries absolute paths (`project_root`, `resolved_state_dir`) that mean
+nothing on anyone else's machine. The file meant to be committed is `mast.config.json` at
+the project root — see the next paragraph.
+
+**If you move the index, move the ignore rule with it.** `.mast/` is the default location,
+not the only one: a `state_dir` in `mast.config.json`, a `MAST_STATE_DIR` in the
+environment, or a `--state-dir` flag all put the index somewhere else, and a `.gitignore`
+naming `.mast/` then protects nothing.
 
 **A custom index location is not remembered between runs.** `--state-dir` applies to the
 one command you pass it to. Path settings are deliberately never read back out of a
@@ -351,9 +363,19 @@ Options:
   -e, --exported         Only exported symbols
   -f, --file <glob>      Restrict to files matching a glob
       --state-dir <dir>  State directory
+      --reindex          Incrementally reindex before searching
       --json             Emit the raw MCP response instead of text
 ```
 
+`--reindex` exists because the CLI has neither freshness mechanism the MCP server has:
+no file watcher, and no cached freshness probe (a one-shot process has no lifetime to
+amortise one over). Without it, a file created since the last index run is invisible —
+JIT staleness only re-parses files the index already knows. It is opt-in because it adds
+a whole incremental index run to what is otherwise a single query, and it reports what it
+did on **stderr**, so `--json` consumers keep a parseable stdout. A reindex that loses
+`structure.lock` to a concurrent writer — likelier now that `mast serve` watches by
+default — warns and queries the existing index rather than failing the search.
+
 **Why:** the fastest way to check what the index actually contains, and the same code path
 the MCP `mast_search` tool uses — it dispatches through the registered handler rather than
 re-implementing ranking, so CLI and assistant results cannot disagree. Staleness and
@@ -390,10 +412,16 @@ Start the MCP server over stdio.
 Options:
   --state-dir <dir>         State directory
   --no-startup-reindex      Skip the startup staleness check (not recommended)
+  --no-watch                Do not watch source files (batch/container use)
   --watch                   Watch source files and incrementally reindex on change
-                             (interactive use; not needed in the container ladder)
+                             (the default; accepted for compatibility)
 ```
 
+Watching is **on by default**. JIT staleness only re-parses files the index already knows,
+so without a watcher a file created during a session stays invisible to every read tool
+until something reindexes — and nothing does. A watcher failure (EMFILE, permissions) logs
+a warning and the server keeps serving, so the default cannot stop `serve` from starting.
+
 The server implements a four-step startup ladder so MCP clients get a usable server in under a second even for large projects. See [Startup Ladder](#startup-ladder) for details.
 
 ---
@@ -442,6 +470,7 @@ Invoke any MCP read tool directly, with byte-identical output to the MCP transpo
 ```
 Options:
   --state-dir <dir>   State directory
+  --reindex           Incrementally reindex before querying (see `mast search --reindex`)
   --json              Emit the exact single-line MCP response (default pretty-prints)
 ```
 
@@ -512,6 +541,31 @@ MAST registers 11 tools with the MCP server. Every read tool includes a `_stats`
 }
 ```
 
+### The signals
+
+Beyond the results, a response carries fields describing what MAST **does not know** about
+the answer it just gave. All but the last are omitted entirely when they do not apply, so
+their absence carries meaning and their presence is never noise. `truncated` is the one
+exception — it is a required field on every `type_context` entry and is present-and-`false`
+in the ordinary case, so read it, don't test for it.
+
+| signal | carried by | means |
+|---|---|---|
+| `stale` | per result of `mast_search`, `mast_implementors` | this result's file changed on disk since it was indexed. The content and line numbers shown may be out of date; no re-parse was attempted (see [JIT Staleness Checks](#jit-staleness-checks)) |
+| `file_busy_returning_stale_cache` | `mast_signature`, `mast_callers`, `mast_exports`, `mast_dependencies`, `mast_rename_impact` | a re-parse *was* attempted and lost to a concurrent writer, so the previous chunk was returned. Contended, not wrong by design — retry shortly |
+| `index_empty` | the empty answer of any of the eight tools that return a result set | nothing is indexed at all. The answer is empty because there was nothing to answer from, not because nothing matched |
+| `unindexed_files` | `mast_search`, `mast_callers`, `mast_implementors`, `mast_rename_impact`, `mast_project_skeleton` (always); `mast_signature`, `mast_exports`, `mast_dependencies` (on an empty answer only) | this many files exist on disk and are not in the index. The answer was computed over an incomplete corpus — an empty or thin result may mean "not indexed" rather than "not present". The tools that claim an exhaustive set report it even when they found something, because a short list reads exactly like a complete one; the three that answer about one named thing report it only when that thing was not found |
+| `results_truncated` | `mast_signature`, `mast_implementors` | the list was capped at `limit`; the field carries the real, uncapped total |
+| `exports_truncated` | `mast_exports` | the same, for a module's export list |
+| `potential_truncated` | `mast_callers`, `mast_rename_impact` | the unresolved-candidate fetch was capped at 50; the field carries the real match count |
+| `truncated` | a `type_context` entry of `mast_signature` | that referenced type's declaration was clipped at 50 lines. The one signal that *is* always present, as a boolean |
+
+Two distinctions that are not flags but carry the same weight: in `mast_callers` and
+`mast_rename_impact`, a **`verified_caller`** carries a `resolution` naming how the edge was
+statically proven and is safe to act on, while a **`potential_match`** carries a `reason` and
+is a name match with no proven edge — review before editing. And an empty result is never
+proof of absence: MAST indexes TypeScript, JavaScript, and Markdown only.
+
 ---
 
 ### `mast_search`
@@ -529,7 +583,7 @@ Lexical BM25 + declaration-exact search over the indexed codebase.
 }
 ```
 
-**Returns:** `{ results[], suggestions?, _stats }`. Each result includes `file_path`, `start_line`, `end_line`, `content`, `chunk_type`, `symbol_name`, `parent_symbol`, `is_exported`, `match_score` (BM25 score, negative; `null` when the hit came only from ranker D), `rank`, `match_snippet`, and an optional `related` hint when a method and its class shell both matched (only the higher-ranked one is returned). `suggestions` is present, possibly empty, only when `results` is empty — a zero-result "did you mean" assist.
+**Returns:** `{ results[], suggestions?, index_empty?, unindexed_files?, _stats }`. Each result includes `file_path`, `start_line`, `end_line`, `content`, `chunk_type`, `symbol_name`, `parent_symbol`, `is_exported`, `match_score` (BM25 score, negative; `null` when the hit came only from ranker D), `rank`, `match_snippet`, an optional `stale` flag, and an optional `related` hint when a method and its class shell both matched (only the higher-ranked one is returned). `suggestions` is present, possibly empty, only when `results` is empty — a zero-result "did you mean" assist. See [The signals](#the-signals) for `stale`, `index_empty`, and `unindexed_files`.
 
 **Why:** `grep` and `glob` find exact strings and require the caller to already know the pattern. `mast_search` ranks by relevance across two signals fused with Reciprocal Rank Fusion:
 
@@ -776,13 +830,34 @@ with default `k = 60`. A chunk appearing at rank 1 in both lists scores twice as
 
 ### JIT Staleness Checks
 
-Every read tool (search, exports, signature, callers, dependencies, implementors) calls `jitRefreshFile` before returning results. This function:
+Read tools handle a file that changed since it was indexed in one of **two** ways. Which one a tool
+uses is fixed per tool, not decided at runtime:
+
+**Re-parse inline** — `mast_signature`, `mast_callers`, `mast_exports`, `mast_dependencies`,
+`mast_rename_impact`. Before returning, each calls `jitRefreshFile`, which:
 
-1. Reads the stored mtime for the file from the `files` table.
+1. Reads the stored mtime for the file from the `files` table. **If there is no row, it returns
+   immediately** — an unindexed file has no stored mtime to compare against, which is the whole of
+   why JIT cannot discover new files.
 2. Calls `stat()` on the file on disk.
-3. If the disk mtime is newer, acquires the `structure.lock` and re-parses the file immediately.
+3. If the disk mtime is newer, acquires the `structure.lock` and re-parses the file immediately,
+   on the request path — the answer waits for it.
+
+If the lock is held by a concurrent writer the previous chunk is returned with
+`file_busy_returning_stale_cache` set, so the caller is told rather than quietly served stale data.
+
+**Stat and flag** — `mast_search`, `mast_implementors`. These call `findStaleFiles`, which stats the
+files behind the results and sets `stale: true` on the affected ones. **No re-parse is attempted**:
+taking a write lock on a ranked result set would serialise the cheapest tools in the package behind
+the most expensive operation in it. The line coordinates on a flagged result may be off; the caller
+is expected to re-read or call a re-parsing tool.
+
+**Neither** — `mast_project_skeleton` (documented exempt) and `mast_efficiency`.
 
-This means an assistant editing a file and immediately querying it will always see the current version, without waiting for a scheduled reindex. (JIT staleness handles files already known to the index; a brand-new file or symbol still needs `mast_reindex` or the next scheduled/watch reindex to be discoverable.)
+Both mechanisms only ever touch files the index **already knows**. Neither can discover a file that
+was never indexed, so a file created during a session is invisible to every read tool until
+something reindexes it — which is why `mast serve` watches by default, and why every
+primary-result tool reports `unindexed_files` when the index is nonetheless behind.
 
 ### Startup Ladder