@spikedpunch/mast 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/MAST_SPEC.md +2926 -0
- package/README.md +842 -0
- package/assets/skill.md +45 -0
- package/dist/ast/extract.d.ts +65 -0
- package/dist/ast/extract.d.ts.map +1 -0
- package/dist/ast/extract.js +147 -0
- package/dist/ast/extract.js.map +1 -0
- package/dist/ast/extractor.d.ts +85 -0
- package/dist/ast/extractor.d.ts.map +1 -0
- package/dist/ast/extractor.js +2 -0
- package/dist/ast/extractor.js.map +1 -0
- package/dist/ast/extractors/markdown.d.ts +40 -0
- package/dist/ast/extractors/markdown.d.ts.map +1 -0
- package/dist/ast/extractors/markdown.js +175 -0
- package/dist/ast/extractors/markdown.js.map +1 -0
- package/dist/ast/extractors/typescript.d.ts +179 -0
- package/dist/ast/extractors/typescript.d.ts.map +1 -0
- package/dist/ast/extractors/typescript.js +1439 -0
- package/dist/ast/extractors/typescript.js.map +1 -0
- package/dist/ast/parser.d.ts +5 -0
- package/dist/ast/parser.d.ts.map +1 -0
- package/dist/ast/parser.js +21 -0
- package/dist/ast/parser.js.map +1 -0
- package/dist/ast/types.d.ts +586 -0
- package/dist/ast/types.d.ts.map +1 -0
- package/dist/ast/types.js +3 -0
- package/dist/ast/types.js.map +1 -0
- package/dist/cli/docs-cmd.d.ts +22 -0
- package/dist/cli/docs-cmd.d.ts.map +1 -0
- package/dist/cli/docs-cmd.js +100 -0
- package/dist/cli/docs-cmd.js.map +1 -0
- package/dist/cli/index-cmd.d.ts +72 -0
- package/dist/cli/index-cmd.d.ts.map +1 -0
- package/dist/cli/index-cmd.js +263 -0
- package/dist/cli/index-cmd.js.map +1 -0
- package/dist/cli/index.d.ts +3 -0
- package/dist/cli/index.d.ts.map +1 -0
- package/dist/cli/index.js +4 -0
- package/dist/cli/index.js.map +1 -0
- package/dist/cli/init.d.ts +14 -0
- package/dist/cli/init.d.ts.map +1 -0
- package/dist/cli/init.js +64 -0
- package/dist/cli/init.js.map +1 -0
- package/dist/cli/install-hooks.d.ts +3 -0
- package/dist/cli/install-hooks.d.ts.map +1 -0
- package/dist/cli/install-hooks.js +34 -0
- package/dist/cli/install-hooks.js.map +1 -0
- package/dist/cli/metrics-cmd.d.ts +3 -0
- package/dist/cli/metrics-cmd.d.ts.map +1 -0
- package/dist/cli/metrics-cmd.js +164 -0
- package/dist/cli/metrics-cmd.js.map +1 -0
- package/dist/cli/program.d.ts +14 -0
- package/dist/cli/program.d.ts.map +1 -0
- package/dist/cli/program.js +44 -0
- package/dist/cli/program.js.map +1 -0
- package/dist/cli/query.d.ts +46 -0
- package/dist/cli/query.d.ts.map +1 -0
- package/dist/cli/query.js +153 -0
- package/dist/cli/query.js.map +1 -0
- package/dist/cli/search-cmd.d.ts +33 -0
- package/dist/cli/search-cmd.d.ts.map +1 -0
- package/dist/cli/search-cmd.js +114 -0
- package/dist/cli/search-cmd.js.map +1 -0
- package/dist/cli/serve.d.ts +3 -0
- package/dist/cli/serve.d.ts.map +1 -0
- package/dist/cli/serve.js +19 -0
- package/dist/cli/serve.js.map +1 -0
- package/dist/cli/skill-install.d.ts +39 -0
- package/dist/cli/skill-install.d.ts.map +1 -0
- package/dist/cli/skill-install.js +57 -0
- package/dist/cli/skill-install.js.map +1 -0
- package/dist/cli/status.d.ts +35 -0
- package/dist/cli/status.d.ts.map +1 -0
- package/dist/cli/status.js +117 -0
- package/dist/cli/status.js.map +1 -0
- package/dist/cli/upgrade-cmd.d.ts +48 -0
- package/dist/cli/upgrade-cmd.d.ts.map +1 -0
- package/dist/cli/upgrade-cmd.js +129 -0
- package/dist/cli/upgrade-cmd.js.map +1 -0
- package/dist/cli/version.d.ts +3 -0
- package/dist/cli/version.d.ts.map +1 -0
- package/dist/cli/version.js +19 -0
- package/dist/cli/version.js.map +1 -0
- package/dist/env.d.ts +17 -0
- package/dist/env.d.ts.map +1 -0
- package/dist/env.js +17 -0
- package/dist/env.js.map +1 -0
- package/dist/graph/checker-resolver.d.ts +156 -0
- package/dist/graph/checker-resolver.d.ts.map +1 -0
- package/dist/graph/checker-resolver.js +458 -0
- package/dist/graph/checker-resolver.js.map +1 -0
- package/dist/graph/db.d.ts +318 -0
- package/dist/graph/db.d.ts.map +1 -0
- package/dist/graph/db.js +266 -0
- package/dist/graph/db.js.map +1 -0
- package/dist/graph/index.d.ts +4 -0
- package/dist/graph/index.d.ts.map +1 -0
- package/dist/graph/index.js +5 -0
- package/dist/graph/index.js.map +1 -0
- package/dist/graph/local-type-env.d.ts +45 -0
- package/dist/graph/local-type-env.d.ts.map +1 -0
- package/dist/graph/local-type-env.js +59 -0
- package/dist/graph/local-type-env.js.map +1 -0
- package/dist/graph/path-range.d.ts +40 -0
- package/dist/graph/path-range.d.ts.map +1 -0
- package/dist/graph/path-range.js +42 -0
- package/dist/graph/path-range.js.map +1 -0
- package/dist/graph/populate.d.ts +263 -0
- package/dist/graph/populate.d.ts.map +1 -0
- package/dist/graph/populate.js +984 -0
- package/dist/graph/populate.js.map +1 -0
- package/dist/graph/queries.d.ts +114 -0
- package/dist/graph/queries.d.ts.map +1 -0
- package/dist/graph/queries.js +415 -0
- package/dist/graph/queries.js.map +1 -0
- package/dist/graph/sqliteBatch.d.ts +48 -0
- package/dist/graph/sqliteBatch.d.ts.map +1 -0
- package/dist/graph/sqliteBatch.js +93 -0
- package/dist/graph/sqliteBatch.js.map +1 -0
- package/dist/indexer/freshness.d.ts +48 -0
- package/dist/indexer/freshness.d.ts.map +1 -0
- package/dist/indexer/freshness.js +72 -0
- package/dist/indexer/freshness.js.map +1 -0
- package/dist/indexer/import-resolver.d.ts +50 -0
- package/dist/indexer/import-resolver.d.ts.map +1 -0
- package/dist/indexer/import-resolver.js +318 -0
- package/dist/indexer/import-resolver.js.map +1 -0
- package/dist/indexer/index.d.ts +249 -0
- package/dist/indexer/index.d.ts.map +1 -0
- package/dist/indexer/index.js +560 -0
- package/dist/indexer/index.js.map +1 -0
- package/dist/indexer/walker.d.ts +60 -0
- package/dist/indexer/walker.d.ts.map +1 -0
- package/dist/indexer/walker.js +128 -0
- package/dist/indexer/walker.js.map +1 -0
- package/dist/indexer/watcher.d.ts +80 -0
- package/dist/indexer/watcher.d.ts.map +1 -0
- package/dist/indexer/watcher.js +151 -0
- package/dist/indexer/watcher.js.map +1 -0
- package/dist/mcp/context.d.ts +23 -0
- package/dist/mcp/context.d.ts.map +1 -0
- package/dist/mcp/context.js +2 -0
- package/dist/mcp/context.js.map +1 -0
- package/dist/mcp/register-tools.d.ts +21 -0
- package/dist/mcp/register-tools.d.ts.map +1 -0
- package/dist/mcp/register-tools.js +42 -0
- package/dist/mcp/register-tools.js.map +1 -0
- package/dist/mcp/server.d.ts +58 -0
- package/dist/mcp/server.d.ts.map +1 -0
- package/dist/mcp/server.js +177 -0
- package/dist/mcp/server.js.map +1 -0
- package/dist/mcp/staleness.d.ts +72 -0
- package/dist/mcp/staleness.d.ts.map +1 -0
- package/dist/mcp/staleness.js +213 -0
- package/dist/mcp/staleness.js.map +1 -0
- package/dist/mcp/startup.d.ts +40 -0
- package/dist/mcp/startup.d.ts.map +1 -0
- package/dist/mcp/startup.js +114 -0
- package/dist/mcp/startup.js.map +1 -0
- package/dist/mcp/tools/_helpers.d.ts +26 -0
- package/dist/mcp/tools/_helpers.d.ts.map +1 -0
- package/dist/mcp/tools/_helpers.js +54 -0
- package/dist/mcp/tools/_helpers.js.map +1 -0
- package/dist/mcp/tools/callers.d.ts +4 -0
- package/dist/mcp/tools/callers.d.ts.map +1 -0
- package/dist/mcp/tools/callers.js +112 -0
- package/dist/mcp/tools/callers.js.map +1 -0
- package/dist/mcp/tools/dependencies.d.ts +4 -0
- package/dist/mcp/tools/dependencies.d.ts.map +1 -0
- package/dist/mcp/tools/dependencies.js +36 -0
- package/dist/mcp/tools/dependencies.js.map +1 -0
- package/dist/mcp/tools/efficiency.d.ts +4 -0
- package/dist/mcp/tools/efficiency.d.ts.map +1 -0
- package/dist/mcp/tools/efficiency.js +39 -0
- package/dist/mcp/tools/efficiency.js.map +1 -0
- package/dist/mcp/tools/exports.d.ts +4 -0
- package/dist/mcp/tools/exports.d.ts.map +1 -0
- package/dist/mcp/tools/exports.js +56 -0
- package/dist/mcp/tools/exports.js.map +1 -0
- package/dist/mcp/tools/implementors.d.ts +4 -0
- package/dist/mcp/tools/implementors.d.ts.map +1 -0
- package/dist/mcp/tools/implementors.js +47 -0
- package/dist/mcp/tools/implementors.js.map +1 -0
- package/dist/mcp/tools/project-skeleton.d.ts +4 -0
- package/dist/mcp/tools/project-skeleton.d.ts.map +1 -0
- package/dist/mcp/tools/project-skeleton.js +61 -0
- package/dist/mcp/tools/project-skeleton.js.map +1 -0
- package/dist/mcp/tools/reindex.d.ts +14 -0
- package/dist/mcp/tools/reindex.d.ts.map +1 -0
- package/dist/mcp/tools/reindex.js +32 -0
- package/dist/mcp/tools/reindex.js.map +1 -0
- package/dist/mcp/tools/rename-impact.d.ts +4 -0
- package/dist/mcp/tools/rename-impact.d.ts.map +1 -0
- package/dist/mcp/tools/rename-impact.js +124 -0
- package/dist/mcp/tools/rename-impact.js.map +1 -0
- package/dist/mcp/tools/search.d.ts +4 -0
- package/dist/mcp/tools/search.d.ts.map +1 -0
- package/dist/mcp/tools/search.js +69 -0
- package/dist/mcp/tools/search.js.map +1 -0
- package/dist/mcp/tools/signature.d.ts +4 -0
- package/dist/mcp/tools/signature.d.ts.map +1 -0
- package/dist/mcp/tools/signature.js +127 -0
- package/dist/mcp/tools/signature.js.map +1 -0
- package/dist/mcp/tools/status.d.ts +4 -0
- package/dist/mcp/tools/status.d.ts.map +1 -0
- package/dist/mcp/tools/status.js +30 -0
- package/dist/mcp/tools/status.js.map +1 -0
- package/dist/search/declex.d.ts +109 -0
- package/dist/search/declex.d.ts.map +1 -0
- package/dist/search/declex.js +242 -0
- package/dist/search/declex.js.map +1 -0
- package/dist/search/fts.d.ts +77 -0
- package/dist/search/fts.d.ts.map +1 -0
- package/dist/search/fts.js +223 -0
- package/dist/search/fts.js.map +1 -0
- package/dist/search/fused.d.ts +110 -0
- package/dist/search/fused.d.ts.map +1 -0
- package/dist/search/fused.js +323 -0
- package/dist/search/fused.js.map +1 -0
- package/dist/search/potential-matches.d.ts +88 -0
- package/dist/search/potential-matches.d.ts.map +1 -0
- package/dist/search/potential-matches.js +83 -0
- package/dist/search/potential-matches.js.map +1 -0
- package/dist/search/scope.d.ts +66 -0
- package/dist/search/scope.d.ts.map +1 -0
- package/dist/search/scope.js +71 -0
- package/dist/search/scope.js.map +1 -0
- package/dist/store/config.d.ts +57 -0
- package/dist/store/config.d.ts.map +1 -0
- package/dist/store/config.js +208 -0
- package/dist/store/config.js.map +1 -0
- package/dist/store/lock.d.ts +101 -0
- package/dist/store/lock.d.ts.map +1 -0
- package/dist/store/lock.js +182 -0
- package/dist/store/lock.js.map +1 -0
- package/dist/store/lockMetrics.d.ts +52 -0
- package/dist/store/lockMetrics.d.ts.map +1 -0
- package/dist/store/lockMetrics.js +29 -0
- package/dist/store/lockMetrics.js.map +1 -0
- package/dist/store/sqliteChunkStore.d.ts +44 -0
- package/dist/store/sqliteChunkStore.d.ts.map +1 -0
- package/dist/store/sqliteChunkStore.js +135 -0
- package/dist/store/sqliteChunkStore.js.map +1 -0
- package/dist/telemetry/lockMetricsSummary.d.ts +42 -0
- package/dist/telemetry/lockMetricsSummary.d.ts.map +1 -0
- package/dist/telemetry/lockMetricsSummary.js +139 -0
- package/dist/telemetry/lockMetricsSummary.js.map +1 -0
- package/dist/telemetry/metrics.d.ts +140 -0
- package/dist/telemetry/metrics.d.ts.map +1 -0
- package/dist/telemetry/metrics.js +312 -0
- package/dist/telemetry/metrics.js.map +1 -0
- package/dist/telemetry/tokenizer.d.ts +125 -0
- package/dist/telemetry/tokenizer.d.ts.map +1 -0
- package/dist/telemetry/tokenizer.js +174 -0
- package/dist/telemetry/tokenizer.js.map +1 -0
- package/package.json +81 -0
package/MAST_SPEC.md
ADDED
|
@@ -0,0 +1,2926 @@
|
|
|
1
|
+
# MAST — Monorepo AST Search Tool
|
|
2
|
+
## Technical Specification
|
|
3
|
+
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
## 1. Overview
|
|
7
|
+
|
|
8
|
+
**mast** is a lexical + declaration-exact code search engine exposed over two surfaces:
|
|
9
|
+
an MCP server (used by the agent inside the claude-runner container) and a CLI (used by
|
|
10
|
+
humans and hooks outside the container). It replaces ad-hoc `Grep`, `Glob`, and
|
|
11
|
+
whole-file `Read` calls with targeted, index-backed queries that return structured
|
|
12
|
+
subsets of code rather than full file contents.
|
|
13
|
+
|
|
14
|
+
A single on-disk index — written to a configurable state directory — is shared by both
|
|
15
|
+
surfaces. The index persists on the mounted workspace volume across container runs,
|
|
16
|
+
so each new container inherits the index built by previous tasks.
|
|
17
|
+
|
|
18
|
+
**Note on prior discussions of a semantic/vector search leg:** an earlier revision of
|
|
19
|
+
this system fused BM25 with a vector-embedding ranker (LanceDB + a local ONNX model).
|
|
20
|
+
That subsystem was removed 2026-08-06 per the M2 decision (`adr/003-2026-08-04-vector-store-deletion.md`);
|
|
21
|
+
the pre-deletion system is preserved at the git tag
|
|
22
|
+
`mast-pre-vector-delete`. Everything below describes the system as it exists today.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## 2. Goals
|
|
27
|
+
|
|
28
|
+
- Return **chunks not files**: every query response contains only the lines the agent
|
|
29
|
+
needs, not the full file.
|
|
30
|
+
- **Zero BT involvement** after init: the MCP server startup check keeps the index
|
|
31
|
+
fresh automatically.
|
|
32
|
+
- **Configurable state directory**: the SDD pipeline writes state under
|
|
33
|
+
`.kluster/.mast`; the default for standalone use is `.mast` at the project root.
|
|
34
|
+
- **Synchronous freshness on demand**: the agent can call `mast_reindex` mid-task after
|
|
35
|
+
writing files and be guaranteed the next query sees those changes.
|
|
36
|
+
- **Single codebase, two surfaces**: the same index and query engine backs both the
|
|
37
|
+
MCP tools and CLI commands.
|
|
38
|
+
|
|
39
|
+
## 3. Non-Goals
|
|
40
|
+
|
|
41
|
+
- Code generation or explanation.
|
|
42
|
+
- PR review, wiki generation, story generation.
|
|
43
|
+
- A persistent background daemon — freshness is handled by the startup check and
|
|
44
|
+
`mast_reindex`. *Narrow carve-out:* `mast serve --watch` (§11.4) is an opt-in
|
|
45
|
+
file watcher for **interactive, non-container** use only. It is scoped to the
|
|
46
|
+
serve process lifetime (not a daemon), the SDD pipeline never uses it, and it
|
|
47
|
+
is a discovery-freshness optimization — it lets new files and symbols become
|
|
48
|
+
searchable sooner than waiting for an explicit `mast_reindex`. JIT staleness
|
|
49
|
+
handling (§9.0) already guarantees line-coordinate and content correctness for
|
|
50
|
+
already-indexed files without it.
|
|
51
|
+
- Support for non-TypeScript/JavaScript projects in v1 (AST layer is extensible but
|
|
52
|
+
v1 targets the SDD stack).
|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## 4. Configuration
|
|
57
|
+
|
|
58
|
+
The state directory itself is resolved first, independent of everything else (it must
|
|
59
|
+
be known before its own persisted config can be loaded from inside it):
|
|
60
|
+
|
|
61
|
+
1. CLI flag `--state-dir <path>` (or env `MAST_STATE_DIR`)
|
|
62
|
+
2. `state_dir` key in `mast.config.json` in the project root
|
|
63
|
+
3. Default: `<project_root>/.mast`
|
|
64
|
+
|
|
65
|
+
Every other config key (`file_extensions`, `exclude_patterns`, `rrf_k`,
|
|
66
|
+
`declaration_exact_ranker`, `chunk_split_threshold`, `context_lines`,
|
|
67
|
+
`markdown_heading_depth`) is then resolved in this priority order, highest first:
|
|
68
|
+
|
|
69
|
+
1. Explicit CLI overrides — `mast init --extensions <ext,...>` / `--exclude <pattern,...>`
|
|
70
|
+
(F9, Stage 3.5)
|
|
71
|
+
2. `mast.config.json` in the project root
|
|
72
|
+
3. The persisted `<state_dir>/config.json` from a previous `mast init` / `mast serve`
|
|
73
|
+
in this state directory (F9 — previously write-only; now read back on every
|
|
74
|
+
resolution)
|
|
75
|
+
4. Built-in defaults
|
|
76
|
+
|
|
77
|
+
**Path keys are never taken from the persisted state config.** `<state_dir>/config.json`
|
|
78
|
+
stores a full resolved config, including the ABSOLUTE `state_dir`/`project_root`/
|
|
79
|
+
`resolved_state_dir`/`resolved_project_root` from whichever process last wrote it. The
|
|
80
|
+
SDD pipeline mounts the same workspace volume at different container paths across runs,
|
|
81
|
+
so an absolute path loaded back from a previous container could silently point the
|
|
82
|
+
resolver at a path that doesn't exist (or belongs to an unrelated project) in the
|
|
83
|
+
current one. Only the customisation keys are read from the persisted file; the four
|
|
84
|
+
path keys always come from the current resolution.
|
|
85
|
+
|
|
86
|
+
### 4.1 `mast.config.json`
|
|
87
|
+
|
|
88
|
+
```json
|
|
89
|
+
{
|
|
90
|
+
"state_dir": ".mast",
|
|
91
|
+
"project_root": ".",
|
|
92
|
+
"file_extensions": [".ts", ".tsx", ".js", ".jsx", ".md"],
|
|
93
|
+
"exclude_patterns": [
|
|
94
|
+
"**/node_modules/**",
|
|
95
|
+
"**/dist/**",
|
|
96
|
+
"**/build/**",
|
|
97
|
+
"**/coverage/**",
|
|
98
|
+
"**/.next/**",
|
|
99
|
+
"**/.turbo/**",
|
|
100
|
+
".mast/**",
|
|
101
|
+
"**/*.test.ts",
|
|
102
|
+
"**/*.spec.ts"
|
|
103
|
+
],
|
|
104
|
+
"rrf_k": 60,
|
|
105
|
+
"declaration_exact_ranker": true,
|
|
106
|
+
"chunk_split_threshold": 100,
|
|
107
|
+
"context_lines": 3,
|
|
108
|
+
"markdown_heading_depth": 2
|
|
109
|
+
}
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
`rrf_k` is the constant in the Reciprocal Rank Fusion formula (see §7.3). The default
|
|
113
|
+
of 60 is standard. Higher values reduce the influence of rank differences.
|
|
114
|
+
|
|
115
|
+
`declaration_exact_ranker` (default `true`) is the F18 kill-switch: it fuses the
|
|
116
|
+
declaration-exact ranker (ranker D — see §7.3) into `mast_search` ranking as an
|
|
117
|
+
additional RRF input. Set `false` to restore pre-F18 ranking without a code change.
|
|
118
|
+
The flag exists because ranker D's harm surface on identifier-free queries is
|
|
119
|
+
monitored, not proven safe (M2 decision memo, condition 3); its D-fire telemetry
|
|
120
|
+
(§14.3 `declex_json`) is the input signal for that monitoring.
|
|
121
|
+
|
|
122
|
+
There is deliberately **no `similarity_threshold` key** — it gated a vector-search leg
|
|
123
|
+
that no longer exists (removed 2026-08-06, §1); no replacement config key was needed.
|
|
124
|
+
|
|
125
|
+
`chunk_split_threshold` is the line count above which a single declaration is split
|
|
126
|
+
into overlapping sub-chunks. Below this threshold, a declaration is always one chunk
|
|
127
|
+
regardless of length.
|
|
128
|
+
|
|
129
|
+
`context_lines` controls how many source lines before and after a chunk's AST
|
|
130
|
+
boundaries are included in the stored `content` field. When a chunk's AST span is
|
|
131
|
+
`[start_line, end_line]`, the stored content covers
|
|
132
|
+
`[max(1, start_line - context_lines), min(file_lines, end_line + context_lines)]`.
|
|
133
|
+
This gives agents surrounding context (e.g., the `const` binding before a function
|
|
134
|
+
expression, or the closing brace of an enclosing block) without requiring a full file
|
|
135
|
+
read. The `start_line` and `end_line` fields in the chunk record always reflect the
|
|
136
|
+
AST declaration boundaries, not the expanded content boundaries.
|
|
137
|
+
|
|
138
|
+
`markdown_heading_depth` is the maximum ATX heading level that starts a new `doc`
|
|
139
|
+
chunk when indexing markdown files (§10.1). Headings deeper than this fold into
|
|
140
|
+
their enclosing section. The default of 2 means one chunk per `##` section.
|
|
141
|
+
|
|
142
|
+
Vendored markdown noise (dependency READMEs and the like) is handled by the
|
|
143
|
+
existing `exclude_patterns` — `node_modules/**` is authoritative; there is no
|
|
144
|
+
markdown-specific exclusion logic.
|
|
145
|
+
|
|
146
|
+
Every field above except `state_dir`/`project_root` is also read back from
|
|
147
|
+
`<state_dir>/config.json` when `mast.config.json` and CLI flags don't override it (§4,
|
|
148
|
+
F9) — see §5 for the file's read/write semantics.
|
|
149
|
+
|
|
150
|
+
### 4.2 SDD Pipeline Configuration
|
|
151
|
+
|
|
152
|
+
The claude-runner passes `--state-dir` at serve time:
|
|
153
|
+
|
|
154
|
+
```json
|
|
155
|
+
// claude-runner/.mcp.json
|
|
156
|
+
{
|
|
157
|
+
"mcpServers": {
|
|
158
|
+
"mast": {
|
|
159
|
+
"command": "mast",
|
|
160
|
+
"args": ["serve", "--state-dir", "/workspace/.kluster/.mast"],
|
|
161
|
+
"type": "stdio"
|
|
162
|
+
}
|
|
163
|
+
}
|
|
164
|
+
}
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
This is the only configuration change needed in the SDD pipeline after `mast init`.
|
|
168
|
+
|
|
169
|
+
---
|
|
170
|
+
|
|
171
|
+
## 5. Storage Layout
|
|
172
|
+
|
|
173
|
+
```
|
|
174
|
+
<state_dir>/
|
|
175
|
+
├── config.json # Resolved active config (written at init/serve; read back
|
|
176
|
+
# on every resolution — §4, F9)
|
|
177
|
+
├── index.json # Index metadata: last_indexed, file_count, schema_version
|
|
178
|
+
├── file_manifest.json # {path: mtime} snapshot from last index run
|
|
179
|
+
├── structure.lock # Advisory write lock for coarse writers (index, mast_reindex, manifest)
|
|
180
|
+
└── graph.db # Knowledge graph, chunks, and FTS5 index (SQLite, WAL mode)
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
`index.json` example:
|
|
184
|
+
```json
|
|
185
|
+
{
|
|
186
|
+
"schema_version": "1.3.0",
|
|
187
|
+
"last_indexed": "2026-05-13T14:22:00Z",
|
|
188
|
+
"file_count": 142,
|
|
189
|
+
"chunk_count": 1840
|
|
190
|
+
}
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
---
|
|
194
|
+
|
|
195
|
+
## 6. Data Model
|
|
196
|
+
|
|
197
|
+
### 6.1 Chunk (`chunks` table — `graph.db`)
|
|
198
|
+
|
|
199
|
+
| Field | Type | Description |
|
|
200
|
+
|---|---|---|
|
|
201
|
+
| `chunk_id` | `str` | `sha256(file_path + ":" + start_line)` |
|
|
202
|
+
| `file_path` | `str` | Relative to `project_root` |
|
|
203
|
+
| `start_line` | `int` | 1-indexed |
|
|
204
|
+
| `end_line` | `int` | 1-indexed, inclusive |
|
|
205
|
+
| `content` | `str` | Raw source text of the chunk |
|
|
206
|
+
| `chunk_type` | `str` | `function` \| `method` \| `class_shell` \| `interface` \| `type` \| `export` \| `block` \| `doc` |
|
|
207
|
+
| `symbol_name` | `str \| None` | Top-level symbol name if applicable. For `method` chunks, qualified as `ClassName.methodName`. For `doc` chunks, the heading path (§10.1). |
|
|
208
|
+
| `parent_symbol` | `str \| None` | For `method` chunks, the enclosing class name (unqualified). `None` for all other chunk types. Enables fast "find all methods of class X" queries against the `chunks` table without joining the graph. |
|
|
209
|
+
| `is_exported` | `bool` | True if the declaration carries an `export` modifier. For `method` chunks, inherited from the enclosing `class_shell`'s `is_exported` *and* the method's accessibility (anything not `private` is treated as exported when the class is exported). |
|
|
210
|
+
| `language` | `str` | `typescript` \| `javascript` \| `markdown` |
|
|
211
|
+
| `file_mtime` | `float` | File mtime at index time — used for staleness detection |
|
|
212
|
+
|
|
213
|
+
`is_exported` enables `mast_search` to filter results to public API surface only,
|
|
214
|
+
which is the correct scope when an agent is looking for a service to call rather than
|
|
215
|
+
an internal utility to modify.
|
|
216
|
+
|
|
217
|
+
**`class_shell` content is synthesized, not raw source.** For a `class_shell` chunk,
|
|
218
|
+
the stored `content` field is the class declaration line followed by every member
|
|
219
|
+
signature (with TSDoc comments), ordered as they appear in source — but with method
|
|
220
|
+
bodies stripped. This is the "outline" view used for orientation and for
|
|
221
|
+
`mast_signature` calls that target a class rather than a specific method. The raw
|
|
222
|
+
class body source is *not* stored as a single chunk; it is decomposed into N
|
|
223
|
+
`method` chunks, each its own row in the `chunks` table (see §10.1).
|
|
224
|
+
|
|
225
|
+
### 6.3 Knowledge Graph (SQLite — `graph.db`)
|
|
226
|
+
|
|
227
|
+
SQLite with WAL mode replaces KuzuDB. WAL mode is correct for containerised
|
|
228
|
+
shared-volume access: it tolerates concurrent readers and a single writer without
|
|
229
|
+
exclusive locks that can deadlock across container boundaries.
|
|
230
|
+
|
|
231
|
+
Recursive CTEs handle multi-hop graph traversal (callers of callers, transitive
|
|
232
|
+
dependency chains) with sub-millisecond latency at monorepo scale.
|
|
233
|
+
|
|
234
|
+
```sql
|
|
235
|
+
PRAGMA journal_mode = WAL;
|
|
236
|
+
PRAGMA foreign_keys = ON;
|
|
237
|
+
|
|
238
|
+
CREATE TABLE IF NOT EXISTS files (
|
|
239
|
+
id INTEGER PRIMARY KEY,
|
|
240
|
+
path TEXT NOT NULL UNIQUE,
|
|
241
|
+
language TEXT NOT NULL,
|
|
242
|
+
mtime REAL NOT NULL
|
|
243
|
+
);
|
|
244
|
+
|
|
245
|
+
CREATE TABLE IF NOT EXISTS symbols (
|
|
246
|
+
id INTEGER PRIMARY KEY,
|
|
247
|
+
name TEXT NOT NULL,
|
|
248
|
+
kind TEXT NOT NULL, -- function | class | interface | type | const
|
|
249
|
+
file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
|
250
|
+
line INTEGER NOT NULL,
|
|
251
|
+
is_exported INTEGER NOT NULL DEFAULT 0, -- boolean
|
|
252
|
+
declaration_hash TEXT, -- sha256 of signature text only (excludes body)
|
|
253
|
+
body_hash TEXT -- sha256 of body text only (excludes signature)
|
|
254
|
+
-- If both declaration_hash and body_hash are unchanged on incremental reindex:
|
|
255
|
+
-- skip the KG rebuild for this symbol (§7.1's file-level stability-hash skip).
|
|
256
|
+
);
|
|
257
|
+
|
|
258
|
+
CREATE TABLE IF NOT EXISTS edges (
|
|
259
|
+
from_id INTEGER NOT NULL REFERENCES symbols(id) ON DELETE CASCADE,
|
|
260
|
+
to_id INTEGER NOT NULL REFERENCES symbols(id) ON DELETE CASCADE,
|
|
261
|
+
edge_type TEXT NOT NULL,
|
|
262
|
+
resolution TEXT, -- POTENTIAL_CALL only: which §10.3.1 rule matched
|
|
263
|
+
-- (import | field_type | parameter_type | new_expression | same_file
|
|
264
|
+
-- | this_method | super_method)
|
|
265
|
+
-- or 'checker' (§10.3.2) — the opt-in `mast index --checker`
|
|
266
|
+
-- pass upgraded a heuristic-unresolved potential match via
|
|
267
|
+
-- the real TypeScript checker. 'this_method'/'super_method' (F4,
|
|
268
|
+
-- Stage 3) are additive values for `this.foo()`/`super.foo()`
|
|
269
|
+
-- call sites — see §10.3.1's "Method calls on super and this".
|
|
270
|
+
-- Additive values, no schema change.
|
|
271
|
+
call_line INTEGER, -- POTENTIAL_CALL only: 1-indexed source line of the call site
|
|
272
|
+
context TEXT, -- POTENTIAL_CALL only: trimmed source text of the call-site line
|
|
273
|
+
-- POTENTIAL_CALL | IMPLEMENTS | EXTENDS | RE_EXPORTS | PARENT_OF
|
|
274
|
+
-- POTENTIAL_CALL: name-resolved reference. The local heuristic resolver (see §10.3)
|
|
275
|
+
-- produces these edges when it can statically link the receiver of a
|
|
276
|
+
-- method call to a known symbol. Coverage is partial by design — see
|
|
277
|
+
-- §10.3 for what the resolver catches and what it doesn't. Tools that
|
|
278
|
+
-- consume these edges (mast_callers) MUST also surface identifier-FTS
|
|
279
|
+
-- matches as "potential" results to compensate for missed edges.
|
|
280
|
+
-- RE_EXPORTS: file A re-exports a symbol from file B via export * or export { x }
|
|
281
|
+
-- PARENT_OF: class symbol → method symbol. Emitted once per method during
|
|
282
|
+
-- class_body decomposition (see §10.1). Enables "list all methods of X"
|
|
283
|
+
-- queries via a single indexed lookup.
|
|
284
|
+
PRIMARY KEY (from_id, to_id, edge_type)
|
|
285
|
+
);
|
|
286
|
+
|
|
287
|
+
CREATE TABLE IF NOT EXISTS re_export_files (
|
|
288
|
+
-- Tracks export * from '...' at the file level (no specific symbol known at parse time)
|
|
289
|
+
from_file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
|
290
|
+
to_file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
|
291
|
+
PRIMARY KEY (from_file_id, to_file_id)
|
|
292
|
+
);
|
|
293
|
+
|
|
294
|
+
CREATE TABLE IF NOT EXISTS imports (
|
|
295
|
+
file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
|
296
|
+
module TEXT NOT NULL,
|
|
297
|
+
symbols TEXT NOT NULL, -- JSON array of imported symbol names
|
|
298
|
+
is_external INTEGER NOT NULL DEFAULT 0,
|
|
299
|
+
resolved_path TEXT -- NULL for external modules; populated by path resolver
|
|
300
|
+
);
|
|
301
|
+
|
|
302
|
+
-- FTS5 with built-in content: stores content directly alongside the index structures.
|
|
303
|
+
-- snippet() works without any external table. Indexing inserts/updates/deletes directly
|
|
304
|
+
-- on chunk_fts; no sync logic required. Content duplication vs the `chunks` table is
|
|
305
|
+
-- acceptable at monorepo scale (~1-2 GB of source) and eliminates a whole class of
|
|
306
|
+
-- consistency bugs.
|
|
307
|
+
-- trigram tokenizer: substring matching for camelCase identifiers, prose, and partial
|
|
308
|
+
-- queries. Used by mast_search BM25 ranking via bm25(chunk_fts) (returns negative scores).
|
|
309
|
+
CREATE VIRTUAL TABLE IF NOT EXISTS chunk_fts USING fts5(
|
|
310
|
+
content,
|
|
311
|
+
symbol_name UNINDEXED, -- stored for retrieval; trigram on content already surfaces symbol names
|
|
312
|
+
chunk_id UNINDEXED,
|
|
313
|
+
tokenize = 'trigram'
|
|
314
|
+
);
|
|
315
|
+
|
|
316
|
+
-- Identifier-exact FTS: a search for "findUserByEmail" must match that identifier
|
|
317
|
+
-- exactly, NOT substrings like "findUser" or "findUserByEmailVerified". The trigram
|
|
318
|
+
-- tokenizer (above) is too noisy for this — it routinely returns dozens of false
|
|
319
|
+
-- positives on common method names. unicode61 with code-aware separators tokenizes
|
|
320
|
+
-- on identifier boundaries (.-_/()[]{}<>:;,=+*&|!?), giving exact-identifier match
|
|
321
|
+
-- semantics needed by mast_callers' "potential_matches" set (see §9 mast_callers).
|
|
322
|
+
--
|
|
323
|
+
-- The `identifiers` column is populated by Phase 1: it stores a whitespace-separated
|
|
324
|
+
-- list of every identifier token found in the chunk (deduplicated). Phase 1
|
|
325
|
+
-- extracts these via tree-sitter — no separate parse pass.
|
|
326
|
+
--
|
|
327
|
+
-- F5 (Stage 3, schema 1.3.0): the bag ALSO carries QUALIFIED compound strings
|
|
328
|
+
-- ("Class.method"), appended after the bare tokens. A phrase-quoted query for a
|
|
329
|
+
-- qualified method name (mast_callers/mast_rename_impact's documented
|
|
330
|
+
-- "Class.method" convention) requires the class and method tokens ADJACENT
|
|
331
|
+
-- under this tokenizer's '.' separator — the bare bag alone essentially never
|
|
332
|
+
-- produces that adjacency, silently emptying the potential set for any method
|
|
333
|
+
-- query. Two sources feed the compounds, both derived from the SAME
|
|
334
|
+
-- `LocalTypeEnvironment` resolution `extractEdges` already computes (no
|
|
335
|
+
-- parallel mechanism): (1) a method chunk's own qualified `symbol_name`
|
|
336
|
+
-- (declaration self-discoverability — constructor/getter/setter forms are
|
|
337
|
+
-- already qualified); (2) for every call site the resolver statically linked
|
|
338
|
+
-- to a receiver type, the resolved `Type.method` string, appended to the
|
|
339
|
+
-- CALLING chunk's bag — this also heals the case where the POTENTIAL_CALL
|
|
340
|
+
-- edge itself is later dropped by `insertEdges`' file-scoped resolution (e.g.
|
|
341
|
+
-- an unresolvable import), since extraction and edge-insertion are
|
|
342
|
+
-- independent. A receiver the resolver could NOT statically link (DI
|
|
343
|
+
-- containers, factories — §10.3.1's documented "does NOT catch" list)
|
|
344
|
+
-- contributes no compound; that residual gap is checker-pass (`--checker`)
|
|
345
|
+
-- territory, not this fix's.
|
|
346
|
+
CREATE VIRTUAL TABLE IF NOT EXISTS identifier_fts USING fts5(
|
|
347
|
+
identifiers,
|
|
348
|
+
chunk_id UNINDEXED,
|
|
349
|
+
tokenize = "unicode61 separators '.-_/()[]{}<>:;,=+*&|!?'"
|
|
350
|
+
);
|
|
351
|
+
|
|
352
|
+
-- Hot-path index: name lookup returns file_id, line, kind directly from the index
|
|
353
|
+
-- tree without touching table rows (covering index for mast_signature / mast_search).
|
|
354
|
+
CREATE INDEX IF NOT EXISTS idx_symbols_lookup ON symbols(name, file_id, line, kind);
|
|
355
|
+
CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_id);
|
|
356
|
+
CREATE INDEX IF NOT EXISTS idx_edges_from ON edges(from_id);
|
|
357
|
+
CREATE INDEX IF NOT EXISTS idx_edges_to ON edges(to_id);
|
|
358
|
+
CREATE INDEX IF NOT EXISTS idx_imports_file ON imports(file_id);
|
|
359
|
+
CREATE INDEX IF NOT EXISTS idx_imports_resolved ON imports(resolved_path);
|
|
360
|
+
```
|
|
361
|
+
|
|
362
|
+
**Recursive CTE example — transitive callers (verified set only):**
|
|
363
|
+
```sql
|
|
364
|
+
WITH RECURSIVE callers(id) AS (
|
|
365
|
+
SELECT from_id FROM edges
|
|
366
|
+
WHERE to_id = :target_id AND edge_type = 'POTENTIAL_CALL'
|
|
367
|
+
UNION
|
|
368
|
+
SELECT e.from_id FROM edges e
|
|
369
|
+
JOIN callers c ON e.to_id = c.id
|
|
370
|
+
WHERE e.edge_type = 'POTENTIAL_CALL'
|
|
371
|
+
)
|
|
372
|
+
SELECT s.name, f.path, s.line
|
|
373
|
+
FROM symbols s
|
|
374
|
+
JOIN files f ON s.file_id = f.id
|
|
375
|
+
JOIN callers c ON s.id = c.id;
|
|
376
|
+
```
|
|
377
|
+
|
|
378
|
+
This CTE only returns callers the heuristic resolver could statically link. It is
|
|
379
|
+
the **verified** set. `mast_callers` MUST also run an `identifier_fts` query for
|
|
380
|
+
the symbol name and present those hits as a separate **potential** set — see §9
|
|
381
|
+
`mast_callers`.
|
|
382
|
+
|
|
383
|
+
**Recursive CTE example — barrel file / transitive re-export resolution:**
|
|
384
|
+
|
|
385
|
+
Used by `mast_signature` when the direct symbol lookup returns no result, indicating
|
|
386
|
+
the symbol may be re-exported through one or more barrel files.
|
|
387
|
+
|
|
388
|
+
Note: `re_export_files` edges are file-to-file (using `files.id`). `edges` with
|
|
389
|
+
`RE_EXPORTS` type are symbol-to-symbol (using `symbols.id`). The CTE below follows
|
|
390
|
+
the file-level chain from `re_export_files`, then locates the symbol in the terminal
|
|
391
|
+
file. This correctly separates file IDs from symbol IDs.
|
|
392
|
+
|
|
393
|
+
```sql
|
|
394
|
+
WITH RECURSIVE re_export_chain(file_id) AS (
|
|
395
|
+
-- Start from the file where the agent queried the symbol
|
|
396
|
+
SELECT :start_file_id
|
|
397
|
+
UNION
|
|
398
|
+
-- Follow export * from '...' edges to source files
|
|
399
|
+
SELECT rf.to_file_id
|
|
400
|
+
FROM re_export_files rf
|
|
401
|
+
JOIN re_export_chain rec ON rf.from_file_id = rec.file_id
|
|
402
|
+
)
|
|
403
|
+
SELECT s.id, s.name, s.kind, s.line, f.path
|
|
404
|
+
FROM symbols s
|
|
405
|
+
JOIN files f ON s.file_id = f.id
|
|
406
|
+
JOIN re_export_chain rec ON s.file_id = rec.file_id
|
|
407
|
+
WHERE s.name = :symbol_name
|
|
408
|
+
AND s.file_id != :start_file_id -- exclude the barrel file itself
|
|
409
|
+
LIMIT 1;
|
|
410
|
+
```
|
|
411
|
+
|
|
412
|
+
---
|
|
413
|
+
|
|
414
|
+
## 7. Index Lifecycle
|
|
415
|
+
|
|
416
|
+
### 7.1 Indexing (Single Phase — Parse → Chunks → Graph → FTS)
|
|
417
|
+
|
|
418
|
+
Indexing is a single phase — there is no separate embedding step. `runIndex`:
|
|
419
|
+
|
|
420
|
+
1. Acquire `structure.lock` — see §7.6. Exit with error if lock cannot be
|
|
421
|
+
acquired within the configured timeout.
|
|
422
|
+
2. Walk project files matching `file_extensions`, respecting `exclude_patterns`.
|
|
423
|
+
Collect `{ path, mtime }` for every file found.
|
|
424
|
+
3. **Deleted file cleanup:** load `file_manifest.json` (previous scan's path set).
|
|
425
|
+
Any path present in the manifest but absent from the current walk has been deleted.
|
|
426
|
+
Remove its rows from `graph.db` `files` table (cascade deletes symbols, edges, and
|
|
427
|
+
imports), and delete all matching rows from the `chunks` table.
|
|
428
|
+
4. For each file to index (all files on full run; only files where
|
|
429
|
+
`mtime > manifest[path]` on incremental run): wrap the parse in try/catch. On
|
|
430
|
+
tree-sitter error, log at `warn` level with the file path and error message,
|
|
431
|
+
increment `parse_errors`, and skip to the next file. Never abort the full run.
|
|
432
|
+
On success: run the two-pass walk and extract chunks with type, symbol, `is_exported`,
|
|
433
|
+
and `declaration_hash` metadata.
|
|
434
|
+
5. Write chunks to the `chunks` table. Delete and replace all existing chunks for the
|
|
435
|
+
same `file_path`. Update `chunk_fts`: `DELETE FROM chunk_fts WHERE chunk_id = ?`
|
|
436
|
+
for removed chunks, `INSERT INTO chunk_fts(content, symbol_name, chunk_id) VALUES
|
|
437
|
+
(?, ?, ?)` for new/changed chunks. No external table or trigger is needed — FTS5
|
|
438
|
+
built-in content handles everything. Multi-row inserts here (and for `symbols`,
|
|
439
|
+
`imports`, `identifier_fts`, and `edges`) are batched under SQLite's 32,766
|
|
440
|
+
bound-parameter ceiling, batch-by-batch inside the same per-file transaction
|
|
441
|
+
(`graph/sqliteBatch.ts`), so file size never caps how many chunks get indexed
|
|
442
|
+
(Stage 4.5 S1 — `adr/011-2026-08-17-indexing-scale.md`).
|
|
443
|
+
6. Populate `graph.db` from AST imports and relationships, wrapped in a single
|
|
444
|
+
transaction per file (delete-and-replace). Record `RE_EXPORTS` edges from
|
|
445
|
+
`export { x }` clauses and `re_export_files` rows from `export * from '...'`
|
|
446
|
+
clauses.
|
|
447
|
+
7. **Stability hash optimisation (incremental only):** the `declaration_hash`
|
|
448
|
+
(signature) and `body_hash` are computed from the AST (signature node vs body
|
|
449
|
+
node), not by splitting chunk text.
|
|
450
|
+
|
|
451
|
+
**File-level skip:** a file whose mtime changed but whose chunked content is
|
|
452
|
+
byte-identical (same chunk-id set AND same per-symbol `declaration_hash`/
|
|
453
|
+
`body_hash` signature, and no `block` chunks) is not re-written at all. A
|
|
454
|
+
true *per-symbol* KG-rebuild skip was rejected: under the per-file
|
|
455
|
+
delete-and-replace model (step 6), symbols are re-inserted with new ids, so
|
|
456
|
+
edges must be rebuilt — preserving them per-symbol would be invasive surgery
|
|
457
|
+
on the hot path for marginal gain.
|
|
458
|
+
|
|
459
|
+
**Class shells use a member-signature hash, not a body hash.** A `class_shell`
|
|
460
|
+
chunk's content is the synthesized outline (declaration + member signatures + TSDoc;
|
|
461
|
+
no method bodies — see §6.1). Its `body_hash` is therefore computed as
|
|
462
|
+
`sha256(sorted(member_signature_text + member_doc_text).join("\n"))`, where each
|
|
463
|
+
`member_signature_text` is the method's signature line stripped of body. This means:
|
|
464
|
+
- Renaming a method → shell content changes → shell `body_hash` changes → shell
|
|
465
|
+
is rewritten. Correct: the outline visible to the agent now lists a new name.
|
|
466
|
+
- Editing a method body without changing its signature → shell `body_hash`
|
|
467
|
+
**unchanged** → shell content is NOT rewritten; only the affected `method` chunk is.
|
|
468
|
+
Correct: the class's public interface didn't change, only its internals.
|
|
469
|
+
- Adding or removing a method → shell content changes → shell is rewritten, plus
|
|
470
|
+
the new/removed `method` chunk is added/deleted.
|
|
471
|
+
|
|
472
|
+
Without this rule, the shell silently drifts out of sync with its members, and
|
|
473
|
+
`mast_search "session validation"` returns a stale outline naming a method that
|
|
474
|
+
no longer exists.
|
|
475
|
+
8. Write `file_manifest.json` with the current `{ path: mtime }` snapshot.
|
|
476
|
+
9. Write/update `index.json` with `last_indexed` and `file_count`.
|
|
477
|
+
|
|
478
|
+
Incremental variant: step 4 skips files where `mtime === manifest[path]`. Steps 3
|
|
479
|
+
(deleted file cleanup) and 8–9 always run.
|
|
480
|
+
|
|
481
|
+
BM25 search is handled by the `chunk_fts` FTS5 virtual table in `graph.db`, populated
|
|
482
|
+
incrementally during indexing (step 5). There is no separate serialization step.
|
|
483
|
+
|
|
484
|
+
### 7.3 Ranked Search with RRF
|
|
485
|
+
|
|
486
|
+
Search combines two rankers — FTS5 BM25 (lexical) and the declaration-exact
|
|
487
|
+
ranker (ranker D, structural) — using **Reciprocal Rank Fusion**:
|
|
488
|
+
|
|
489
|
+
$$Score(d) = \sum_{r \in R} \frac{1}{k + r(d)}$$
|
|
490
|
+
|
|
491
|
+
Where $r(d)$ is the rank of document $d$ in ranker $R$, and $k$ is `rrf_k` (default 60).
|
|
492
|
+
|
|
493
|
+
RRF is used instead of weighted score addition because the two rankers' scores are on
|
|
494
|
+
incomparable scales (a BM25 score and a structural match are not directly comparable).
|
|
495
|
+
Rank position, not the underlying score magnitude, is what RRF fuses.
|
|
496
|
+
|
|
497
|
+
Implementation: run FTS5 BM25 search (and, when enabled, ranker D) independently over a
|
|
498
|
+
candidate pool of **4× `limit` per ranker** (40 for the default `limit: 10`),
|
|
499
|
+
then apply RRF to produce a unified ranked list. Return the top `limit` results.
|
|
500
|
+
|
|
501
|
+
**Second RRF input — the declaration-exact ranker (ranker D, F18).** When
|
|
502
|
+
`declaration_exact_ranker` is enabled (§4.1, default on), a second ranked list
|
|
503
|
+
joins the fusion. Ranker D (`src/search/declex.ts`, the Q1/DECLEX-measured
|
|
504
|
+
construction) is symbol-gated and purely lexical-structural:
|
|
505
|
+
|
|
506
|
+
- The raw query is split on `/[A-Za-z0-9_$]+/` (no camelCase decomposition, no
|
|
507
|
+
lowercasing); only **symbol-shaped** tokens survive the eligibility gate
|
|
508
|
+
(contains an uppercase letter, `_`, `$`, or a digit adjacent to a letter) —
|
|
509
|
+
bare lowercase prose words never reach the ranker.
|
|
510
|
+
- Each eligible token matches chunks whose own `symbol_name` **equals the
|
|
511
|
+
token** (full-name match) or **ends with `.` + token** (final-dot-segment
|
|
512
|
+
match — reaches `Class.method` chunks by their method name), both
|
|
513
|
+
case-insensitive, via a direct SQL predicate against `chunks.symbol_name`
|
|
514
|
+
(not FTS — the rule is a structural string comparison).
|
|
515
|
+
- Ordering is deterministic: full-name matches before segment-only matches,
|
|
516
|
+
then ascending same-matched-name multiplicity (a uniquely-named match beats
|
|
517
|
+
one of 140 `toJSON`s), then ascending `chunk_id`. The pool is capped at
|
|
518
|
+
4× `limit` like BM25, and the list enters RRF **by rank** with the same
|
|
519
|
+
`rrf_k`.
|
|
520
|
+
- Ranker D applies the caller's `file_pattern`/`language` scope to its candidate
|
|
521
|
+
pool **before** ordering and capping, so a scoped search behaves as though the
|
|
522
|
+
index held only the files in scope. Both filters are compiled once per call by
|
|
523
|
+
`search/scope.ts` and applied identically by every ranker RRF fuses — a ranker
|
|
524
|
+
that skipped them would put out-of-scope chunks into a scoped result set, which
|
|
525
|
+
is what D did until 2026-08-20 (`docs/defects/LEDGER.md` D031; this bullet read
|
|
526
|
+
"applies no `file_pattern`/`language` pre-filter (same semantics as BM25)",
|
|
527
|
+
and the parenthetical was wrong about BM25 too — BM25 pre-filtered throughout).
|
|
528
|
+
`chunk_type`/`only_exported` post-filters apply downstream unchanged. When
|
|
529
|
+
`declaration_exact_ranker` is off, `mast_search` is BM25-only.
|
|
530
|
+
- `file_pattern` is a **glob**, compiled by `globToRegex` (`indexer/walker.ts`) —
|
|
531
|
+
the same primitive that matches `exclude_patterns` at index time and backs
|
|
532
|
+
`mast_project_skeleton`'s own `file_pattern`, so one pattern selects one set of
|
|
533
|
+
files everywhere. `*` does not cross `/`, `**` does, `?` is a single non-`/`
|
|
534
|
+
character, matching is case-sensitive, and every other character is literal.
|
|
535
|
+
|
|
536
|
+
Provenance: pre-registered and measured as Q1/DECLEX; shipped per the M2 decision
|
|
537
|
+
memo as F18 (`adr/004-2026-08-06-ranker-d.md`). The measured **escape variant**
|
|
538
|
+
(lowercase-token recovery under a match-count cap) is deliberately NOT shipped —
|
|
539
|
+
it is measured harmful off-stratum and requires a fresh pre-registration.
|
|
540
|
+
Per-call firing telemetry is persisted to `metrics.declex_json` (§14.3).
|
|
541
|
+
|
|
542
|
+
**FTS5 sign convention:** SQLite's `bm25(chunk_fts)` returns negative scores — more
|
|
543
|
+
negative means a better match. When sorting the FTS5 result set, sort ascending
|
|
544
|
+
(most negative first) before applying RRF ranks. Do not negate the scores; rank
|
|
545
|
+
position is all that RRF uses.
|
|
546
|
+
|
|
547
|
+
### 7.4 Startup Reindex (Primary Pipeline Hook)
|
|
548
|
+
|
|
549
|
+
When `mast serve` starts, the goal is **time-to-first-query in single-digit
|
|
550
|
+
seconds**. All 11 tools are registered and ready to serve as soon as Step 3
|
|
551
|
+
completes — there is no reduced-capability window and no discriminator on tool
|
|
552
|
+
responses to track. Cold-start dead time is the single biggest UX risk to MAST
|
|
553
|
+
adoption — see Failure 4 in the design review — so this ladder is structured to
|
|
554
|
+
eliminate it.
|
|
555
|
+
|
|
556
|
+
```
|
|
557
|
+
startup
|
|
558
|
+
├─ STEP 1 (sync, < 1s): bootstrap state directory
|
|
559
|
+
│ ├─ if <state_dir> is missing or empty:
|
|
560
|
+
│ │ └─ if /opt/mast-seed exists (Docker-baked seed, see §13.8):
|
|
561
|
+
│ │ copy /opt/mast-seed → <state_dir>
|
|
562
|
+
│ │ └─ else:
|
|
563
|
+
│ │ run `mast init --no-index` to create config + empty state
|
|
564
|
+
│ ├─ ensure lock markers exist; persist the resolved config
|
|
565
|
+
│ └─ best-effort remove orphaned pre-vector-store state (`lance/`,
|
|
566
|
+
│ `embed_cache/`, `vectors.lock`) left behind by a pre-2026-08-06
|
|
567
|
+
│ install — logged, never fatal, runs on every startup
|
|
568
|
+
│
|
|
569
|
+
├─ STEP 2 (sync, < 2s): schema version + open database
|
|
570
|
+
│ ├─ if index.json.schema_version != CURRENT_SCHEMA_VERSION:
|
|
571
|
+
│ │ wipe all derived state (graph.db, file_manifest.json, and any
|
|
572
|
+
│ │ remaining orphaned state)
|
|
573
|
+
│ │ set needs_full_reindex = true
|
|
574
|
+
│ │ write new index.json with updated schema_version
|
|
575
|
+
│ ├─ open graph.db (better-sqlite3, WAL mode)
|
|
576
|
+
│ └─ verify chunk_fts and identifier_fts tables exist (created on first init)
|
|
577
|
+
│
|
|
578
|
+
├─ STEP 3 (sync, < 1s): open MCP transport — SERVER READY
|
|
579
|
+
│ ├─ register all 11 tools (mast_search, mast_project_skeleton,
|
|
580
|
+
│ │ mast_exports, mast_signature, mast_callers, mast_dependencies,
|
|
581
|
+
│ │ mast_implementors, mast_reindex, mast_status, mast_efficiency,
|
|
582
|
+
│ │ mast_rename_impact)
|
|
583
|
+
│ └─ accept incoming MCP connections
|
|
584
|
+
│
|
|
585
|
+
└─ STEP 4 (async): background incremental reindex
|
|
586
|
+
├─ scan filesystem: collect {path, mtime} for all matched files
|
|
587
|
+
├─ deleted_files = manifest_paths - scanned_paths
|
|
588
|
+
│ └─ for each: delete chunks/symbols (acquire structure.lock briefly)
|
|
589
|
+
├─ stale_files = [f for f in scanned if f.mtime > index.last_indexed
|
|
590
|
+
│ OR needs_full_reindex]
|
|
591
|
+
├─ acquire structure.lock
|
|
592
|
+
│ ├─ run the indexer (§7.1) for stale_files
|
|
593
|
+
│ ├─ update file_manifest.json + index.json.last_indexed
|
|
594
|
+
│ └─ release structure.lock
|
|
595
|
+
└─ FROM THIS POINT: the index is up-to-date for stale files
|
|
596
|
+
(mast_search, mast_callers verified+potential, etc.)
|
|
597
|
+
```
|
|
598
|
+
|
|
599
|
+
If `--watch` was passed to `mast serve`, the file watcher (§11.4) starts
|
|
600
|
+
immediately after Step 3's transport opens, independent of Step 4.
|
|
601
|
+
|
|
602
|
+
**`--no-startup-reindex` refusal (M6 Part A).** The empty-during-Step-4 window
|
|
603
|
+
above is legitimate and by design — but `--no-startup-reindex` disables Step 4
|
|
604
|
+
entirely, and a state dir that has never completed an index run under that
|
|
605
|
+
flag would then answer every query `{"results":[]}` forever, with nothing
|
|
606
|
+
left to ever fill it in (`eval/GITNEXUS_COMPARISON.md` §13.8 item 4). To catch
|
|
607
|
+
only that unrecoverable case, `mast serve` calls `assertServableIndex`
|
|
608
|
+
(`mcp/server.ts`) after Step 1 and before Step 2 opens `graph.db`: if
|
|
609
|
+
`--no-startup-reindex` was passed AND the state dir is never-indexed
|
|
610
|
+
(`graph.db` absent, or `index.json` reports `chunk_count: 0` with
|
|
611
|
+
`last_indexed` null/absent), the process exits with an error naming the state
|
|
612
|
+
dir and suggesting `mast init`/`mast index` or dropping the flag. A state dir
|
|
613
|
+
indexed over a genuinely empty file set (`last_indexed` set, `chunk_count: 0`)
|
|
614
|
+
is NOT refused — see §9.0's "Empty-index signal" for how that legitimate
|
|
615
|
+
empty-index case is surfaced to callers instead. With the startup reindex
|
|
616
|
+
enabled (the default), this check is a no-op and Step 3 opens the transport
|
|
617
|
+
exactly as described above.
|
|
618
|
+
|
|
619
|
+
`CURRENT_SCHEMA_VERSION` is a constant in the mast binary (currently `"1.3.0"`). A
|
|
620
|
+
version bump is required any time the SQLite schema or `index.json` fields change
|
|
621
|
+
in a way that makes old on-disk state unreadable by the new code. Incrementing
|
|
622
|
+
without a state wipe causes a corrupt or partial index; wiping without
|
|
623
|
+
incrementing loses the protection. Both are bugs — treat the version as a
|
|
624
|
+
migration guard, not a display string.
|
|
625
|
+
|
|
626
|
+
(Backward-compatible additions that do not break reading an old table — e.g. the
|
|
627
|
+
`edges.resolution`/`call_line`/`context` columns added via `ALTER TABLE … ADD
|
|
628
|
+
COLUMN` — do NOT require a bump, since `openDatabase` migrates them in place.) On
|
|
629
|
+
schema bump the seed index in `/opt/mast-seed` is also invalidated and a full
|
|
630
|
+
reindex runs in the background.
|
|
631
|
+
|
|
632
|
+
**Fast first-task latency.** With a baked seed (§13.8), Steps 1–3 typically complete
|
|
633
|
+
in **2–4 seconds** on a cold container. Step 4 then catches up any files changed
|
|
634
|
+
since the seed was built in the background — the agent can begin useful work as
|
|
635
|
+
soon as Step 3 completes; JIT staleness handling (§9.0) guarantees any individual
|
|
636
|
+
file it queries is correct even before Step 4 reaches it.
|
|
637
|
+
|
|
638
|
+
This is the **only hook required for the SDD pipeline**. The BT orchestrator needs no
|
|
639
|
+
reindex calls. Files committed by the previous task are picked up by Step 4's
|
|
640
|
+
filesystem scan. JIT staleness handling (§9) covers files modified mid-task before
|
|
641
|
+
Step 4 has caught up to them.
|
|
642
|
+
|
|
643
|
+
### 7.5 Mid-Task Reindex (`mast_reindex` MCP tool)
|
|
644
|
+
|
|
645
|
+
The agent calls `mast_reindex` immediately after writing files, before querying for
|
|
646
|
+
symbols it just created. This is synchronous — the tool does not return until the index
|
|
647
|
+
is updated. Incremental by default — only files with changed mtimes are touched. For a
|
|
648
|
+
typical single-file write this completes in <500ms.
|
|
649
|
+
|
|
650
|
+
### 7.6 Write Locking
|
|
651
|
+
|
|
652
|
+
Coarse writes are coordinated by **one advisory file lock**, managed by `proper-lockfile`:
|
|
653
|
+
|
|
654
|
+
- **`<state_dir>/structure.lock`** — held during chunk parsing, graph population, and
|
|
655
|
+
FTS index writes (`chunks` table, `graph.db`, `chunk_fts`, `identifier_fts`) for
|
|
656
|
+
**coarse writers only**: `mast index`, the startup full/incremental reindex, and
|
|
657
|
+
`mast_reindex`. It also coordinates the manifest/`index.json` phase, which SQLite
|
|
658
|
+
itself can never protect (plain `writeFileSync`, not a database write).
|
|
659
|
+
|
|
660
|
+
**Why `proper-lockfile`:** it writes the acquiring process's PID into the lock file and
|
|
661
|
+
checks liveness on encounter. If a container is killed mid-index and a `.lock` file is
|
|
662
|
+
left on the shared volume, the next process that tries to acquire the lock sees a PID
|
|
663
|
+
that is no longer running and breaks the stale lock automatically. No manual heartbeat
|
|
664
|
+
or timestamp logic is needed. Always pass `{ stale: 10000 }` (10 seconds) when calling
|
|
665
|
+
`lockfile.lock()`; this is the maximum age a lock file is considered valid without a
|
|
666
|
+
live PID, and ensures clean recovery from abrupt container exits on shared volumes.
|
|
667
|
+
|
|
668
|
+
**Behaviour by caller (structure.lock):**
|
|
669
|
+
- **CLI commands** (`mast index`): non-blocking — if the lock cannot be acquired within
|
|
670
|
+
2 seconds, exit with a clear error message naming the operation that holds it.
|
|
671
|
+
- **`mast_reindex` MCP tool**: blocking with retries — attempt acquisition up to 5
|
|
672
|
+
times with 1-second backoff, then return an error to the agent with the suggestion
|
|
673
|
+
to retry after the current index run completes.
|
|
674
|
+
- **`mast serve` startup reindex**: blocking, same retry policy as `mast_reindex`.
|
|
675
|
+
|
|
676
|
+
**If the lock is compromised while held** — another process judged it stale and took it,
|
|
677
|
+
or the lock directory was removed — the operation fails with `LockCompromisedError` and
|
|
678
|
+
must be re-run; its result is not trustworthy, because the run was not exclusive.
|
|
679
|
+
`proper-lockfile` detects this on its refresh timer, inside an `fs.stat` callback with no
|
|
680
|
+
caller on the stack, so `onCompromised` must record rather than rethrow: the library's
|
|
681
|
+
default rethrow reached the process as an uncaught exception and killed it
|
|
682
|
+
(`docs/defects/LEDGER.md` D036). The error is raised at release instead. **Work already in
|
|
683
|
+
flight is not interrupted** — that would require an `AbortSignal` through `withLock` and is
|
|
684
|
+
not implemented. The exposure is bounded: SQLite writes stay serialised by `populateFile`'s
|
|
685
|
+
`BEGIN IMMEDIATE` regardless of this advisory lock, so what is at risk is the plain-JSON
|
|
686
|
+
manifest/`index.json` phase, which the re-run rewrites.
|
|
687
|
+
|
|
688
|
+
**JIT re-parse from a read tool does NOT acquire `structure.lock`.** `structure.lock`
|
|
689
|
+
is one global lock per state dir with no per-file component, so — measured directly
|
|
690
|
+
(`eval/e7-concurrency.json`) — it made a JIT re-parse of file A block a JIT re-parse of
|
|
691
|
+
file B despite the two touching disjoint rows, driving JIT failure rates as high as
|
|
692
|
+
88.5% under pure reader-vs-reader concurrency. Instead, the JIT write goes straight to
|
|
693
|
+
`populateFile` (§9.0), which opens its own transaction with `BEGIN IMMEDIATE` and a
|
|
694
|
+
**dedicated, short `busy_timeout` of 200ms** (`IMMEDIATE_WRITE_BUSY_TIMEOUT_MS`,
|
|
695
|
+
`graph/populate.ts`) — distinct from `graph.db`'s shared 5000ms connection default
|
|
696
|
+
(set once at `openDatabase`, `graph/db.ts`) — set immediately before the transaction and restored immediately after, so no
|
|
697
|
+
other statement on the connection ever inherits the short value. `BEGIN IMMEDIATE`
|
|
698
|
+
takes the write reservation up front instead of discovering contention on commit (the
|
|
699
|
+
`SQLITE_BUSY_SNAPSHOT` failure mode a plain deferred `BEGIN` is prone to), and its
|
|
700
|
+
`busy_timeout` wait — capped at 200ms rather than inheriting the shared 5000ms default
|
|
701
|
+
— is what bounds how long a genuinely contended write can hold up the calling tool. On
|
|
702
|
+
exhaustion (`SQLITE_BUSY`/`SQLITE_BUSY_SNAPSHOT`), fall through to the TOCTOU policy
|
|
703
|
+
below and return the stale chunk with a `file_busy_returning_stale_cache` flag rather
|
|
704
|
+
than blocking the agent indefinitely — the same contract the old lock-retry exhaustion
|
|
705
|
+
path used to produce, just reached by a different mechanism.
|
|
706
|
+
|
|
707
|
+
Coarse writers serialize against each other and against the manifest phase on
|
|
708
|
+
`structure.lock`. JIT writes serialize against each other cheaply within one `mast
|
|
709
|
+
serve` process (Kysely's SQLite dialect guards every connection acquisition on a `Db`
|
|
710
|
+
instance with an in-process mutex) and against coarse writers or other processes via
|
|
711
|
+
`BEGIN IMMEDIATE`'s own write-reservation semantics — not via `structure.lock`.
|
|
712
|
+
Concurrent readers (all MCP query tools) acquire no lock and take no part in either
|
|
713
|
+
mechanism — they only `stat()` files for staleness detection (§9).
|
|
714
|
+
|
|
715
|
+
`store/lockMetrics.ts`'s JSONL sink (`<state_dir>/lock-metrics.jsonl`) is the standing
|
|
716
|
+
instrument for this section's timing claims; `mast metrics --locks` (§14.6) summarizes
|
|
717
|
+
it into per-caller hold/wait percentiles and failed-acquisition counts.
|
|
718
|
+
|
|
719
|
+
---
|
|
720
|
+
|
|
721
|
+
## 8. CLI Interface
|
|
722
|
+
|
|
723
|
+
### `mast init [path] [options]`
|
|
724
|
+
|
|
725
|
+
Initialise mast for a project.
|
|
726
|
+
|
|
727
|
+
```
|
|
728
|
+
Options:
|
|
729
|
+
--state-dir <dir> Where to write index state (default: <path>/.mast)
|
|
730
|
+
--extensions <ext,...> File extensions to index (default: .ts,.tsx,.js,.jsx)
|
|
731
|
+
--exclude <pattern,...> Glob patterns to exclude
|
|
732
|
+
--no-index Create config only, skip initial indexing
|
|
733
|
+
```
|
|
734
|
+
|
|
735
|
+
`--extensions` and `--exclude` are honoured (F9, Stage 3.5): each is a comma-separated
|
|
736
|
+
list, trimmed and with empty entries dropped; `--extensions` additionally normalizes
|
|
737
|
+
bare names to leading-dot form (`py` and `.py` are both accepted). They take priority
|
|
738
|
+
over `mast.config.json` and any previously-persisted `<state_dir>/config.json` — see §4
|
|
739
|
+
for the full priority chain.
|
|
740
|
+
|
|
741
|
+
Creates `<state_dir>/`, writes `config.json`, runs a full index. On every subsequent
|
|
742
|
+
`mast init`/`mast index`/`mast serve`/`mast status`/`mast query`/`mast metrics` call
|
|
743
|
+
against the same state directory, the customisation keys in `config.json` are read back
|
|
744
|
+
and applied unless a higher-priority source (CLI flags, `mast.config.json`) overrides
|
|
745
|
+
them (§4).
|
|
746
|
+
|
|
747
|
+
---
|
|
748
|
+
|
|
749
|
+
### `mast index [path] [options]`
|
|
750
|
+
|
|
751
|
+
Build or update the index.
|
|
752
|
+
|
|
753
|
+
```
|
|
754
|
+
Options:
|
|
755
|
+
--state-dir <dir> State directory (resolved from config if omitted)
|
|
756
|
+
--incremental Only reindex files changed since last index run
|
|
757
|
+
--show-progress Print indexing progress to stderr
|
|
758
|
+
--checker Opt-in TypeScript-checker enrichment pass (§10.3.2) —
|
|
759
|
+
upgrades potential_matches into verified 'checker'
|
|
760
|
+
edges or drops non-call-site/wrong-declaration noise.
|
|
761
|
+
Holds one ts.Program at a time; can take tens of
|
|
762
|
+
seconds on a large monorepo — not part of the default
|
|
763
|
+
index path.
|
|
764
|
+
```
|
|
765
|
+
|
|
766
|
+
---
|
|
767
|
+
|
|
768
|
+
### `mast serve [options]`
|
|
769
|
+
|
|
770
|
+
Start the MCP server over stdio.
|
|
771
|
+
|
|
772
|
+
```
|
|
773
|
+
Options:
|
|
774
|
+
--state-dir <dir> State directory
|
|
775
|
+
--no-startup-reindex Skip the startup staleness check (not recommended)
|
|
776
|
+
--watch Watch source files and incrementally reindex on change
|
|
777
|
+
(interactive use — see §11.4)
|
|
778
|
+
```
|
|
779
|
+
|
|
780
|
+
The server runs until the parent process (Claude CLI) closes stdin.
|
|
781
|
+
|
|
782
|
+
`--no-startup-reindex` combined with a never-indexed state dir is refused at
|
|
783
|
+
startup (M6 Part A, §7.4) — that combination disables the one mechanism that
|
|
784
|
+
would ever fill the index, so `mast serve` exits with an error instead of
|
|
785
|
+
silently answering every query `{"results":[]}` forever. A never-indexed
|
|
786
|
+
state dir with the startup reindex left enabled (the default) is unaffected
|
|
787
|
+
and starts normally, as does `--no-startup-reindex` against an already-indexed
|
|
788
|
+
state dir (including one indexed over a genuinely empty file set).
|
|
789
|
+
|
|
790
|
+
`--watch` is opt-in and intended for interactive local development; the SDD
|
|
791
|
+
container does not use it (§3, §11.4). The watcher is closed on stdin close,
|
|
792
|
+
SIGTERM, and SIGINT; a watcher startup failure logs a warning and the server
|
|
793
|
+
continues without watch.
|
|
794
|
+
|
|
795
|
+
---
|
|
796
|
+
|
|
797
|
+
### `mast status [path] [options]`
|
|
798
|
+
|
|
799
|
+
Print index health.
|
|
800
|
+
|
|
801
|
+
```
|
|
802
|
+
Options:
|
|
803
|
+
--state-dir <dir> State directory
|
|
804
|
+
--json Output as JSON
|
|
805
|
+
```
|
|
806
|
+
|
|
807
|
+
Output:
|
|
808
|
+
```
|
|
809
|
+
state_dir: /workspace/.kluster/.mast
|
|
810
|
+
schema_version: 1.3.0
|
|
811
|
+
last_indexed: 2026-05-13T14:22:00Z (3 minutes ago)
|
|
812
|
+
indexed_files: 142
|
|
813
|
+
chunk_count: 1840
|
|
814
|
+
stale_files: 0
|
|
815
|
+
parse_errors: 0
|
|
816
|
+
write_errors: 0
|
|
817
|
+
index_fresh: true
|
|
818
|
+
freshness_cause: none
|
|
819
|
+
```
|
|
820
|
+
|
|
821
|
+
`freshness_cause` carries the same semantics as the `mast_status` MCP tool (§9) —
|
|
822
|
+
it prints `none` in human output when the JSON value would be `null`; the JSON
|
|
823
|
+
value is `"phase1_stale"` when `stale_files > 0`. On a never-indexed project the
|
|
824
|
+
state directory is not created as a side effect of running `status`.
|
|
825
|
+
|
|
826
|
+
---
|
|
827
|
+
|
|
828
|
+
### `mast query <tool> [json] [path]`
|
|
829
|
+
|
|
830
|
+
Invoke any MCP tool (§9) directly from the CLI — the read tools (`mast_search`,
|
|
831
|
+
`mast_project_skeleton`, `mast_exports`, `mast_signature`, `mast_callers`,
|
|
832
|
+
`mast_dependencies`, `mast_implementors`, `mast_rename_impact`, `mast_status`,
|
|
833
|
+
`mast_efficiency`) and `mast_reindex`, by their exact MCP names.
|
|
834
|
+
|
|
835
|
+
```
|
|
836
|
+
Options:
|
|
837
|
+
--state-dir <dir> State directory
|
|
838
|
+
--json Emit the exact single-line MCP response text (machine
|
|
839
|
+
use); default pretty-prints the parsed response with
|
|
840
|
+
2-space indent for humans
|
|
841
|
+
```
|
|
842
|
+
|
|
843
|
+
`json` (positional, default `'{}'`) is the tool's argument object as a JSON
|
|
844
|
+
string; `path` is the project root (same resolution as every other command's
|
|
845
|
+
`[path]`).
|
|
846
|
+
|
|
847
|
+
Identical-output-by-construction: `mast query` dispatches through the exact
|
|
848
|
+
same registered tool handler an MCP client's call would invoke — the same
|
|
849
|
+
schema validation, the same JIT/staleness handling, the same `_stats`
|
|
850
|
+
block — so CLI output can never drift from the MCP transport's, by
|
|
851
|
+
construction rather than by keeping two implementations in sync.
|
|
852
|
+
|
|
853
|
+
Examples:
|
|
854
|
+
```
|
|
855
|
+
mast query mast_status '{}' /path/to/project
|
|
856
|
+
mast query mast_search '{"query":"add","limit":5}' /path/to/project --json
|
|
857
|
+
mast query mast_exports '{"file_path":"src/math.ts"}'
|
|
858
|
+
```
|
|
859
|
+
|
|
860
|
+
Error behavior (all exit 1, message to stderr):
|
|
861
|
+
- **Unknown tool** — lists every registered tool name.
|
|
862
|
+
- **Malformed JSON argument** — names the parse failure.
|
|
863
|
+
- **Args that fail the tool's own zod schema** — the zod issues.
|
|
864
|
+
- **State dir with no `graph.db`** (never-indexed project) —
|
|
865
|
+
``no index found at <state_dir>; run `mast init` / `mast index` first``.
|
|
866
|
+
This is `mast query`'s own fail-fast guard, parallel to `mast serve`'s
|
|
867
|
+
`--no-startup-reindex` refusal (M6 Part A, §7.4). Because `mast query`
|
|
868
|
+
dispatches through the same registered tool handlers, the M6 Part B
|
|
869
|
+
`index_empty` signal (§9.0) appears on its responses automatically — an
|
|
870
|
+
indexed-but-empty corpus queries fine and says so.
|
|
871
|
+
|
|
872
|
+
---
|
|
873
|
+
|
|
874
|
+
### `mast install-hooks [path]`
|
|
875
|
+
|
|
876
|
+
Write git hooks into `<path>/.git/hooks/`:
|
|
877
|
+
- `post-commit` → `mast index <path> --incremental`
|
|
878
|
+
- `post-checkout` → `mast index <path> --incremental`
|
|
879
|
+
|
|
880
|
+
For developer ergonomics outside the SDD pipeline. Not required for the automated
|
|
881
|
+
pipeline.
|
|
882
|
+
|
|
883
|
+
---
|
|
884
|
+
|
|
885
|
+
## 9. MCP Tools
|
|
886
|
+
|
|
887
|
+
All tools are exposed on the `mast` MCP server. Tool names follow the convention
|
|
888
|
+
`mast_<action>`.
|
|
889
|
+
|
|
890
|
+
---
|
|
891
|
+
|
|
892
|
+
### 9.0 Staleness Handling (All Read Tools)
|
|
893
|
+
|
|
894
|
+
Every read tool that returns line coordinates (`mast_search`, `mast_signature`,
|
|
895
|
+
`mast_exports`, `mast_callers`, `mast_dependencies`, `mast_implementors`,
|
|
896
|
+
`mast_rename_impact`) performs a **mandatory server-side staleness check**
|
|
897
|
+
before returning. This is
|
|
898
|
+
not optional and is not controlled by the caller — the index is responsible for
|
|
899
|
+
its own consistency, not the agent. `mast_project_skeleton` is exempt: its
|
|
900
|
+
response is a directory map of exported symbol names with no line coordinates
|
|
901
|
+
to go stale (P3).
|
|
902
|
+
|
|
903
|
+
**The agent must never see a chunk whose line coordinates do not match the
|
|
904
|
+
current file on disk without being told.** Returning stale line numbers
|
|
905
|
+
unflagged leads directly to agent-assisted corruption: the agent issues an
|
|
906
|
+
`Edit` against the stale range and overwrites unrelated logic. This class of
|
|
907
|
+
failure does not surface as an error — it surfaces as silent, hard-to-attribute
|
|
908
|
+
breakage downstream. Two different mechanisms enforce this, chosen per tool by
|
|
909
|
+
how many files a single call's results can span:
|
|
910
|
+
|
|
911
|
+
- **Just-In-Time (JIT) re-parse** (`mast_signature`, `mast_exports`,
|
|
912
|
+
`mast_callers`, `mast_dependencies`, `mast_rename_impact`) — these tools'
|
|
913
|
+
results are scoped to one file, or a small, explicitly-named set, so a
|
|
914
|
+
stale result can be transparently refreshed in place. See below.
|
|
915
|
+
- **Stat-and-flag** (`mast_search`, `mast_implementors`) — these tools can
|
|
916
|
+
return results spanning dozens of files in one call, so JIT re-parsing
|
|
917
|
+
every result file would mean up to ~50 tree-sitter re-parses and write
|
|
918
|
+
transactions per call, and re-parsing a result file mid-response could
|
|
919
|
+
shift its rank, gain or lose a match, or change its chunk boundaries —
|
|
920
|
+
invalidating the ranking/query that already selected the result being
|
|
921
|
+
"refreshed". Instead,
|
|
922
|
+
after results are computed, each **unique** result `file_path` is
|
|
923
|
+
`statSync`'d (no lock, no re-parse, no DB write) and its disk mtime
|
|
924
|
+
compared against the indexed `files.mtime`. Newer-on-disk, or a failed
|
|
925
|
+
stat (file deleted/renamed since indexing — its coordinates are
|
|
926
|
+
definitely untrustworthy), sets `stale: true` on that result; a file
|
|
927
|
+
absent from the `files` table (nothing indexed to be stale against) is
|
|
928
|
+
left unflagged. **C1** split this signal into its own `stale` field,
|
|
929
|
+
distinct from the JIT-refresh tools' `file_busy_returning_stale_cache`
|
|
930
|
+
below — no lock is ever taken here, so nothing is ever "busy" in that
|
|
931
|
+
sense; see the "Confidence signals (C1)" table below for the full
|
|
932
|
+
picture.
|
|
933
|
+
|
|
934
|
+
**Just-In-Time (JIT) re-parse.** For every result a JIT-refresh tool is about to return:
|
|
935
|
+
|
|
936
|
+
1. `fs.stat()` the `file_path`. Compare disk `mtime` against the chunk's stored
|
|
937
|
+
`file_mtime`.
|
|
938
|
+
2. If `disk_mtime <= stored_mtime` → return the result unchanged. Fast path.
|
|
939
|
+
3. If `disk_mtime > stored_mtime` → the chunk is stale. Re-index **this file
|
|
940
|
+
only** (one tree-sitter parse, one `BEGIN IMMEDIATE` transactional
|
|
941
|
+
delete-and-replace against the `chunks` table, `graph.db`, `chunk_fts`,
|
|
942
|
+
`identifier_fts` — see §7.6; no `structure.lock` acquisition on this path).
|
|
943
|
+
Re-resolve the tool's result against the refreshed chunks. A single-file
|
|
944
|
+
re-parse typically completes in 10–50ms; the transactional write itself is
|
|
945
|
+
bounded by a dedicated 200ms `busy_timeout` (§7.6), not the connection's
|
|
946
|
+
shared 5000ms default a genuinely contended write would otherwise wait
|
|
947
|
+
out.
|
|
948
|
+
|
|
949
|
+
JIT re-parse covers files already known to the index. It does not discover a
|
|
950
|
+
brand-new file or a newly-created symbol — those become searchable via the next
|
|
951
|
+
`mast_reindex` call or the background/`--watch` reindex (§7.4/§11.4) reaching
|
|
952
|
+
them. The agent prompt should still recommend `mast_reindex` after writing new
|
|
953
|
+
files or symbols — not because JIT leaves existing files stale (it doesn't), but
|
|
954
|
+
because discovery of new ones requires an actual indexing pass.
|
|
955
|
+
|
|
956
|
+
**TOCTOU Policy (file mid-write).** Between `stat()` and `parse()`, the file
|
|
957
|
+
may be in the middle of being written by another process (e.g., the agent's
|
|
958
|
+
own `Edit` tool, an editor save, a `git checkout`). Tree-sitter on partial
|
|
959
|
+
source either throws or produces a truncated tree. The policy:
|
|
960
|
+
|
|
961
|
+
1. On parse error or empty tree, sleep **50ms** and retry **once**.
|
|
962
|
+
2. If the second attempt also fails, return the **previous (stale) chunk** with
|
|
963
|
+
a `file_busy_returning_stale_cache: true` flag on the result. Do not throw
|
|
964
|
+
— the agent has no recovery for a thrown error, but it can interpret the
|
|
965
|
+
flag.
|
|
966
|
+
3. If the file no longer exists (deleted), remove its rows from the graph and
|
|
967
|
+
chunk stores and return whatever results remain (possibly empty).
|
|
968
|
+
|
|
969
|
+
**`identifier_fts` and `chunk_fts` updates.** The JIT re-parse rewrites the
|
|
970
|
+
FTS5 rows for the affected file in the same transaction as the chunk/graph
|
|
971
|
+
rewrite. There is no separate sync step.
|
|
972
|
+
|
|
973
|
+
**Concurrency.** Two simultaneous read tools targeting different stale files
|
|
974
|
+
no longer serialize on any lock (§7.6) — each parses its own file fully in
|
|
975
|
+
parallel, and only briefly contends on the transactional write. That write
|
|
976
|
+
is bounded by the dedicated 200ms `busy_timeout`
|
|
977
|
+
(`IMMEDIATE_WRITE_BUSY_TIMEOUT_MS`, §7.6): under real contention, `BEGIN
|
|
978
|
+
IMMEDIATE`'s `busy_timeout` wait is a **synchronous** hold on the whole
|
|
979
|
+
`mast serve` process's event loop (better-sqlite3's busy-wait is native and
|
|
980
|
+
blocks the process, not just the calling request) — 200ms is the accepted
|
|
981
|
+
trade at this magnitude, comparable to the 3×100ms lock-retry budget the
|
|
982
|
+
pre-F11 JIT path used to pay, and far below the connection's shared 5000ms
|
|
983
|
+
default, which would otherwise freeze the entire process for up to 5 seconds
|
|
984
|
+
per contended write.
|
|
985
|
+
|
|
986
|
+
**Result shape.** Every JIT-refresh read tool's result objects MAY include
|
|
987
|
+
`file_busy_returning_stale_cache: true` (omitted when false). Result schemas
|
|
988
|
+
in the per-tool sections below document only the steady-state shape; this flag
|
|
989
|
+
is implicit on all of them. Tools whose response is a single-file envelope
|
|
990
|
+
(`mast_exports`, `mast_dependencies`) or whose staleness taints the whole
|
|
991
|
+
answer (`mast_callers`, `mast_rename_impact`) carry the flag at the envelope
|
|
992
|
+
level instead of per-entry. `mast_signature` carries it per-result — except
|
|
993
|
+
when a `file_path`-narrowed query returns **zero** results while that file's
|
|
994
|
+
JIT re-parse could not acquire the lock: with no result objects to carry the
|
|
995
|
+
signal, the flag appears on the response envelope (F14), so "no results" from
|
|
996
|
+
a stale, un-refreshable file never reads as "symbol doesn't exist".
|
|
997
|
+
`mast_search` and `mast_implementors` carry a **different** field, `stale`,
|
|
998
|
+
per-result — via stat-and-flag rather than JIT re-parse (F7, see above; split
|
|
999
|
+
into its own field by C1, see the table below) — each result's `stale`
|
|
1000
|
+
reflects that result's own `file_path` statting newer-on-disk or failing to
|
|
1001
|
+
stat, independent of every other result in the same response.
|
|
1002
|
+
|
|
1003
|
+
**Empty-index signal (M6 Part B).** Every read tool with a primary result
|
|
1004
|
+
array — `mast_search`, `mast_project_skeleton`, `mast_exports`,
|
|
1005
|
+
`mast_signature`, `mast_callers`, `mast_dependencies`, `mast_implementors`,
|
|
1006
|
+
`mast_rename_impact` — MAY attach `index_empty: true` to its response
|
|
1007
|
+
envelope. Present only when BOTH (a) that tool's primary result set came back
|
|
1008
|
+
empty (for `mast_callers`, both `verified_callers` AND `potential_matches`;
|
|
1009
|
+
for `mast_rename_impact`, all four of `declaration_sites`, `verified_callers`,
|
|
1010
|
+
`potential_matches`, and `barrel_exports`) AND (b) the `chunks` table has zero
|
|
1011
|
+
rows at that moment (`mcp/tools/_helpers.ts`'s `isIndexEmpty`, checked only on
|
|
1012
|
+
the already-empty-result path — a populated response never pays for this
|
|
1013
|
+
check). Distinguishes "`[]` because nothing is indexed yet, or you pointed at
|
|
1014
|
+
the wrong state dir" from "`[]` because no match" — the exact ambiguity M6
|
|
1015
|
+
(`eval/GITNEXUS_COMPARISON.md` §13.8 item 4) names, and the legitimate empty
|
|
1016
|
+
window §7.4's startup ladder deliberately leaves servable (see the
|
|
1017
|
+
`--no-startup-reindex` refusal note in §7.4, which catches only the
|
|
1018
|
+
never-recoverable case, not this one). Omitted entirely when false, same
|
|
1019
|
+
present-only-when-true convention as `file_busy_returning_stale_cache` above
|
|
1020
|
+
— never present-and-false. Independent of `mast_search`'s `suggestions` field:
|
|
1021
|
+
a truly empty index yields no suggestions either, but the two are not
|
|
1022
|
+
coupled — either may be present without the other. `mast_status` is
|
|
1023
|
+
unaffected (it already reports `chunk_count`/`index_fresh` directly — it IS
|
|
1024
|
+
the diagnostic surface); `mast_efficiency` and `mast_reindex` have no primary
|
|
1025
|
+
result array and never carry this flag.
|
|
1026
|
+
|
|
1027
|
+
#### Confidence signals (C1)
|
|
1028
|
+
|
|
1029
|
+
MAST does not compute a single scalar "confidence score" for a result — it
|
|
1030
|
+
never has, and C1 does not introduce one (`eval/GITNEXUS_COMPARISON.md`
|
|
1031
|
+
§13.8 item 5 / §14.8 item 5: "frame as unification, not a new feature").
|
|
1032
|
+
What existed before C1 was a set of independently-evolved signals, computed
|
|
1033
|
+
in different tools for different reasons, some sharing one misleading field
|
|
1034
|
+
name. C1's only change is documentation and one field split — no new enum,
|
|
1035
|
+
no wrapper object, no field beyond the rename (see the F7 result's
|
|
1036
|
+
"Known naming tension" note above, and `adr/008-2026-08-09-honest-surfaces.md`;
|
|
1037
|
+
the C1 result block is `adr/proposals/honest-surfaces/PLAN-EXCERPT.md` § "C1 result").
|
|
1038
|
+
This table is the single place that lists every signal an agent may see
|
|
1039
|
+
across all MCP tools, and what to do with each one:
|
|
1040
|
+
|
|
1041
|
+
| Field | Carried by | Meaning | Agent action |
|
|
1042
|
+
|---|---|---|---|
|
|
1043
|
+
| `resolution` | `VerifiedCaller` entries (`mast_callers`, `mast_rename_impact`) | How this call site was statically resolved to the queried declaration — one of eight values (`import`, `field_type`, `parameter_type`, `new_expression`, `same_file`, `checker`, `this_method`, `super_method` — the last two added by F4, Stage 3, for `this.foo()`/`super.foo()` call sites). | High confidence. Safe to act on directly (e.g. as a rename/refactor site) without further verification. |
|
|
1044
|
+
| `reason` | `PotentialMatch` entries (`mast_callers`, `mast_rename_impact`) | Why this call site could **not** be statically resolved — currently always `identifier_match_no_resolved_edge`. | Mandatory review. This is a name-match, not a verified edge; confirm it is a real call site before acting on it. |
|
|
1045
|
+
| `file_busy_returning_stale_cache` | JIT-refresh tools' results/envelopes (`mast_signature`, `mast_exports`, `mast_callers`, `mast_dependencies`, `mast_rename_impact`) | A refresh **was attempted** (this file's JIT re-parse) and lost to genuine write contention (`populateFile`'s `BEGIN IMMEDIATE` exhausted its `busy_timeout`), so the previous, possibly-stale chunk was returned instead. | Contended, not wrong-by-design. Retry shortly — the contention is expected to clear (§7.6). |
|
|
1046
|
+
| `stale` | `mast_search` / `mast_implementors` per-result (F7) | This result's `file_path` stat'd newer-on-disk than its indexed mtime, or the stat failed — **no refresh was attempted by design** (stat-and-flag, not JIT re-parse; see above). | Treat this result's line coordinates as untrustworthy. A `mast_reindex` call, or any JIT-refreshing tool call against the file, heals it. |
|
|
1047
|
+
| `index_empty` | Every primary-result read tool's envelope (M6) | Nothing is indexed at all — the empty result set is not "no match", it is "no index (yet)". | Run `mast init`/`mast index`, or — if a startup reindex is in progress — wait and retry. |
|
|
1048
|
+
| `truncated` | `TypeContextEntry` (`mast_signature`'s `type_context`) | This referenced type's declaration was clipped at the 50-line cap. | Re-read the file directly (or call `mast_exports`/a narrower `mast_signature` query) for the full declaration if the clipped portion matters. |
|
|
1049
|
+
| `potential_truncated` | `CallersResponse.summary` / `RenameImpactResponse.summary` (`mast_callers`, `mast_rename_impact`) | The `identifier_fts` fetch behind `potential_matches` is capped at 50 entries; this carries the real, uncapped match count when the cap is hit (F10, Stage 3). Reports RAW fetch truncation only — `potential_matches` may still be smaller than the cap even when this field is present, because verified-overlap exclusion and checker-verdict filtering run AFTER the capped fetch (already visible via `checker_classified_*`). | The potential set is incomplete — narrow the query, or run `mast index --checker` to classify candidates away. |
|
|
1050
|
+
|
|
1051
|
+
`file_busy_returning_stale_cache`, `stale`, `index_empty`, and
|
|
1052
|
+
`potential_truncated` all follow the same **omitted-when-false /
|
|
1053
|
+
present-only-when-true** convention (never present-and-false) established
|
|
1054
|
+
above — `potential_truncated`'s "false" case is "the fetch came back under
|
|
1055
|
+
the cap," where the fetch count already IS the real count. `resolution` and
|
|
1056
|
+
`reason` are always
|
|
1057
|
+
present on their respective entry types (a `VerifiedCaller`/`PotentialMatch`
|
|
1058
|
+
without one would be meaningless). `truncated` is the one exception: it is
|
|
1059
|
+
an always-present `boolean` on every `TypeContextEntry`, not an optional
|
|
1060
|
+
flag — callers check its value rather than its presence.
|
|
1061
|
+
|
|
1062
|
+
---
|
|
1063
|
+
|
|
1064
|
+
### `mast_search`
|
|
1065
|
+
|
|
1066
|
+
Lexical BM25 + declaration-exact search via RRF (§7.3). Returns chunks, not full files.
|
|
1067
|
+
|
|
1068
|
+
**Input:**
|
|
1069
|
+
```json
|
|
1070
|
+
{
|
|
1071
|
+
"query": "string",
|
|
1072
|
+
"limit": 10,
|
|
1073
|
+
"language": "typescript | javascript | markdown | null",
|
|
1074
|
+
"file_pattern": "glob pattern | null",
|
|
1075
|
+
"chunk_type": "function | method | class_shell | interface | type | export | block | doc | null",
|
|
1076
|
+
"only_exported": false
|
|
1077
|
+
}
|
|
1078
|
+
```
|
|
1079
|
+
|
|
1080
|
+
`only_exported: true` restricts results to chunks where `is_exported = true`. Use
|
|
1081
|
+
this when looking for a service or utility to call into — it eliminates internal
|
|
1082
|
+
implementation details from results.
|
|
1083
|
+
|
|
1084
|
+
**Output:** `SearchResponse`
|
|
1085
|
+
```json
|
|
1086
|
+
{
|
|
1087
|
+
"results": [
|
|
1088
|
+
{
|
|
1089
|
+
"file_path": "api/services/auth/src/handler.ts",
|
|
1090
|
+
"start_line": 42,
|
|
1091
|
+
"end_line": 58,
|
|
1092
|
+
"content": "export async function handleLogin(...) {",
|
|
1093
|
+
"chunk_type": "function",
|
|
1094
|
+
"symbol_name": "handleLogin",
|
|
1095
|
+
"parent_symbol": null,
|
|
1096
|
+
"is_exported": true,
|
|
1097
|
+
"match_score": -4.21,
|
|
1098
|
+
"rank": 1,
|
|
1099
|
+
"match_snippet": "...async function [1mhandleLogin[0m(req: [1mLoginRequest[0m..."
|
|
1100
|
+
}
|
|
1101
|
+
]
|
|
1102
|
+
}
|
|
1103
|
+
```
|
|
1104
|
+
|
|
1105
|
+
`match_snippet` is produced by SQLite's `snippet(chunk_fts, 0, '**', '**', '...', 12)`
|
|
1106
|
+
function. It returns a short fragment of the chunk content with matched terms marked,
|
|
1107
|
+
so the agent can see immediately why the result was returned without reading the full
|
|
1108
|
+
`content` field. The fragment length (12 tokens) is configurable.
|
|
1109
|
+
|
|
1110
|
+
`match_score` carries the BM25 score (negative — §7.3's FTS5 sign convention) when the
|
|
1111
|
+
FTS ranker produced a hit for this chunk, and `null` when the chunk reached the result
|
|
1112
|
+
set only through ranker D (declaration-exact, §7.3). `rank` is the chunk's position in
|
|
1113
|
+
the RRF-fused list and is always present, starting at 1.
|
|
1114
|
+
|
|
1115
|
+
`parent_symbol` is populated only on `method` chunks (carries the enclosing
|
|
1116
|
+
class name); `null` for all other chunk types.
|
|
1117
|
+
|
|
1118
|
+
**Zero-result assist (`suggestions`).** When a search returns no results — no
|
|
1119
|
+
FTS or ranker-D hit at all, or the `chunk_type` / `only_exported` filters emptied
|
|
1120
|
+
the set — the tool does not return a bare dead end. It runs a relaxation pass
|
|
1121
|
+
and attaches a `suggestions` array of `{ symbol, file_path, reason }` "did you
|
|
1122
|
+
mean" candidates.
|
|
1123
|
+
|
|
1124
|
+
```json
|
|
1125
|
+
{
|
|
1126
|
+
"results": [],
|
|
1127
|
+
"suggestions": [
|
|
1128
|
+
{ "symbol": "handleLogin", "file_path": "api/services/auth/src/handler.ts", "reason": "similar symbol name" },
|
|
1129
|
+
{ "symbol": "handleLogout", "file_path": "api/services/auth/src/handler.ts", "reason": "matched split query terms" }
|
|
1130
|
+
]
|
|
1131
|
+
}
|
|
1132
|
+
```
|
|
1133
|
+
|
|
1134
|
+
Candidates are gathered from three complementary passes, de-duplicated by
|
|
1135
|
+
`(symbol, file_path)` and capped at `limit`:
|
|
1136
|
+
|
|
1137
|
+
- **Trigram symbol-name similarity** against the `symbols` table (Dice
|
|
1138
|
+
coefficient over character trigrams; SQLite ships no `pg_trgm`, so the ranking
|
|
1139
|
+
is computed in-process). `reason: "similar symbol name"`.
|
|
1140
|
+
- **FTS retry over split query terms** — the query is split on camelCase,
|
|
1141
|
+
acronym, and snake/kebab boundaries (`getUserById` → `get`, `user`), then
|
|
1142
|
+
re-run against `chunk_fts`. `reason: "matched split query terms"`.
|
|
1143
|
+
- **`identifier_fts` near-miss** — the same split terms are OR-matched against
|
|
1144
|
+
the identifier index. `reason: "identifier near-miss"`.
|
|
1145
|
+
|
|
1146
|
+
**Trigger and contract.** `suggestions` is present (possibly empty) **only when
|
|
1147
|
+
`results` is empty**, and is omitted from the response entirely when results
|
|
1148
|
+
were found. Suggestions are advisory: they are **never** promoted into
|
|
1149
|
+
`results`, so `results` stays `[]` on the assist path. The agent should treat
|
|
1150
|
+
them as vocabulary hints to re-query with, not as answers.
|
|
1151
|
+
|
|
1152
|
+
**Shell/method dedup (`related`).** Class decomposition (§10.1) means one query
|
|
1153
|
+
can match both a `method` chunk and its parent `class_shell`, whose synthesized
|
|
1154
|
+
outline repeats the same signature — returning both charges the agent twice for
|
|
1155
|
+
one fact. A post-RRF presentation pass therefore keeps only the higher-ranked
|
|
1156
|
+
half of every shell↔method pair and attaches a `related` hint to the survivor:
|
|
1157
|
+
|
|
1158
|
+
- surviving `method` (shell suppressed): `"related": { "parent_symbol": "AuthService" }`
|
|
1159
|
+
— the class outline also matched; query the class name for the full picture.
|
|
1160
|
+
When several methods of the class survive, the hint lands on the
|
|
1161
|
+
highest-ranked one only.
|
|
1162
|
+
- surviving `class_shell` (methods suppressed): `"related": { "methods_matched":
|
|
1163
|
+
["AuthService.validateSession", "AuthService.refresh"] }` — these specific
|
|
1164
|
+
members also matched; read them next instead of re-searching.
|
|
1165
|
+
|
|
1166
|
+
Rules: shell and method are paired by `parent_symbol` + the same `file_path`
|
|
1167
|
+
(same-named classes in different files never collapse); methods never suppress
|
|
1168
|
+
each other; ranking math is untouched — suppression frees slots that are
|
|
1169
|
+
backfilled from the RRF candidate tail, so the response still returns up to
|
|
1170
|
+
`limit` distinct results, and `rank` values are re-assigned after dedup so they
|
|
1171
|
+
remain contiguous from 1. `related` is absent when no collision occurred.
|
|
1172
|
+
|
|
1173
|
+
**When used:** primary code discovery — replaces `Grep`, `Glob`, and exploratory `Read`.
|
|
1174
|
+
|
|
1175
|
+
---
|
|
1176
|
+
|
|
1177
|
+
### `mast_project_skeleton`
|
|
1178
|
+
|
|
1179
|
+
All file paths in the project with their exported top-level symbol names. Returns
|
|
1180
|
+
names only — no signatures, no bodies. Designed to give the agent a compressed
|
|
1181
|
+
architectural overview in a single call.
|
|
1182
|
+
|
|
1183
|
+
**Input:**
|
|
1184
|
+
```json
|
|
1185
|
+
{
|
|
1186
|
+
"directory": "api/services/auth | null",
|
|
1187
|
+
"max_depth": 2,
|
|
1188
|
+
"file_pattern": "glob pattern | null"
|
|
1189
|
+
}
|
|
1190
|
+
```
|
|
1191
|
+
|
|
1192
|
+
`directory` scopes the skeleton to a subtree of the project — essential for large
|
|
1193
|
+
monorepos where a full skeleton would be noisy. `max_depth` limits directory traversal
|
|
1194
|
+
depth relative to `directory` (or `project_root` if omitted). Both are optional;
|
|
1195
|
+
omitting both returns the full project skeleton.
|
|
1196
|
+
|
|
1197
|
+
**Output:** array of `FileSkeleton`
|
|
1198
|
+
```json
|
|
1199
|
+
[
|
|
1200
|
+
{
|
|
1201
|
+
"file_path": "api/services/auth/src/index.ts",
|
|
1202
|
+
"exports": ["handleLogin", "handleLogout", "AuthPlugin"]
|
|
1203
|
+
},
|
|
1204
|
+
{
|
|
1205
|
+
"file_path": "api/services/auth/src/repository.ts",
|
|
1206
|
+
"exports": ["AuthRepository"]
|
|
1207
|
+
}
|
|
1208
|
+
]
|
|
1209
|
+
```
|
|
1210
|
+
|
|
1211
|
+
Sourced entirely from the `chunks` table where `is_exported = true` — no tree-sitter
|
|
1212
|
+
reparsing at query time.
|
|
1213
|
+
|
|
1214
|
+
**When used:** early in a task for orientation — "what services exist and what do they
|
|
1215
|
+
expose?" before deciding which files to query further. Replaces opening multiple
|
|
1216
|
+
`index.ts` barrel files. Scope with `directory` when working within a known service
|
|
1217
|
+
subtree to avoid noise from unrelated packages.
|
|
1218
|
+
|
|
1219
|
+
---
|
|
1220
|
+
|
|
1221
|
+
### `mast_exports`
|
|
1222
|
+
|
|
1223
|
+
All exported symbols from a single file with type signatures. No function bodies.
|
|
1224
|
+
|
|
1225
|
+
**Input:**
|
|
1226
|
+
```json
|
|
1227
|
+
{
|
|
1228
|
+
"file_path": "api/services/auth/src/index.ts"
|
|
1229
|
+
}
|
|
1230
|
+
```
|
|
1231
|
+
|
|
1232
|
+
**Output:** array of `Export`
|
|
1233
|
+
```json
|
|
1234
|
+
[
|
|
1235
|
+
{
|
|
1236
|
+
"name": "handleLogin",
|
|
1237
|
+
"kind": "function",
|
|
1238
|
+
"signature": "export async function handleLogin(req: LoginRequest, reply: FastifyReply): Promise<void>",
|
|
1239
|
+
"line": 42,
|
|
1240
|
+
"doc": "Validates credentials and issues a session token. Throws UnauthorizedError on failure."
|
|
1241
|
+
},
|
|
1242
|
+
{
|
|
1243
|
+
"name": "AuthPlugin",
|
|
1244
|
+
"kind": "class",
|
|
1245
|
+
"signature": "export class AuthPlugin implements FastifyPlugin",
|
|
1246
|
+
"line": 12,
|
|
1247
|
+
"doc": null
|
|
1248
|
+
}
|
|
1249
|
+
]
|
|
1250
|
+
```
|
|
1251
|
+
|
|
1252
|
+
**When used:** "What does this file expose?" before deciding whether to open it.
|
|
1253
|
+
Replaces a full-file `Read` when the agent only needs the public API surface.
|
|
1254
|
+
|
|
1255
|
+
**Implementation:** tree-sitter AST walk — extract `export_statement` nodes and
|
|
1256
|
+
`export_clause` nodes. For each exported declaration, extract the declaration node
|
|
1257
|
+
up to (not including) the body `{ ... }` block. Extract leading TSDoc/JSDoc comment.
|
|
1258
|
+
|
|
1259
|
+
---
|
|
1260
|
+
|
|
1261
|
+
### `mast_signature`
|
|
1262
|
+
|
|
1263
|
+
Declaration, TSDoc, and resolved parameter type context for a named symbol.
|
|
1264
|
+
|
|
1265
|
+
**Input:**
|
|
1266
|
+
```json
|
|
1267
|
+
{
|
|
1268
|
+
"symbol": "handleLogin",
|
|
1269
|
+
"file_path": "api/services/auth/src/handler.ts"
|
|
1270
|
+
}
|
|
1271
|
+
```
|
|
1272
|
+
|
|
1273
|
+
`file_path` is optional. When omitted, all matching symbols across the codebase are
|
|
1274
|
+
returned. If multiple matches are found and the caller only expects one, pass `file_path`
|
|
1275
|
+
to disambiguate. There is no "first match" shortcut — an ambiguous query always returns
|
|
1276
|
+
the full match set so the agent can choose.
|
|
1277
|
+
|
|
1278
|
+
**Output:** `SignatureResult[]` — always an array, even for a single match.
|
|
1279
|
+
```json
|
|
1280
|
+
[
|
|
1281
|
+
{
|
|
1282
|
+
"symbol": "handleLogin",
|
|
1283
|
+
"file_path": "api/services/auth/src/handler.ts",
|
|
1284
|
+
"line": 42,
|
|
1285
|
+
"signature": "export async function handleLogin(req: LoginRequest, reply: FastifyReply): Promise<void>",
|
|
1286
|
+
"doc": "Validates credentials and issues a session token. Throws UnauthorizedError on failure.",
|
|
1287
|
+
"params": [
|
|
1288
|
+
{ "name": "req", "type": "LoginRequest" },
|
|
1289
|
+
{ "name": "reply", "type": "FastifyReply" }
|
|
1290
|
+
],
|
|
1291
|
+
"return_type": "Promise<void>",
|
|
1292
|
+
"type_context": [
|
|
1293
|
+
{
|
|
1294
|
+
"name": "LoginRequest",
|
|
1295
|
+
"signature": "interface LoginRequest { email: string; password: string; }",
|
|
1296
|
+
"file_path": "api/services/auth/src/types.ts",
|
|
1297
|
+
"line": 3,
|
|
1298
|
+
"truncated": false
|
|
1299
|
+
}
|
|
1300
|
+
]
|
|
1301
|
+
}
|
|
1302
|
+
]
|
|
1303
|
+
```
|
|
1304
|
+
|
|
1305
|
+
When `file_path` is omitted and multiple files define a symbol with the same name,
|
|
1306
|
+
`type_context` is still resolved per entry using that entry's containing file as the
|
|
1307
|
+
resolution root. Entries are ordered by `is_exported` descending (exported symbols
|
|
1308
|
+
first), then by `file_path` ascending.
|
|
1309
|
+
|
|
1310
|
+
**Transitive export resolution:** when looking up a symbol by name, the `symbols`
|
|
1311
|
+
table is queried first. If the symbol is not found directly but a `RE_EXPORTS` edge
|
|
1312
|
+
or `re_export_files` row exists pointing from the queried file to another file, the
|
|
1313
|
+
lookup recurses through that chain until the source declaration is found. This handles
|
|
1314
|
+
barrel file patterns where `packages/shared/src/index.ts` re-exports everything from
|
|
1315
|
+
internal modules three levels deep. The recursive CTE for this lookup mirrors the
|
|
1316
|
+
callers pattern already in §6.3.
|
|
1317
|
+
|
|
1318
|
+
**`type_context` resolution rules:**
|
|
1319
|
+
1. Extract all named types from `params` and `return_type`.
|
|
1320
|
+
2. For each type name, resolve the declaration using this priority order:
|
|
1321
|
+
a. **Same file first:** query `symbols` where `file_id = <containing file's id>` and
|
|
1322
|
+
`name = <type name>`. This handles types defined alongside the function.
|
|
1323
|
+
b. **Imports:** query `imports` for the containing file; find the row where `symbols`
|
|
1324
|
+
contains the type name. Resolve `resolved_path` using the path resolver (see
|
|
1325
|
+
§13.7) for tsconfig aliases and workspace packages.
|
|
1326
|
+
c. **RE_EXPORTS chain:** if not found via imports, walk the `re_export_files` chain
|
|
1327
|
+
recursively (same CTE as §6.3 barrel resolution) to find a file that defines the
|
|
1328
|
+
type. This handles barrel files that re-export types without explicit `import`.
|
|
1329
|
+
d. **Not found:** treat as external; omit from `type_context` (do not error).
|
|
1330
|
+
3. If found in the monorepo: extract the declaration from `graph.db` `symbols` table —
|
|
1331
|
+
signature only, no body. Do not re-parse the file with tree-sitter.
|
|
1332
|
+
4. Include as a `type_context` entry.
|
|
1333
|
+
5. **One level deep only.** Do not recurse into the fields of resolved types.
|
|
1334
|
+
6. **Monorepo types only.** Types from `node_modules` (external) are not resolved.
|
|
1335
|
+
7. **Simple type references only.** Generic parameters (`Promise<T>`) are resolved
|
|
1336
|
+
for their type argument (`T`) if it is a named monorepo type. Union members
|
|
1337
|
+
(`string | LoginRequest`) are resolved individually.
|
|
1338
|
+
8. **Line limit:** if a resolved type declaration exceeds 50 lines, truncate to the
|
|
1339
|
+
first 50 lines and set `truncated: true` on the entry. This prevents a large
|
|
1340
|
+
generated interface from bloating the response.
|
|
1341
|
+
|
|
1342
|
+
This saves the agent a predictable second tool call when it needs to understand a
|
|
1343
|
+
function's contract in full.
|
|
1344
|
+
|
|
1345
|
+
**When used:** "What does this function accept and return, and what shape are those
|
|
1346
|
+
types?" Replaces opening the file plus following imports to type definitions.
|
|
1347
|
+
|
|
1348
|
+
---
|
|
1349
|
+
|
|
1350
|
+
### `mast_callers`
|
|
1351
|
+
|
|
1352
|
+
Call sites of a named symbol, partitioned into **verified** and **potential** sets.
|
|
1353
|
+
The split is fundamental to the tool's contract — see §10.3 for why.
|
|
1354
|
+
|
|
1355
|
+
**Input:**
|
|
1356
|
+
```json
|
|
1357
|
+
{
|
|
1358
|
+
"symbol": "handleLogin",
|
|
1359
|
+
"file_path": "api/services/auth/src/handler.ts",
|
|
1360
|
+
"transitive": false,
|
|
1361
|
+
"include_potential": true
|
|
1362
|
+
}
|
|
1363
|
+
```
|
|
1364
|
+
|
|
1365
|
+
- `transitive: true` returns callers of callers via recursive CTE on `POTENTIAL_CALL`
|
|
1366
|
+
edges (the verified set only — the recursion does not traverse `potential_matches`).
|
|
1367
|
+
Default is direct callers only.
|
|
1368
|
+
- `include_potential: false` skips the `identifier_fts` query entirely. Useful when
|
|
1369
|
+
the agent wants only edges the resolver could statically link, accepting that the
|
|
1370
|
+
result is incomplete. Default is `true`.
|
|
1371
|
+
|
|
1372
|
+
**Output:** `CallersResponse`
|
|
1373
|
+
```json
|
|
1374
|
+
{
|
|
1375
|
+
"verified_callers": [
|
|
1376
|
+
{
|
|
1377
|
+
"file_path": "api/server/src/routes/auth.ts",
|
|
1378
|
+
"line": 28,
|
|
1379
|
+
"caller_symbol": "registerAuthRoutes",
|
|
1380
|
+
"context": " return handleLogin(request, reply)",
|
|
1381
|
+
"resolution": "import"
|
|
1382
|
+
},
|
|
1383
|
+
{
|
|
1384
|
+
"file_path": "api/services/auth/src/middleware.ts",
|
|
1385
|
+
"line": 64,
|
|
1386
|
+
"caller_symbol": "AuthMiddleware.authenticate",
|
|
1387
|
+
"context": " await this.handler.handleLogin(req, reply)",
|
|
1388
|
+
"resolution": "field_type"
|
|
1389
|
+
}
|
|
1390
|
+
],
|
|
1391
|
+
"potential_matches": [
|
|
1392
|
+
{
|
|
1393
|
+
"file_path": "api/services/admin/src/audit-log.ts",
|
|
1394
|
+
"line": 142,
|
|
1395
|
+
"context": " // TODO: emit event when handleLogin is called",
|
|
1396
|
+
"reason": "identifier_match_no_resolved_edge"
|
|
1397
|
+
}
|
|
1398
|
+
],
|
|
1399
|
+
"summary": {
|
|
1400
|
+
"verified_count": 2,
|
|
1401
|
+
"potential_count": 1,
|
|
1402
|
+
"transitive": false,
|
|
1403
|
+
"checker_classified_non_call_site": 0,
|
|
1404
|
+
"checker_classified_different_declaration": 0
|
|
1405
|
+
}
|
|
1406
|
+
}
|
|
1407
|
+
```
|
|
1408
|
+
|
|
1409
|
+
`summary.potential_truncated` is omitted above because this query's `identifier_fts`
|
|
1410
|
+
fetch came back under the 50-entry cap. When it doesn't — e.g. the `isUndefined`
|
|
1411
|
+
query that motivated F10, which had 71 real matches — the fetch is capped at 50 and
|
|
1412
|
+
`summary` instead carries:
|
|
1413
|
+
```json
|
|
1414
|
+
{
|
|
1415
|
+
"potential_count": 50,
|
|
1416
|
+
"potential_truncated": 71
|
|
1417
|
+
}
|
|
1418
|
+
```
|
|
1419
|
+
`potential_count` (50, capped) and `potential_truncated` (71, real) diverging is the
|
|
1420
|
+
signal: the potential set is known-incomplete, not merely large.
|
|
1421
|
+
|
|
1422
|
+
**The two sets have different meanings.** Tools and prompts must treat them
|
|
1423
|
+
differently:
|
|
1424
|
+
|
|
1425
|
+
- **`verified_callers`** — a call site the graph statically linked to the queried
|
|
1426
|
+
symbol. The `resolution` field names how: five values come from the local heuristic
|
|
1427
|
+
resolver (§10.3) — `import` (top-level named import), `field_type` (`this.x` where
|
|
1428
|
+
`x` is a class field with a known type annotation), `parameter_type` (parameter
|
|
1429
|
+
property or annotated parameter), `new_expression` (`new Foo()`-style construction),
|
|
1430
|
+
`same_file` (call site and definition in the same file) — and one, `checker`, comes
|
|
1431
|
+
from the opt-in `mast index --checker` pass (§10.3.2): a call site the heuristic left
|
|
1432
|
+
as `potential` that `ts.TypeChecker.getSymbolAtLocation` resolved to the queried
|
|
1433
|
+
declaration. All six are high confidence; safe to act on.
|
|
1434
|
+
|
|
1435
|
+
- **`potential_matches`** — `identifier_fts` matched the symbol name exactly inside a
|
|
1436
|
+
chunk, but neither the heuristic resolver nor (if it has run) the checker pass could
|
|
1437
|
+
statically link it. These are *candidates that require human or agent review* before
|
|
1438
|
+
any refactor proceeds. Common causes: factory patterns, DI container lookups,
|
|
1439
|
+
inferred types, dynamic dispatch, comments and string literals containing the
|
|
1440
|
+
identifier. The `reason` field is informational; v1 always returns
|
|
1441
|
+
`identifier_match_no_resolved_edge`. **A qualified `"Class.method"` query (§9's
|
|
1442
|
+
documented convention) now actually matches** (F5, schema 1.3.0): `identifier_fts`
|
|
1443
|
+
rows carry qualified compounds (see §6.3's DDL comment), so this set typically
|
|
1444
|
+
includes the declaration chunk itself plus any call site the resolver linked to a
|
|
1445
|
+
receiver type but whose edge was later dropped by file-scoped resolution (e.g. an
|
|
1446
|
+
unresolvable import) — before F5 this set was confidently empty for essentially
|
|
1447
|
+
every method query (58% of symbols in the eval corpus), independent of whether a
|
|
1448
|
+
real caller existed.
|
|
1449
|
+
|
|
1450
|
+
**`summary.checker_classified_non_call_site` / `checker_classified_different_declaration`**
|
|
1451
|
+
count candidates the checker pass classified away — not a real call site (comment,
|
|
1452
|
+
string, type position) or a same-name collision resolving to a different declaration
|
|
1453
|
+
— that would otherwise still be sitting in `potential_matches` as unresolved review
|
|
1454
|
+
noise. Both are `0` when `mast index --checker` has never run against this index; a
|
|
1455
|
+
nonzero value is direct evidence the pass ran and is doing its job (§10.3.2).
|
|
1456
|
+
|
|
1457
|
+
**`summary.potential_truncated`** (F10, Stage 3) carries the real, uncapped
|
|
1458
|
+
`identifier_fts` match count — present only when that fetch (capped at 50 entries)
|
|
1459
|
+
came back full and the true count exceeds it. It is a **raw fetch** signal, computed
|
|
1460
|
+
BEFORE the verified-overlap exclusion and checker-verdict filtering that produce
|
|
1461
|
+
`potential_matches`/`potential_count` — so `potential_matches` can still be smaller
|
|
1462
|
+
than 50 even when `potential_truncated` is present; that's filtering (already visible
|
|
1463
|
+
via `checker_classified_*` above), not truncation. See the Confidence signals (C1)
|
|
1464
|
+
table (§9.0) for the full contract.
|
|
1465
|
+
|
|
1466
|
+
**Why partition rather than merge?** Mixing the two sets would force the agent to
|
|
1467
|
+
treat every result as low-confidence, defeating the value of the verified set. Mixing
|
|
1468
|
+
also means a single false-positive in the FTS hits can derail a refactor. The split
|
|
1469
|
+
preserves "I know this is a caller" as a distinct, actionable category.
|
|
1470
|
+
|
|
1471
|
+
**Agent prompt guidance.** The implement-task prompt (§12) MUST instruct the agent:
|
|
1472
|
+
"`verified_callers` are the impact set you can rely on. `potential_matches` are
|
|
1473
|
+
mandatory review sites — open each one and confirm whether it is a real caller before
|
|
1474
|
+
shipping a refactor."
|
|
1475
|
+
|
|
1476
|
+
**When used:** understanding impact before modifying a function. Sourced from
|
|
1477
|
+
`graph.db` `edges` table (verified) + `identifier_fts` (potential). When
|
|
1478
|
+
`transitive: true`, only the verified set recurses.
|
|
1479
|
+
|
|
1480
|
+
---
|
|
1481
|
+
|
|
1482
|
+
### `mast_dependencies`
|
|
1483
|
+
|
|
1484
|
+
Import graph for a file — what it imports and from where.
|
|
1485
|
+
|
|
1486
|
+
**Input:**
|
|
1487
|
+
```json
|
|
1488
|
+
{
|
|
1489
|
+
"file_path": "api/services/auth/src/handler.ts"
|
|
1490
|
+
}
|
|
1491
|
+
```
|
|
1492
|
+
|
|
1493
|
+
**Output:** `DependencyResult`
|
|
1494
|
+
```json
|
|
1495
|
+
{
|
|
1496
|
+
"file_path": "api/services/auth/src/handler.ts",
|
|
1497
|
+
"imports": [
|
|
1498
|
+
{
|
|
1499
|
+
"module": "@kluser-kinetic-01/shared",
|
|
1500
|
+
"symbols": ["UserId", "SessionToken"],
|
|
1501
|
+
"is_external": true
|
|
1502
|
+
},
|
|
1503
|
+
{
|
|
1504
|
+
"module": "../repository",
|
|
1505
|
+
"symbols": ["AuthRepository"],
|
|
1506
|
+
"is_external": false,
|
|
1507
|
+
"resolved_path": "api/services/auth/src/repository.ts"
|
|
1508
|
+
}
|
|
1509
|
+
]
|
|
1510
|
+
}
|
|
1511
|
+
```
|
|
1512
|
+
|
|
1513
|
+
**When used:** understanding what a file pulls in before modifying it. Sourced from
|
|
1514
|
+
`graph.db` `imports` table.
|
|
1515
|
+
|
|
1516
|
+
---
|
|
1517
|
+
|
|
1518
|
+
### `mast_implementors`
|
|
1519
|
+
|
|
1520
|
+
Concrete classes that implement a given interface.
|
|
1521
|
+
|
|
1522
|
+
**Input:**
|
|
1523
|
+
```json
|
|
1524
|
+
{
|
|
1525
|
+
"interface_name": "AuthRepository"
|
|
1526
|
+
}
|
|
1527
|
+
```
|
|
1528
|
+
|
|
1529
|
+
**Output:** array of `ImplementorResult`
|
|
1530
|
+
```json
|
|
1531
|
+
[
|
|
1532
|
+
{
|
|
1533
|
+
"class_name": "PostgresAuthRepository",
|
|
1534
|
+
"file_path": "api/services/auth/src/postgres-repository.ts",
|
|
1535
|
+
"line": 8,
|
|
1536
|
+
"methods": ["findByEmail", "createSession", "invalidateSession"]
|
|
1537
|
+
}
|
|
1538
|
+
]
|
|
1539
|
+
```
|
|
1540
|
+
|
|
1541
|
+
**When used:** finding the concrete implementation to follow as a pattern or extend.
|
|
1542
|
+
Sourced from `graph.db` `edges` where `edge_type = 'IMPLEMENTS'`.
|
|
1543
|
+
|
|
1544
|
+
---
|
|
1545
|
+
|
|
1546
|
+
### `mast_rename_impact`
|
|
1547
|
+
|
|
1548
|
+
Composed refactor checklist for renaming a symbol. One call packages what an
|
|
1549
|
+
agent would otherwise stitch together from `mast_callers` + manual barrel-file
|
|
1550
|
+
inspection: every section reuses an existing query capability — no new
|
|
1551
|
+
resolution logic.
|
|
1552
|
+
|
|
1553
|
+
**Input:**
|
|
1554
|
+
```json
|
|
1555
|
+
{
|
|
1556
|
+
"symbol": "createPolicyGroup",
|
|
1557
|
+
"file_path": "api/services/policy/src/service.ts | null"
|
|
1558
|
+
}
|
|
1559
|
+
```
|
|
1560
|
+
|
|
1561
|
+
`file_path` disambiguates duplicate names, with the same semantics as
|
|
1562
|
+
`mast_signature`. Methods are addressed by qualified name
|
|
1563
|
+
(`ClassName.methodName`), exactly as `mast_callers` accepts them.
|
|
1564
|
+
|
|
1565
|
+
**Output:** `RenameImpactResponse`
|
|
1566
|
+
```json
|
|
1567
|
+
{
|
|
1568
|
+
"symbol": "createPolicyGroup",
|
|
1569
|
+
"declaration_sites": [
|
|
1570
|
+
{ "file_path": "api/services/policy/src/service.ts", "line": 42, "kind": "function", "is_exported": true }
|
|
1571
|
+
],
|
|
1572
|
+
"verified_callers": [
|
|
1573
|
+
{ "file_path": "api/routes/policy.ts", "line": 18, "caller_symbol": "registerPolicyRoutes", "context": "const group = await createPolicyGroup(input);", "resolution": "import" }
|
|
1574
|
+
],
|
|
1575
|
+
"potential_matches": [
|
|
1576
|
+
{ "file_path": "api/services/policy/src/service.ts", "line": 42, "context": "createPolicyGroup", "reason": "identifier_match_no_resolved_edge" }
|
|
1577
|
+
],
|
|
1578
|
+
"barrel_exports": [
|
|
1579
|
+
{ "file_path": "api/services/policy/index.ts", "line": 1, "exported_as": "createPolicyGroup", "via": "named" },
|
|
1580
|
+
{ "file_path": "api/index.ts", "line": null, "exported_as": "createPolicyGroup", "via": "star" }
|
|
1581
|
+
],
|
|
1582
|
+
"summary": {
|
|
1583
|
+
"declaration_count": 1,
|
|
1584
|
+
"verified_count": 1,
|
|
1585
|
+
"potential_count": 1,
|
|
1586
|
+
"barrel_count": 2,
|
|
1587
|
+
"checklist": "1 verified call site(s) to update, 1 review-required identifier match(es), 2 barrel export(s) to update.",
|
|
1588
|
+
"checker_classified_non_call_site": 0,
|
|
1589
|
+
"checker_classified_different_declaration": 0
|
|
1590
|
+
}
|
|
1591
|
+
}
|
|
1592
|
+
```
|
|
1593
|
+
|
|
1594
|
+
`summary.potential_truncated` is omitted above for the same reason as in
|
|
1595
|
+
`mast_callers` (this query's `identifier_fts` fetch came back under the cap) — see
|
|
1596
|
+
§9 `mast_callers`' `potential_truncated` example for what it looks like when present.
|
|
1597
|
+
|
|
1598
|
+
Section sources and semantics:
|
|
1599
|
+
|
|
1600
|
+
- `declaration_sites` — the `symbols` table (multiple entries when the name is
|
|
1601
|
+
ambiguous and no `file_path` was given). Impact below is computed against the
|
|
1602
|
+
first match, same convention as `mast_callers`; the full list keeps an
|
|
1603
|
+
ambiguous rename visible.
|
|
1604
|
+
- `verified_callers` — direct `POTENTIAL_CALL` edges, identical to
|
|
1605
|
+
`mast_callers`' verified set. **Direct callers only** — a rename edits call
|
|
1606
|
+
sites, and every call site is a direct caller; there is no `transitive`
|
|
1607
|
+
option (deliberate v1 scope).
|
|
1608
|
+
- `potential_matches` — `identifier_fts` hits not covered by a verified edge,
|
|
1609
|
+
identical to `mast_callers`' potential set (shared implementation, including
|
|
1610
|
+
checker-verdict filtering when `mast index --checker` has run, §10.3.2). These
|
|
1611
|
+
are **mandatory review sites**: the graph could not prove them, so the agent
|
|
1612
|
+
must check each before declaring the rename complete. The declaration chunk
|
|
1613
|
+
itself typically appears here — correctly, since it must be edited.
|
|
1614
|
+
`summary.checker_classified_non_call_site`/`checker_classified_different_declaration`
|
|
1615
|
+
carry the same meaning as in `mast_callers`, as does `summary.potential_truncated`
|
|
1616
|
+
(F10) — the real, uncapped `identifier_fts` match count, present only when the
|
|
1617
|
+
capped fetch came back full; a raw-fetch signal, not a `potential_matches` size
|
|
1618
|
+
guarantee (§9 `mast_callers`'s `potential_truncated` paragraph has the full
|
|
1619
|
+
precision note).
|
|
1620
|
+
- `barrel_exports` — files that re-export the symbol: `via: "named"` rows come
|
|
1621
|
+
from `RE_EXPORTS` edges (the export statement names the symbol —
|
|
1622
|
+
`exported_as` carries the alias — and must be edited); `via: "star"` rows
|
|
1623
|
+
come from a recursive walk of `re_export_files` (`export *` statements need
|
|
1624
|
+
no edit, but every downstream consumer reaches the symbol through them, so
|
|
1625
|
+
they are surfaced for awareness; `line` is null — star rows are file-level).
|
|
1626
|
+
|
|
1627
|
+
**When used:** before renaming any exported symbol — replaces the
|
|
1628
|
+
callers-then-grep-then-barrel-hunt sequence with one call, and again after the
|
|
1629
|
+
rename (the checklist should come back empty for the old name).
|
|
1630
|
+
|
|
1631
|
+
---
|
|
1632
|
+
|
|
1633
|
+
### `mast_reindex`
|
|
1634
|
+
|
|
1635
|
+
Synchronous incremental reindex. Does not return until the index reflects the current
|
|
1636
|
+
state of the filesystem.
|
|
1637
|
+
|
|
1638
|
+
**Input:**
|
|
1639
|
+
```json
|
|
1640
|
+
{
|
|
1641
|
+
"full": false
|
|
1642
|
+
}
|
|
1643
|
+
```
|
|
1644
|
+
|
|
1645
|
+
`full: true` forces a complete reindex regardless of mtimes.
|
|
1646
|
+
|
|
1647
|
+
**Output:** `ReindexResult`
|
|
1648
|
+
```json
|
|
1649
|
+
{
|
|
1650
|
+
"files_indexed": 3,
|
|
1651
|
+
"files_skipped": 139,
|
|
1652
|
+
"chunks_added": 24,
|
|
1653
|
+
"chunks_removed": 18,
|
|
1654
|
+
"parse_errors": 0,
|
|
1655
|
+
"write_errors": 0,
|
|
1656
|
+
"duration_ms": 380
|
|
1657
|
+
}
|
|
1658
|
+
```
|
|
1659
|
+
|
|
1660
|
+
`parse_errors > 0` means one or more files were skipped due to tree-sitter parse failures. `write_errors > 0` means a file parsed successfully but its chunk/graph/FTS write failed — a distinct failure mode from a parse error (a chunk-store write failure must never be conflated with an unparseable file). The agent should call `mast_status` for details, or check the mast server log for the specific file paths.
|
|
1661
|
+
|
|
1662
|
+
**When used:** immediately after the agent writes or edits files, before querying for
|
|
1663
|
+
symbols it just created. Called explicitly by the agent — not automatic.
|
|
1664
|
+
|
|
1665
|
+
---
|
|
1666
|
+
|
|
1667
|
+
### `mast_status`
|
|
1668
|
+
|
|
1669
|
+
Index health snapshot.
|
|
1670
|
+
|
|
1671
|
+
**Input:** none
|
|
1672
|
+
|
|
1673
|
+
**Output:** `StatusResult`
|
|
1674
|
+
```json
|
|
1675
|
+
{
|
|
1676
|
+
"state_dir": "/workspace/.kluster/.mast",
|
|
1677
|
+
"schema_version": "1.3.0",
|
|
1678
|
+
"last_indexed": "2026-05-13T14:22:00Z",
|
|
1679
|
+
"indexed_files": 142,
|
|
1680
|
+
"chunk_count": 1840,
|
|
1681
|
+
"stale_files": 0,
|
|
1682
|
+
"parse_errors": 0,
|
|
1683
|
+
"write_errors": 0,
|
|
1684
|
+
"index_fresh": true,
|
|
1685
|
+
"freshness_cause": null,
|
|
1686
|
+
"seed_commit": "abc1234"
|
|
1687
|
+
}
|
|
1688
|
+
```
|
|
1689
|
+
|
|
1690
|
+
`schema_version` is `CURRENT_SCHEMA_VERSION` **as compiled into the running
|
|
1691
|
+
binary**, not the value stored in `index.json`. After a normal startup the two are
|
|
1692
|
+
identical, because §7.4 Step 2's guard wipes derived state on a mismatch; they
|
|
1693
|
+
diverge in exactly the situation this field exists to expose — a long-lived process
|
|
1694
|
+
still executing an older build while the state directory it holds open has since
|
|
1695
|
+
been migrated by a newer one. That case is invisible to every other field (the index
|
|
1696
|
+
looks healthy, because it *is* healthy — it is the server that is stale), and the
|
|
1697
|
+
startup guard cannot catch it because the guard only runs at startup. Reading the
|
|
1698
|
+
value off disk would report the migrated version and hide the divergence, so it is
|
|
1699
|
+
deliberately sourced from the binary's own constant. See
|
|
1700
|
+
`adr/proposals/measurement-harness/PLAN-EXCERPT.md` § "D8 result" for the incident that
|
|
1701
|
+
motivated it, and `adr/009-2026-08-10-measurement-harness.md` for the decision it produced.
|
|
1702
|
+
|
|
1703
|
+
`parse_errors` is the count of files skipped during the last index run due to tree-sitter
|
|
1704
|
+
parse failures; `write_errors` is the count skipped due to a chunk/graph/FTS write
|
|
1705
|
+
failure after a successful parse (see `mast_reindex`, above — the two are never
|
|
1706
|
+
conflated). Non-zero in either indicates files the agent should investigate.
|
|
1707
|
+
`seed_commit` is present only when the state directory was bootstrapped from a
|
|
1708
|
+
Docker-baked seed (§13.8) and reports the git revision the seed was built from.
|
|
1709
|
+
|
|
1710
|
+
**Freshness diagnostics.** `freshness_cause` is `"phase1_stale"` when `stale_files > 0`
|
|
1711
|
+
(chunk line coordinates lag disk — corrected by JIT re-parse on read, §9.0, or by
|
|
1712
|
+
running `mast_reindex`) and `null` when the index is fully fresh. `index_fresh` is
|
|
1713
|
+
`true` only when `stale_files === 0` and the index has been run at least once.
|
|
1714
|
+
|
|
1715
|
+
`stale_files` counts three things, not one: files whose content changed since they
|
|
1716
|
+
were indexed, files on disk that are **not in the index at all**, and files the index
|
|
1717
|
+
still lists that are gone from disk. It is computed by `indexer/freshness.ts`
|
|
1718
|
+
`measureFreshness`, which `mast status` and `mast_status` both call — one producer,
|
|
1719
|
+
because they answer one question. It reads the manifest and the `files.mtime` stamps
|
|
1720
|
+
and takes the union: only the manifest can see a file that was never indexed, and only
|
|
1721
|
+
the stamps can see a manifest entry with no row, or an edit that landed mid-run (the
|
|
1722
|
+
manifest is stamped from a finalise-time re-stat; the row carries the pre-parse stamp).
|
|
1723
|
+
|
|
1724
|
+
**A file that fails to parse or write is not recorded in the manifest**, so the next
|
|
1725
|
+
run retries it, and an incremental run's work set includes any walked file with no
|
|
1726
|
+
`files` row — otherwise a hole left by an older build would be reported stale forever
|
|
1727
|
+
with no run willing to fix it.
|
|
1728
|
+
|
|
1729
|
+
**When used:** diagnostic — agent checks this when search returns unexpected
|
|
1730
|
+
results, or before a long agentic workflow to confirm the index is current.
|
|
1731
|
+
|
|
1732
|
+
---
|
|
1733
|
+
|
|
1734
|
+
### `mast_efficiency`
|
|
1735
|
+
|
|
1736
|
+
Rolling per-session and global token-savings telemetry. Designed to be called by the
|
|
1737
|
+
agent periodically so it can self-correct when its tool usage is inefficient (e.g.,
|
|
1738
|
+
falling back to full-file `Read` instead of `mast_search`).
|
|
1739
|
+
|
|
1740
|
+
**Input:**
|
|
1741
|
+
```json
|
|
1742
|
+
{
|
|
1743
|
+
"scope": "session | global",
|
|
1744
|
+
"since_minutes": 60
|
|
1745
|
+
}
|
|
1746
|
+
```
|
|
1747
|
+
|
|
1748
|
+
- `scope: "session"` aggregates only this `mast serve` session's calls. `scope: "global"`
|
|
1749
|
+
aggregates the persistent metrics table on the shared volume (see §14).
|
|
1750
|
+
- `since_minutes` bounds the window for `scope: "global"`. Ignored when `scope` is
|
|
1751
|
+
`"session"`. Defaults to 60.
|
|
1752
|
+
|
|
1753
|
+
**Output:** `EfficiencyResult`
|
|
1754
|
+
```json
|
|
1755
|
+
{
|
|
1756
|
+
"scope": "session",
|
|
1757
|
+
"window_started_at": "2026-05-13T14:00:00Z",
|
|
1758
|
+
"tokens_returned": 18420,
|
|
1759
|
+
"tokens_full_file_upper_bound": 142880,
|
|
1760
|
+
"efficiency_ratio": 0.871,
|
|
1761
|
+
"calls_total": 47,
|
|
1762
|
+
"calls_by_tool": {
|
|
1763
|
+
"mast_search": 18,
|
|
1764
|
+
"mast_signature": 12,
|
|
1765
|
+
"mast_exports": 9,
|
|
1766
|
+
"mast_callers": 5,
|
|
1767
|
+
"mast_project_skeleton": 3
|
|
1768
|
+
},
|
|
1769
|
+
"tokenizer": "@anthropic-ai/tokenizer (claude-2 era, approximate for current models)",
|
|
1770
|
+
"counterfactual": "Saved vs. full file Read (upper bound — overstates savings against a smart agent that would have used Grep)"
|
|
1771
|
+
}
|
|
1772
|
+
```
|
|
1773
|
+
|
|
1774
|
+
The `tokenizer` field carries the honest label from §14.5 verbatim — token
|
|
1775
|
+
counts are approximate for current models, and consumers should treat the
|
|
1776
|
+
savings *ratio*, not the absolute counts, as the robust number.
|
|
1777
|
+
|
|
1778
|
+
`efficiency_ratio` is `1 - (tokens_returned / tokens_full_file_upper_bound)`. Higher
|
|
1779
|
+
is better; 0.871 means 87.1% fewer tokens than the "naive `Read` every result file"
|
|
1780
|
+
counterfactual.
|
|
1781
|
+
|
|
1782
|
+
**Honest framing.** The counterfactual is explicitly labelled as an upper bound. An
|
|
1783
|
+
agent using `Grep -A 10 -B 10` instead of `Read` would have used fewer tokens than
|
|
1784
|
+
the counterfactual but more than MAST — the real savings sit between zero and the
|
|
1785
|
+
reported ratio. The label exists so this number can survive a "is MAST worth it?"
|
|
1786
|
+
review without being challenged as cherry-picked. See §14 for the methodology and
|
|
1787
|
+
the SQLite metrics table schema.
|
|
1788
|
+
|
|
1789
|
+
**When used:** the agent calls this once per task (typically near the end) to see
|
|
1790
|
+
whether its tool usage was efficient. The implement-task prompt (§12) instructs:
|
|
1791
|
+
"If `efficiency_ratio < 0.30`, you are reading more than you should — prefer
|
|
1792
|
+
`mast_search` over `Read` for the next task." This gives the agent an in-loop
|
|
1793
|
+
feedback signal without humans needing to inspect logs.
|
|
1794
|
+
|
|
1795
|
+
---
|
|
1796
|
+
|
|
1797
|
+
## 10. AST Extraction
|
|
1798
|
+
|
|
1799
|
+
mast uses **tree-sitter** with `tree-sitter-typescript` for all AST operations.
|
|
1800
|
+
tree-sitter is chosen over the TypeScript Compiler API for speed (C library, no
|
|
1801
|
+
tsc overhead) and extensibility to other languages via grammar plugins.
|
|
1802
|
+
|
|
1803
|
+
### 10.1 Chunking Strategy
|
|
1804
|
+
|
|
1805
|
+
Chunks are **declaration-first**: each chunk maps to exactly one top-level AST
|
|
1806
|
+
declaration. Line-based splitting is a fallback, not the primary strategy.
|
|
1807
|
+
|
|
1808
|
+
| Node type | `chunk_type` | `symbol_name` | `parent_symbol` | `is_exported` |
|
|
1809
|
+
|---|---|---|---|---|
|
|
1810
|
+
| `function_declaration` | `function` | function name | `null` | has `export` modifier |
|
|
1811
|
+
| `arrow_function` → `const` | `function` | variable name | `null` | has `export` modifier |
|
|
1812
|
+
| `class_declaration` | `class_shell` (synthesized) | class name | `null` | has `export` modifier |
|
|
1813
|
+
| `method_definition` (inside class) | `method` | `ClassName.methodName` | class name | inherits from class **and** non-`private` |
|
|
1814
|
+
| `interface_declaration` | `interface` | interface name | `null` | has `export` modifier |
|
|
1815
|
+
| `type_alias_declaration` | `type` | type name | `null` | has `export` modifier |
|
|
1816
|
+
| `export_statement` wrapping any above | inherits inner | inherits inner | inherits inner | `true` |
|
|
1817
|
+
| Everything else at top level | `block` | `null` | `null` | `false` |
|
|
1818
|
+
|
|
1819
|
+
**Class decomposition: shell + methods.** A `class_declaration` node does NOT
|
|
1820
|
+
become one chunk. The chunker emits:
|
|
1821
|
+
|
|
1822
|
+
1. **One `class_shell` chunk** whose `content` is the synthesized class outline:
|
|
1823
|
+
the class declaration line, all member signatures (methods, properties, getters,
|
|
1824
|
+
setters, constructors) with their leading TSDoc, but **no method bodies**. This
|
|
1825
|
+
is the "outline" view used for orientation and for `mast_signature` calls that
|
|
1826
|
+
target the class itself rather than a specific member.
|
|
1827
|
+
|
|
1828
|
+
2. **N `method` chunks**, one per `method_definition` (and `constructor`, getter,
|
|
1829
|
+
setter) inside the class body. Each method chunk has:
|
|
1830
|
+
- `symbol_name` qualified as `ClassName.methodName` (e.g. `AuthService.validateSession`,
|
|
1831
|
+
`AuthService.constructor`).
|
|
1832
|
+
- `parent_symbol` set to the enclosing class name (unqualified).
|
|
1833
|
+
- `is_exported` = class's `is_exported` **AND** method is not `private`. (Methods
|
|
1834
|
+
marked `protected` or with no accessibility modifier are treated as exported when
|
|
1835
|
+
the class is exported. This mirrors how callers from outside the class can reach
|
|
1836
|
+
them; `private` members are intentionally hidden from `only_exported: true`
|
|
1837
|
+
queries.)
|
|
1838
|
+
- Its own `content`, the full method source (signature + body).
|
|
1839
|
+
- A `PARENT_OF` edge from the class's `symbols` row to the method's `symbols` row
|
|
1840
|
+
in `graph.db` (see §6.3).
|
|
1841
|
+
|
|
1842
|
+
3. **A `body_hash` for the class_shell** computed over the sorted concatenation of
|
|
1843
|
+
member signatures + member TSDoc (see §7.1 stability hash rule). This ensures
|
|
1844
|
+
the shell content is rewritten when methods are renamed, added, or removed — but
|
|
1845
|
+
**not** when method bodies change.
|
|
1846
|
+
|
|
1847
|
+
**Why this matters.** A 400-line service class becomes one ~30-line outline chunk
|
|
1848
|
+
plus ~12 small method chunks (~30 lines each). `mast_search "validate session"`
|
|
1849
|
+
returns the matching `method` chunk (~30 lines) instead of the whole class (400
|
|
1850
|
+
lines). For class-heavy codebases the token savings move from "marginal" to
|
|
1851
|
+
"material." Chunk count for a class-heavy 5K-file repo grows from ~6K to ~20–30K;
|
|
1852
|
+
SQLite (chunks table + FTS5) handles this fine at sub-100MB index size.
|
|
1853
|
+
|
|
1854
|
+
**Interfaces and type aliases are NOT decomposed.** Their members are signatures
|
|
1855
|
+
already (no bodies to split), so the interface or type alias remains a single chunk.
|
|
1856
|
+
|
|
1857
|
+
**Anonymous default exports** (`export default function () {}`, `export default {}`):
|
|
1858
|
+
`symbol_name` is set to the filename without extension (e.g., `handler` for
|
|
1859
|
+
`handler.ts`). This is a heuristic and `mast_search` will surface these via FTS
|
|
1860
|
+
on the filename. A future v2 may resolve the alias from importers, but v1 keeps it
|
|
1861
|
+
simple.
|
|
1862
|
+
|
|
1863
|
+
**Re-export aliases** (`export { foo as bar } from './x'`): the extractor
|
|
1864
|
+
records `bar` as an exported **marker symbol** (kind `export`, no hashes) in
|
|
1865
|
+
the barrel's `symbols` rows plus a `RE_EXPORTS` edge from the marker to `foo`'s
|
|
1866
|
+
declaration; `export * from './x'` becomes a `re_export_files` row instead
|
|
1867
|
+
(file-level — stars name no symbols). Marker rows exist to anchor the edge for
|
|
1868
|
+
`mast_rename_impact`'s barrel checklist and are **excluded from symbol lookups**
|
|
1869
|
+
(`querySymbolByName` filters kind `export`), so `mast_signature`/`mast_callers`
|
|
1870
|
+
keep resolving to the real declaration rather than the barrel. Import
|
|
1871
|
+
specifiers are resolved with the same §13.7 resolver used for `import`
|
|
1872
|
+
statements.
|
|
1873
|
+
|
|
1874
|
+
**Implementation note — local aliases.** For a *local* alias
|
|
1875
|
+
(`export { foo as bar }`, no `from`), the chunker does not use the
|
|
1876
|
+
`RE_EXPORTS`-edge path above. It instead emits an extra chunk for `bar` that
|
|
1877
|
+
mirrors `foo`'s declaration (own `chunk_id`, marked exported), and
|
|
1878
|
+
`extractSignatures` emits a matching `bar` signature entry. This makes `bar`
|
|
1879
|
+
discoverable (`mast_exports`/`mast_search`) and resolvable (`mast_signature`)
|
|
1880
|
+
without a chain walk — same observable result, simpler mechanism. The aliased
|
|
1881
|
+
local name (`foo`) is NOT itself marked exported, since the export name is `bar`.
|
|
1882
|
+
|
|
1883
|
+
**Two-pass walk for `is_exported`:**
|
|
1884
|
+
|
|
1885
|
+
TypeScript allows declarations to be exported separately from their definition:
|
|
1886
|
+
|
|
1887
|
+
```typescript
|
|
1888
|
+
function internalHandle() { ... }
|
|
1889
|
+
export { internalHandle as handleLogin };
|
|
1890
|
+
export * from './other-module';
|
|
1891
|
+
```
|
|
1892
|
+
|
|
1893
|
+
A single-pass walk that only checks for the `export` modifier on declarations would
|
|
1894
|
+
mark `internalHandle` as not exported, which is wrong. The chunker uses a two-pass
|
|
1895
|
+
walk:
|
|
1896
|
+
|
|
1897
|
+
- **Pass 1:** map all top-level symbol names to their declaration nodes and initial
|
|
1898
|
+
`is_exported` state (based on `export` modifier presence).
|
|
1899
|
+
- **Pass 2:** walk all `export_clause` nodes (`export { ... }`) and
|
|
1900
|
+
`export_all_clause` nodes (`export * from ...`). For each named re-export, find
|
|
1901
|
+
the symbol in the Pass 1 map and set `is_exported = true`. For `export *`, mark the
|
|
1902
|
+
source module path for resolution — the graph populator will follow the edge.
|
|
1903
|
+
|
|
1904
|
+
**Split rule:** if a declaration spans more than `chunk_split_threshold` lines
|
|
1905
|
+
(default: 100), split into overlapping sub-chunks with 10-line overlap. The first
|
|
1906
|
+
sub-chunk always includes the full declaration header so signature extraction is
|
|
1907
|
+
always possible from sub-chunk 0.
|
|
1908
|
+
|
|
1909
|
+
Most functions will not trigger this split — the default of 100 lines comfortably
|
|
1910
|
+
covers ordinary function and method bodies. The threshold exists for edge cases
|
|
1911
|
+
(large generated files, data-heavy switch statements) where a single chunk would
|
|
1912
|
+
otherwise be too large to be a useful, self-contained search result.
|
|
1913
|
+
|
|
1914
|
+
**Markdown documents (`chunk_type: "doc"`).** `.md` files are chunked by ATX
|
|
1915
|
+
heading, not by AST — one chunk per heading of level ≤ `markdown_heading_depth`
|
|
1916
|
+
(default 2: one chunk per `##` section). Doc chunks get
|
|
1917
|
+
`chunk_fts` rows like any other chunk, but **no graph presence**: no `symbols`
|
|
1918
|
+
rows, no `imports`, no `edges`, and no `identifier_fts` rows (that index feeds
|
|
1919
|
+
`mast_callers` potential_matches, where a doc that merely *mentions* a symbol
|
|
1920
|
+
name is noise, not a call site).
|
|
1921
|
+
|
|
1922
|
+
Rules:
|
|
1923
|
+
|
|
1924
|
+
- `symbol_name` is the heading path — the file name, every ancestor heading,
|
|
1925
|
+
and the section's own heading joined with `" > "`, e.g.
|
|
1926
|
+
`MAST_SPEC.md > Technical Specification > 7. Index Lifecycle`. Skipped
|
|
1927
|
+
heading levels are omitted from the path.
|
|
1928
|
+
- Headings deeper than `markdown_heading_depth` fold into their enclosing
|
|
1929
|
+
section's content.
|
|
1930
|
+
- Content before the first boundary heading becomes a preamble chunk whose
|
|
1931
|
+
`symbol_name` is the file name alone. A file with no headings is one
|
|
1932
|
+
preamble chunk.
|
|
1933
|
+
- `#` lines inside fenced code blocks (``` or ~~~) are not headings. Setext
|
|
1934
|
+
headings (`===`/`---` underlines) are not recognised — this repo's docs use
|
|
1935
|
+
ATX exclusively.
|
|
1936
|
+
- `is_exported` is always `false` — `only_exported: true` searches exclude
|
|
1937
|
+
docs by construction. `parent_symbol` is always `null`; `language` is
|
|
1938
|
+
`markdown`.
|
|
1939
|
+
- The split rule above applies to oversized sections (same window, overlap,
|
|
1940
|
+
and sub-chunk ID scheme as declarations).
|
|
1941
|
+
- No `context_lines` expansion: sections are self-delimiting, and expansion
|
|
1942
|
+
would duplicate neighbouring sections' text into every chunk.
|
|
1943
|
+
- Doc chunks carry no stability hashes, so the incremental "unchanged file"
|
|
1944
|
+
fast path conservatively rewrites a markdown file whose mtime changed
|
|
1945
|
+
(same treatment as files containing `block` chunks).
|
|
1946
|
+
|
|
1947
|
+
### 10.2 Signature Extraction
|
|
1948
|
+
|
|
1949
|
+
For `mast_signature` and the signature field in `mast_exports`:
|
|
1950
|
+
|
|
1951
|
+
1. Locate the declaration node by symbol name via tree-sitter query.
|
|
1952
|
+
2. Extract node text up to (not including) the `statement_block` child (`{ ... }` body).
|
|
1953
|
+
3. For interfaces and type aliases: the full declaration is the signature — no body
|
|
1954
|
+
exists to strip.
|
|
1955
|
+
4. Walk backwards from the declaration's start byte to find the immediately preceding
|
|
1956
|
+
`comment` node. Accept `/** ... */` (TSDoc) or `// ...` (line comment). Include
|
|
1957
|
+
as `doc`.
|
|
1958
|
+
|
|
1959
|
+
### 10.3 Knowledge Graph Population
|
|
1960
|
+
|
|
1961
|
+
After Phase 1 chunking, mast populates `graph.db` for each indexed file:
|
|
1962
|
+
|
|
1963
|
+
- **`imports`**: parse each `import_statement`. Extract module specifier and named
|
|
1964
|
+
imports. Resolve relative paths against `project_root` for local modules. The
|
|
1965
|
+
resolved imports double as the input to the **local type environment** (see
|
|
1966
|
+
§10.3.1) used by the `POTENTIAL_CALL` resolver.
|
|
1967
|
+
- **`symbols` rows**: insert a row for each top-level declaration AND each
|
|
1968
|
+
`method_definition` inside a class. Method symbols carry qualified names
|
|
1969
|
+
(`ClassName.methodName`) and a `PARENT_OF` edge from the class symbol — see §10.1
|
|
1970
|
+
class decomposition.
|
|
1971
|
+
- **`POTENTIAL_CALL` edges**: within function and method bodies, find `call_expression`
|
|
1972
|
+
nodes. Run the local heuristic resolver (§10.3.1) against the receiver to identify
|
|
1973
|
+
the callee symbol. Only insert an edge when the resolver returns a single known
|
|
1974
|
+
indexed symbol; skip external library calls and unresolved receivers (those are
|
|
1975
|
+
surfaced as `potential_matches` in `mast_callers` via `identifier_fts`, not as
|
|
1976
|
+
edges).
|
|
1977
|
+
- **`IMPLEMENTS` / `EXTENDS` edges**: from `class_declaration` nodes with
|
|
1978
|
+
`implements_clause` or `extends_clause`, resolve the named type and insert the edge.
|
|
1979
|
+
- **`identifier_fts` rows**: extract every identifier token in the chunk (function
|
|
1980
|
+
names, method names, type names, variable references) via tree-sitter `identifier`
|
|
1981
|
+
node enumeration. Deduplicate per chunk. Insert one row per chunk with
|
|
1982
|
+
whitespace-joined identifiers — this is what `mast_callers`'s `potential_matches`
|
|
1983
|
+
query hits. **F5 (schema 1.3.0):** after the bare-token bag, append QUALIFIED
|
|
1984
|
+
compound strings (`"Class.method"`) — a method chunk's own qualified name, plus
|
|
1985
|
+
the resolved `Type.method` string for every call site the local type environment
|
|
1986
|
+
(§10.3.1) linked to a receiver type, appended to the CALLING chunk's row. This is
|
|
1987
|
+
what makes the phrase-quoted `"Class.method"` query `mast_callers`/
|
|
1988
|
+
`mast_rename_impact` issue (§9) actually hit a row: identifier_fts' `unicode61`
|
|
1989
|
+
tokenizer treats `.` as a separator, so the literal compound string tokenizes into
|
|
1990
|
+
two ADJACENT tokens, matching the phrase directly. The bare bag alone essentially
|
|
1991
|
+
never has that adjacency for a method name. A receiver the resolver could not
|
|
1992
|
+
statically link (§10.3.1's "does NOT catch" list) contributes no compound.
|
|
1993
|
+
|
|
1994
|
+
All inserts for a file are wrapped in a single transaction. On incremental reindex,
|
|
1995
|
+
existing rows for the file are deleted before reinsertion (delete-and-replace, not
|
|
1996
|
+
upsert), which keeps the graph consistent with renames and deletions.
|
|
1997
|
+
|
|
1998
|
+
### 10.3.1 Local Type Environment (POTENTIAL_CALL Resolver)
|
|
1999
|
+
|
|
2000
|
+
The `POTENTIAL_CALL` edge type is named for what it actually is: a name-resolved
|
|
2001
|
+
reference whose receiver was statically linkable to a known symbol. There is no full
|
|
2002
|
+
TypeScript type-checker in mast; the resolver is a deliberately scoped set of
|
|
2003
|
+
heuristics that catches the high-frequency cases without bringing `tsserver` into
|
|
2004
|
+
the indexer.
|
|
2005
|
+
|
|
2006
|
+
**What the resolver catches (will produce a `POTENTIAL_CALL` edge):**
|
|
2007
|
+
|
|
2008
|
+
1. **Top-level named imports.** `import { handleLogin } from './handler'; handleLogin(req, reply)`
|
|
2009
|
+
→ resolves `handleLogin` via the `imports` table to a known symbol.
|
|
2010
|
+
2. **Class field types** (annotated). `private userRepo: UserRepository` followed
|
|
2011
|
+
somewhere in the class body by `this.userRepo.findByEmail(email)` → resolves
|
|
2012
|
+
`findByEmail` to `UserRepository.findByEmail`. The field's type annotation must
|
|
2013
|
+
be a named type the resolver can find via `imports` or same-file `symbols`.
|
|
2014
|
+
3. **Constructor parameter properties.** `constructor(private readonly users: UserRepository)`
|
|
2015
|
+
creates an implicit field; `this.users.create(input)` resolves the same way as (2).
|
|
2016
|
+
4. **Annotated parameters in any function/method.** `function foo(repo: UserRepository) { repo.findById(id) }`
|
|
2017
|
+
→ resolves `findById` to `UserRepository.findById`.
|
|
2018
|
+
5. **`new` expressions.** `const repo = new UserRepository(); repo.findById(id)` →
|
|
2019
|
+
the resolver tracks `repo`'s inferred type as `UserRepository` for the rest of the
|
|
2020
|
+
block (or until shadowed) and resolves the chained call.
|
|
2021
|
+
6. **Same-file function calls.** A function calling another function in the same file
|
|
2022
|
+
resolves directly via the local `symbols` table.
|
|
2023
|
+
|
|
2024
|
+
**What the resolver does NOT catch (will NOT produce a `POTENTIAL_CALL` edge — but
|
|
2025
|
+
the identifier match still lands in `identifier_fts` and surfaces as
|
|
2026
|
+
`potential_matches`):**
|
|
2027
|
+
|
|
2028
|
+
- **Factory return types without annotation.** `const repo = makeRepository(); repo.findById(id)`
|
|
2029
|
+
— `repo`'s type is inferred and the resolver does not run inference.
|
|
2030
|
+
- **DI container lookups.** `container.get(UserRepository).findById(id)` — the
|
|
2031
|
+
generic erases at the resolver level.
|
|
2032
|
+
- **Chained calls without intermediate binding.** `getUserService().findById(id)`.
|
|
2033
|
+
- **Dynamic dispatch.** `repos[name].findById(id)`.
|
|
2034
|
+
- **Re-exported types not yet resolved through the `re_export_files` chain at edge
|
|
2035
|
+
time.** Resolution can be deferred: edges are inserted on a second pass after all
|
|
2036
|
+
symbols are populated (see "Two-pass edge insertion" below).
|
|
2037
|
+
- **Generic type parameters.** `class Repo<T> { find(id: ID): T }` — the resolver
|
|
2038
|
+
treats `T` as opaque.
|
|
2039
|
+
|
|
2040
|
+
**F5 honesty note on "still lands in `identifier_fts`" above.** That claim was
|
|
2041
|
+
historically true only for BARE identifier tokens (e.g. `findById`), which
|
|
2042
|
+
`mast_callers`/`mast_rename_impact` never actually query — they query the
|
|
2043
|
+
QUALIFIED `"Class.method"` form (§9), which (pre-F5) essentially never phrase-
|
|
2044
|
+
matched anything, silently emptying `potential_matches` for methods regardless
|
|
2045
|
+
of this list. Post-F5, the distinction that matters is whether
|
|
2046
|
+
`LocalTypeEnvironment.resolveCall` (the SAME resolution `POTENTIAL_CALL`
|
|
2047
|
+
edges use) succeeded, not whether the edge survived to `graph.db`:
|
|
2048
|
+
- **Re-exported types not yet resolved through the `re_export_files` chain**
|
|
2049
|
+
— extraction still resolves the receiver's type name and emits the
|
|
2050
|
+
qualified compound; only the DB-layer edge is dropped. This case IS now
|
|
2051
|
+
healed: the mention surfaces in `potential_matches`.
|
|
2052
|
+
- **Factory return types, DI container lookups, chained calls without
|
|
2053
|
+
intermediate binding, dynamic dispatch, generic type parameters** — the
|
|
2054
|
+
receiver is unresolvable at EXTRACTION time (`resolveCall` itself returns
|
|
2055
|
+
null), so no qualified compound is ever produced. These remain a genuine
|
|
2056
|
+
coverage gap for qualified queries specifically — closing it (if ever) is
|
|
2057
|
+
the opt-in `mast index --checker` pass's (§10.3.2) territory, not this
|
|
2058
|
+
fix's; F5 deliberately does not attempt to guess a qualified name for an
|
|
2059
|
+
unresolvable receiver.
|
|
2060
|
+
|
|
2061
|
+
**Coverage characterisation.** In a Fastify + DI service codebase, the resolver
|
|
2062
|
+
catches roughly the field/parameter/import cases — typically 60–80% of real call
|
|
2063
|
+
sites depending on how heavily the codebase uses factories and containers. The
|
|
2064
|
+
intentional design choice is: when in doubt, do NOT produce a `POTENTIAL_CALL` edge,
|
|
2065
|
+
and rely on `identifier_fts` + the `mast_callers` `potential_matches` set to catch
|
|
2066
|
+
the rest. False negatives in the verified set are acceptable; false positives would
|
|
2067
|
+
poison the contract.
|
|
2068
|
+
|
|
2069
|
+
**Two-pass edge insertion.** Because cross-file references depend on all symbols
|
|
2070
|
+
being in the table first, edge insertion runs as a second pass after Phase 1's
|
|
2071
|
+
symbol-population pass completes for the entire indexed file set. This means a full
|
|
2072
|
+
reindex has the ordering: walk all files → insert all chunks + symbols → re-walk
|
|
2073
|
+
all files → insert all edges. On incremental reindex, only the affected files are
|
|
2074
|
+
re-walked in pass two, but all of `graph.db`'s `symbols` table is queryable so cross-
|
|
2075
|
+
file references resolve correctly.
|
|
2076
|
+
|
|
2077
|
+
**Method calls on `super` and `this` without receiver.** `this.foo()` resolves to
|
|
2078
|
+
the enclosing class's `foo` method via the qualified `symbols` row. `super.foo()`
|
|
2079
|
+
resolves to the parent class via the `EXTENDS` edge. Implemented as ordinary
|
|
2080
|
+
receiver bindings (F4, Stage 3): `emitClassEdges` seeds `this` → the enclosing
|
|
2081
|
+
class name and, only when an `extends` clause names a parent, `super` → that
|
|
2082
|
+
parent's name, riding the same `LocalTypeEnvironment` receiver-binding path
|
|
2083
|
+
`field_type` etc. use — no parallel resolution mechanism. The resulting
|
|
2084
|
+
`resolution` values are `this_method` and `super_method` respectively (see the
|
|
2085
|
+
`edges` table comment above and §9.0's confidence-signals table); a class with
|
|
2086
|
+
no `extends` clause never produces a `super_method` edge — an unresolvable
|
|
2087
|
+
`super.foo()` call falls through to the `identifier_fts` potential set instead
|
|
2088
|
+
of guessing. `this.foo()` inside a nested non-arrow function/method/generator
|
|
2089
|
+
body is NOT the class instance and is excluded before it ever reaches the
|
|
2090
|
+
resolver — arrow functions inherit the enclosing `this` and are not excluded.
|
|
2091
|
+
|
|
2092
|
+
(Non-normative, D7/Stage 4) `extractEdges`/`emitCallEdges` also accept an optional
|
|
2093
|
+
`onCallSite` diagnostics callback, invoked once per visited call site with a closed
|
|
2094
|
+
outcome union; it exists solely as a test seam for `call-oracle.test.ts`'s corpus
|
|
2095
|
+
accounting invariant and is not configurable, persisted, or surfaced in any tool
|
|
2096
|
+
response.
|
|
2097
|
+
|
|
2098
|
+
**Name resolution is file-scoped, not name-only.** When two files export a
|
|
2099
|
+
same-named symbol, `insertEdges`' name→id resolution (§10.3) uses the
|
|
2100
|
+
resolution rule's own file evidence to pick the target: `same_file` is scoped
|
|
2101
|
+
to the calling file itself; `import` is scoped to the import's own
|
|
2102
|
+
`resolved_path` (following the `re_export_files`/`RE_EXPORTS` chain into a
|
|
2103
|
+
barrel when the resolved file doesn't declare the symbol directly, per §6.3);
|
|
2104
|
+
`field_type`/`parameter_type`/`new_expression` are scoped the same way via
|
|
2105
|
+
the receiver's type name, when that type name is itself traceable to an
|
|
2106
|
+
import or a same-file declaration. Only when a rule has no such evidence
|
|
2107
|
+
(e.g. a default/namespace import, which is not tracked as a named import) does
|
|
2108
|
+
resolution fall back to a global name match — a known, narrow coverage gap,
|
|
2109
|
+
not the general case. Prior to 2026-07-15 every rule fell back to the global
|
|
2110
|
+
match unconditionally, so a same-named symbol in an earlier-indexed file could
|
|
2111
|
+
silently win a `verified_callers` edge that belonged to a different file
|
|
2112
|
+
(IMPLEMENTATION_PLAN_VEXP.md §P, "Shipped-resolver finding").
|
|
2113
|
+
|
|
2114
|
+
The same file-scoping applies to `RE_EXPORTS` edges (`export { x } from './y'`,
|
|
2115
|
+
§6.3): the extractor resolves the re-export's own `from`-clause module
|
|
2116
|
+
specifier to a real file (`EdgeRecord.toResolvedPath`) at parse time, and
|
|
2117
|
+
`insertEdges` uses that path — not a bare name match across the whole graph —
|
|
2118
|
+
to pick the target when two files export a same-named symbol. An unresolved
|
|
2119
|
+
module (external, or a relative specifier that doesn't probe to a real file)
|
|
2120
|
+
produces no edge rather than a name-only guess. This closed the sibling of the
|
|
2121
|
+
same false-green class for `mast_rename_impact`'s `barrel_exports`
|
|
2122
|
+
(IMPLEMENTATION_PLAN_VEXP.md §P, "Sibling false-green").
|
|
2123
|
+
|
|
2124
|
+
### 10.3.2 TypeScript-Checker Enrichment Pass (`mast index --checker`)
|
|
2125
|
+
|
|
2126
|
+
An **opt-in** CLI pass (`src/graph/checker-resolver.ts`) that uses the real
|
|
2127
|
+
TypeScript compiler to upgrade `potential_matches` the §10.3.1 heuristic
|
|
2128
|
+
resolver could not statically link. Reshaped from an originally-planned
|
|
2129
|
+
always-on background worker by a spike measurement: holding every workspace
|
|
2130
|
+
project's `ts.Program` alive at once peaked at 2.45 GB RSS, over a 2 GB gate
|
|
2131
|
+
(IMPLEMENTATION_PLAN_VEXP.md Stage 1.1, `eval/spikes/checker-edges/REPORT.md`).
|
|
2132
|
+
The default `mast index` path is behaviourally untouched — this only runs with
|
|
2133
|
+
the `--checker` flag.
|
|
2134
|
+
|
|
2135
|
+
**What it guarantees:**
|
|
2136
|
+
|
|
2137
|
+
- Holds exactly **one** `ts.Program` at a time, disposed before the next
|
|
2138
|
+
tsconfig project loads (the spike's cautionary tale: holding all programs
|
|
2139
|
+
alive made a "warm" re-check *slower* than cold, via GC pressure).
|
|
2140
|
+
- Every candidate is one of the shipped `potential_matches` pool
|
|
2141
|
+
(`collectPotentialMatchCandidates`, shared with `mast_callers` — never a
|
|
2142
|
+
second definition of "what counts as a potential match").
|
|
2143
|
+
- `getAliasedSymbol` alias-chain following (bounded at 8 hops) on every
|
|
2144
|
+
resolution — without it, resolution collapses from ~38% to ~2% (spike
|
|
2145
|
+
finding; an import binding otherwise resolves to itself, not its target).
|
|
2146
|
+
- **False-green gate:** a checker edge is written ONLY when the resolved
|
|
2147
|
+
declaration's `(file, line)` matches the queried symbol's own recorded
|
|
2148
|
+
`(file, line)` (±3 lines, for decorator/JSDoc/overload offsets). A same-name
|
|
2149
|
+
collision (two unrelated declarations sharing a method name, an
|
|
2150
|
+
interface-typed receiver with multiple implementors, a shadowed import)
|
|
2151
|
+
resolves to a DIFFERENT declaration and is classified `resolves_to_different`
|
|
2152
|
+
— never written as a `checker` edge. A wrong "verified" edge is worse than no
|
|
2153
|
+
edge (adversarial fixtures: `src/graph/__tests__/checker-resolver.test.ts`).
|
|
2154
|
+
|
|
2155
|
+
**What it does to each candidate:**
|
|
2156
|
+
|
|
2157
|
+
| Classification | Effect |
|
|
2158
|
+
|---|---|
|
|
2159
|
+
| Resolves to the queried declaration | A `POTENTIAL_CALL` edge is written with `resolution: 'checker'` — joins `verified_callers` exactly like a heuristic edge, and dedupes with one on the same `(from_id, to_id, edge_type)` triple. |
|
|
2160
|
+
| Resolves to a DIFFERENT declaration | Recorded in `checker_verdicts`; drops out of `potential_matches` and is counted in `summary.checker_classified_different_declaration`. |
|
|
2161
|
+
| Not a call site (comment, string, type position) | Recorded in `checker_verdicts`; drops out of `potential_matches` and is counted in `summary.checker_classified_non_call_site`. This residue was 30–44% of the sampled potential pool in the Stage 1.1 spike — classifying it away is itself a major token win, independent of edge upgrades. |
|
|
2162
|
+
| Unresolvable (dynamic dispatch, DI lookup, etc.) | No edge, no verdict — stays a genuine `potential_match`, exactly as before `--checker` ran. |
|
|
2163
|
+
|
|
2164
|
+
**What stays potential — the checker pass does not close every gap:**
|
|
2165
|
+
|
|
2166
|
+
- Files outside every discovered tsconfig project (roughly 22% of the
|
|
2167
|
+
monorepo sample) are left completely untouched; the CLI summary reports the
|
|
2168
|
+
count (`outside_ts_scope`), never a silent cap.
|
|
2169
|
+
`discoverTsConfigProjects` finds every `tsconfig.json` under the project
|
|
2170
|
+
root whose `parseJsonConfigFileContent` resolves at least one file — a base
|
|
2171
|
+
config meant to be `extends`-ed (no own `"include"`) is skipped as
|
|
2172
|
+
`no_include_base_config`, and an unparseable config is skipped with the
|
|
2173
|
+
parser's own error text. Generic and project-shape-agnostic (unlike the
|
|
2174
|
+
Stage 1.1 spike's hardcoded 25-project list for this monorepo specifically).
|
|
2175
|
+
- A call site with no enclosing declared symbol (e.g. top-level script code,
|
|
2176
|
+
a `block` chunk) still gets classified — so it never re-surfaces as review
|
|
2177
|
+
noise — but no `checker` edge is written, since there is no valid `from_id`
|
|
2178
|
+
to attach one to.
|
|
2179
|
+
- Cross-package calls where the target resolves into another workspace
|
|
2180
|
+
package's compiled `.d.ts` output (not the `.ts` source `mast` indexed) fail
|
|
2181
|
+
the `(file, line)` match safely — classified `resolves_to_different`, not a
|
|
2182
|
+
false positive, just a missed upgrade.
|
|
2183
|
+
|
|
2184
|
+
**Verdict staleness — the severity-zero invariant.** A verdict must not
|
|
2185
|
+
outlive the file content it was computed against: a stale verdict silently
|
|
2186
|
+
suppressing a genuinely new call site is worse than never having run the pass.
|
|
2187
|
+
`checker_verdicts.call_site_file_id REFERENCES files(id) ON DELETE CASCADE`
|
|
2188
|
+
ties a verdict's lifetime to the file row it was computed against —
|
|
2189
|
+
`populateFile`'s delete-and-replace on ANY content change (both full
|
|
2190
|
+
`mast index` and the JIT re-parse triggered by staleness detection, §9.0)
|
|
2191
|
+
cascades away every verdict for that file automatically, exactly like
|
|
2192
|
+
`symbols`/`edges`/`imports` already do. `checker_verdicts.call_site_mtime` is
|
|
2193
|
+
checked again at read time (`queryCheckerVerdicts`) as a second, independent
|
|
2194
|
+
guard. Proven directly against the real Phase 1 pipeline (edit a fixture file,
|
|
2195
|
+
reindex, assert the verdict no longer applies) in
|
|
2196
|
+
`src/graph/__tests__/checker-resolver.test.ts`.
|
|
2197
|
+
|
|
2198
|
+
**Persistence.** `checker_verdicts` is a brand-new, additive table (§7.4 — no
|
|
2199
|
+
`CURRENT_SCHEMA_VERSION` bump). Writes are flushed one tsconfig project at a
|
|
2200
|
+
time under `structure.lock` (§7.6), kept as a short batch strictly separate
|
|
2201
|
+
from the compiler-heavy classification loop (which holds no lock). A JIT
|
|
2202
|
+
re-parse from a concurrent read tool is never starved behind either phase —
|
|
2203
|
+
it no longer acquires `structure.lock` at all (§7.6, §9.0), so this
|
|
2204
|
+
separation now matters only for coarse-writer-vs-coarse-writer contention
|
|
2205
|
+
(e.g. a concurrent `mast_reindex`).
|
|
2206
|
+
|
|
2207
|
+
**Consumption.** `mast_callers` and `mast_rename_impact` (via the shared
|
|
2208
|
+
`collectPotentialMatches`) filter `non_call_site`/`resolves_to_different`
|
|
2209
|
+
candidates out of `potential_matches` and report honest counts in
|
|
2210
|
+
`summary.checker_classified_non_call_site` /
|
|
2211
|
+
`summary.checker_classified_different_declaration` (§9) — both `0` until
|
|
2212
|
+
`--checker` has run.
|
|
2213
|
+
|
|
2214
|
+
---
|
|
2215
|
+
|
|
2216
|
+
## 11. Hook Architecture
|
|
2217
|
+
|
|
2218
|
+
### 11.1 Primary Hook — `mast serve` Startup
|
|
2219
|
+
|
|
2220
|
+
Defined in full in §7.4. Summary: a four-step ladder that brings the whole index
|
|
2221
|
+
(graph + FTS) online in 2–4 seconds via a Docker-baked seed index (§13.8), with all
|
|
2222
|
+
11 tools registered and ready to serve as soon as Step 3 completes — there is no
|
|
2223
|
+
reduced-capability warm-up window. Step 4 then catches up any files changed since
|
|
2224
|
+
the seed was built, in the background.
|
|
2225
|
+
|
|
2226
|
+
This is the **only hook required for the SDD pipeline**.
|
|
2227
|
+
|
|
2228
|
+
### 11.2 Mid-Task Hook — `mast_reindex` (agent-controlled)
|
|
2229
|
+
|
|
2230
|
+
The agent calls this explicitly after writes. JIT staleness handling (§9.0) already
|
|
2231
|
+
keeps already-indexed files correct on read; `mast_reindex` is what makes a
|
|
2232
|
+
**brand-new** file or symbol discoverable by `mast_search`/`mast_callers`/etc. before
|
|
2233
|
+
the next scheduled or `--watch` reindex reaches it. The implement prompt instructs:
|
|
2234
|
+
|
|
2235
|
+
> After writing or editing files, call `mast_reindex` before any search query that
|
|
2236
|
+
> depends on symbols you just created. This is the only way to guarantee the index
|
|
2237
|
+
> reflects your changes within this task.
|
|
2238
|
+
|
|
2239
|
+
### 11.3 Optional Developer Hooks — Git
|
|
2240
|
+
|
|
2241
|
+
Installed by `mast install-hooks`:
|
|
2242
|
+
|
|
2243
|
+
**`.git/hooks/post-commit`**
|
|
2244
|
+
```bash
|
|
2245
|
+
#!/bin/sh
|
|
2246
|
+
mast index "$(git rev-parse --show-toplevel)" --incremental
|
|
2247
|
+
```
|
|
2248
|
+
|
|
2249
|
+
**`.git/hooks/post-checkout`**
|
|
2250
|
+
```bash
|
|
2251
|
+
#!/bin/sh
|
|
2252
|
+
mast index "$(git rev-parse --show-toplevel)" --incremental
|
|
2253
|
+
```
|
|
2254
|
+
|
|
2255
|
+
Not required for the automated SDD pipeline — the startup hook covers the same
|
|
2256
|
+
scenario (files changed since last index).
|
|
2257
|
+
|
|
2258
|
+
### 11.4 Optional Interactive Hook — `mast serve --watch`
|
|
2259
|
+
|
|
2260
|
+
Local interactive development has no equivalent of the container's startup
|
|
2261
|
+
ladder: git hooks are opt-in and fire only on commit/checkout, so a long-lived
|
|
2262
|
+
interactive session can leave newly-created files and symbols undiscoverable
|
|
2263
|
+
between explicit `mast_reindex` calls, even though JIT re-parse (§9.0) keeps
|
|
2264
|
+
every already-indexed file correct on read. `--watch` closes that gap as an
|
|
2265
|
+
**opt-in** flag — it is a discovery-freshness optimization, never a correctness
|
|
2266
|
+
mechanism, and it does not reopen the §3 no-daemon non-goal (it lives and dies
|
|
2267
|
+
with the serve process).
|
|
2268
|
+
|
|
2269
|
+
Behaviour:
|
|
2270
|
+
|
|
2271
|
+
- A chokidar watcher covers `file_extensions` under the project root,
|
|
2272
|
+
respecting `exclude_patterns` **and the state directory itself** — watching
|
|
2273
|
+
the state dir would self-trigger on every index write.
|
|
2274
|
+
- Events are debounced (~500ms) and coalesced: rapid saves of one file collapse
|
|
2275
|
+
to a single entry; distinct files within the window share one batch.
|
|
2276
|
+
- Each batch runs the existing **incremental indexer** (§7.1) (acquiring
|
|
2277
|
+
`structure.lock` exactly as `mast_reindex` does — deleted files are cleaned
|
|
2278
|
+
up by the manifest diff).
|
|
2279
|
+
- **Single-flight:** events arriving during an in-flight run queue a follow-up
|
|
2280
|
+
run; runs never overlap.
|
|
2281
|
+
- **Lock contention:** a failed run (e.g. `structure.lock` held by
|
|
2282
|
+
`mast_reindex`) is logged and the batch requeued for the next debounce tick;
|
|
2283
|
+
after 3 consecutive failures the batch is dropped **with a warning** (JIT
|
|
2284
|
+
keeps existing-file reads correct, so a drop only delays discovery of new
|
|
2285
|
+
files/symbols).
|
|
2286
|
+
- **Degradation:** watcher construction failure (EMFILE, permissions) or
|
|
2287
|
+
runtime watcher errors log a warning and the server keeps serving without
|
|
2288
|
+
watch. `--watch` can never take down MCP serving.
|
|
2289
|
+
- Shutdown: the watcher closes on stdin close, SIGTERM, and SIGINT.
|
|
2290
|
+
|
|
2291
|
+
---
|
|
2292
|
+
|
|
2293
|
+
## 12. SDD Pipeline Integration
|
|
2294
|
+
|
|
2295
|
+
### 12.1 One-Time Setup
|
|
2296
|
+
|
|
2297
|
+
Add to `kluster init` or run manually once per project:
|
|
2298
|
+
|
|
2299
|
+
```bash
|
|
2300
|
+
mast init /path/to/app --state-dir .kluster/.mast
|
|
2301
|
+
```
|
|
2302
|
+
|
|
2303
|
+
### 12.2 `implement-task.md` Prompt Changes
|
|
2304
|
+
|
|
2305
|
+
Replace the existing `codemogger` MANDATORY FIRST ACTION block with:
|
|
2306
|
+
|
|
2307
|
+
```markdown
|
|
2308
|
+
## MANDATORY FIRST ACTION — NO EXCEPTIONS
|
|
2309
|
+
|
|
2310
|
+
Before writing any code, before opening any file:
|
|
2311
|
+
|
|
2312
|
+
1. Get a project overview:
|
|
2313
|
+
mast_project_skeleton — see all files and their exported symbols in one call
|
|
2314
|
+
|
|
2315
|
+
2. Run at least 2 targeted searches:
|
|
2316
|
+
mast_search: { "query": "<relevant symbols or concepts>", "only_exported": true }
|
|
2317
|
+
mast_search: { "query": "existing patterns conventions types" }
|
|
2318
|
+
|
|
2319
|
+
3. Use mast_exports to inspect a file's API before opening it:
|
|
2320
|
+
mast_exports: { "file_path": "src/services/auth/index.ts" }
|
|
2321
|
+
|
|
2322
|
+
4. Use mast_signature to get a function's contract (includes parameter type shapes):
|
|
2323
|
+
mast_signature: { "symbol": "handleLogin" }
|
|
2324
|
+
|
|
2325
|
+
After writing or editing files, call mast_reindex before searching for symbols
|
|
2326
|
+
you just created:
|
|
2327
|
+
mast_reindex: {}
|
|
2328
|
+
```
|
|
2329
|
+
|
|
2330
|
+
### 12.3 What the BT Pipeline Does Not Need to Change
|
|
2331
|
+
|
|
2332
|
+
- No new `execute-command` nodes in any YAML.
|
|
2333
|
+
- No changes to `run-task.yaml` or `implement.yaml`.
|
|
2334
|
+
- The only addition is the `mast` entry in the claude-runner `.mcp.json`.
|
|
2335
|
+
|
|
2336
|
+
---
|
|
2337
|
+
|
|
2338
|
+
## 13. Implementation Notes
|
|
2339
|
+
|
|
2340
|
+
### 13.1 Language
|
|
2341
|
+
|
|
2342
|
+
TypeScript (Node.js LTS). Rationale:
|
|
2343
|
+
|
|
2344
|
+
- Fits the existing monorepo stack — no second language in the container.
|
|
2345
|
+
- The MCP TypeScript SDK (`@modelcontextprotocol/sdk`) is Anthropic's primary SDK
|
|
2346
|
+
and has the best type safety and first-class support.
|
|
2347
|
+
- `better-sqlite3`'s synchronous API removes async complexity from graph queries;
|
|
2348
|
+
the recursive CTEs are blocking operations anyway.
|
|
2349
|
+
|
|
2350
|
+
### 13.2 Dependency Map
|
|
2351
|
+
|
|
2352
|
+
| Concern | Package | Notes |
|
|
2353
|
+
|---|---|---|
|
|
2354
|
+
| MCP server | `@modelcontextprotocol/sdk` | Official TS SDK, stdio transport |
|
|
2355
|
+
| BM25 | SQLite FTS5 (built-in) | Replaces external BM25 dependency entirely; `trigram` tokenizer for code identifiers |
|
|
2356
|
+
| Knowledge graph | `better-sqlite3` + `@types/better-sqlite3` | Synchronous API, WAL mode, recursive CTEs |
|
|
2357
|
+
| Query builder | `kysely` | Typed SQL query builder over the `better-sqlite3` connection |
|
|
2358
|
+
| AST parsing | `tree-sitter` + `tree-sitter-typescript` | Official Node.js bindings + TypeScript grammar |
|
|
2359
|
+
| Path resolution | `tsconfig-paths` | Resolves tsconfig `paths` aliases at index time |
|
|
2360
|
+
| Validation | `zod` | MCP tool inputs cross a trust boundary; validate `symbol`, `file_path`, `max_depth`, etc. before hitting the DB |
|
|
2361
|
+
| Locking | `proper-lockfile` | PID-based advisory lock; set `stale: 10000` (10s) to handle abrupt container exits |
|
|
2362
|
+
| CLI | `commander` | Standard TS CLI |
|
|
2363
|
+
| File walking | `fast-glob` | Glob pattern support for `exclude_patterns` |
|
|
2364
|
+
| File watching | `chokidar` | Powers `mast serve --watch` (§11.4) |
|
|
2365
|
+
| Token counting | `@anthropic-ai/tokenizer` | Counts `tokens_returned`/`tokens_full_file_upper_bound` for `_stats` (§14.5) |
|
|
2366
|
+
| Identifiers | `uuid` | Per-`mast serve`-session `session_id` for metrics attribution |
|
|
2367
|
+
|
|
2368
|
+
### 13.4 Project Structure
|
|
2369
|
+
|
|
2370
|
+
```
|
|
2371
|
+
packages/mast/
|
|
2372
|
+
├── src/
|
|
2373
|
+
│ ├── cli/
|
|
2374
|
+
│ │ ├── index.ts # commander entry point — mast <command>
|
|
2375
|
+
│ │ ├── init.ts
|
|
2376
|
+
│ │ ├── index-cmd.ts # `mast index` (avoids conflict with src/index.ts)
|
|
2377
|
+
│ │ ├── serve.ts
|
|
2378
|
+
│ │ ├── status.ts
|
|
2379
|
+
│ │ ├── metrics-cmd.ts # `mast metrics --since|--rollup|--vacuum` (§14)
|
|
2380
|
+
│ │ └── install-hooks.ts
|
|
2381
|
+
│ ├── mcp/
|
|
2382
|
+
│ │ ├── server.ts # MCP server setup, tool registration (§7.4 Steps 3-4)
|
|
2383
|
+
│ │ ├── startup.ts # bootstrap + schema-version guard + orphan-state cleanup (§7.4 Steps 1-2)
|
|
2384
|
+
│ │ ├── staleness.ts # stat-and-sync wrapper for all read tools (§9.0)
|
|
2385
|
+
│ │ ├── context.ts # AppContext (db, chunkStore, config, sessionId) shared by every tool
|
|
2386
|
+
│ │ └── tools/
|
|
2387
|
+
│ │ ├── search.ts # §9 mast_search — fused BM25 + ranker D
|
|
2388
|
+
│ │ ├── project-skeleton.ts
|
|
2389
|
+
│ │ ├── exports.ts
|
|
2390
|
+
│ │ ├── signature.ts
|
|
2391
|
+
│ │ ├── callers.ts # verified + potential partition (§9 mast_callers)
|
|
2392
|
+
│ │ ├── dependencies.ts
|
|
2393
|
+
│ │ ├── implementors.ts
|
|
2394
|
+
│ │ ├── rename-impact.ts # §9 mast_rename_impact
|
|
2395
|
+
│ │ ├── reindex.ts
|
|
2396
|
+
│ │ ├── status.ts
|
|
2397
|
+
│ │ └── efficiency.ts # mast_efficiency telemetry tool (§9, §14)
|
|
2398
|
+
│ ├── indexer/
|
|
2399
|
+
│ │ ├── index.ts # orchestrates the single indexing pass (§7.1)
|
|
2400
|
+
│ │ ├── walker.ts # file discovery, exclude pattern matching, manifest diff
|
|
2401
|
+
│ │ ├── watcher.ts # chokidar-backed `mast serve --watch` (§11.4)
|
|
2402
|
+
│ │ └── import-resolver.ts # tsconfig paths + pnpm workspace resolution (§13.7)
|
|
2403
|
+
│ ├── graph/
|
|
2404
|
+
│ │ ├── db.ts # better-sqlite3 + Kysely connection, schema init
|
|
2405
|
+
│ │ ├── populate.ts # AST → graph.db inserts (two-pass edge insertion, §10.3)
|
|
2406
|
+
│ │ ├── queries.ts # callers, implementors, dependencies, type-context
|
|
2407
|
+
│ │ ├── local-type-env.ts # POTENTIAL_CALL resolver heuristics (§10.3.1)
|
|
2408
|
+
│ │ └── checker-resolver.ts # opt-in `mast index --checker` pass (§10.3.2)
|
|
2409
|
+
│ ├── ast/
|
|
2410
|
+
│ │ ├── parser.ts # tree-sitter setup, parse file → AST
|
|
2411
|
+
│ │ ├── extractor.ts # LanguageExtractor contract + FileExtraction types
|
|
2412
|
+
│ │ ├── extract.ts # extension dispatch → per-language extractor
|
|
2413
|
+
│ │ ├── extractors/
|
|
2414
|
+
│ │ │ ├── typescript.ts # TS/JS: class-shell synth, method walk, hashes, splitting (§10.1)
|
|
2415
|
+
│ │ │ └── markdown.ts # heading-based doc chunking (§10.1)
|
|
2416
|
+
│ │ └── types.ts # Chunk, Export, SignatureResult, config, MCP I/O shared types
|
|
2417
|
+
│ ├── store/
|
|
2418
|
+
│ │ ├── sqliteChunkStore.ts # ChunkStore: chunk CRUD against graph.db's `chunks` table
|
|
2419
|
+
│ │ ├── config.ts # config resolution, index.json read/write
|
|
2420
|
+
│ │ ├── lock.ts # structure.lock manager (§7.6)
|
|
2421
|
+
│ │ └── lockMetrics.ts # JSONL lock-hold telemetry sink
|
|
2422
|
+
│ ├── search/
|
|
2423
|
+
│ │ ├── fused.ts # RRF fusion of BM25 + ranker D (§7.3); D-fire telemetry
|
|
2424
|
+
│ │ ├── declex.ts # ranker D — declaration-exact match (§7.3)
|
|
2425
|
+
│ │ ├── fts.ts # FTS5 queries (chunk_fts BM25 + identifier_fts exact match)
|
|
2426
|
+
│ │ └── potential-matches.ts # shared candidate collection for mast_callers/mast_rename_impact
|
|
2427
|
+
│ └── telemetry/
|
|
2428
|
+
│ ├── metrics.ts # metrics table writes, _stats meta builder (§14)
|
|
2429
|
+
│ └── tokenizer.ts # @anthropic-ai/tokenizer wrapper for counterfactuals
|
|
2430
|
+
├── package.json
|
|
2431
|
+
└── tsconfig.json
|
|
2432
|
+
```
|
|
2433
|
+
|
|
2434
|
+
**Language extensibility pattern:** `extract.ts` dispatches by file extension to a
|
|
2435
|
+
per-language extractor module in `ast/extractors/`. Each extractor implements the
|
|
2436
|
+
`LanguageExtractor` contract (defined in `ast/extractor.ts`) and owns its **full**
|
|
2437
|
+
extraction story — parsing strategy included:
|
|
2438
|
+
|
|
2439
|
+
```typescript
|
|
2440
|
+
interface LanguageExtractor {
|
|
2441
|
+
language: Language; // "typescript" | "markdown" | ...
|
|
2442
|
+
extensions: readonly string[]; // [".ts", ".tsx"]
|
|
2443
|
+
extract(src: string, filePath: string, fileMtime: number,
|
|
2444
|
+
options: ExtractorOptions): FileExtraction;
|
|
2445
|
+
}
|
|
2446
|
+
|
|
2447
|
+
interface FileExtraction {
|
|
2448
|
+
language: Language; // concrete language of THIS file
|
|
2449
|
+
chunks: readonly Chunk[];
|
|
2450
|
+
symbols: readonly SymbolRecord[]; // empty for graph-less languages
|
|
2451
|
+
imports: readonly ImportRecord[];
|
|
2452
|
+
edges: readonly EdgeRecord[];
|
|
2453
|
+
identifierRows: readonly IdentifierRow[]; // identifier_fts rows per chunk
|
|
2454
|
+
}
|
|
2455
|
+
```
|
|
2456
|
+
|
|
2457
|
+
The contract is deliberately parser-agnostic: the TypeScript extractor parses with
|
|
2458
|
+
tree-sitter internally (and keeps `declarationHash`/`bodyHash` as its own methods),
|
|
2459
|
+
while the markdown extractor line-scans — the pipeline never sees a `Tree` and
|
|
2460
|
+
never branches on language. `identifierRows` are produced by the extractor rather
|
|
2461
|
+
than the graph layer because what counts as an "identifier" is a language-level
|
|
2462
|
+
judgment: markdown contributes none, since `identifier_fts` feeds `mast_callers`
|
|
2463
|
+
potential_matches and prose mentions are not call sites.
|
|
2464
|
+
|
|
2465
|
+
Two extractors ship today: `typescript.ts` (`.ts`, `.tsx`, `.js`, `.jsx`) and
|
|
2466
|
+
`markdown.ts` (`.md`, §10.1 doc chunking). Adding Go or Python means adding a new
|
|
2467
|
+
extractor module and a `tree-sitter-<lang>` package — no changes to core indexer
|
|
2468
|
+
logic. Do not use tree-sitter `.scm` query files; the extractor function approach
|
|
2469
|
+
is sufficient and keeps the build simple.
|
|
2470
|
+
|
|
2471
|
+
### 13.7 Path Resolution
|
|
2472
|
+
|
|
2473
|
+
Monorepo imports use two alias systems that both need resolving to physical file paths
|
|
2474
|
+
for the knowledge graph edges and `type_context` lookups to work:
|
|
2475
|
+
|
|
2476
|
+
**0. NodeNext `.js` specifier substitution** (e.g. `./repo.js` → `./repo.ts`)
|
|
2477
|
+
|
|
2478
|
+
TypeScript ESM/NodeNext code writes the *compiled* extension in relative specifiers
|
|
2479
|
+
(`import { Repo } from './repo.js'`) even though the on-disk source is `./repo.ts`.
|
|
2480
|
+
When a relative specifier carries a JS-family extension, the resolver looks up the
|
|
2481
|
+
TypeScript source first and only falls back to the literal file, matching tsc's
|
|
2482
|
+
"file extension substitution" lookup order:
|
|
2483
|
+
|
|
2484
|
+
| Specifier ext | Lookup order |
|
|
2485
|
+
| ------------- | ----------------------------------- |
|
|
2486
|
+
| `.js` | `.ts`, then `.tsx`, then `.js` |
|
|
2487
|
+
| `.jsx` | `.tsx`, then `.jsx` |
|
|
2488
|
+
| `.mjs` | `.mts`, then `.mjs` |
|
|
2489
|
+
| `.cjs` | `.cts`, then `.cjs` |
|
|
2490
|
+
|
|
2491
|
+
The source-first precedence means that when both `x.ts` and a real `x.js` exist,
|
|
2492
|
+
`./x.js` resolves to `x.ts` (the `.js` names the *output*). A genuine `.js` file with
|
|
2493
|
+
no TypeScript source still resolves to itself. Declaration files (`.d.ts`) are out of
|
|
2494
|
+
scope — MAST indexes implementation files. Without this rule, ESM `.js` specifiers left
|
|
2495
|
+
`resolved_path` NULL and star re-export barrels written with `.js` produced no
|
|
2496
|
+
`re_export_files` rows. See the TypeScript Modules Reference, "File extension
|
|
2497
|
+
substitution".
|
|
2498
|
+
|
|
2499
|
+
**1. tsconfig `paths` aliases** (e.g. `@api/types` → `./src/types/index.ts`)
|
|
2500
|
+
|
|
2501
|
+
Read the nearest `tsconfig.json` at `mast init` time using `tsconfig-paths`. Build a
|
|
2502
|
+
resolver function `resolveAlias(alias: string, fromFile: string): string | null` that
|
|
2503
|
+
is passed into Phase 1 and used wherever `resolved_path` is written to the `imports`
|
|
2504
|
+
table.
|
|
2505
|
+
|
|
2506
|
+
**2. pnpm workspace package names** (e.g. `@kluster-kinetic-01/shared`)
|
|
2507
|
+
|
|
2508
|
+
Walk the workspace root `pnpm-workspace.yaml` (or `workspaces` field in root
|
|
2509
|
+
`package.json`) at startup. For each matched package directory, read its
|
|
2510
|
+
`package.json` `name` field. Build a map `{ packageName → packageDir }`. When an
|
|
2511
|
+
import module matches a package name, resolve to `<packageDir>/src/index.ts` (or the
|
|
2512
|
+
`main`/`exports` field in that package's `package.json`).
|
|
2513
|
+
|
|
2514
|
+
**pnpm symlink handling:** pnpm links workspace packages into `node_modules` as
|
|
2515
|
+
symlinks. A naive resolver may return a path under `node_modules/@pkg/shared` (the
|
|
2516
|
+
symlink) rather than `packages/shared/src/index.ts` (the real file). The `files`
|
|
2517
|
+
table indexes real paths, so a symlink path would produce a graph edge pointing to a
|
|
2518
|
+
path that does not exist in the index. Always call `fs.realpathSync()` on the resolved
|
|
2519
|
+
path before writing it to `resolved_path`. This collapses symlinks to their canonical
|
|
2520
|
+
source paths, ensuring graph edges connect correctly.
|
|
2521
|
+
|
|
2522
|
+
Both resolvers are composed: alias resolution runs first, workspace resolution second,
|
|
2523
|
+
`realpathSync` applied last. External modules (no match in either resolver) leave
|
|
2524
|
+
`resolved_path = NULL` in the `imports` table.
|
|
2525
|
+
|
|
2526
|
+
This resolver is initialised once at `mast serve` startup and at the start of each
|
|
2527
|
+
`mast index` run.
|
|
2528
|
+
|
|
2529
|
+
### 13.8 Dockerfile Pre-Warming
|
|
2530
|
+
|
|
2531
|
+
The index is pre-warmed during `docker build` to eliminate cold-start dead time on
|
|
2532
|
+
the first container run. Without this, the first task can stall for minutes while
|
|
2533
|
+
the index builds from scratch — a UX risk identified in the design review
|
|
2534
|
+
(Failure 4).
|
|
2535
|
+
|
|
2536
|
+
#### 13.8.1 Seed Index Pre-Warm
|
|
2537
|
+
|
|
2538
|
+
For a 5K-file class-heavy repo, indexing from scratch takes on the order of a
|
|
2539
|
+
minute or more. During that window the agent has a cold, empty index. The seed
|
|
2540
|
+
index moves this work into the Docker build, so the runtime container starts with
|
|
2541
|
+
a fully-warmed index for the build-time commit.
|
|
2542
|
+
|
|
2543
|
+
```dockerfile
|
|
2544
|
+
# After the application source is copied into the image and dependencies installed:
|
|
2545
|
+
RUN mast init /workspace --state-dir /opt/mast-seed --no-index \
|
|
2546
|
+
&& mast index /workspace --state-dir /opt/mast-seed
|
|
2547
|
+
```
|
|
2548
|
+
|
|
2549
|
+
Two important properties of the seed:
|
|
2550
|
+
|
|
2551
|
+
1. **The index runs fully at build time.** The seed contains a fully-populated
|
|
2552
|
+
`graph.db` (chunks, symbols, edges, `chunk_fts`, `identifier_fts`). The runtime
|
|
2553
|
+
container is ready to serve at full capability immediately (Step 3 of §7.4) — no
|
|
2554
|
+
warm-up window.
|
|
2555
|
+
2. **Frozen at build commit.** The seed reflects whatever code was in the image at
|
|
2556
|
+
`docker build` time. Files modified since the build commit are picked up by
|
|
2557
|
+
§7.4 Step 4's filesystem scan (a few seconds for typical incremental staleness),
|
|
2558
|
+
and uncommitted-tree edits trigger JIT re-parse on first query (§9.0).
|
|
2559
|
+
|
|
2560
|
+
**Runtime copy.** The container entrypoint runs:
|
|
2561
|
+
|
|
2562
|
+
```bash
|
|
2563
|
+
#!/bin/sh
|
|
2564
|
+
if [ -z "$(ls -A /workspace/.kluster/.mast 2>/dev/null)" ]; then
|
|
2565
|
+
cp -r /opt/mast-seed/. /workspace/.kluster/.mast/
|
|
2566
|
+
fi
|
|
2567
|
+
exec mast serve --state-dir /workspace/.kluster/.mast
|
|
2568
|
+
```
|
|
2569
|
+
|
|
2570
|
+
The copy is conditional on the workspace state being empty. Subsequent container
|
|
2571
|
+
starts on the same mounted volume reuse the existing state (which may have
|
|
2572
|
+
already-applied incremental updates from prior tasks).
|
|
2573
|
+
|
|
2574
|
+
**Seed commit field on `mast_status`.** When the seed is copied, `index.json`
|
|
2575
|
+
records `seed_commit: "<git-rev>"` so `mast_status` can report:
|
|
2576
|
+
|
|
2577
|
+
```
|
|
2578
|
+
seed_commit: abc1234 (built 2026-05-10T12:00:00Z)
|
|
2579
|
+
last_indexed: 2026-05-13T14:22:00Z
|
|
2580
|
+
files_since_seed: 47
|
|
2581
|
+
```
|
|
2582
|
+
|
|
2583
|
+
This makes "why does the index look stale?" debuggable without opening
|
|
2584
|
+
`file_manifest.json` by hand.
|
|
2585
|
+
|
|
2586
|
+
**Schema-bump invalidation.** On startup (§7.4 Step 2), if
|
|
2587
|
+
`index.json.schema_version != CURRENT_SCHEMA_VERSION`, both the runtime state
|
|
2588
|
+
AND the seed are considered invalid. The state is wiped and a full reindex runs
|
|
2589
|
+
in the background (Step 4). The seed in `/opt/mast-seed` is not re-built —
|
|
2590
|
+
that's the next image build's job — but it is ignored on this run.
|
|
2591
|
+
|
|
2592
|
+
### 13.10 What to Keep from mcp-vector-search (as reference only)
|
|
2593
|
+
|
|
2594
|
+
mcp-vector-search is a Python codebase — nothing is reused directly. It serves as a
|
|
2595
|
+
reference for:
|
|
2596
|
+
|
|
2597
|
+
- RRF fusion logic
|
|
2598
|
+
- Incremental indexing strategy (mtime-based staleness)
|
|
2599
|
+
|
|
2600
|
+
### 13.11 Distribution
|
|
2601
|
+
|
|
2602
|
+
```
|
|
2603
|
+
pip install # not applicable
|
|
2604
|
+
npm install -g mast-search # installs CLI + MCP server
|
|
2605
|
+
```
|
|
2606
|
+
|
|
2607
|
+
For the SDD pipeline, mast is installed into the claude-runner Docker image. The
|
|
2608
|
+
Docker-baked seed index (§13.8) is pre-built into the image layer to avoid a
|
|
2609
|
+
from-scratch index build inside a task container.
|
|
2610
|
+
|
|
2611
|
+
For external developers:
|
|
2612
|
+
|
|
2613
|
+
```bash
|
|
2614
|
+
npm install -g mast-search
|
|
2615
|
+
mast init . # initialise index
|
|
2616
|
+
claude mcp add mast -- mast serve # wire into Claude Code
|
|
2617
|
+
```
|
|
2618
|
+
|
|
2619
|
+
There is only one install shape — lexical + declaration-exact search is the whole
|
|
2620
|
+
product, so no `--no-embeddings`-style variant is needed.
|
|
2621
|
+
|
|
2622
|
+
---
|
|
2623
|
+
|
|
2624
|
+
## 14. Telemetry & Measurement
|
|
2625
|
+
|
|
2626
|
+
The MAST thesis is **chunks not files → fewer tokens per task**. If that claim is
|
|
2627
|
+
not measurable, the index and the Docker layers that support it are a complexity
|
|
2628
|
+
budget the project cannot defend. This section
|
|
2629
|
+
specifies the instrumentation that makes the savings legible to humans (`mast metrics`),
|
|
2630
|
+
visible to the agent (`mast_efficiency`), and persistent across sessions (the SQLite
|
|
2631
|
+
`metrics` table).
|
|
2632
|
+
|
|
2633
|
+
### 14.1 Goals
|
|
2634
|
+
|
|
2635
|
+
- **Defensible savings number.** When asked "is MAST worth it?", produce a chart-backed
|
|
2636
|
+
answer with a documented methodology, not a vibe.
|
|
2637
|
+
- **In-loop agent feedback.** The agent can call `mast_efficiency` to see whether its
|
|
2638
|
+
own tool usage was efficient, and self-correct on subsequent tasks.
|
|
2639
|
+
- **Per-task attribution.** The SDD pipeline (§12) captures one `mast metrics --session`
|
|
2640
|
+
snapshot per task so savings can be analysed by task type.
|
|
2641
|
+
- **Negligible overhead.** Instrumentation must add < 1ms per tool call; write-heavy,
|
|
2642
|
+
read-light. The one documented exception is `tokens_full_file_upper_bound` for
|
|
2643
|
+
whole-project-scale callers (`mast_project_skeleton`, §14.2): exact tokenization
|
|
2644
|
+
alone already exceeds 1ms per file, so this counterfactual is bounded instead by a
|
|
2645
|
+
per-call tokenization budget (F8, §14.2) — worst case ~0.7-0.8s on a first call
|
|
2646
|
+
against an uncached project, converging toward the < 1ms goal as the cache warms and
|
|
2647
|
+
degenerating to true negligible overhead (cache hits only) once fully warm.
|
|
2648
|
+
|
|
2649
|
+
### 14.2 The `_stats` Meta Field
|
|
2650
|
+
|
|
2651
|
+
Every read tool (`mast_search`, `mast_signature`, `mast_exports`,
|
|
2652
|
+
`mast_project_skeleton`, `mast_callers`, `mast_dependencies`, `mast_implementors`)
|
|
2653
|
+
attaches a `_stats` object to its response:
|
|
2654
|
+
|
|
2655
|
+
```json
|
|
2656
|
+
{
|
|
2657
|
+
"results": [ /* ... tool-specific shape ... */ ],
|
|
2658
|
+
"_stats": {
|
|
2659
|
+
"tool": "mast_search",
|
|
2660
|
+
"tokens_returned": 412,
|
|
2661
|
+
"tokens_full_file_upper_bound": 3187,
|
|
2662
|
+
"files_referenced": ["api/services/auth/src/handler.ts", "api/services/auth/src/repository.ts"],
|
|
2663
|
+
"efficiency_ratio": 0.871,
|
|
2664
|
+
"duration_ms": 38
|
|
2665
|
+
}
|
|
2666
|
+
}
|
|
2667
|
+
```
|
|
2668
|
+
|
|
2669
|
+
- `tokens_returned` — token count of the serialised response body using
|
|
2670
|
+
`@anthropic-ai/tokenizer` (see §14.5).
|
|
2671
|
+
- `tokens_full_file_upper_bound` — sum of `@anthropic-ai/tokenizer` counts over the
|
|
2672
|
+
full contents of every unique file referenced in the results. This is the "what
|
|
2673
|
+
a naive `Read` of every result file would have cost" counterfactual.
|
|
2674
|
+
- `efficiency_ratio` — `1 - (tokens_returned / tokens_full_file_upper_bound)`. Always
|
|
2675
|
+
in `[0, 1]`. Higher is better.
|
|
2676
|
+
- `duration_ms` — wall-clock time for the tool call, including JIT staleness
|
|
2677
|
+
re-parse if triggered.
|
|
2678
|
+
|
|
2679
|
+
**Honest framing.** `tokens_full_file_upper_bound` is explicitly labelled as an upper
|
|
2680
|
+
bound. A smart agent using `Grep -A 10 -B 10` would have used fewer tokens than the
|
|
2681
|
+
upper bound but more than MAST — the real saving sits between zero and `efficiency_ratio`.
|
|
2682
|
+
Reporting the upper bound is defensible because the methodology is documented and
|
|
2683
|
+
the label is honest. Reporting "X% savings" with no upper-bound qualifier would not
|
|
2684
|
+
survive scrutiny.
|
|
2685
|
+
|
|
2686
|
+
**Implementation status.** `tokens_full_file_upper_bound` is computed — for each
|
|
2687
|
+
unique file in `files_referenced`, `estimateFullFileBound` (`telemetry/tokenizer.ts`)
|
|
2688
|
+
reads the file's full contents from the project root and sums `countTokens` over
|
|
2689
|
+
them, with an mtime-keyed cache so repeated calls against an unchanged file don't
|
|
2690
|
+
re-tokenize. It previously shipped as an unimplemented stub that always returned 0,
|
|
2691
|
+
which made `efficiency_ratio` a constant 0 across every recorded row (see the
|
|
2692
|
+
Promotion Log, 2026-07-15) — that regression is what this fixes.
|
|
2693
|
+
|
|
2694
|
+
**Per-call work cap (F8, 2026-08-07).** Reading and tokenizing every referenced file
|
|
2695
|
+
does not scale to a caller like `mast_project_skeleton`, which references every file
|
|
2696
|
+
in the project — `estimateFullFileBound` measured ~28s/call on a 1,334-file project,
|
|
2697
|
+
99% of it here. Beyond a per-call budget (`FULL_FILE_TOKENIZE_BUDGET_PER_CALL = 32`
|
|
2698
|
+
exact reads per call, cache hits excluded), further cache-miss files are **not** read
|
|
2699
|
+
or tokenized; they are size-estimated instead, at
|
|
2700
|
+
`Math.ceil(sizeBytes / BYTES_PER_TOKEN_ESTIMATE)` bytes-per-token
|
|
2701
|
+
(`BYTES_PER_TOKEN_ESTIMATE = 4`, the standard heuristic for source text) and are not
|
|
2702
|
+
cached — an estimate must never masquerade as an exact cached count. Successive calls
|
|
2703
|
+
over the same file set progressively convert estimates to exact, cached counts as the
|
|
2704
|
+
budget reaches further into the set, converging to fully-exact after enough calls with
|
|
2705
|
+
zero cache thrash (`FULL_FILE_BOUND_CACHE_LIMIT` raised from 200 to 8192 alongside the
|
|
2706
|
+
budget — see `telemetry/tokenizer.ts` doc comments). The upper-bound counterfactual was
|
|
2707
|
+
already explicitly approximate (see "Honest framing" above and §14.5); the size
|
|
2708
|
+
estimate for budget-exceeding files is an additional, documented layer of the same
|
|
2709
|
+
approximation, not a departure from it.
|
|
2710
|
+
|
|
2711
|
+
### 14.3 The `metrics` Table
|
|
2712
|
+
|
|
2713
|
+
Telemetry persists in `graph.db` (same SQLite database as the knowledge graph; one
|
|
2714
|
+
fewer connection to manage):
|
|
2715
|
+
|
|
2716
|
+
```sql
|
|
2717
|
+
CREATE TABLE IF NOT EXISTS metrics (
|
|
2718
|
+
id INTEGER PRIMARY KEY,
|
|
2719
|
+
tool_name TEXT NOT NULL,
|
|
2720
|
+
call_timestamp REAL NOT NULL, -- unix epoch seconds (REAL for sub-second precision)
|
|
2721
|
+
tokens_returned INTEGER NOT NULL,
|
|
2722
|
+
tokens_full_file_upper_bound INTEGER NOT NULL,
|
|
2723
|
+
duration_ms INTEGER NOT NULL,
|
|
2724
|
+
mode TEXT, -- historical: pre-2026-08-06 rows only ("hybrid"|"lexical"); new rows NULL
|
|
2725
|
+
session_id TEXT NOT NULL, -- uuid set at mast serve startup
|
|
2726
|
+
status TEXT NOT NULL, -- "ok" | "stale_returned" | "error"
|
|
2727
|
+
args_json TEXT, -- salient tool arguments, capped at 1,000 chars
|
|
2728
|
+
results_json TEXT, -- {file_path, symbol_name} identity pairs, capped at 20 entries
|
|
2729
|
+
declex_json TEXT -- ranker D fire telemetry (mast_search only), NULL when D silent
|
|
2730
|
+
);
|
|
2731
|
+
|
|
2732
|
+
CREATE INDEX IF NOT EXISTS idx_metrics_timestamp ON metrics(call_timestamp);
|
|
2733
|
+
CREATE INDEX IF NOT EXISTS idx_metrics_tool ON metrics(tool_name);
|
|
2734
|
+
CREATE INDEX IF NOT EXISTS idx_metrics_session ON metrics(session_id);
|
|
2735
|
+
|
|
2736
|
+
CREATE TABLE IF NOT EXISTS metrics_daily (
|
|
2737
|
+
-- Pre-aggregated rollup; populated by `mast metrics --rollup`.
|
|
2738
|
+
day TEXT PRIMARY KEY, -- ISO date, e.g. "2026-05-13"
|
|
2739
|
+
tool_name TEXT NOT NULL,
|
|
2740
|
+
calls INTEGER NOT NULL,
|
|
2741
|
+
tokens_returned_total INTEGER NOT NULL,
|
|
2742
|
+
tokens_full_file_total INTEGER NOT NULL,
|
|
2743
|
+
avg_duration_ms REAL NOT NULL,
|
|
2744
|
+
PRIMARY KEY (day, tool_name)
|
|
2745
|
+
);
|
|
2746
|
+
```
|
|
2747
|
+
|
|
2748
|
+
**Writes are non-blocking.** `metrics.ts` enqueues writes on a per-tick batch
|
|
2749
|
+
(flushed every 1s or every 100 rows, whichever comes first) so the metrics path
|
|
2750
|
+
never blocks a tool response. Worst-case data loss on abrupt container exit is one
|
|
2751
|
+
flush window's worth of rows; acceptable for a savings metric.
|
|
2752
|
+
|
|
2753
|
+
**Argument/result identity columns (`args_json`, `results_json`).** Added additively
|
|
2754
|
+
(`ALTER TABLE metrics ADD COLUMN`, same precedent as `edges.resolution`/`call_line`/
|
|
2755
|
+
`context` — no `CURRENT_SCHEMA_VERSION` bump) to make the "linked chain" measurable:
|
|
2756
|
+
did a later `mast_signature`/`mast_exports`/`mast_callers` call target a symbol or
|
|
2757
|
+
file that an earlier `mast_search` returned in the same session? This was the missing
|
|
2758
|
+
instrumentation the `mast_capsule` v2 hold identified (Promotion Log, 2026-07-15) —
|
|
2759
|
+
without it, a capsule chain-rate measurement can only be an argument-blind upper
|
|
2760
|
+
bound. `args_json` carries the salient tool arguments (query + filters for search;
|
|
2761
|
+
symbol and/or file_path for signature/exports/callers), capped at 1,000 characters;
|
|
2762
|
+
`results_json` carries the tool's returned `{file_path, symbol_name}` identity pairs
|
|
2763
|
+
in rank order, capped at 20 entries. Both caps are stated honestly in the payload
|
|
2764
|
+
when hit (`_truncated`) rather than silently cut. Wired for `mast_search`,
|
|
2765
|
+
`mast_signature`, `mast_exports`, and `mast_callers` — the chain-analysis tools the
|
|
2766
|
+
capsule decision depends on; both columns are `NULL` for every other tool and for
|
|
2767
|
+
rows recorded before this migration.
|
|
2768
|
+
|
|
2769
|
+
**`declex_json` (F18 D-fire telemetry — M2 decision memo condition 3).** Added
|
|
2770
|
+
via the same additive `ALTER TABLE` precedent (no schema bump). Populated only
|
|
2771
|
+
on `mast_search` calls where ranker D (§7.3) actually fired; `NULL` when D was
|
|
2772
|
+
silent, when `declaration_exact_ranker` is off, for every other tool, and for
|
|
2773
|
+
pre-migration rows. Shape: `{fired: true, top_match_channel: "full"|"segment",
|
|
2774
|
+
candidate_count, window_effects: [{chunk_id, symbol_name, rank_with_d,
|
|
2775
|
+
rank_without_d}], _truncated?}`. `window_effects` is a dual-fusion diff computed
|
|
2776
|
+
in-memory per call — the fused (pre-dedup) rank of each affected chunk with D's
|
|
2777
|
+
list included vs excluded from RRF; ranks are the actual positions in each list
|
|
2778
|
+
(`null` only when the chunk is absent from that list entirely, e.g. a D-only
|
|
2779
|
+
anchor has `rank_without_d: null`), capped at 10 entries with a top-level
|
|
2780
|
+
`_truncated` count. This column is the input signal for the F18 kill-switch and
|
|
2781
|
+
the M2 re-entry criteria: fire rate on real queries, and whether D demotes
|
|
2782
|
+
in-window targets, are both answerable from it without re-instrumenting.
|
|
2783
|
+
|
|
2784
|
+
### 14.4 Rotation Policy
|
|
2785
|
+
|
|
2786
|
+
A 5K-file repo with ~50 tool calls per task × ~30 tasks per day produces ~1,500
|
|
2787
|
+
metrics rows per day. The table needs rotation to keep `SUM()` queries fast.
|
|
2788
|
+
|
|
2789
|
+
- `mast metrics --rollup` (run weekly via a cron-style hook, or manually): aggregates
|
|
2790
|
+
rows older than 30 days into the `metrics_daily` table, then deletes the raw rows.
|
|
2791
|
+
- `mast metrics --vacuum`: runs `VACUUM` on `graph.db` after a rollup to reclaim
|
|
2792
|
+
space.
|
|
2793
|
+
- The rollup is non-destructive at the daily level: 90-day or 1-year historical
|
|
2794
|
+
analysis still works against `metrics_daily`.
|
|
2795
|
+
|
|
2796
|
+
### 14.5 Tokenizer Choice
|
|
2797
|
+
|
|
2798
|
+
`@anthropic-ai/tokenizer` (`^0.0.4`) counts `tokens_returned` and
|
|
2799
|
+
`tokens_full_file_upper_bound` — but it is **approximate, not ground truth**.
|
|
2800
|
+
The package implements the Claude 2-era tokenizer, and Anthropic never
|
|
2801
|
+
published the Claude 3+ vocabularies, so absolute counts drift for every model
|
|
2802
|
+
that actually consumes MAST output today.
|
|
2803
|
+
|
|
2804
|
+
```typescript
|
|
2805
|
+
import { countTokens } from "@anthropic-ai/tokenizer";
|
|
2806
|
+
const n = countTokens(responseBodyAsString);
|
|
2807
|
+
```
|
|
2808
|
+
|
|
2809
|
+
**Why this is still the right mechanism.** §14.2's headline number is the
|
|
2810
|
+
savings *ratio*, and both its numerator and denominator are counted with the
|
|
2811
|
+
same tokenizer — the per-count error mostly cancels, so the ratio is robust
|
|
2812
|
+
even though the absolute counts are not. The same honesty rule that governs
|
|
2813
|
+
the upper-bound counterfactual (§14.2) applies here: report the limitation,
|
|
2814
|
+
don't paper over it.
|
|
2815
|
+
|
|
2816
|
+
**The label.** The active tokenizer is reported verbatim in `mast_efficiency`'s
|
|
2817
|
+
`tokenizer` field and in `mast metrics`'s footer as:
|
|
2818
|
+
|
|
2819
|
+
```
|
|
2820
|
+
@anthropic-ai/tokenizer (claude-2 era, approximate for current models)
|
|
2821
|
+
```
|
|
2822
|
+
|
|
2823
|
+
The string has one definition (`TOKENIZER_LABEL` in `src/telemetry/tokenizer.ts`)
|
|
2824
|
+
that every consumer reads, so the wording cannot drift between surfaces.
|
|
2825
|
+
|
|
2826
|
+
**Future seams (not implemented).** An exact mode via the Anthropic API's
|
|
2827
|
+
`count_tokens` endpoint (opt-in, requires an API key) and a `tiktoken
|
|
2828
|
+
cl100k_base` fallback for non-Claude consumers are both documented options;
|
|
2829
|
+
neither ships today, and if either is added the reported label must change to
|
|
2830
|
+
match the active counter.
|
|
2831
|
+
|
|
2832
|
+
### 14.6 CLI: `mast metrics`
|
|
2833
|
+
|
|
2834
|
+
```
|
|
2835
|
+
Usage: mast metrics [options]
|
|
2836
|
+
|
|
2837
|
+
Options:
|
|
2838
|
+
--since <duration> e.g. "1h", "24h", "7d", "30d" (default: 7d)
|
|
2839
|
+
--by-tool Break down by tool_name (default: true)
|
|
2840
|
+
--rollup Aggregate rows older than --keep-days into metrics_daily, then delete raw rows
|
|
2841
|
+
--vacuum Delete daily roll-up rows older than --keep-days
|
|
2842
|
+
--keep-days <n> Retention days for --rollup / --vacuum (default: 7 for rollup, 90 for vacuum)
|
|
2843
|
+
--locks Summarize structure-lock hold/wait timing by caller (D6) — reads
|
|
2844
|
+
store/lockMetrics.ts's `<state_dir>/lock-metrics.jsonl`; does not
|
|
2845
|
+
touch graph.db. Prints "No lock metrics recorded." (exit 0) when
|
|
2846
|
+
the file is missing or empty; malformed lines are skipped and
|
|
2847
|
+
counted, never fatal.
|
|
2848
|
+
--json Machine-readable output
|
|
2849
|
+
--state-dir <dir> State directory override
|
|
2850
|
+
```
|
|
2851
|
+
|
|
2852
|
+
Sample output (`mast metrics --since 7d --by-tool`):
|
|
2853
|
+
|
|
2854
|
+
```
|
|
2855
|
+
Tool Calls Tokens Avg ms p50 ms p95 ms Efficiency
|
|
2856
|
+
──────────────────────────────────────────────────────────────────────────────
|
|
2857
|
+
mast_search 1,847 742,103 41.2 32 89 87.4%
|
|
2858
|
+
mast_signature 912 91,820 18.7 15 47 95.6%
|
|
2859
|
+
mast_exports 408 33,041 12.4 10 31 96.3%
|
|
2860
|
+
mast_project_skeleton 127 8,209 9.1 8 22 98.0%
|
|
2861
|
+
mast_callers 321 42,118 22.5 19 58 95.7%
|
|
2862
|
+
mast_dependencies 156 12,047 14.0 11 35 97.6%
|
|
2863
|
+
mast_implementors 88 6,213 10.8 9 26 96.9%
|
|
2864
|
+
|
|
2865
|
+
Tokenizer: @anthropic-ai/tokenizer (claude-2 era, approximate for current models)
|
|
2866
|
+
```
|
|
2867
|
+
|
|
2868
|
+
`p50`/`p95` (D6 — `adr/009-2026-08-10-measurement-harness.md`) are nearest-rank percentiles (sort
|
|
2869
|
+
ascending, take the value at rank `ceil(P/100 * N)`, no interpolation) computed in JS
|
|
2870
|
+
over the window's raw `duration_ms` values — `--by-tool`'s existing `Avg ms` column
|
|
2871
|
+
alone hides tail latency (this is the column that would have caught F8's 28 s
|
|
2872
|
+
outlier). `--json` emits the same rows as an array, each carrying
|
|
2873
|
+
`p50_duration_ms`/`p95_duration_ms` alongside the existing `avg_duration_ms`.
|
|
2874
|
+
|
|
2875
|
+
Sample output (`mast metrics --locks`):
|
|
2876
|
+
|
|
2877
|
+
```
|
|
2878
|
+
Caller Count Hold p50 Hold p95 Hold max Wait p50 Wait p95 Wait max Failed
|
|
2879
|
+
────────────────────────────────────────────────────────────────────────────────────────
|
|
2880
|
+
index-run 42 18 47 61 2 6 9 0
|
|
2881
|
+
jit-staleness 301 1 4 9 0 1 3 2
|
|
2882
|
+
|
|
2883
|
+
Warning: skipped 1 malformed line(s) in lock-metrics.jsonl.
|
|
2884
|
+
```
|
|
2885
|
+
|
|
2886
|
+
`Hold *` comes from `LockEvent`'s `released.holdMs`; `Wait *` from `acquired.waitMs`;
|
|
2887
|
+
`Failed` counts `failed` events (acquisition attempts that exhausted retries). The
|
|
2888
|
+
malformed-line warning only appears when at least one JSONL line failed to parse or
|
|
2889
|
+
didn't match the `LockEvent` shape — those lines are skipped, never fatal.
|
|
2890
|
+
|
|
2891
|
+
### 14.7 SDD Pipeline Integration
|
|
2892
|
+
|
|
2893
|
+
The SDD task pipeline (§12) adds one step to its task wrap-up:
|
|
2894
|
+
|
|
2895
|
+
```yaml
|
|
2896
|
+
# packages/workbench/sdd/flows/strategies/task-pipeline/run-task.yaml (excerpt)
|
|
2897
|
+
- execute-command:
|
|
2898
|
+
name: capture_mast_metrics
|
|
2899
|
+
cmd: mast metrics --session --json > {task_history_dir}/mast-metrics.json
|
|
2900
|
+
run_after: [implement_task, review_task]
|
|
2901
|
+
```
|
|
2902
|
+
|
|
2903
|
+
This produces one `mast-metrics.json` per task in the per-task history directory.
|
|
2904
|
+
Retrospective analysis ("does MAST save more on bug fixes than on feature additions?")
|
|
2905
|
+
becomes a `jq` over those files, not a re-run of the entire pipeline.
|
|
2906
|
+
|
|
2907
|
+
### 14.8 Agent Feedback Loop
|
|
2908
|
+
|
|
2909
|
+
The `mast_efficiency` MCP tool (§9) exposes `_stats` aggregates back to the agent
|
|
2910
|
+
within the conversation. The implement-task prompt (§12) instructs:
|
|
2911
|
+
|
|
2912
|
+
> Near the end of your task, call `mast_efficiency { "scope": "session" }`. If
|
|
2913
|
+
> `efficiency_ratio < 0.30`, you have been reading more than you should — prefer
|
|
2914
|
+
> `mast_search` over `Read` for the next task. If `efficiency_ratio > 0.85`, you
|
|
2915
|
+
> are using MAST well; keep going.
|
|
2916
|
+
|
|
2917
|
+
This is the load-bearing reason for `mast_efficiency`'s existence as an MCP tool
|
|
2918
|
+
rather than a CLI-only command: **the instrumentation has to have at least one
|
|
2919
|
+
consumer inside the agent loop**, or it rots from disuse.
|
|
2920
|
+
|
|
2921
|
+
### 14.9 What Is Deliberately Not Measured
|
|
2922
|
+
|
|
2923
|
+
- **Latency p99.** Tool call duration is captured per-row, but no SLA is asserted
|
|
2924
|
+
on it. Latency optimisation comes after the savings thesis is validated.
|
|
2925
|
+
- **Per-user / per-agent attribution.** v1 has one agent per `mast serve` session;
|
|
2926
|
+
multi-tenancy is out of scope.
|