@spikedpunch/mast 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (257) hide show
  1. package/LICENSE +21 -0
  2. package/MAST_SPEC.md +2926 -0
  3. package/README.md +842 -0
  4. package/assets/skill.md +45 -0
  5. package/dist/ast/extract.d.ts +65 -0
  6. package/dist/ast/extract.d.ts.map +1 -0
  7. package/dist/ast/extract.js +147 -0
  8. package/dist/ast/extract.js.map +1 -0
  9. package/dist/ast/extractor.d.ts +85 -0
  10. package/dist/ast/extractor.d.ts.map +1 -0
  11. package/dist/ast/extractor.js +2 -0
  12. package/dist/ast/extractor.js.map +1 -0
  13. package/dist/ast/extractors/markdown.d.ts +40 -0
  14. package/dist/ast/extractors/markdown.d.ts.map +1 -0
  15. package/dist/ast/extractors/markdown.js +175 -0
  16. package/dist/ast/extractors/markdown.js.map +1 -0
  17. package/dist/ast/extractors/typescript.d.ts +179 -0
  18. package/dist/ast/extractors/typescript.d.ts.map +1 -0
  19. package/dist/ast/extractors/typescript.js +1439 -0
  20. package/dist/ast/extractors/typescript.js.map +1 -0
  21. package/dist/ast/parser.d.ts +5 -0
  22. package/dist/ast/parser.d.ts.map +1 -0
  23. package/dist/ast/parser.js +21 -0
  24. package/dist/ast/parser.js.map +1 -0
  25. package/dist/ast/types.d.ts +586 -0
  26. package/dist/ast/types.d.ts.map +1 -0
  27. package/dist/ast/types.js +3 -0
  28. package/dist/ast/types.js.map +1 -0
  29. package/dist/cli/docs-cmd.d.ts +22 -0
  30. package/dist/cli/docs-cmd.d.ts.map +1 -0
  31. package/dist/cli/docs-cmd.js +100 -0
  32. package/dist/cli/docs-cmd.js.map +1 -0
  33. package/dist/cli/index-cmd.d.ts +72 -0
  34. package/dist/cli/index-cmd.d.ts.map +1 -0
  35. package/dist/cli/index-cmd.js +263 -0
  36. package/dist/cli/index-cmd.js.map +1 -0
  37. package/dist/cli/index.d.ts +3 -0
  38. package/dist/cli/index.d.ts.map +1 -0
  39. package/dist/cli/index.js +4 -0
  40. package/dist/cli/index.js.map +1 -0
  41. package/dist/cli/init.d.ts +14 -0
  42. package/dist/cli/init.d.ts.map +1 -0
  43. package/dist/cli/init.js +64 -0
  44. package/dist/cli/init.js.map +1 -0
  45. package/dist/cli/install-hooks.d.ts +3 -0
  46. package/dist/cli/install-hooks.d.ts.map +1 -0
  47. package/dist/cli/install-hooks.js +34 -0
  48. package/dist/cli/install-hooks.js.map +1 -0
  49. package/dist/cli/metrics-cmd.d.ts +3 -0
  50. package/dist/cli/metrics-cmd.d.ts.map +1 -0
  51. package/dist/cli/metrics-cmd.js +164 -0
  52. package/dist/cli/metrics-cmd.js.map +1 -0
  53. package/dist/cli/program.d.ts +14 -0
  54. package/dist/cli/program.d.ts.map +1 -0
  55. package/dist/cli/program.js +44 -0
  56. package/dist/cli/program.js.map +1 -0
  57. package/dist/cli/query.d.ts +46 -0
  58. package/dist/cli/query.d.ts.map +1 -0
  59. package/dist/cli/query.js +153 -0
  60. package/dist/cli/query.js.map +1 -0
  61. package/dist/cli/search-cmd.d.ts +33 -0
  62. package/dist/cli/search-cmd.d.ts.map +1 -0
  63. package/dist/cli/search-cmd.js +114 -0
  64. package/dist/cli/search-cmd.js.map +1 -0
  65. package/dist/cli/serve.d.ts +3 -0
  66. package/dist/cli/serve.d.ts.map +1 -0
  67. package/dist/cli/serve.js +19 -0
  68. package/dist/cli/serve.js.map +1 -0
  69. package/dist/cli/skill-install.d.ts +39 -0
  70. package/dist/cli/skill-install.d.ts.map +1 -0
  71. package/dist/cli/skill-install.js +57 -0
  72. package/dist/cli/skill-install.js.map +1 -0
  73. package/dist/cli/status.d.ts +35 -0
  74. package/dist/cli/status.d.ts.map +1 -0
  75. package/dist/cli/status.js +117 -0
  76. package/dist/cli/status.js.map +1 -0
  77. package/dist/cli/upgrade-cmd.d.ts +48 -0
  78. package/dist/cli/upgrade-cmd.d.ts.map +1 -0
  79. package/dist/cli/upgrade-cmd.js +129 -0
  80. package/dist/cli/upgrade-cmd.js.map +1 -0
  81. package/dist/cli/version.d.ts +3 -0
  82. package/dist/cli/version.d.ts.map +1 -0
  83. package/dist/cli/version.js +19 -0
  84. package/dist/cli/version.js.map +1 -0
  85. package/dist/env.d.ts +17 -0
  86. package/dist/env.d.ts.map +1 -0
  87. package/dist/env.js +17 -0
  88. package/dist/env.js.map +1 -0
  89. package/dist/graph/checker-resolver.d.ts +156 -0
  90. package/dist/graph/checker-resolver.d.ts.map +1 -0
  91. package/dist/graph/checker-resolver.js +458 -0
  92. package/dist/graph/checker-resolver.js.map +1 -0
  93. package/dist/graph/db.d.ts +318 -0
  94. package/dist/graph/db.d.ts.map +1 -0
  95. package/dist/graph/db.js +266 -0
  96. package/dist/graph/db.js.map +1 -0
  97. package/dist/graph/index.d.ts +4 -0
  98. package/dist/graph/index.d.ts.map +1 -0
  99. package/dist/graph/index.js +5 -0
  100. package/dist/graph/index.js.map +1 -0
  101. package/dist/graph/local-type-env.d.ts +45 -0
  102. package/dist/graph/local-type-env.d.ts.map +1 -0
  103. package/dist/graph/local-type-env.js +59 -0
  104. package/dist/graph/local-type-env.js.map +1 -0
  105. package/dist/graph/path-range.d.ts +40 -0
  106. package/dist/graph/path-range.d.ts.map +1 -0
  107. package/dist/graph/path-range.js +42 -0
  108. package/dist/graph/path-range.js.map +1 -0
  109. package/dist/graph/populate.d.ts +263 -0
  110. package/dist/graph/populate.d.ts.map +1 -0
  111. package/dist/graph/populate.js +984 -0
  112. package/dist/graph/populate.js.map +1 -0
  113. package/dist/graph/queries.d.ts +114 -0
  114. package/dist/graph/queries.d.ts.map +1 -0
  115. package/dist/graph/queries.js +415 -0
  116. package/dist/graph/queries.js.map +1 -0
  117. package/dist/graph/sqliteBatch.d.ts +48 -0
  118. package/dist/graph/sqliteBatch.d.ts.map +1 -0
  119. package/dist/graph/sqliteBatch.js +93 -0
  120. package/dist/graph/sqliteBatch.js.map +1 -0
  121. package/dist/indexer/freshness.d.ts +48 -0
  122. package/dist/indexer/freshness.d.ts.map +1 -0
  123. package/dist/indexer/freshness.js +72 -0
  124. package/dist/indexer/freshness.js.map +1 -0
  125. package/dist/indexer/import-resolver.d.ts +50 -0
  126. package/dist/indexer/import-resolver.d.ts.map +1 -0
  127. package/dist/indexer/import-resolver.js +318 -0
  128. package/dist/indexer/import-resolver.js.map +1 -0
  129. package/dist/indexer/index.d.ts +249 -0
  130. package/dist/indexer/index.d.ts.map +1 -0
  131. package/dist/indexer/index.js +560 -0
  132. package/dist/indexer/index.js.map +1 -0
  133. package/dist/indexer/walker.d.ts +60 -0
  134. package/dist/indexer/walker.d.ts.map +1 -0
  135. package/dist/indexer/walker.js +128 -0
  136. package/dist/indexer/walker.js.map +1 -0
  137. package/dist/indexer/watcher.d.ts +80 -0
  138. package/dist/indexer/watcher.d.ts.map +1 -0
  139. package/dist/indexer/watcher.js +151 -0
  140. package/dist/indexer/watcher.js.map +1 -0
  141. package/dist/mcp/context.d.ts +23 -0
  142. package/dist/mcp/context.d.ts.map +1 -0
  143. package/dist/mcp/context.js +2 -0
  144. package/dist/mcp/context.js.map +1 -0
  145. package/dist/mcp/register-tools.d.ts +21 -0
  146. package/dist/mcp/register-tools.d.ts.map +1 -0
  147. package/dist/mcp/register-tools.js +42 -0
  148. package/dist/mcp/register-tools.js.map +1 -0
  149. package/dist/mcp/server.d.ts +58 -0
  150. package/dist/mcp/server.d.ts.map +1 -0
  151. package/dist/mcp/server.js +177 -0
  152. package/dist/mcp/server.js.map +1 -0
  153. package/dist/mcp/staleness.d.ts +72 -0
  154. package/dist/mcp/staleness.d.ts.map +1 -0
  155. package/dist/mcp/staleness.js +213 -0
  156. package/dist/mcp/staleness.js.map +1 -0
  157. package/dist/mcp/startup.d.ts +40 -0
  158. package/dist/mcp/startup.d.ts.map +1 -0
  159. package/dist/mcp/startup.js +114 -0
  160. package/dist/mcp/startup.js.map +1 -0
  161. package/dist/mcp/tools/_helpers.d.ts +26 -0
  162. package/dist/mcp/tools/_helpers.d.ts.map +1 -0
  163. package/dist/mcp/tools/_helpers.js +54 -0
  164. package/dist/mcp/tools/_helpers.js.map +1 -0
  165. package/dist/mcp/tools/callers.d.ts +4 -0
  166. package/dist/mcp/tools/callers.d.ts.map +1 -0
  167. package/dist/mcp/tools/callers.js +112 -0
  168. package/dist/mcp/tools/callers.js.map +1 -0
  169. package/dist/mcp/tools/dependencies.d.ts +4 -0
  170. package/dist/mcp/tools/dependencies.d.ts.map +1 -0
  171. package/dist/mcp/tools/dependencies.js +36 -0
  172. package/dist/mcp/tools/dependencies.js.map +1 -0
  173. package/dist/mcp/tools/efficiency.d.ts +4 -0
  174. package/dist/mcp/tools/efficiency.d.ts.map +1 -0
  175. package/dist/mcp/tools/efficiency.js +39 -0
  176. package/dist/mcp/tools/efficiency.js.map +1 -0
  177. package/dist/mcp/tools/exports.d.ts +4 -0
  178. package/dist/mcp/tools/exports.d.ts.map +1 -0
  179. package/dist/mcp/tools/exports.js +56 -0
  180. package/dist/mcp/tools/exports.js.map +1 -0
  181. package/dist/mcp/tools/implementors.d.ts +4 -0
  182. package/dist/mcp/tools/implementors.d.ts.map +1 -0
  183. package/dist/mcp/tools/implementors.js +47 -0
  184. package/dist/mcp/tools/implementors.js.map +1 -0
  185. package/dist/mcp/tools/project-skeleton.d.ts +4 -0
  186. package/dist/mcp/tools/project-skeleton.d.ts.map +1 -0
  187. package/dist/mcp/tools/project-skeleton.js +61 -0
  188. package/dist/mcp/tools/project-skeleton.js.map +1 -0
  189. package/dist/mcp/tools/reindex.d.ts +14 -0
  190. package/dist/mcp/tools/reindex.d.ts.map +1 -0
  191. package/dist/mcp/tools/reindex.js +32 -0
  192. package/dist/mcp/tools/reindex.js.map +1 -0
  193. package/dist/mcp/tools/rename-impact.d.ts +4 -0
  194. package/dist/mcp/tools/rename-impact.d.ts.map +1 -0
  195. package/dist/mcp/tools/rename-impact.js +124 -0
  196. package/dist/mcp/tools/rename-impact.js.map +1 -0
  197. package/dist/mcp/tools/search.d.ts +4 -0
  198. package/dist/mcp/tools/search.d.ts.map +1 -0
  199. package/dist/mcp/tools/search.js +69 -0
  200. package/dist/mcp/tools/search.js.map +1 -0
  201. package/dist/mcp/tools/signature.d.ts +4 -0
  202. package/dist/mcp/tools/signature.d.ts.map +1 -0
  203. package/dist/mcp/tools/signature.js +127 -0
  204. package/dist/mcp/tools/signature.js.map +1 -0
  205. package/dist/mcp/tools/status.d.ts +4 -0
  206. package/dist/mcp/tools/status.d.ts.map +1 -0
  207. package/dist/mcp/tools/status.js +30 -0
  208. package/dist/mcp/tools/status.js.map +1 -0
  209. package/dist/search/declex.d.ts +109 -0
  210. package/dist/search/declex.d.ts.map +1 -0
  211. package/dist/search/declex.js +242 -0
  212. package/dist/search/declex.js.map +1 -0
  213. package/dist/search/fts.d.ts +77 -0
  214. package/dist/search/fts.d.ts.map +1 -0
  215. package/dist/search/fts.js +223 -0
  216. package/dist/search/fts.js.map +1 -0
  217. package/dist/search/fused.d.ts +110 -0
  218. package/dist/search/fused.d.ts.map +1 -0
  219. package/dist/search/fused.js +323 -0
  220. package/dist/search/fused.js.map +1 -0
  221. package/dist/search/potential-matches.d.ts +88 -0
  222. package/dist/search/potential-matches.d.ts.map +1 -0
  223. package/dist/search/potential-matches.js +83 -0
  224. package/dist/search/potential-matches.js.map +1 -0
  225. package/dist/search/scope.d.ts +66 -0
  226. package/dist/search/scope.d.ts.map +1 -0
  227. package/dist/search/scope.js +71 -0
  228. package/dist/search/scope.js.map +1 -0
  229. package/dist/store/config.d.ts +57 -0
  230. package/dist/store/config.d.ts.map +1 -0
  231. package/dist/store/config.js +208 -0
  232. package/dist/store/config.js.map +1 -0
  233. package/dist/store/lock.d.ts +101 -0
  234. package/dist/store/lock.d.ts.map +1 -0
  235. package/dist/store/lock.js +182 -0
  236. package/dist/store/lock.js.map +1 -0
  237. package/dist/store/lockMetrics.d.ts +52 -0
  238. package/dist/store/lockMetrics.d.ts.map +1 -0
  239. package/dist/store/lockMetrics.js +29 -0
  240. package/dist/store/lockMetrics.js.map +1 -0
  241. package/dist/store/sqliteChunkStore.d.ts +44 -0
  242. package/dist/store/sqliteChunkStore.d.ts.map +1 -0
  243. package/dist/store/sqliteChunkStore.js +135 -0
  244. package/dist/store/sqliteChunkStore.js.map +1 -0
  245. package/dist/telemetry/lockMetricsSummary.d.ts +42 -0
  246. package/dist/telemetry/lockMetricsSummary.d.ts.map +1 -0
  247. package/dist/telemetry/lockMetricsSummary.js +139 -0
  248. package/dist/telemetry/lockMetricsSummary.js.map +1 -0
  249. package/dist/telemetry/metrics.d.ts +140 -0
  250. package/dist/telemetry/metrics.d.ts.map +1 -0
  251. package/dist/telemetry/metrics.js +312 -0
  252. package/dist/telemetry/metrics.js.map +1 -0
  253. package/dist/telemetry/tokenizer.d.ts +125 -0
  254. package/dist/telemetry/tokenizer.d.ts.map +1 -0
  255. package/dist/telemetry/tokenizer.js +174 -0
  256. package/dist/telemetry/tokenizer.js.map +1 -0
  257. package/package.json +81 -0
package/MAST_SPEC.md ADDED
@@ -0,0 +1,2926 @@
1
+ # MAST — Monorepo AST Search Tool
2
+ ## Technical Specification
3
+
4
+ ---
5
+
6
+ ## 1. Overview
7
+
8
+ **mast** is a lexical + declaration-exact code search engine exposed over two surfaces:
9
+ an MCP server (used by the agent inside the claude-runner container) and a CLI (used by
10
+ humans and hooks outside the container). It replaces ad-hoc `Grep`, `Glob`, and
11
+ whole-file `Read` calls with targeted, index-backed queries that return structured
12
+ subsets of code rather than full file contents.
13
+
14
+ A single on-disk index — written to a configurable state directory — is shared by both
15
+ surfaces. The index persists on the mounted workspace volume across container runs,
16
+ so each new container inherits the index built by previous tasks.
17
+
18
+ **Note on prior discussions of a semantic/vector search leg:** an earlier revision of
19
+ this system fused BM25 with a vector-embedding ranker (LanceDB + a local ONNX model).
20
+ That subsystem was removed 2026-08-06 per the M2 decision (`adr/003-2026-08-04-vector-store-deletion.md`);
21
+ the pre-deletion system is preserved at the git tag
22
+ `mast-pre-vector-delete`. Everything below describes the system as it exists today.
23
+
24
+ ---
25
+
26
+ ## 2. Goals
27
+
28
+ - Return **chunks not files**: every query response contains only the lines the agent
29
+ needs, not the full file.
30
+ - **Zero BT involvement** after init: the MCP server startup check keeps the index
31
+ fresh automatically.
32
+ - **Configurable state directory**: the SDD pipeline writes state under
33
+ `.kluster/.mast`; the default for standalone use is `.mast` at the project root.
34
+ - **Synchronous freshness on demand**: the agent can call `mast_reindex` mid-task after
35
+ writing files and be guaranteed the next query sees those changes.
36
+ - **Single codebase, two surfaces**: the same index and query engine backs both the
37
+ MCP tools and CLI commands.
38
+
39
+ ## 3. Non-Goals
40
+
41
+ - Code generation or explanation.
42
+ - PR review, wiki generation, story generation.
43
+ - A persistent background daemon — freshness is handled by the startup check and
44
+ `mast_reindex`. *Narrow carve-out:* `mast serve --watch` (§11.4) is an opt-in
45
+ file watcher for **interactive, non-container** use only. It is scoped to the
46
+ serve process lifetime (not a daemon), the SDD pipeline never uses it, and it
47
+ is a discovery-freshness optimization — it lets new files and symbols become
48
+ searchable sooner than waiting for an explicit `mast_reindex`. JIT staleness
49
+ handling (§9.0) already guarantees line-coordinate and content correctness for
50
+ already-indexed files without it.
51
+ - Support for non-TypeScript/JavaScript projects in v1 (AST layer is extensible but
52
+ v1 targets the SDD stack).
53
+
54
+ ---
55
+
56
+ ## 4. Configuration
57
+
58
+ The state directory itself is resolved first, independent of everything else (it must
59
+ be known before its own persisted config can be loaded from inside it):
60
+
61
+ 1. CLI flag `--state-dir <path>` (or env `MAST_STATE_DIR`)
62
+ 2. `state_dir` key in `mast.config.json` in the project root
63
+ 3. Default: `<project_root>/.mast`
64
+
65
+ Every other config key (`file_extensions`, `exclude_patterns`, `rrf_k`,
66
+ `declaration_exact_ranker`, `chunk_split_threshold`, `context_lines`,
67
+ `markdown_heading_depth`) is then resolved in this priority order, highest first:
68
+
69
+ 1. Explicit CLI overrides — `mast init --extensions <ext,...>` / `--exclude <pattern,...>`
70
+ (F9, Stage 3.5)
71
+ 2. `mast.config.json` in the project root
72
+ 3. The persisted `<state_dir>/config.json` from a previous `mast init` / `mast serve`
73
+ in this state directory (F9 — previously write-only; now read back on every
74
+ resolution)
75
+ 4. Built-in defaults
76
+
77
+ **Path keys are never taken from the persisted state config.** `<state_dir>/config.json`
78
+ stores a full resolved config, including the ABSOLUTE `state_dir`/`project_root`/
79
+ `resolved_state_dir`/`resolved_project_root` from whichever process last wrote it. The
80
+ SDD pipeline mounts the same workspace volume at different container paths across runs,
81
+ so an absolute path loaded back from a previous container could silently point the
82
+ resolver at a path that doesn't exist (or belongs to an unrelated project) in the
83
+ current one. Only the customisation keys are read from the persisted file; the four
84
+ path keys always come from the current resolution.
85
+
86
+ ### 4.1 `mast.config.json`
87
+
88
+ ```json
89
+ {
90
+ "state_dir": ".mast",
91
+ "project_root": ".",
92
+ "file_extensions": [".ts", ".tsx", ".js", ".jsx", ".md"],
93
+ "exclude_patterns": [
94
+ "**/node_modules/**",
95
+ "**/dist/**",
96
+ "**/build/**",
97
+ "**/coverage/**",
98
+ "**/.next/**",
99
+ "**/.turbo/**",
100
+ ".mast/**",
101
+ "**/*.test.ts",
102
+ "**/*.spec.ts"
103
+ ],
104
+ "rrf_k": 60,
105
+ "declaration_exact_ranker": true,
106
+ "chunk_split_threshold": 100,
107
+ "context_lines": 3,
108
+ "markdown_heading_depth": 2
109
+ }
110
+ ```
111
+
112
+ `rrf_k` is the constant in the Reciprocal Rank Fusion formula (see §7.3). The default
113
+ of 60 is standard. Higher values reduce the influence of rank differences.
114
+
115
+ `declaration_exact_ranker` (default `true`) is the F18 kill-switch: it fuses the
116
+ declaration-exact ranker (ranker D — see §7.3) into `mast_search` ranking as an
117
+ additional RRF input. Set `false` to restore pre-F18 ranking without a code change.
118
+ The flag exists because ranker D's harm surface on identifier-free queries is
119
+ monitored, not proven safe (M2 decision memo, condition 3); its D-fire telemetry
120
+ (§14.3 `declex_json`) is the input signal for that monitoring.
121
+
122
+ There is deliberately **no `similarity_threshold` key** — it gated a vector-search leg
123
+ that no longer exists (removed 2026-08-06, §1); no replacement config key was needed.
124
+
125
+ `chunk_split_threshold` is the line count above which a single declaration is split
126
+ into overlapping sub-chunks. Below this threshold, a declaration is always one chunk
127
+ regardless of length.
128
+
129
+ `context_lines` controls how many source lines before and after a chunk's AST
130
+ boundaries are included in the stored `content` field. When a chunk's AST span is
131
+ `[start_line, end_line]`, the stored content covers
132
+ `[max(1, start_line - context_lines), min(file_lines, end_line + context_lines)]`.
133
+ This gives agents surrounding context (e.g., the `const` binding before a function
134
+ expression, or the closing brace of an enclosing block) without requiring a full file
135
+ read. The `start_line` and `end_line` fields in the chunk record always reflect the
136
+ AST declaration boundaries, not the expanded content boundaries.
137
+
138
+ `markdown_heading_depth` is the maximum ATX heading level that starts a new `doc`
139
+ chunk when indexing markdown files (§10.1). Headings deeper than this fold into
140
+ their enclosing section. The default of 2 means one chunk per `##` section.
141
+
142
+ Vendored markdown noise (dependency READMEs and the like) is handled by the
143
+ existing `exclude_patterns` — `node_modules/**` is authoritative; there is no
144
+ markdown-specific exclusion logic.
145
+
146
+ Every field above except `state_dir`/`project_root` is also read back from
147
+ `<state_dir>/config.json` when `mast.config.json` and CLI flags don't override it (§4,
148
+ F9) — see §5 for the file's read/write semantics.
149
+
150
+ ### 4.2 SDD Pipeline Configuration
151
+
152
+ The claude-runner passes `--state-dir` at serve time:
153
+
154
+ ```json
155
+ // claude-runner/.mcp.json
156
+ {
157
+ "mcpServers": {
158
+ "mast": {
159
+ "command": "mast",
160
+ "args": ["serve", "--state-dir", "/workspace/.kluster/.mast"],
161
+ "type": "stdio"
162
+ }
163
+ }
164
+ }
165
+ ```
166
+
167
+ This is the only configuration change needed in the SDD pipeline after `mast init`.
168
+
169
+ ---
170
+
171
+ ## 5. Storage Layout
172
+
173
+ ```
174
+ <state_dir>/
175
+ ├── config.json # Resolved active config (written at init/serve; read back
176
+ # on every resolution — §4, F9)
177
+ ├── index.json # Index metadata: last_indexed, file_count, schema_version
178
+ ├── file_manifest.json # {path: mtime} snapshot from last index run
179
+ ├── structure.lock # Advisory write lock for coarse writers (index, mast_reindex, manifest)
180
+ └── graph.db # Knowledge graph, chunks, and FTS5 index (SQLite, WAL mode)
181
+ ```
182
+
183
+ `index.json` example:
184
+ ```json
185
+ {
186
+ "schema_version": "1.3.0",
187
+ "last_indexed": "2026-05-13T14:22:00Z",
188
+ "file_count": 142,
189
+ "chunk_count": 1840
190
+ }
191
+ ```
192
+
193
+ ---
194
+
195
+ ## 6. Data Model
196
+
197
+ ### 6.1 Chunk (`chunks` table — `graph.db`)
198
+
199
+ | Field | Type | Description |
200
+ |---|---|---|
201
+ | `chunk_id` | `str` | `sha256(file_path + ":" + start_line)` |
202
+ | `file_path` | `str` | Relative to `project_root` |
203
+ | `start_line` | `int` | 1-indexed |
204
+ | `end_line` | `int` | 1-indexed, inclusive |
205
+ | `content` | `str` | Raw source text of the chunk |
206
+ | `chunk_type` | `str` | `function` \| `method` \| `class_shell` \| `interface` \| `type` \| `export` \| `block` \| `doc` |
207
+ | `symbol_name` | `str \| None` | Top-level symbol name if applicable. For `method` chunks, qualified as `ClassName.methodName`. For `doc` chunks, the heading path (§10.1). |
208
+ | `parent_symbol` | `str \| None` | For `method` chunks, the enclosing class name (unqualified). `None` for all other chunk types. Enables fast "find all methods of class X" queries against the `chunks` table without joining the graph. |
209
+ | `is_exported` | `bool` | True if the declaration carries an `export` modifier. For `method` chunks, inherited from the enclosing `class_shell`'s `is_exported` *and* the method's accessibility (anything not `private` is treated as exported when the class is exported). |
210
+ | `language` | `str` | `typescript` \| `javascript` \| `markdown` |
211
+ | `file_mtime` | `float` | File mtime at index time — used for staleness detection |
212
+
213
+ `is_exported` enables `mast_search` to filter results to public API surface only,
214
+ which is the correct scope when an agent is looking for a service to call rather than
215
+ an internal utility to modify.
216
+
217
+ **`class_shell` content is synthesized, not raw source.** For a `class_shell` chunk,
218
+ the stored `content` field is the class declaration line followed by every member
219
+ signature (with TSDoc comments), ordered as they appear in source — but with method
220
+ bodies stripped. This is the "outline" view used for orientation and for
221
+ `mast_signature` calls that target a class rather than a specific method. The raw
222
+ class body source is *not* stored as a single chunk; it is decomposed into N
223
+ `method` chunks, each its own row in the `chunks` table (see §10.1).
224
+
225
+ ### 6.3 Knowledge Graph (SQLite — `graph.db`)
226
+
227
+ SQLite with WAL mode replaces KuzuDB. WAL mode is correct for containerised
228
+ shared-volume access: it tolerates concurrent readers and a single writer without
229
+ exclusive locks that can deadlock across container boundaries.
230
+
231
+ Recursive CTEs handle multi-hop graph traversal (callers of callers, transitive
232
+ dependency chains) with sub-millisecond latency at monorepo scale.
233
+
234
+ ```sql
235
+ PRAGMA journal_mode = WAL;
236
+ PRAGMA foreign_keys = ON;
237
+
238
+ CREATE TABLE IF NOT EXISTS files (
239
+ id INTEGER PRIMARY KEY,
240
+ path TEXT NOT NULL UNIQUE,
241
+ language TEXT NOT NULL,
242
+ mtime REAL NOT NULL
243
+ );
244
+
245
+ CREATE TABLE IF NOT EXISTS symbols (
246
+ id INTEGER PRIMARY KEY,
247
+ name TEXT NOT NULL,
248
+ kind TEXT NOT NULL, -- function | class | interface | type | const
249
+ file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
250
+ line INTEGER NOT NULL,
251
+ is_exported INTEGER NOT NULL DEFAULT 0, -- boolean
252
+ declaration_hash TEXT, -- sha256 of signature text only (excludes body)
253
+ body_hash TEXT -- sha256 of body text only (excludes signature)
254
+ -- If both declaration_hash and body_hash are unchanged on incremental reindex:
255
+ -- skip the KG rebuild for this symbol (§7.1's file-level stability-hash skip).
256
+ );
257
+
258
+ CREATE TABLE IF NOT EXISTS edges (
259
+ from_id INTEGER NOT NULL REFERENCES symbols(id) ON DELETE CASCADE,
260
+ to_id INTEGER NOT NULL REFERENCES symbols(id) ON DELETE CASCADE,
261
+ edge_type TEXT NOT NULL,
262
+ resolution TEXT, -- POTENTIAL_CALL only: which §10.3.1 rule matched
263
+ -- (import | field_type | parameter_type | new_expression | same_file
264
+ -- | this_method | super_method)
265
+ -- or 'checker' (§10.3.2) — the opt-in `mast index --checker`
266
+ -- pass upgraded a heuristic-unresolved potential match via
267
+ -- the real TypeScript checker. 'this_method'/'super_method' (F4,
268
+ -- Stage 3) are additive values for `this.foo()`/`super.foo()`
269
+ -- call sites — see §10.3.1's "Method calls on super and this".
270
+ -- Additive values, no schema change.
271
+ call_line INTEGER, -- POTENTIAL_CALL only: 1-indexed source line of the call site
272
+ context TEXT, -- POTENTIAL_CALL only: trimmed source text of the call-site line
273
+ -- POTENTIAL_CALL | IMPLEMENTS | EXTENDS | RE_EXPORTS | PARENT_OF
274
+ -- POTENTIAL_CALL: name-resolved reference. The local heuristic resolver (see §10.3)
275
+ -- produces these edges when it can statically link the receiver of a
276
+ -- method call to a known symbol. Coverage is partial by design — see
277
+ -- §10.3 for what the resolver catches and what it doesn't. Tools that
278
+ -- consume these edges (mast_callers) MUST also surface identifier-FTS
279
+ -- matches as "potential" results to compensate for missed edges.
280
+ -- RE_EXPORTS: file A re-exports a symbol from file B via export * or export { x }
281
+ -- PARENT_OF: class symbol → method symbol. Emitted once per method during
282
+ -- class_body decomposition (see §10.1). Enables "list all methods of X"
283
+ -- queries via a single indexed lookup.
284
+ PRIMARY KEY (from_id, to_id, edge_type)
285
+ );
286
+
287
+ CREATE TABLE IF NOT EXISTS re_export_files (
288
+ -- Tracks export * from '...' at the file level (no specific symbol known at parse time)
289
+ from_file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
290
+ to_file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
291
+ PRIMARY KEY (from_file_id, to_file_id)
292
+ );
293
+
294
+ CREATE TABLE IF NOT EXISTS imports (
295
+ file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
296
+ module TEXT NOT NULL,
297
+ symbols TEXT NOT NULL, -- JSON array of imported symbol names
298
+ is_external INTEGER NOT NULL DEFAULT 0,
299
+ resolved_path TEXT -- NULL for external modules; populated by path resolver
300
+ );
301
+
302
+ -- FTS5 with built-in content: stores content directly alongside the index structures.
303
+ -- snippet() works without any external table. Indexing inserts/updates/deletes directly
304
+ -- on chunk_fts; no sync logic required. Content duplication vs the `chunks` table is
305
+ -- acceptable at monorepo scale (~1-2 GB of source) and eliminates a whole class of
306
+ -- consistency bugs.
307
+ -- trigram tokenizer: substring matching for camelCase identifiers, prose, and partial
308
+ -- queries. Used by mast_search BM25 ranking via bm25(chunk_fts) (returns negative scores).
309
+ CREATE VIRTUAL TABLE IF NOT EXISTS chunk_fts USING fts5(
310
+ content,
311
+ symbol_name UNINDEXED, -- stored for retrieval; trigram on content already surfaces symbol names
312
+ chunk_id UNINDEXED,
313
+ tokenize = 'trigram'
314
+ );
315
+
316
+ -- Identifier-exact FTS: a search for "findUserByEmail" must match that identifier
317
+ -- exactly, NOT substrings like "findUser" or "findUserByEmailVerified". The trigram
318
+ -- tokenizer (above) is too noisy for this — it routinely returns dozens of false
319
+ -- positives on common method names. unicode61 with code-aware separators tokenizes
320
+ -- on identifier boundaries (.-_/()[]{}<>:;,=+*&|!?), giving exact-identifier match
321
+ -- semantics needed by mast_callers' "potential_matches" set (see §9 mast_callers).
322
+ --
323
+ -- The `identifiers` column is populated by Phase 1: it stores a whitespace-separated
324
+ -- list of every identifier token found in the chunk (deduplicated). Phase 1
325
+ -- extracts these via tree-sitter — no separate parse pass.
326
+ --
327
+ -- F5 (Stage 3, schema 1.3.0): the bag ALSO carries QUALIFIED compound strings
328
+ -- ("Class.method"), appended after the bare tokens. A phrase-quoted query for a
329
+ -- qualified method name (mast_callers/mast_rename_impact's documented
330
+ -- "Class.method" convention) requires the class and method tokens ADJACENT
331
+ -- under this tokenizer's '.' separator — the bare bag alone essentially never
332
+ -- produces that adjacency, silently emptying the potential set for any method
333
+ -- query. Two sources feed the compounds, both derived from the SAME
334
+ -- `LocalTypeEnvironment` resolution `extractEdges` already computes (no
335
+ -- parallel mechanism): (1) a method chunk's own qualified `symbol_name`
336
+ -- (declaration self-discoverability — constructor/getter/setter forms are
337
+ -- already qualified); (2) for every call site the resolver statically linked
338
+ -- to a receiver type, the resolved `Type.method` string, appended to the
339
+ -- CALLING chunk's bag — this also heals the case where the POTENTIAL_CALL
340
+ -- edge itself is later dropped by `insertEdges`' file-scoped resolution (e.g.
341
+ -- an unresolvable import), since extraction and edge-insertion are
342
+ -- independent. A receiver the resolver could NOT statically link (DI
343
+ -- containers, factories — §10.3.1's documented "does NOT catch" list)
344
+ -- contributes no compound; that residual gap is checker-pass (`--checker`)
345
+ -- territory, not this fix's.
346
+ CREATE VIRTUAL TABLE IF NOT EXISTS identifier_fts USING fts5(
347
+ identifiers,
348
+ chunk_id UNINDEXED,
349
+ tokenize = "unicode61 separators '.-_/()[]{}<>:;,=+*&|!?'"
350
+ );
351
+
352
+ -- Hot-path index: name lookup returns file_id, line, kind directly from the index
353
+ -- tree without touching table rows (covering index for mast_signature / mast_search).
354
+ CREATE INDEX IF NOT EXISTS idx_symbols_lookup ON symbols(name, file_id, line, kind);
355
+ CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_id);
356
+ CREATE INDEX IF NOT EXISTS idx_edges_from ON edges(from_id);
357
+ CREATE INDEX IF NOT EXISTS idx_edges_to ON edges(to_id);
358
+ CREATE INDEX IF NOT EXISTS idx_imports_file ON imports(file_id);
359
+ CREATE INDEX IF NOT EXISTS idx_imports_resolved ON imports(resolved_path);
360
+ ```
361
+
362
+ **Recursive CTE example — transitive callers (verified set only):**
363
+ ```sql
364
+ WITH RECURSIVE callers(id) AS (
365
+ SELECT from_id FROM edges
366
+ WHERE to_id = :target_id AND edge_type = 'POTENTIAL_CALL'
367
+ UNION
368
+ SELECT e.from_id FROM edges e
369
+ JOIN callers c ON e.to_id = c.id
370
+ WHERE e.edge_type = 'POTENTIAL_CALL'
371
+ )
372
+ SELECT s.name, f.path, s.line
373
+ FROM symbols s
374
+ JOIN files f ON s.file_id = f.id
375
+ JOIN callers c ON s.id = c.id;
376
+ ```
377
+
378
+ This CTE only returns callers the heuristic resolver could statically link. It is
379
+ the **verified** set. `mast_callers` MUST also run an `identifier_fts` query for
380
+ the symbol name and present those hits as a separate **potential** set — see §9
381
+ `mast_callers`.
382
+
383
+ **Recursive CTE example — barrel file / transitive re-export resolution:**
384
+
385
+ Used by `mast_signature` when the direct symbol lookup returns no result, indicating
386
+ the symbol may be re-exported through one or more barrel files.
387
+
388
+ Note: `re_export_files` edges are file-to-file (using `files.id`). `edges` with
389
+ `RE_EXPORTS` type are symbol-to-symbol (using `symbols.id`). The CTE below follows
390
+ the file-level chain from `re_export_files`, then locates the symbol in the terminal
391
+ file. This correctly separates file IDs from symbol IDs.
392
+
393
+ ```sql
394
+ WITH RECURSIVE re_export_chain(file_id) AS (
395
+ -- Start from the file where the agent queried the symbol
396
+ SELECT :start_file_id
397
+ UNION
398
+ -- Follow export * from '...' edges to source files
399
+ SELECT rf.to_file_id
400
+ FROM re_export_files rf
401
+ JOIN re_export_chain rec ON rf.from_file_id = rec.file_id
402
+ )
403
+ SELECT s.id, s.name, s.kind, s.line, f.path
404
+ FROM symbols s
405
+ JOIN files f ON s.file_id = f.id
406
+ JOIN re_export_chain rec ON s.file_id = rec.file_id
407
+ WHERE s.name = :symbol_name
408
+ AND s.file_id != :start_file_id -- exclude the barrel file itself
409
+ LIMIT 1;
410
+ ```
411
+
412
+ ---
413
+
414
+ ## 7. Index Lifecycle
415
+
416
+ ### 7.1 Indexing (Single Phase — Parse → Chunks → Graph → FTS)
417
+
418
+ Indexing is a single phase — there is no separate embedding step. `runIndex`:
419
+
420
+ 1. Acquire `structure.lock` — see §7.6. Exit with error if lock cannot be
421
+ acquired within the configured timeout.
422
+ 2. Walk project files matching `file_extensions`, respecting `exclude_patterns`.
423
+ Collect `{ path, mtime }` for every file found.
424
+ 3. **Deleted file cleanup:** load `file_manifest.json` (previous scan's path set).
425
+ Any path present in the manifest but absent from the current walk has been deleted.
426
+ Remove its rows from `graph.db` `files` table (cascade deletes symbols, edges, and
427
+ imports), and delete all matching rows from the `chunks` table.
428
+ 4. For each file to index (all files on full run; only files where
429
+ `mtime > manifest[path]` on incremental run): wrap the parse in try/catch. On
430
+ tree-sitter error, log at `warn` level with the file path and error message,
431
+ increment `parse_errors`, and skip to the next file. Never abort the full run.
432
+ On success: run the two-pass walk and extract chunks with type, symbol, `is_exported`,
433
+ and `declaration_hash` metadata.
434
+ 5. Write chunks to the `chunks` table. Delete and replace all existing chunks for the
435
+ same `file_path`. Update `chunk_fts`: `DELETE FROM chunk_fts WHERE chunk_id = ?`
436
+ for removed chunks, `INSERT INTO chunk_fts(content, symbol_name, chunk_id) VALUES
437
+ (?, ?, ?)` for new/changed chunks. No external table or trigger is needed — FTS5
438
+ built-in content handles everything. Multi-row inserts here (and for `symbols`,
439
+ `imports`, `identifier_fts`, and `edges`) are batched under SQLite's 32,766
440
+ bound-parameter ceiling, batch-by-batch inside the same per-file transaction
441
+ (`graph/sqliteBatch.ts`), so file size never caps how many chunks get indexed
442
+ (Stage 4.5 S1 — `adr/011-2026-08-17-indexing-scale.md`).
443
+ 6. Populate `graph.db` from AST imports and relationships, wrapped in a single
444
+ transaction per file (delete-and-replace). Record `RE_EXPORTS` edges from
445
+ `export { x }` clauses and `re_export_files` rows from `export * from '...'`
446
+ clauses.
447
+ 7. **Stability hash optimisation (incremental only):** the `declaration_hash`
448
+ (signature) and `body_hash` are computed from the AST (signature node vs body
449
+ node), not by splitting chunk text.
450
+
451
+ **File-level skip:** a file whose mtime changed but whose chunked content is
452
+ byte-identical (same chunk-id set AND same per-symbol `declaration_hash`/
453
+ `body_hash` signature, and no `block` chunks) is not re-written at all. A
454
+ true *per-symbol* KG-rebuild skip was rejected: under the per-file
455
+ delete-and-replace model (step 6), symbols are re-inserted with new ids, so
456
+ edges must be rebuilt — preserving them per-symbol would be invasive surgery
457
+ on the hot path for marginal gain.
458
+
459
+ **Class shells use a member-signature hash, not a body hash.** A `class_shell`
460
+ chunk's content is the synthesized outline (declaration + member signatures + TSDoc;
461
+ no method bodies — see §6.1). Its `body_hash` is therefore computed as
462
+ `sha256(sorted(member_signature_text + member_doc_text).join("\n"))`, where each
463
+ `member_signature_text` is the method's signature line stripped of body. This means:
464
+ - Renaming a method → shell content changes → shell `body_hash` changes → shell
465
+ is rewritten. Correct: the outline visible to the agent now lists a new name.
466
+ - Editing a method body without changing its signature → shell `body_hash`
467
+ **unchanged** → shell content is NOT rewritten; only the affected `method` chunk is.
468
+ Correct: the class's public interface didn't change, only its internals.
469
+ - Adding or removing a method → shell content changes → shell is rewritten, plus
470
+ the new/removed `method` chunk is added/deleted.
471
+
472
+ Without this rule, the shell silently drifts out of sync with its members, and
473
+ `mast_search "session validation"` returns a stale outline naming a method that
474
+ no longer exists.
475
+ 8. Write `file_manifest.json` with the current `{ path: mtime }` snapshot.
476
+ 9. Write/update `index.json` with `last_indexed` and `file_count`.
477
+
478
+ Incremental variant: step 4 skips files where `mtime === manifest[path]`. Steps 3
479
+ (deleted file cleanup) and 8–9 always run.
480
+
481
+ BM25 search is handled by the `chunk_fts` FTS5 virtual table in `graph.db`, populated
482
+ incrementally during indexing (step 5). There is no separate serialization step.
483
+
484
+ ### 7.3 Ranked Search with RRF
485
+
486
+ Search combines two rankers — FTS5 BM25 (lexical) and the declaration-exact
487
+ ranker (ranker D, structural) — using **Reciprocal Rank Fusion**:
488
+
489
+ $$Score(d) = \sum_{r \in R} \frac{1}{k + r(d)}$$
490
+
491
+ Where $r(d)$ is the rank of document $d$ in ranker $R$, and $k$ is `rrf_k` (default 60).
492
+
493
+ RRF is used instead of weighted score addition because the two rankers' scores are on
494
+ incomparable scales (a BM25 score and a structural match are not directly comparable).
495
+ Rank position, not the underlying score magnitude, is what RRF fuses.
496
+
497
+ Implementation: run FTS5 BM25 search (and, when enabled, ranker D) independently over a
498
+ candidate pool of **4× `limit` per ranker** (40 for the default `limit: 10`),
499
+ then apply RRF to produce a unified ranked list. Return the top `limit` results.
500
+
501
+ **Second RRF input — the declaration-exact ranker (ranker D, F18).** When
502
+ `declaration_exact_ranker` is enabled (§4.1, default on), a second ranked list
503
+ joins the fusion. Ranker D (`src/search/declex.ts`, the Q1/DECLEX-measured
504
+ construction) is symbol-gated and purely lexical-structural:
505
+
506
+ - The raw query is split on `/[A-Za-z0-9_$]+/` (no camelCase decomposition, no
507
+ lowercasing); only **symbol-shaped** tokens survive the eligibility gate
508
+ (contains an uppercase letter, `_`, `$`, or a digit adjacent to a letter) —
509
+ bare lowercase prose words never reach the ranker.
510
+ - Each eligible token matches chunks whose own `symbol_name` **equals the
511
+ token** (full-name match) or **ends with `.` + token** (final-dot-segment
512
+ match — reaches `Class.method` chunks by their method name), both
513
+ case-insensitive, via a direct SQL predicate against `chunks.symbol_name`
514
+ (not FTS — the rule is a structural string comparison).
515
+ - Ordering is deterministic: full-name matches before segment-only matches,
516
+ then ascending same-matched-name multiplicity (a uniquely-named match beats
517
+ one of 140 `toJSON`s), then ascending `chunk_id`. The pool is capped at
518
+ 4× `limit` like BM25, and the list enters RRF **by rank** with the same
519
+ `rrf_k`.
520
+ - Ranker D applies the caller's `file_pattern`/`language` scope to its candidate
521
+ pool **before** ordering and capping, so a scoped search behaves as though the
522
+ index held only the files in scope. Both filters are compiled once per call by
523
+ `search/scope.ts` and applied identically by every ranker RRF fuses — a ranker
524
+ that skipped them would put out-of-scope chunks into a scoped result set, which
525
+ is what D did until 2026-08-20 (`docs/defects/LEDGER.md` D031; this bullet read
526
+ "applies no `file_pattern`/`language` pre-filter (same semantics as BM25)",
527
+ and the parenthetical was wrong about BM25 too — BM25 pre-filtered throughout).
528
+ `chunk_type`/`only_exported` post-filters apply downstream unchanged. When
529
+ `declaration_exact_ranker` is off, `mast_search` is BM25-only.
530
+ - `file_pattern` is a **glob**, compiled by `globToRegex` (`indexer/walker.ts`) —
531
+ the same primitive that matches `exclude_patterns` at index time and backs
532
+ `mast_project_skeleton`'s own `file_pattern`, so one pattern selects one set of
533
+ files everywhere. `*` does not cross `/`, `**` does, `?` is a single non-`/`
534
+ character, matching is case-sensitive, and every other character is literal.
535
+
536
+ Provenance: pre-registered and measured as Q1/DECLEX; shipped per the M2 decision
537
+ memo as F18 (`adr/004-2026-08-06-ranker-d.md`). The measured **escape variant**
538
+ (lowercase-token recovery under a match-count cap) is deliberately NOT shipped —
539
+ it is measured harmful off-stratum and requires a fresh pre-registration.
540
+ Per-call firing telemetry is persisted to `metrics.declex_json` (§14.3).
541
+
542
+ **FTS5 sign convention:** SQLite's `bm25(chunk_fts)` returns negative scores — more
543
+ negative means a better match. When sorting the FTS5 result set, sort ascending
544
+ (most negative first) before applying RRF ranks. Do not negate the scores; rank
545
+ position is all that RRF uses.
546
+
547
+ ### 7.4 Startup Reindex (Primary Pipeline Hook)
548
+
549
+ When `mast serve` starts, the goal is **time-to-first-query in single-digit
550
+ seconds**. All 11 tools are registered and ready to serve as soon as Step 3
551
+ completes — there is no reduced-capability window and no discriminator on tool
552
+ responses to track. Cold-start dead time is the single biggest UX risk to MAST
553
+ adoption — see Failure 4 in the design review — so this ladder is structured to
554
+ eliminate it.
555
+
556
+ ```
557
+ startup
558
+ ├─ STEP 1 (sync, < 1s): bootstrap state directory
559
+ │ ├─ if <state_dir> is missing or empty:
560
+ │ │ └─ if /opt/mast-seed exists (Docker-baked seed, see §13.8):
561
+ │ │ copy /opt/mast-seed → <state_dir>
562
+ │ │ └─ else:
563
+ │ │ run `mast init --no-index` to create config + empty state
564
+ │ ├─ ensure lock markers exist; persist the resolved config
565
+ │ └─ best-effort remove orphaned pre-vector-store state (`lance/`,
566
+ │ `embed_cache/`, `vectors.lock`) left behind by a pre-2026-08-06
567
+ │ install — logged, never fatal, runs on every startup
568
+
569
+ ├─ STEP 2 (sync, < 2s): schema version + open database
570
+ │ ├─ if index.json.schema_version != CURRENT_SCHEMA_VERSION:
571
+ │ │ wipe all derived state (graph.db, file_manifest.json, and any
572
+ │ │ remaining orphaned state)
573
+ │ │ set needs_full_reindex = true
574
+ │ │ write new index.json with updated schema_version
575
+ │ ├─ open graph.db (better-sqlite3, WAL mode)
576
+ │ └─ verify chunk_fts and identifier_fts tables exist (created on first init)
577
+
578
+ ├─ STEP 3 (sync, < 1s): open MCP transport — SERVER READY
579
+ │ ├─ register all 11 tools (mast_search, mast_project_skeleton,
580
+ │ │ mast_exports, mast_signature, mast_callers, mast_dependencies,
581
+ │ │ mast_implementors, mast_reindex, mast_status, mast_efficiency,
582
+ │ │ mast_rename_impact)
583
+ │ └─ accept incoming MCP connections
584
+
585
+ └─ STEP 4 (async): background incremental reindex
586
+ ├─ scan filesystem: collect {path, mtime} for all matched files
587
+ ├─ deleted_files = manifest_paths - scanned_paths
588
+ │ └─ for each: delete chunks/symbols (acquire structure.lock briefly)
589
+ ├─ stale_files = [f for f in scanned if f.mtime > index.last_indexed
590
+ │ OR needs_full_reindex]
591
+ ├─ acquire structure.lock
592
+ │ ├─ run the indexer (§7.1) for stale_files
593
+ │ ├─ update file_manifest.json + index.json.last_indexed
594
+ │ └─ release structure.lock
595
+ └─ FROM THIS POINT: the index is up-to-date for stale files
596
+ (mast_search, mast_callers verified+potential, etc.)
597
+ ```
598
+
599
+ If `--watch` was passed to `mast serve`, the file watcher (§11.4) starts
600
+ immediately after Step 3's transport opens, independent of Step 4.
601
+
602
+ **`--no-startup-reindex` refusal (M6 Part A).** The empty-during-Step-4 window
603
+ above is legitimate and by design — but `--no-startup-reindex` disables Step 4
604
+ entirely, and a state dir that has never completed an index run under that
605
+ flag would then answer every query `{"results":[]}` forever, with nothing
606
+ left to ever fill it in (`eval/GITNEXUS_COMPARISON.md` §13.8 item 4). To catch
607
+ only that unrecoverable case, `mast serve` calls `assertServableIndex`
608
+ (`mcp/server.ts`) after Step 1 and before Step 2 opens `graph.db`: if
609
+ `--no-startup-reindex` was passed AND the state dir is never-indexed
610
+ (`graph.db` absent, or `index.json` reports `chunk_count: 0` with
611
+ `last_indexed` null/absent), the process exits with an error naming the state
612
+ dir and suggesting `mast init`/`mast index` or dropping the flag. A state dir
613
+ indexed over a genuinely empty file set (`last_indexed` set, `chunk_count: 0`)
614
+ is NOT refused — see §9.0's "Empty-index signal" for how that legitimate
615
+ empty-index case is surfaced to callers instead. With the startup reindex
616
+ enabled (the default), this check is a no-op and Step 3 opens the transport
617
+ exactly as described above.
618
+
619
+ `CURRENT_SCHEMA_VERSION` is a constant in the mast binary (currently `"1.3.0"`). A
620
+ version bump is required any time the SQLite schema or `index.json` fields change
621
+ in a way that makes old on-disk state unreadable by the new code. Incrementing
622
+ without a state wipe causes a corrupt or partial index; wiping without
623
+ incrementing loses the protection. Both are bugs — treat the version as a
624
+ migration guard, not a display string.
625
+
626
+ (Backward-compatible additions that do not break reading an old table — e.g. the
627
+ `edges.resolution`/`call_line`/`context` columns added via `ALTER TABLE … ADD
628
+ COLUMN` — do NOT require a bump, since `openDatabase` migrates them in place.) On
629
+ schema bump the seed index in `/opt/mast-seed` is also invalidated and a full
630
+ reindex runs in the background.
631
+
632
+ **Fast first-task latency.** With a baked seed (§13.8), Steps 1–3 typically complete
633
+ in **2–4 seconds** on a cold container. Step 4 then catches up any files changed
634
+ since the seed was built in the background — the agent can begin useful work as
635
+ soon as Step 3 completes; JIT staleness handling (§9.0) guarantees any individual
636
+ file it queries is correct even before Step 4 reaches it.
637
+
638
+ This is the **only hook required for the SDD pipeline**. The BT orchestrator needs no
639
+ reindex calls. Files committed by the previous task are picked up by Step 4's
640
+ filesystem scan. JIT staleness handling (§9) covers files modified mid-task before
641
+ Step 4 has caught up to them.
642
+
643
+ ### 7.5 Mid-Task Reindex (`mast_reindex` MCP tool)
644
+
645
+ The agent calls `mast_reindex` immediately after writing files, before querying for
646
+ symbols it just created. This is synchronous — the tool does not return until the index
647
+ is updated. Incremental by default — only files with changed mtimes are touched. For a
648
+ typical single-file write this completes in <500ms.
649
+
650
+ ### 7.6 Write Locking
651
+
652
+ Coarse writes are coordinated by **one advisory file lock**, managed by `proper-lockfile`:
653
+
654
+ - **`<state_dir>/structure.lock`** — held during chunk parsing, graph population, and
655
+ FTS index writes (`chunks` table, `graph.db`, `chunk_fts`, `identifier_fts`) for
656
+ **coarse writers only**: `mast index`, the startup full/incremental reindex, and
657
+ `mast_reindex`. It also coordinates the manifest/`index.json` phase, which SQLite
658
+ itself can never protect (plain `writeFileSync`, not a database write).
659
+
660
+ **Why `proper-lockfile`:** it writes the acquiring process's PID into the lock file and
661
+ checks liveness on encounter. If a container is killed mid-index and a `.lock` file is
662
+ left on the shared volume, the next process that tries to acquire the lock sees a PID
663
+ that is no longer running and breaks the stale lock automatically. No manual heartbeat
664
+ or timestamp logic is needed. Always pass `{ stale: 10000 }` (10 seconds) when calling
665
+ `lockfile.lock()`; this is the maximum age a lock file is considered valid without a
666
+ live PID, and ensures clean recovery from abrupt container exits on shared volumes.
667
+
668
+ **Behaviour by caller (structure.lock):**
669
+ - **CLI commands** (`mast index`): non-blocking — if the lock cannot be acquired within
670
+ 2 seconds, exit with a clear error message naming the operation that holds it.
671
+ - **`mast_reindex` MCP tool**: blocking with retries — attempt acquisition up to 5
672
+ times with 1-second backoff, then return an error to the agent with the suggestion
673
+ to retry after the current index run completes.
674
+ - **`mast serve` startup reindex**: blocking, same retry policy as `mast_reindex`.
675
+
676
+ **If the lock is compromised while held** — another process judged it stale and took it,
677
+ or the lock directory was removed — the operation fails with `LockCompromisedError` and
678
+ must be re-run; its result is not trustworthy, because the run was not exclusive.
679
+ `proper-lockfile` detects this on its refresh timer, inside an `fs.stat` callback with no
680
+ caller on the stack, so `onCompromised` must record rather than rethrow: the library's
681
+ default rethrow reached the process as an uncaught exception and killed it
682
+ (`docs/defects/LEDGER.md` D036). The error is raised at release instead. **Work already in
683
+ flight is not interrupted** — that would require an `AbortSignal` through `withLock` and is
684
+ not implemented. The exposure is bounded: SQLite writes stay serialised by `populateFile`'s
685
+ `BEGIN IMMEDIATE` regardless of this advisory lock, so what is at risk is the plain-JSON
686
+ manifest/`index.json` phase, which the re-run rewrites.
687
+
688
+ **JIT re-parse from a read tool does NOT acquire `structure.lock`.** `structure.lock`
689
+ is one global lock per state dir with no per-file component, so — measured directly
690
+ (`eval/e7-concurrency.json`) — it made a JIT re-parse of file A block a JIT re-parse of
691
+ file B despite the two touching disjoint rows, driving JIT failure rates as high as
692
+ 88.5% under pure reader-vs-reader concurrency. Instead, the JIT write goes straight to
693
+ `populateFile` (§9.0), which opens its own transaction with `BEGIN IMMEDIATE` and a
694
+ **dedicated, short `busy_timeout` of 200ms** (`IMMEDIATE_WRITE_BUSY_TIMEOUT_MS`,
695
+ `graph/populate.ts`) — distinct from `graph.db`'s shared 5000ms connection default
696
+ (set once at `openDatabase`, `graph/db.ts`) — set immediately before the transaction and restored immediately after, so no
697
+ other statement on the connection ever inherits the short value. `BEGIN IMMEDIATE`
698
+ takes the write reservation up front instead of discovering contention on commit (the
699
+ `SQLITE_BUSY_SNAPSHOT` failure mode a plain deferred `BEGIN` is prone to), and its
700
+ `busy_timeout` wait — capped at 200ms rather than inheriting the shared 5000ms default
701
+ — is what bounds how long a genuinely contended write can hold up the calling tool. On
702
+ exhaustion (`SQLITE_BUSY`/`SQLITE_BUSY_SNAPSHOT`), fall through to the TOCTOU policy
703
+ below and return the stale chunk with a `file_busy_returning_stale_cache` flag rather
704
+ than blocking the agent indefinitely — the same contract the old lock-retry exhaustion
705
+ path used to produce, just reached by a different mechanism.
706
+
707
+ Coarse writers serialize against each other and against the manifest phase on
708
+ `structure.lock`. JIT writes serialize against each other cheaply within one `mast
709
+ serve` process (Kysely's SQLite dialect guards every connection acquisition on a `Db`
710
+ instance with an in-process mutex) and against coarse writers or other processes via
711
+ `BEGIN IMMEDIATE`'s own write-reservation semantics — not via `structure.lock`.
712
+ Concurrent readers (all MCP query tools) acquire no lock and take no part in either
713
+ mechanism — they only `stat()` files for staleness detection (§9).
714
+
715
+ `store/lockMetrics.ts`'s JSONL sink (`<state_dir>/lock-metrics.jsonl`) is the standing
716
+ instrument for this section's timing claims; `mast metrics --locks` (§14.6) summarizes
717
+ it into per-caller hold/wait percentiles and failed-acquisition counts.
718
+
719
+ ---
720
+
721
+ ## 8. CLI Interface
722
+
723
+ ### `mast init [path] [options]`
724
+
725
+ Initialise mast for a project.
726
+
727
+ ```
728
+ Options:
729
+ --state-dir <dir> Where to write index state (default: <path>/.mast)
730
+ --extensions <ext,...> File extensions to index (default: .ts,.tsx,.js,.jsx)
731
+ --exclude <pattern,...> Glob patterns to exclude
732
+ --no-index Create config only, skip initial indexing
733
+ ```
734
+
735
+ `--extensions` and `--exclude` are honoured (F9, Stage 3.5): each is a comma-separated
736
+ list, trimmed and with empty entries dropped; `--extensions` additionally normalizes
737
+ bare names to leading-dot form (`py` and `.py` are both accepted). They take priority
738
+ over `mast.config.json` and any previously-persisted `<state_dir>/config.json` — see §4
739
+ for the full priority chain.
740
+
741
+ Creates `<state_dir>/`, writes `config.json`, runs a full index. On every subsequent
742
+ `mast init`/`mast index`/`mast serve`/`mast status`/`mast query`/`mast metrics` call
743
+ against the same state directory, the customisation keys in `config.json` are read back
744
+ and applied unless a higher-priority source (CLI flags, `mast.config.json`) overrides
745
+ them (§4).
746
+
747
+ ---
748
+
749
+ ### `mast index [path] [options]`
750
+
751
+ Build or update the index.
752
+
753
+ ```
754
+ Options:
755
+ --state-dir <dir> State directory (resolved from config if omitted)
756
+ --incremental Only reindex files changed since last index run
757
+ --show-progress Print indexing progress to stderr
758
+ --checker Opt-in TypeScript-checker enrichment pass (§10.3.2) —
759
+ upgrades potential_matches into verified 'checker'
760
+ edges or drops non-call-site/wrong-declaration noise.
761
+ Holds one ts.Program at a time; can take tens of
762
+ seconds on a large monorepo — not part of the default
763
+ index path.
764
+ ```
765
+
766
+ ---
767
+
768
+ ### `mast serve [options]`
769
+
770
+ Start the MCP server over stdio.
771
+
772
+ ```
773
+ Options:
774
+ --state-dir <dir> State directory
775
+ --no-startup-reindex Skip the startup staleness check (not recommended)
776
+ --watch Watch source files and incrementally reindex on change
777
+ (interactive use — see §11.4)
778
+ ```
779
+
780
+ The server runs until the parent process (Claude CLI) closes stdin.
781
+
782
+ `--no-startup-reindex` combined with a never-indexed state dir is refused at
783
+ startup (M6 Part A, §7.4) — that combination disables the one mechanism that
784
+ would ever fill the index, so `mast serve` exits with an error instead of
785
+ silently answering every query `{"results":[]}` forever. A never-indexed
786
+ state dir with the startup reindex left enabled (the default) is unaffected
787
+ and starts normally, as does `--no-startup-reindex` against an already-indexed
788
+ state dir (including one indexed over a genuinely empty file set).
789
+
790
+ `--watch` is opt-in and intended for interactive local development; the SDD
791
+ container does not use it (§3, §11.4). The watcher is closed on stdin close,
792
+ SIGTERM, and SIGINT; a watcher startup failure logs a warning and the server
793
+ continues without watch.
794
+
795
+ ---
796
+
797
+ ### `mast status [path] [options]`
798
+
799
+ Print index health.
800
+
801
+ ```
802
+ Options:
803
+ --state-dir <dir> State directory
804
+ --json Output as JSON
805
+ ```
806
+
807
+ Output:
808
+ ```
809
+ state_dir: /workspace/.kluster/.mast
810
+ schema_version: 1.3.0
811
+ last_indexed: 2026-05-13T14:22:00Z (3 minutes ago)
812
+ indexed_files: 142
813
+ chunk_count: 1840
814
+ stale_files: 0
815
+ parse_errors: 0
816
+ write_errors: 0
817
+ index_fresh: true
818
+ freshness_cause: none
819
+ ```
820
+
821
+ `freshness_cause` carries the same semantics as the `mast_status` MCP tool (§9) —
822
+ it prints `none` in human output when the JSON value would be `null`; the JSON
823
+ value is `"phase1_stale"` when `stale_files > 0`. On a never-indexed project the
824
+ state directory is not created as a side effect of running `status`.
825
+
826
+ ---
827
+
828
+ ### `mast query <tool> [json] [path]`
829
+
830
+ Invoke any MCP tool (§9) directly from the CLI — the read tools (`mast_search`,
831
+ `mast_project_skeleton`, `mast_exports`, `mast_signature`, `mast_callers`,
832
+ `mast_dependencies`, `mast_implementors`, `mast_rename_impact`, `mast_status`,
833
+ `mast_efficiency`) and `mast_reindex`, by their exact MCP names.
834
+
835
+ ```
836
+ Options:
837
+ --state-dir <dir> State directory
838
+ --json Emit the exact single-line MCP response text (machine
839
+ use); default pretty-prints the parsed response with
840
+ 2-space indent for humans
841
+ ```
842
+
843
+ `json` (positional, default `'{}'`) is the tool's argument object as a JSON
844
+ string; `path` is the project root (same resolution as every other command's
845
+ `[path]`).
846
+
847
+ Identical-output-by-construction: `mast query` dispatches through the exact
848
+ same registered tool handler an MCP client's call would invoke — the same
849
+ schema validation, the same JIT/staleness handling, the same `_stats`
850
+ block — so CLI output can never drift from the MCP transport's, by
851
+ construction rather than by keeping two implementations in sync.
852
+
853
+ Examples:
854
+ ```
855
+ mast query mast_status '{}' /path/to/project
856
+ mast query mast_search '{"query":"add","limit":5}' /path/to/project --json
857
+ mast query mast_exports '{"file_path":"src/math.ts"}'
858
+ ```
859
+
860
+ Error behavior (all exit 1, message to stderr):
861
+ - **Unknown tool** — lists every registered tool name.
862
+ - **Malformed JSON argument** — names the parse failure.
863
+ - **Args that fail the tool's own zod schema** — the zod issues.
864
+ - **State dir with no `graph.db`** (never-indexed project) —
865
+ ``no index found at <state_dir>; run `mast init` / `mast index` first``.
866
+ This is `mast query`'s own fail-fast guard, parallel to `mast serve`'s
867
+ `--no-startup-reindex` refusal (M6 Part A, §7.4). Because `mast query`
868
+ dispatches through the same registered tool handlers, the M6 Part B
869
+ `index_empty` signal (§9.0) appears on its responses automatically — an
870
+ indexed-but-empty corpus queries fine and says so.
871
+
872
+ ---
873
+
874
+ ### `mast install-hooks [path]`
875
+
876
+ Write git hooks into `<path>/.git/hooks/`:
877
+ - `post-commit` → `mast index <path> --incremental`
878
+ - `post-checkout` → `mast index <path> --incremental`
879
+
880
+ For developer ergonomics outside the SDD pipeline. Not required for the automated
881
+ pipeline.
882
+
883
+ ---
884
+
885
+ ## 9. MCP Tools
886
+
887
+ All tools are exposed on the `mast` MCP server. Tool names follow the convention
888
+ `mast_<action>`.
889
+
890
+ ---
891
+
892
+ ### 9.0 Staleness Handling (All Read Tools)
893
+
894
+ Every read tool that returns line coordinates (`mast_search`, `mast_signature`,
895
+ `mast_exports`, `mast_callers`, `mast_dependencies`, `mast_implementors`,
896
+ `mast_rename_impact`) performs a **mandatory server-side staleness check**
897
+ before returning. This is
898
+ not optional and is not controlled by the caller — the index is responsible for
899
+ its own consistency, not the agent. `mast_project_skeleton` is exempt: its
900
+ response is a directory map of exported symbol names with no line coordinates
901
+ to go stale (P3).
902
+
903
+ **The agent must never see a chunk whose line coordinates do not match the
904
+ current file on disk without being told.** Returning stale line numbers
905
+ unflagged leads directly to agent-assisted corruption: the agent issues an
906
+ `Edit` against the stale range and overwrites unrelated logic. This class of
907
+ failure does not surface as an error — it surfaces as silent, hard-to-attribute
908
+ breakage downstream. Two different mechanisms enforce this, chosen per tool by
909
+ how many files a single call's results can span:
910
+
911
+ - **Just-In-Time (JIT) re-parse** (`mast_signature`, `mast_exports`,
912
+ `mast_callers`, `mast_dependencies`, `mast_rename_impact`) — these tools'
913
+ results are scoped to one file, or a small, explicitly-named set, so a
914
+ stale result can be transparently refreshed in place. See below.
915
+ - **Stat-and-flag** (`mast_search`, `mast_implementors`) — these tools can
916
+ return results spanning dozens of files in one call, so JIT re-parsing
917
+ every result file would mean up to ~50 tree-sitter re-parses and write
918
+ transactions per call, and re-parsing a result file mid-response could
919
+ shift its rank, gain or lose a match, or change its chunk boundaries —
920
+ invalidating the ranking/query that already selected the result being
921
+ "refreshed". Instead,
922
+ after results are computed, each **unique** result `file_path` is
923
+ `statSync`'d (no lock, no re-parse, no DB write) and its disk mtime
924
+ compared against the indexed `files.mtime`. Newer-on-disk, or a failed
925
+ stat (file deleted/renamed since indexing — its coordinates are
926
+ definitely untrustworthy), sets `stale: true` on that result; a file
927
+ absent from the `files` table (nothing indexed to be stale against) is
928
+ left unflagged. **C1** split this signal into its own `stale` field,
929
+ distinct from the JIT-refresh tools' `file_busy_returning_stale_cache`
930
+ below — no lock is ever taken here, so nothing is ever "busy" in that
931
+ sense; see the "Confidence signals (C1)" table below for the full
932
+ picture.
933
+
934
+ **Just-In-Time (JIT) re-parse.** For every result a JIT-refresh tool is about to return:
935
+
936
+ 1. `fs.stat()` the `file_path`. Compare disk `mtime` against the chunk's stored
937
+ `file_mtime`.
938
+ 2. If `disk_mtime <= stored_mtime` → return the result unchanged. Fast path.
939
+ 3. If `disk_mtime > stored_mtime` → the chunk is stale. Re-index **this file
940
+ only** (one tree-sitter parse, one `BEGIN IMMEDIATE` transactional
941
+ delete-and-replace against the `chunks` table, `graph.db`, `chunk_fts`,
942
+ `identifier_fts` — see §7.6; no `structure.lock` acquisition on this path).
943
+ Re-resolve the tool's result against the refreshed chunks. A single-file
944
+ re-parse typically completes in 10–50ms; the transactional write itself is
945
+ bounded by a dedicated 200ms `busy_timeout` (§7.6), not the connection's
946
+ shared 5000ms default a genuinely contended write would otherwise wait
947
+ out.
948
+
949
+ JIT re-parse covers files already known to the index. It does not discover a
950
+ brand-new file or a newly-created symbol — those become searchable via the next
951
+ `mast_reindex` call or the background/`--watch` reindex (§7.4/§11.4) reaching
952
+ them. The agent prompt should still recommend `mast_reindex` after writing new
953
+ files or symbols — not because JIT leaves existing files stale (it doesn't), but
954
+ because discovery of new ones requires an actual indexing pass.
955
+
956
+ **TOCTOU Policy (file mid-write).** Between `stat()` and `parse()`, the file
957
+ may be in the middle of being written by another process (e.g., the agent's
958
+ own `Edit` tool, an editor save, a `git checkout`). Tree-sitter on partial
959
+ source either throws or produces a truncated tree. The policy:
960
+
961
+ 1. On parse error or empty tree, sleep **50ms** and retry **once**.
962
+ 2. If the second attempt also fails, return the **previous (stale) chunk** with
963
+ a `file_busy_returning_stale_cache: true` flag on the result. Do not throw
964
+ — the agent has no recovery for a thrown error, but it can interpret the
965
+ flag.
966
+ 3. If the file no longer exists (deleted), remove its rows from the graph and
967
+ chunk stores and return whatever results remain (possibly empty).
968
+
969
+ **`identifier_fts` and `chunk_fts` updates.** The JIT re-parse rewrites the
970
+ FTS5 rows for the affected file in the same transaction as the chunk/graph
971
+ rewrite. There is no separate sync step.
972
+
973
+ **Concurrency.** Two simultaneous read tools targeting different stale files
974
+ no longer serialize on any lock (§7.6) — each parses its own file fully in
975
+ parallel, and only briefly contends on the transactional write. That write
976
+ is bounded by the dedicated 200ms `busy_timeout`
977
+ (`IMMEDIATE_WRITE_BUSY_TIMEOUT_MS`, §7.6): under real contention, `BEGIN
978
+ IMMEDIATE`'s `busy_timeout` wait is a **synchronous** hold on the whole
979
+ `mast serve` process's event loop (better-sqlite3's busy-wait is native and
980
+ blocks the process, not just the calling request) — 200ms is the accepted
981
+ trade at this magnitude, comparable to the 3×100ms lock-retry budget the
982
+ pre-F11 JIT path used to pay, and far below the connection's shared 5000ms
983
+ default, which would otherwise freeze the entire process for up to 5 seconds
984
+ per contended write.
985
+
986
+ **Result shape.** Every JIT-refresh read tool's result objects MAY include
987
+ `file_busy_returning_stale_cache: true` (omitted when false). Result schemas
988
+ in the per-tool sections below document only the steady-state shape; this flag
989
+ is implicit on all of them. Tools whose response is a single-file envelope
990
+ (`mast_exports`, `mast_dependencies`) or whose staleness taints the whole
991
+ answer (`mast_callers`, `mast_rename_impact`) carry the flag at the envelope
992
+ level instead of per-entry. `mast_signature` carries it per-result — except
993
+ when a `file_path`-narrowed query returns **zero** results while that file's
994
+ JIT re-parse could not acquire the lock: with no result objects to carry the
995
+ signal, the flag appears on the response envelope (F14), so "no results" from
996
+ a stale, un-refreshable file never reads as "symbol doesn't exist".
997
+ `mast_search` and `mast_implementors` carry a **different** field, `stale`,
998
+ per-result — via stat-and-flag rather than JIT re-parse (F7, see above; split
999
+ into its own field by C1, see the table below) — each result's `stale`
1000
+ reflects that result's own `file_path` statting newer-on-disk or failing to
1001
+ stat, independent of every other result in the same response.
1002
+
1003
+ **Empty-index signal (M6 Part B).** Every read tool with a primary result
1004
+ array — `mast_search`, `mast_project_skeleton`, `mast_exports`,
1005
+ `mast_signature`, `mast_callers`, `mast_dependencies`, `mast_implementors`,
1006
+ `mast_rename_impact` — MAY attach `index_empty: true` to its response
1007
+ envelope. Present only when BOTH (a) that tool's primary result set came back
1008
+ empty (for `mast_callers`, both `verified_callers` AND `potential_matches`;
1009
+ for `mast_rename_impact`, all four of `declaration_sites`, `verified_callers`,
1010
+ `potential_matches`, and `barrel_exports`) AND (b) the `chunks` table has zero
1011
+ rows at that moment (`mcp/tools/_helpers.ts`'s `isIndexEmpty`, checked only on
1012
+ the already-empty-result path — a populated response never pays for this
1013
+ check). Distinguishes "`[]` because nothing is indexed yet, or you pointed at
1014
+ the wrong state dir" from "`[]` because no match" — the exact ambiguity M6
1015
+ (`eval/GITNEXUS_COMPARISON.md` §13.8 item 4) names, and the legitimate empty
1016
+ window §7.4's startup ladder deliberately leaves servable (see the
1017
+ `--no-startup-reindex` refusal note in §7.4, which catches only the
1018
+ never-recoverable case, not this one). Omitted entirely when false, same
1019
+ present-only-when-true convention as `file_busy_returning_stale_cache` above
1020
+ — never present-and-false. Independent of `mast_search`'s `suggestions` field:
1021
+ a truly empty index yields no suggestions either, but the two are not
1022
+ coupled — either may be present without the other. `mast_status` is
1023
+ unaffected (it already reports `chunk_count`/`index_fresh` directly — it IS
1024
+ the diagnostic surface); `mast_efficiency` and `mast_reindex` have no primary
1025
+ result array and never carry this flag.
1026
+
1027
+ #### Confidence signals (C1)
1028
+
1029
+ MAST does not compute a single scalar "confidence score" for a result — it
1030
+ never has, and C1 does not introduce one (`eval/GITNEXUS_COMPARISON.md`
1031
+ §13.8 item 5 / §14.8 item 5: "frame as unification, not a new feature").
1032
+ What existed before C1 was a set of independently-evolved signals, computed
1033
+ in different tools for different reasons, some sharing one misleading field
1034
+ name. C1's only change is documentation and one field split — no new enum,
1035
+ no wrapper object, no field beyond the rename (see the F7 result's
1036
+ "Known naming tension" note above, and `adr/008-2026-08-09-honest-surfaces.md`;
1037
+ the C1 result block is `adr/proposals/honest-surfaces/PLAN-EXCERPT.md` § "C1 result").
1038
+ This table is the single place that lists every signal an agent may see
1039
+ across all MCP tools, and what to do with each one:
1040
+
1041
+ | Field | Carried by | Meaning | Agent action |
1042
+ |---|---|---|---|
1043
+ | `resolution` | `VerifiedCaller` entries (`mast_callers`, `mast_rename_impact`) | How this call site was statically resolved to the queried declaration — one of eight values (`import`, `field_type`, `parameter_type`, `new_expression`, `same_file`, `checker`, `this_method`, `super_method` — the last two added by F4, Stage 3, for `this.foo()`/`super.foo()` call sites). | High confidence. Safe to act on directly (e.g. as a rename/refactor site) without further verification. |
1044
+ | `reason` | `PotentialMatch` entries (`mast_callers`, `mast_rename_impact`) | Why this call site could **not** be statically resolved — currently always `identifier_match_no_resolved_edge`. | Mandatory review. This is a name-match, not a verified edge; confirm it is a real call site before acting on it. |
1045
+ | `file_busy_returning_stale_cache` | JIT-refresh tools' results/envelopes (`mast_signature`, `mast_exports`, `mast_callers`, `mast_dependencies`, `mast_rename_impact`) | A refresh **was attempted** (this file's JIT re-parse) and lost to genuine write contention (`populateFile`'s `BEGIN IMMEDIATE` exhausted its `busy_timeout`), so the previous, possibly-stale chunk was returned instead. | Contended, not wrong-by-design. Retry shortly — the contention is expected to clear (§7.6). |
1046
+ | `stale` | `mast_search` / `mast_implementors` per-result (F7) | This result's `file_path` stat'd newer-on-disk than its indexed mtime, or the stat failed — **no refresh was attempted by design** (stat-and-flag, not JIT re-parse; see above). | Treat this result's line coordinates as untrustworthy. A `mast_reindex` call, or any JIT-refreshing tool call against the file, heals it. |
1047
+ | `index_empty` | Every primary-result read tool's envelope (M6) | Nothing is indexed at all — the empty result set is not "no match", it is "no index (yet)". | Run `mast init`/`mast index`, or — if a startup reindex is in progress — wait and retry. |
1048
+ | `truncated` | `TypeContextEntry` (`mast_signature`'s `type_context`) | This referenced type's declaration was clipped at the 50-line cap. | Re-read the file directly (or call `mast_exports`/a narrower `mast_signature` query) for the full declaration if the clipped portion matters. |
1049
+ | `potential_truncated` | `CallersResponse.summary` / `RenameImpactResponse.summary` (`mast_callers`, `mast_rename_impact`) | The `identifier_fts` fetch behind `potential_matches` is capped at 50 entries; this carries the real, uncapped match count when the cap is hit (F10, Stage 3). Reports RAW fetch truncation only — `potential_matches` may still be smaller than the cap even when this field is present, because verified-overlap exclusion and checker-verdict filtering run AFTER the capped fetch (already visible via `checker_classified_*`). | The potential set is incomplete — narrow the query, or run `mast index --checker` to classify candidates away. |
1050
+
1051
+ `file_busy_returning_stale_cache`, `stale`, `index_empty`, and
1052
+ `potential_truncated` all follow the same **omitted-when-false /
1053
+ present-only-when-true** convention (never present-and-false) established
1054
+ above — `potential_truncated`'s "false" case is "the fetch came back under
1055
+ the cap," where the fetch count already IS the real count. `resolution` and
1056
+ `reason` are always
1057
+ present on their respective entry types (a `VerifiedCaller`/`PotentialMatch`
1058
+ without one would be meaningless). `truncated` is the one exception: it is
1059
+ an always-present `boolean` on every `TypeContextEntry`, not an optional
1060
+ flag — callers check its value rather than its presence.
1061
+
1062
+ ---
1063
+
1064
+ ### `mast_search`
1065
+
1066
+ Lexical BM25 + declaration-exact search via RRF (§7.3). Returns chunks, not full files.
1067
+
1068
+ **Input:**
1069
+ ```json
1070
+ {
1071
+ "query": "string",
1072
+ "limit": 10,
1073
+ "language": "typescript | javascript | markdown | null",
1074
+ "file_pattern": "glob pattern | null",
1075
+ "chunk_type": "function | method | class_shell | interface | type | export | block | doc | null",
1076
+ "only_exported": false
1077
+ }
1078
+ ```
1079
+
1080
+ `only_exported: true` restricts results to chunks where `is_exported = true`. Use
1081
+ this when looking for a service or utility to call into — it eliminates internal
1082
+ implementation details from results.
1083
+
1084
+ **Output:** `SearchResponse`
1085
+ ```json
1086
+ {
1087
+ "results": [
1088
+ {
1089
+ "file_path": "api/services/auth/src/handler.ts",
1090
+ "start_line": 42,
1091
+ "end_line": 58,
1092
+ "content": "export async function handleLogin(...) {",
1093
+ "chunk_type": "function",
1094
+ "symbol_name": "handleLogin",
1095
+ "parent_symbol": null,
1096
+ "is_exported": true,
1097
+ "match_score": -4.21,
1098
+ "rank": 1,
1099
+ "match_snippet": "...async function handleLogin(req: LoginRequest..."
1100
+ }
1101
+ ]
1102
+ }
1103
+ ```
1104
+
1105
+ `match_snippet` is produced by SQLite's `snippet(chunk_fts, 0, '**', '**', '...', 12)`
1106
+ function. It returns a short fragment of the chunk content with matched terms marked,
1107
+ so the agent can see immediately why the result was returned without reading the full
1108
+ `content` field. The fragment length (12 tokens) is configurable.
1109
+
1110
+ `match_score` carries the BM25 score (negative — §7.3's FTS5 sign convention) when the
1111
+ FTS ranker produced a hit for this chunk, and `null` when the chunk reached the result
1112
+ set only through ranker D (declaration-exact, §7.3). `rank` is the chunk's position in
1113
+ the RRF-fused list and is always present, starting at 1.
1114
+
1115
+ `parent_symbol` is populated only on `method` chunks (carries the enclosing
1116
+ class name); `null` for all other chunk types.
1117
+
1118
+ **Zero-result assist (`suggestions`).** When a search returns no results — no
1119
+ FTS or ranker-D hit at all, or the `chunk_type` / `only_exported` filters emptied
1120
+ the set — the tool does not return a bare dead end. It runs a relaxation pass
1121
+ and attaches a `suggestions` array of `{ symbol, file_path, reason }` "did you
1122
+ mean" candidates.
1123
+
1124
+ ```json
1125
+ {
1126
+ "results": [],
1127
+ "suggestions": [
1128
+ { "symbol": "handleLogin", "file_path": "api/services/auth/src/handler.ts", "reason": "similar symbol name" },
1129
+ { "symbol": "handleLogout", "file_path": "api/services/auth/src/handler.ts", "reason": "matched split query terms" }
1130
+ ]
1131
+ }
1132
+ ```
1133
+
1134
+ Candidates are gathered from three complementary passes, de-duplicated by
1135
+ `(symbol, file_path)` and capped at `limit`:
1136
+
1137
+ - **Trigram symbol-name similarity** against the `symbols` table (Dice
1138
+ coefficient over character trigrams; SQLite ships no `pg_trgm`, so the ranking
1139
+ is computed in-process). `reason: "similar symbol name"`.
1140
+ - **FTS retry over split query terms** — the query is split on camelCase,
1141
+ acronym, and snake/kebab boundaries (`getUserById` → `get`, `user`), then
1142
+ re-run against `chunk_fts`. `reason: "matched split query terms"`.
1143
+ - **`identifier_fts` near-miss** — the same split terms are OR-matched against
1144
+ the identifier index. `reason: "identifier near-miss"`.
1145
+
1146
+ **Trigger and contract.** `suggestions` is present (possibly empty) **only when
1147
+ `results` is empty**, and is omitted from the response entirely when results
1148
+ were found. Suggestions are advisory: they are **never** promoted into
1149
+ `results`, so `results` stays `[]` on the assist path. The agent should treat
1150
+ them as vocabulary hints to re-query with, not as answers.
1151
+
1152
+ **Shell/method dedup (`related`).** Class decomposition (§10.1) means one query
1153
+ can match both a `method` chunk and its parent `class_shell`, whose synthesized
1154
+ outline repeats the same signature — returning both charges the agent twice for
1155
+ one fact. A post-RRF presentation pass therefore keeps only the higher-ranked
1156
+ half of every shell↔method pair and attaches a `related` hint to the survivor:
1157
+
1158
+ - surviving `method` (shell suppressed): `"related": { "parent_symbol": "AuthService" }`
1159
+ — the class outline also matched; query the class name for the full picture.
1160
+ When several methods of the class survive, the hint lands on the
1161
+ highest-ranked one only.
1162
+ - surviving `class_shell` (methods suppressed): `"related": { "methods_matched":
1163
+ ["AuthService.validateSession", "AuthService.refresh"] }` — these specific
1164
+ members also matched; read them next instead of re-searching.
1165
+
1166
+ Rules: shell and method are paired by `parent_symbol` + the same `file_path`
1167
+ (same-named classes in different files never collapse); methods never suppress
1168
+ each other; ranking math is untouched — suppression frees slots that are
1169
+ backfilled from the RRF candidate tail, so the response still returns up to
1170
+ `limit` distinct results, and `rank` values are re-assigned after dedup so they
1171
+ remain contiguous from 1. `related` is absent when no collision occurred.
1172
+
1173
+ **When used:** primary code discovery — replaces `Grep`, `Glob`, and exploratory `Read`.
1174
+
1175
+ ---
1176
+
1177
+ ### `mast_project_skeleton`
1178
+
1179
+ All file paths in the project with their exported top-level symbol names. Returns
1180
+ names only — no signatures, no bodies. Designed to give the agent a compressed
1181
+ architectural overview in a single call.
1182
+
1183
+ **Input:**
1184
+ ```json
1185
+ {
1186
+ "directory": "api/services/auth | null",
1187
+ "max_depth": 2,
1188
+ "file_pattern": "glob pattern | null"
1189
+ }
1190
+ ```
1191
+
1192
+ `directory` scopes the skeleton to a subtree of the project — essential for large
1193
+ monorepos where a full skeleton would be noisy. `max_depth` limits directory traversal
1194
+ depth relative to `directory` (or `project_root` if omitted). Both are optional;
1195
+ omitting both returns the full project skeleton.
1196
+
1197
+ **Output:** array of `FileSkeleton`
1198
+ ```json
1199
+ [
1200
+ {
1201
+ "file_path": "api/services/auth/src/index.ts",
1202
+ "exports": ["handleLogin", "handleLogout", "AuthPlugin"]
1203
+ },
1204
+ {
1205
+ "file_path": "api/services/auth/src/repository.ts",
1206
+ "exports": ["AuthRepository"]
1207
+ }
1208
+ ]
1209
+ ```
1210
+
1211
+ Sourced entirely from the `chunks` table where `is_exported = true` — no tree-sitter
1212
+ reparsing at query time.
1213
+
1214
+ **When used:** early in a task for orientation — "what services exist and what do they
1215
+ expose?" before deciding which files to query further. Replaces opening multiple
1216
+ `index.ts` barrel files. Scope with `directory` when working within a known service
1217
+ subtree to avoid noise from unrelated packages.
1218
+
1219
+ ---
1220
+
1221
+ ### `mast_exports`
1222
+
1223
+ All exported symbols from a single file with type signatures. No function bodies.
1224
+
1225
+ **Input:**
1226
+ ```json
1227
+ {
1228
+ "file_path": "api/services/auth/src/index.ts"
1229
+ }
1230
+ ```
1231
+
1232
+ **Output:** array of `Export`
1233
+ ```json
1234
+ [
1235
+ {
1236
+ "name": "handleLogin",
1237
+ "kind": "function",
1238
+ "signature": "export async function handleLogin(req: LoginRequest, reply: FastifyReply): Promise<void>",
1239
+ "line": 42,
1240
+ "doc": "Validates credentials and issues a session token. Throws UnauthorizedError on failure."
1241
+ },
1242
+ {
1243
+ "name": "AuthPlugin",
1244
+ "kind": "class",
1245
+ "signature": "export class AuthPlugin implements FastifyPlugin",
1246
+ "line": 12,
1247
+ "doc": null
1248
+ }
1249
+ ]
1250
+ ```
1251
+
1252
+ **When used:** "What does this file expose?" before deciding whether to open it.
1253
+ Replaces a full-file `Read` when the agent only needs the public API surface.
1254
+
1255
+ **Implementation:** tree-sitter AST walk — extract `export_statement` nodes and
1256
+ `export_clause` nodes. For each exported declaration, extract the declaration node
1257
+ up to (not including) the body `{ ... }` block. Extract leading TSDoc/JSDoc comment.
1258
+
1259
+ ---
1260
+
1261
+ ### `mast_signature`
1262
+
1263
+ Declaration, TSDoc, and resolved parameter type context for a named symbol.
1264
+
1265
+ **Input:**
1266
+ ```json
1267
+ {
1268
+ "symbol": "handleLogin",
1269
+ "file_path": "api/services/auth/src/handler.ts"
1270
+ }
1271
+ ```
1272
+
1273
+ `file_path` is optional. When omitted, all matching symbols across the codebase are
1274
+ returned. If multiple matches are found and the caller only expects one, pass `file_path`
1275
+ to disambiguate. There is no "first match" shortcut — an ambiguous query always returns
1276
+ the full match set so the agent can choose.
1277
+
1278
+ **Output:** `SignatureResult[]` — always an array, even for a single match.
1279
+ ```json
1280
+ [
1281
+ {
1282
+ "symbol": "handleLogin",
1283
+ "file_path": "api/services/auth/src/handler.ts",
1284
+ "line": 42,
1285
+ "signature": "export async function handleLogin(req: LoginRequest, reply: FastifyReply): Promise<void>",
1286
+ "doc": "Validates credentials and issues a session token. Throws UnauthorizedError on failure.",
1287
+ "params": [
1288
+ { "name": "req", "type": "LoginRequest" },
1289
+ { "name": "reply", "type": "FastifyReply" }
1290
+ ],
1291
+ "return_type": "Promise<void>",
1292
+ "type_context": [
1293
+ {
1294
+ "name": "LoginRequest",
1295
+ "signature": "interface LoginRequest { email: string; password: string; }",
1296
+ "file_path": "api/services/auth/src/types.ts",
1297
+ "line": 3,
1298
+ "truncated": false
1299
+ }
1300
+ ]
1301
+ }
1302
+ ]
1303
+ ```
1304
+
1305
+ When `file_path` is omitted and multiple files define a symbol with the same name,
1306
+ `type_context` is still resolved per entry using that entry's containing file as the
1307
+ resolution root. Entries are ordered by `is_exported` descending (exported symbols
1308
+ first), then by `file_path` ascending.
1309
+
1310
+ **Transitive export resolution:** when looking up a symbol by name, the `symbols`
1311
+ table is queried first. If the symbol is not found directly but a `RE_EXPORTS` edge
1312
+ or `re_export_files` row exists pointing from the queried file to another file, the
1313
+ lookup recurses through that chain until the source declaration is found. This handles
1314
+ barrel file patterns where `packages/shared/src/index.ts` re-exports everything from
1315
+ internal modules three levels deep. The recursive CTE for this lookup mirrors the
1316
+ callers pattern already in §6.3.
1317
+
1318
+ **`type_context` resolution rules:**
1319
+ 1. Extract all named types from `params` and `return_type`.
1320
+ 2. For each type name, resolve the declaration using this priority order:
1321
+ a. **Same file first:** query `symbols` where `file_id = <containing file's id>` and
1322
+ `name = <type name>`. This handles types defined alongside the function.
1323
+ b. **Imports:** query `imports` for the containing file; find the row where `symbols`
1324
+ contains the type name. Resolve `resolved_path` using the path resolver (see
1325
+ §13.7) for tsconfig aliases and workspace packages.
1326
+ c. **RE_EXPORTS chain:** if not found via imports, walk the `re_export_files` chain
1327
+ recursively (same CTE as §6.3 barrel resolution) to find a file that defines the
1328
+ type. This handles barrel files that re-export types without explicit `import`.
1329
+ d. **Not found:** treat as external; omit from `type_context` (do not error).
1330
+ 3. If found in the monorepo: extract the declaration from `graph.db` `symbols` table —
1331
+ signature only, no body. Do not re-parse the file with tree-sitter.
1332
+ 4. Include as a `type_context` entry.
1333
+ 5. **One level deep only.** Do not recurse into the fields of resolved types.
1334
+ 6. **Monorepo types only.** Types from `node_modules` (external) are not resolved.
1335
+ 7. **Simple type references only.** Generic parameters (`Promise<T>`) are resolved
1336
+ for their type argument (`T`) if it is a named monorepo type. Union members
1337
+ (`string | LoginRequest`) are resolved individually.
1338
+ 8. **Line limit:** if a resolved type declaration exceeds 50 lines, truncate to the
1339
+ first 50 lines and set `truncated: true` on the entry. This prevents a large
1340
+ generated interface from bloating the response.
1341
+
1342
+ This saves the agent a predictable second tool call when it needs to understand a
1343
+ function's contract in full.
1344
+
1345
+ **When used:** "What does this function accept and return, and what shape are those
1346
+ types?" Replaces opening the file plus following imports to type definitions.
1347
+
1348
+ ---
1349
+
1350
+ ### `mast_callers`
1351
+
1352
+ Call sites of a named symbol, partitioned into **verified** and **potential** sets.
1353
+ The split is fundamental to the tool's contract — see §10.3 for why.
1354
+
1355
+ **Input:**
1356
+ ```json
1357
+ {
1358
+ "symbol": "handleLogin",
1359
+ "file_path": "api/services/auth/src/handler.ts",
1360
+ "transitive": false,
1361
+ "include_potential": true
1362
+ }
1363
+ ```
1364
+
1365
+ - `transitive: true` returns callers of callers via recursive CTE on `POTENTIAL_CALL`
1366
+ edges (the verified set only — the recursion does not traverse `potential_matches`).
1367
+ Default is direct callers only.
1368
+ - `include_potential: false` skips the `identifier_fts` query entirely. Useful when
1369
+ the agent wants only edges the resolver could statically link, accepting that the
1370
+ result is incomplete. Default is `true`.
1371
+
1372
+ **Output:** `CallersResponse`
1373
+ ```json
1374
+ {
1375
+ "verified_callers": [
1376
+ {
1377
+ "file_path": "api/server/src/routes/auth.ts",
1378
+ "line": 28,
1379
+ "caller_symbol": "registerAuthRoutes",
1380
+ "context": " return handleLogin(request, reply)",
1381
+ "resolution": "import"
1382
+ },
1383
+ {
1384
+ "file_path": "api/services/auth/src/middleware.ts",
1385
+ "line": 64,
1386
+ "caller_symbol": "AuthMiddleware.authenticate",
1387
+ "context": " await this.handler.handleLogin(req, reply)",
1388
+ "resolution": "field_type"
1389
+ }
1390
+ ],
1391
+ "potential_matches": [
1392
+ {
1393
+ "file_path": "api/services/admin/src/audit-log.ts",
1394
+ "line": 142,
1395
+ "context": " // TODO: emit event when handleLogin is called",
1396
+ "reason": "identifier_match_no_resolved_edge"
1397
+ }
1398
+ ],
1399
+ "summary": {
1400
+ "verified_count": 2,
1401
+ "potential_count": 1,
1402
+ "transitive": false,
1403
+ "checker_classified_non_call_site": 0,
1404
+ "checker_classified_different_declaration": 0
1405
+ }
1406
+ }
1407
+ ```
1408
+
1409
+ `summary.potential_truncated` is omitted above because this query's `identifier_fts`
1410
+ fetch came back under the 50-entry cap. When it doesn't — e.g. the `isUndefined`
1411
+ query that motivated F10, which had 71 real matches — the fetch is capped at 50 and
1412
+ `summary` instead carries:
1413
+ ```json
1414
+ {
1415
+ "potential_count": 50,
1416
+ "potential_truncated": 71
1417
+ }
1418
+ ```
1419
+ `potential_count` (50, capped) and `potential_truncated` (71, real) diverging is the
1420
+ signal: the potential set is known-incomplete, not merely large.
1421
+
1422
+ **The two sets have different meanings.** Tools and prompts must treat them
1423
+ differently:
1424
+
1425
+ - **`verified_callers`** — a call site the graph statically linked to the queried
1426
+ symbol. The `resolution` field names how: five values come from the local heuristic
1427
+ resolver (§10.3) — `import` (top-level named import), `field_type` (`this.x` where
1428
+ `x` is a class field with a known type annotation), `parameter_type` (parameter
1429
+ property or annotated parameter), `new_expression` (`new Foo()`-style construction),
1430
+ `same_file` (call site and definition in the same file) — and one, `checker`, comes
1431
+ from the opt-in `mast index --checker` pass (§10.3.2): a call site the heuristic left
1432
+ as `potential` that `ts.TypeChecker.getSymbolAtLocation` resolved to the queried
1433
+ declaration. All six are high confidence; safe to act on.
1434
+
1435
+ - **`potential_matches`** — `identifier_fts` matched the symbol name exactly inside a
1436
+ chunk, but neither the heuristic resolver nor (if it has run) the checker pass could
1437
+ statically link it. These are *candidates that require human or agent review* before
1438
+ any refactor proceeds. Common causes: factory patterns, DI container lookups,
1439
+ inferred types, dynamic dispatch, comments and string literals containing the
1440
+ identifier. The `reason` field is informational; v1 always returns
1441
+ `identifier_match_no_resolved_edge`. **A qualified `"Class.method"` query (§9's
1442
+ documented convention) now actually matches** (F5, schema 1.3.0): `identifier_fts`
1443
+ rows carry qualified compounds (see §6.3's DDL comment), so this set typically
1444
+ includes the declaration chunk itself plus any call site the resolver linked to a
1445
+ receiver type but whose edge was later dropped by file-scoped resolution (e.g. an
1446
+ unresolvable import) — before F5 this set was confidently empty for essentially
1447
+ every method query (58% of symbols in the eval corpus), independent of whether a
1448
+ real caller existed.
1449
+
1450
+ **`summary.checker_classified_non_call_site` / `checker_classified_different_declaration`**
1451
+ count candidates the checker pass classified away — not a real call site (comment,
1452
+ string, type position) or a same-name collision resolving to a different declaration
1453
+ — that would otherwise still be sitting in `potential_matches` as unresolved review
1454
+ noise. Both are `0` when `mast index --checker` has never run against this index; a
1455
+ nonzero value is direct evidence the pass ran and is doing its job (§10.3.2).
1456
+
1457
+ **`summary.potential_truncated`** (F10, Stage 3) carries the real, uncapped
1458
+ `identifier_fts` match count — present only when that fetch (capped at 50 entries)
1459
+ came back full and the true count exceeds it. It is a **raw fetch** signal, computed
1460
+ BEFORE the verified-overlap exclusion and checker-verdict filtering that produce
1461
+ `potential_matches`/`potential_count` — so `potential_matches` can still be smaller
1462
+ than 50 even when `potential_truncated` is present; that's filtering (already visible
1463
+ via `checker_classified_*` above), not truncation. See the Confidence signals (C1)
1464
+ table (§9.0) for the full contract.
1465
+
1466
+ **Why partition rather than merge?** Mixing the two sets would force the agent to
1467
+ treat every result as low-confidence, defeating the value of the verified set. Mixing
1468
+ also means a single false-positive in the FTS hits can derail a refactor. The split
1469
+ preserves "I know this is a caller" as a distinct, actionable category.
1470
+
1471
+ **Agent prompt guidance.** The implement-task prompt (§12) MUST instruct the agent:
1472
+ "`verified_callers` are the impact set you can rely on. `potential_matches` are
1473
+ mandatory review sites — open each one and confirm whether it is a real caller before
1474
+ shipping a refactor."
1475
+
1476
+ **When used:** understanding impact before modifying a function. Sourced from
1477
+ `graph.db` `edges` table (verified) + `identifier_fts` (potential). When
1478
+ `transitive: true`, only the verified set recurses.
1479
+
1480
+ ---
1481
+
1482
+ ### `mast_dependencies`
1483
+
1484
+ Import graph for a file — what it imports and from where.
1485
+
1486
+ **Input:**
1487
+ ```json
1488
+ {
1489
+ "file_path": "api/services/auth/src/handler.ts"
1490
+ }
1491
+ ```
1492
+
1493
+ **Output:** `DependencyResult`
1494
+ ```json
1495
+ {
1496
+ "file_path": "api/services/auth/src/handler.ts",
1497
+ "imports": [
1498
+ {
1499
+ "module": "@kluser-kinetic-01/shared",
1500
+ "symbols": ["UserId", "SessionToken"],
1501
+ "is_external": true
1502
+ },
1503
+ {
1504
+ "module": "../repository",
1505
+ "symbols": ["AuthRepository"],
1506
+ "is_external": false,
1507
+ "resolved_path": "api/services/auth/src/repository.ts"
1508
+ }
1509
+ ]
1510
+ }
1511
+ ```
1512
+
1513
+ **When used:** understanding what a file pulls in before modifying it. Sourced from
1514
+ `graph.db` `imports` table.
1515
+
1516
+ ---
1517
+
1518
+ ### `mast_implementors`
1519
+
1520
+ Concrete classes that implement a given interface.
1521
+
1522
+ **Input:**
1523
+ ```json
1524
+ {
1525
+ "interface_name": "AuthRepository"
1526
+ }
1527
+ ```
1528
+
1529
+ **Output:** array of `ImplementorResult`
1530
+ ```json
1531
+ [
1532
+ {
1533
+ "class_name": "PostgresAuthRepository",
1534
+ "file_path": "api/services/auth/src/postgres-repository.ts",
1535
+ "line": 8,
1536
+ "methods": ["findByEmail", "createSession", "invalidateSession"]
1537
+ }
1538
+ ]
1539
+ ```
1540
+
1541
+ **When used:** finding the concrete implementation to follow as a pattern or extend.
1542
+ Sourced from `graph.db` `edges` where `edge_type = 'IMPLEMENTS'`.
1543
+
1544
+ ---
1545
+
1546
+ ### `mast_rename_impact`
1547
+
1548
+ Composed refactor checklist for renaming a symbol. One call packages what an
1549
+ agent would otherwise stitch together from `mast_callers` + manual barrel-file
1550
+ inspection: every section reuses an existing query capability — no new
1551
+ resolution logic.
1552
+
1553
+ **Input:**
1554
+ ```json
1555
+ {
1556
+ "symbol": "createPolicyGroup",
1557
+ "file_path": "api/services/policy/src/service.ts | null"
1558
+ }
1559
+ ```
1560
+
1561
+ `file_path` disambiguates duplicate names, with the same semantics as
1562
+ `mast_signature`. Methods are addressed by qualified name
1563
+ (`ClassName.methodName`), exactly as `mast_callers` accepts them.
1564
+
1565
+ **Output:** `RenameImpactResponse`
1566
+ ```json
1567
+ {
1568
+ "symbol": "createPolicyGroup",
1569
+ "declaration_sites": [
1570
+ { "file_path": "api/services/policy/src/service.ts", "line": 42, "kind": "function", "is_exported": true }
1571
+ ],
1572
+ "verified_callers": [
1573
+ { "file_path": "api/routes/policy.ts", "line": 18, "caller_symbol": "registerPolicyRoutes", "context": "const group = await createPolicyGroup(input);", "resolution": "import" }
1574
+ ],
1575
+ "potential_matches": [
1576
+ { "file_path": "api/services/policy/src/service.ts", "line": 42, "context": "createPolicyGroup", "reason": "identifier_match_no_resolved_edge" }
1577
+ ],
1578
+ "barrel_exports": [
1579
+ { "file_path": "api/services/policy/index.ts", "line": 1, "exported_as": "createPolicyGroup", "via": "named" },
1580
+ { "file_path": "api/index.ts", "line": null, "exported_as": "createPolicyGroup", "via": "star" }
1581
+ ],
1582
+ "summary": {
1583
+ "declaration_count": 1,
1584
+ "verified_count": 1,
1585
+ "potential_count": 1,
1586
+ "barrel_count": 2,
1587
+ "checklist": "1 verified call site(s) to update, 1 review-required identifier match(es), 2 barrel export(s) to update.",
1588
+ "checker_classified_non_call_site": 0,
1589
+ "checker_classified_different_declaration": 0
1590
+ }
1591
+ }
1592
+ ```
1593
+
1594
+ `summary.potential_truncated` is omitted above for the same reason as in
1595
+ `mast_callers` (this query's `identifier_fts` fetch came back under the cap) — see
1596
+ §9 `mast_callers`' `potential_truncated` example for what it looks like when present.
1597
+
1598
+ Section sources and semantics:
1599
+
1600
+ - `declaration_sites` — the `symbols` table (multiple entries when the name is
1601
+ ambiguous and no `file_path` was given). Impact below is computed against the
1602
+ first match, same convention as `mast_callers`; the full list keeps an
1603
+ ambiguous rename visible.
1604
+ - `verified_callers` — direct `POTENTIAL_CALL` edges, identical to
1605
+ `mast_callers`' verified set. **Direct callers only** — a rename edits call
1606
+ sites, and every call site is a direct caller; there is no `transitive`
1607
+ option (deliberate v1 scope).
1608
+ - `potential_matches` — `identifier_fts` hits not covered by a verified edge,
1609
+ identical to `mast_callers`' potential set (shared implementation, including
1610
+ checker-verdict filtering when `mast index --checker` has run, §10.3.2). These
1611
+ are **mandatory review sites**: the graph could not prove them, so the agent
1612
+ must check each before declaring the rename complete. The declaration chunk
1613
+ itself typically appears here — correctly, since it must be edited.
1614
+ `summary.checker_classified_non_call_site`/`checker_classified_different_declaration`
1615
+ carry the same meaning as in `mast_callers`, as does `summary.potential_truncated`
1616
+ (F10) — the real, uncapped `identifier_fts` match count, present only when the
1617
+ capped fetch came back full; a raw-fetch signal, not a `potential_matches` size
1618
+ guarantee (§9 `mast_callers`'s `potential_truncated` paragraph has the full
1619
+ precision note).
1620
+ - `barrel_exports` — files that re-export the symbol: `via: "named"` rows come
1621
+ from `RE_EXPORTS` edges (the export statement names the symbol —
1622
+ `exported_as` carries the alias — and must be edited); `via: "star"` rows
1623
+ come from a recursive walk of `re_export_files` (`export *` statements need
1624
+ no edit, but every downstream consumer reaches the symbol through them, so
1625
+ they are surfaced for awareness; `line` is null — star rows are file-level).
1626
+
1627
+ **When used:** before renaming any exported symbol — replaces the
1628
+ callers-then-grep-then-barrel-hunt sequence with one call, and again after the
1629
+ rename (the checklist should come back empty for the old name).
1630
+
1631
+ ---
1632
+
1633
+ ### `mast_reindex`
1634
+
1635
+ Synchronous incremental reindex. Does not return until the index reflects the current
1636
+ state of the filesystem.
1637
+
1638
+ **Input:**
1639
+ ```json
1640
+ {
1641
+ "full": false
1642
+ }
1643
+ ```
1644
+
1645
+ `full: true` forces a complete reindex regardless of mtimes.
1646
+
1647
+ **Output:** `ReindexResult`
1648
+ ```json
1649
+ {
1650
+ "files_indexed": 3,
1651
+ "files_skipped": 139,
1652
+ "chunks_added": 24,
1653
+ "chunks_removed": 18,
1654
+ "parse_errors": 0,
1655
+ "write_errors": 0,
1656
+ "duration_ms": 380
1657
+ }
1658
+ ```
1659
+
1660
+ `parse_errors > 0` means one or more files were skipped due to tree-sitter parse failures. `write_errors > 0` means a file parsed successfully but its chunk/graph/FTS write failed — a distinct failure mode from a parse error (a chunk-store write failure must never be conflated with an unparseable file). The agent should call `mast_status` for details, or check the mast server log for the specific file paths.
1661
+
1662
+ **When used:** immediately after the agent writes or edits files, before querying for
1663
+ symbols it just created. Called explicitly by the agent — not automatic.
1664
+
1665
+ ---
1666
+
1667
+ ### `mast_status`
1668
+
1669
+ Index health snapshot.
1670
+
1671
+ **Input:** none
1672
+
1673
+ **Output:** `StatusResult`
1674
+ ```json
1675
+ {
1676
+ "state_dir": "/workspace/.kluster/.mast",
1677
+ "schema_version": "1.3.0",
1678
+ "last_indexed": "2026-05-13T14:22:00Z",
1679
+ "indexed_files": 142,
1680
+ "chunk_count": 1840,
1681
+ "stale_files": 0,
1682
+ "parse_errors": 0,
1683
+ "write_errors": 0,
1684
+ "index_fresh": true,
1685
+ "freshness_cause": null,
1686
+ "seed_commit": "abc1234"
1687
+ }
1688
+ ```
1689
+
1690
+ `schema_version` is `CURRENT_SCHEMA_VERSION` **as compiled into the running
1691
+ binary**, not the value stored in `index.json`. After a normal startup the two are
1692
+ identical, because §7.4 Step 2's guard wipes derived state on a mismatch; they
1693
+ diverge in exactly the situation this field exists to expose — a long-lived process
1694
+ still executing an older build while the state directory it holds open has since
1695
+ been migrated by a newer one. That case is invisible to every other field (the index
1696
+ looks healthy, because it *is* healthy — it is the server that is stale), and the
1697
+ startup guard cannot catch it because the guard only runs at startup. Reading the
1698
+ value off disk would report the migrated version and hide the divergence, so it is
1699
+ deliberately sourced from the binary's own constant. See
1700
+ `adr/proposals/measurement-harness/PLAN-EXCERPT.md` § "D8 result" for the incident that
1701
+ motivated it, and `adr/009-2026-08-10-measurement-harness.md` for the decision it produced.
1702
+
1703
+ `parse_errors` is the count of files skipped during the last index run due to tree-sitter
1704
+ parse failures; `write_errors` is the count skipped due to a chunk/graph/FTS write
1705
+ failure after a successful parse (see `mast_reindex`, above — the two are never
1706
+ conflated). Non-zero in either indicates files the agent should investigate.
1707
+ `seed_commit` is present only when the state directory was bootstrapped from a
1708
+ Docker-baked seed (§13.8) and reports the git revision the seed was built from.
1709
+
1710
+ **Freshness diagnostics.** `freshness_cause` is `"phase1_stale"` when `stale_files > 0`
1711
+ (chunk line coordinates lag disk — corrected by JIT re-parse on read, §9.0, or by
1712
+ running `mast_reindex`) and `null` when the index is fully fresh. `index_fresh` is
1713
+ `true` only when `stale_files === 0` and the index has been run at least once.
1714
+
1715
+ `stale_files` counts three things, not one: files whose content changed since they
1716
+ were indexed, files on disk that are **not in the index at all**, and files the index
1717
+ still lists that are gone from disk. It is computed by `indexer/freshness.ts`
1718
+ `measureFreshness`, which `mast status` and `mast_status` both call — one producer,
1719
+ because they answer one question. It reads the manifest and the `files.mtime` stamps
1720
+ and takes the union: only the manifest can see a file that was never indexed, and only
1721
+ the stamps can see a manifest entry with no row, or an edit that landed mid-run (the
1722
+ manifest is stamped from a finalise-time re-stat; the row carries the pre-parse stamp).
1723
+
1724
+ **A file that fails to parse or write is not recorded in the manifest**, so the next
1725
+ run retries it, and an incremental run's work set includes any walked file with no
1726
+ `files` row — otherwise a hole left by an older build would be reported stale forever
1727
+ with no run willing to fix it.
1728
+
1729
+ **When used:** diagnostic — agent checks this when search returns unexpected
1730
+ results, or before a long agentic workflow to confirm the index is current.
1731
+
1732
+ ---
1733
+
1734
+ ### `mast_efficiency`
1735
+
1736
+ Rolling per-session and global token-savings telemetry. Designed to be called by the
1737
+ agent periodically so it can self-correct when its tool usage is inefficient (e.g.,
1738
+ falling back to full-file `Read` instead of `mast_search`).
1739
+
1740
+ **Input:**
1741
+ ```json
1742
+ {
1743
+ "scope": "session | global",
1744
+ "since_minutes": 60
1745
+ }
1746
+ ```
1747
+
1748
+ - `scope: "session"` aggregates only this `mast serve` session's calls. `scope: "global"`
1749
+ aggregates the persistent metrics table on the shared volume (see §14).
1750
+ - `since_minutes` bounds the window for `scope: "global"`. Ignored when `scope` is
1751
+ `"session"`. Defaults to 60.
1752
+
1753
+ **Output:** `EfficiencyResult`
1754
+ ```json
1755
+ {
1756
+ "scope": "session",
1757
+ "window_started_at": "2026-05-13T14:00:00Z",
1758
+ "tokens_returned": 18420,
1759
+ "tokens_full_file_upper_bound": 142880,
1760
+ "efficiency_ratio": 0.871,
1761
+ "calls_total": 47,
1762
+ "calls_by_tool": {
1763
+ "mast_search": 18,
1764
+ "mast_signature": 12,
1765
+ "mast_exports": 9,
1766
+ "mast_callers": 5,
1767
+ "mast_project_skeleton": 3
1768
+ },
1769
+ "tokenizer": "@anthropic-ai/tokenizer (claude-2 era, approximate for current models)",
1770
+ "counterfactual": "Saved vs. full file Read (upper bound — overstates savings against a smart agent that would have used Grep)"
1771
+ }
1772
+ ```
1773
+
1774
+ The `tokenizer` field carries the honest label from §14.5 verbatim — token
1775
+ counts are approximate for current models, and consumers should treat the
1776
+ savings *ratio*, not the absolute counts, as the robust number.
1777
+
1778
+ `efficiency_ratio` is `1 - (tokens_returned / tokens_full_file_upper_bound)`. Higher
1779
+ is better; 0.871 means 87.1% fewer tokens than the "naive `Read` every result file"
1780
+ counterfactual.
1781
+
1782
+ **Honest framing.** The counterfactual is explicitly labelled as an upper bound. An
1783
+ agent using `Grep -A 10 -B 10` instead of `Read` would have used fewer tokens than
1784
+ the counterfactual but more than MAST — the real savings sit between zero and the
1785
+ reported ratio. The label exists so this number can survive a "is MAST worth it?"
1786
+ review without being challenged as cherry-picked. See §14 for the methodology and
1787
+ the SQLite metrics table schema.
1788
+
1789
+ **When used:** the agent calls this once per task (typically near the end) to see
1790
+ whether its tool usage was efficient. The implement-task prompt (§12) instructs:
1791
+ "If `efficiency_ratio < 0.30`, you are reading more than you should — prefer
1792
+ `mast_search` over `Read` for the next task." This gives the agent an in-loop
1793
+ feedback signal without humans needing to inspect logs.
1794
+
1795
+ ---
1796
+
1797
+ ## 10. AST Extraction
1798
+
1799
+ mast uses **tree-sitter** with `tree-sitter-typescript` for all AST operations.
1800
+ tree-sitter is chosen over the TypeScript Compiler API for speed (C library, no
1801
+ tsc overhead) and extensibility to other languages via grammar plugins.
1802
+
1803
+ ### 10.1 Chunking Strategy
1804
+
1805
+ Chunks are **declaration-first**: each chunk maps to exactly one top-level AST
1806
+ declaration. Line-based splitting is a fallback, not the primary strategy.
1807
+
1808
+ | Node type | `chunk_type` | `symbol_name` | `parent_symbol` | `is_exported` |
1809
+ |---|---|---|---|---|
1810
+ | `function_declaration` | `function` | function name | `null` | has `export` modifier |
1811
+ | `arrow_function` → `const` | `function` | variable name | `null` | has `export` modifier |
1812
+ | `class_declaration` | `class_shell` (synthesized) | class name | `null` | has `export` modifier |
1813
+ | `method_definition` (inside class) | `method` | `ClassName.methodName` | class name | inherits from class **and** non-`private` |
1814
+ | `interface_declaration` | `interface` | interface name | `null` | has `export` modifier |
1815
+ | `type_alias_declaration` | `type` | type name | `null` | has `export` modifier |
1816
+ | `export_statement` wrapping any above | inherits inner | inherits inner | inherits inner | `true` |
1817
+ | Everything else at top level | `block` | `null` | `null` | `false` |
1818
+
1819
+ **Class decomposition: shell + methods.** A `class_declaration` node does NOT
1820
+ become one chunk. The chunker emits:
1821
+
1822
+ 1. **One `class_shell` chunk** whose `content` is the synthesized class outline:
1823
+ the class declaration line, all member signatures (methods, properties, getters,
1824
+ setters, constructors) with their leading TSDoc, but **no method bodies**. This
1825
+ is the "outline" view used for orientation and for `mast_signature` calls that
1826
+ target the class itself rather than a specific member.
1827
+
1828
+ 2. **N `method` chunks**, one per `method_definition` (and `constructor`, getter,
1829
+ setter) inside the class body. Each method chunk has:
1830
+ - `symbol_name` qualified as `ClassName.methodName` (e.g. `AuthService.validateSession`,
1831
+ `AuthService.constructor`).
1832
+ - `parent_symbol` set to the enclosing class name (unqualified).
1833
+ - `is_exported` = class's `is_exported` **AND** method is not `private`. (Methods
1834
+ marked `protected` or with no accessibility modifier are treated as exported when
1835
+ the class is exported. This mirrors how callers from outside the class can reach
1836
+ them; `private` members are intentionally hidden from `only_exported: true`
1837
+ queries.)
1838
+ - Its own `content`, the full method source (signature + body).
1839
+ - A `PARENT_OF` edge from the class's `symbols` row to the method's `symbols` row
1840
+ in `graph.db` (see §6.3).
1841
+
1842
+ 3. **A `body_hash` for the class_shell** computed over the sorted concatenation of
1843
+ member signatures + member TSDoc (see §7.1 stability hash rule). This ensures
1844
+ the shell content is rewritten when methods are renamed, added, or removed — but
1845
+ **not** when method bodies change.
1846
+
1847
+ **Why this matters.** A 400-line service class becomes one ~30-line outline chunk
1848
+ plus ~12 small method chunks (~30 lines each). `mast_search "validate session"`
1849
+ returns the matching `method` chunk (~30 lines) instead of the whole class (400
1850
+ lines). For class-heavy codebases the token savings move from "marginal" to
1851
+ "material." Chunk count for a class-heavy 5K-file repo grows from ~6K to ~20–30K;
1852
+ SQLite (chunks table + FTS5) handles this fine at sub-100MB index size.
1853
+
1854
+ **Interfaces and type aliases are NOT decomposed.** Their members are signatures
1855
+ already (no bodies to split), so the interface or type alias remains a single chunk.
1856
+
1857
+ **Anonymous default exports** (`export default function () {}`, `export default {}`):
1858
+ `symbol_name` is set to the filename without extension (e.g., `handler` for
1859
+ `handler.ts`). This is a heuristic and `mast_search` will surface these via FTS
1860
+ on the filename. A future v2 may resolve the alias from importers, but v1 keeps it
1861
+ simple.
1862
+
1863
+ **Re-export aliases** (`export { foo as bar } from './x'`): the extractor
1864
+ records `bar` as an exported **marker symbol** (kind `export`, no hashes) in
1865
+ the barrel's `symbols` rows plus a `RE_EXPORTS` edge from the marker to `foo`'s
1866
+ declaration; `export * from './x'` becomes a `re_export_files` row instead
1867
+ (file-level — stars name no symbols). Marker rows exist to anchor the edge for
1868
+ `mast_rename_impact`'s barrel checklist and are **excluded from symbol lookups**
1869
+ (`querySymbolByName` filters kind `export`), so `mast_signature`/`mast_callers`
1870
+ keep resolving to the real declaration rather than the barrel. Import
1871
+ specifiers are resolved with the same §13.7 resolver used for `import`
1872
+ statements.
1873
+
1874
+ **Implementation note — local aliases.** For a *local* alias
1875
+ (`export { foo as bar }`, no `from`), the chunker does not use the
1876
+ `RE_EXPORTS`-edge path above. It instead emits an extra chunk for `bar` that
1877
+ mirrors `foo`'s declaration (own `chunk_id`, marked exported), and
1878
+ `extractSignatures` emits a matching `bar` signature entry. This makes `bar`
1879
+ discoverable (`mast_exports`/`mast_search`) and resolvable (`mast_signature`)
1880
+ without a chain walk — same observable result, simpler mechanism. The aliased
1881
+ local name (`foo`) is NOT itself marked exported, since the export name is `bar`.
1882
+
1883
+ **Two-pass walk for `is_exported`:**
1884
+
1885
+ TypeScript allows declarations to be exported separately from their definition:
1886
+
1887
+ ```typescript
1888
+ function internalHandle() { ... }
1889
+ export { internalHandle as handleLogin };
1890
+ export * from './other-module';
1891
+ ```
1892
+
1893
+ A single-pass walk that only checks for the `export` modifier on declarations would
1894
+ mark `internalHandle` as not exported, which is wrong. The chunker uses a two-pass
1895
+ walk:
1896
+
1897
+ - **Pass 1:** map all top-level symbol names to their declaration nodes and initial
1898
+ `is_exported` state (based on `export` modifier presence).
1899
+ - **Pass 2:** walk all `export_clause` nodes (`export { ... }`) and
1900
+ `export_all_clause` nodes (`export * from ...`). For each named re-export, find
1901
+ the symbol in the Pass 1 map and set `is_exported = true`. For `export *`, mark the
1902
+ source module path for resolution — the graph populator will follow the edge.
1903
+
1904
+ **Split rule:** if a declaration spans more than `chunk_split_threshold` lines
1905
+ (default: 100), split into overlapping sub-chunks with 10-line overlap. The first
1906
+ sub-chunk always includes the full declaration header so signature extraction is
1907
+ always possible from sub-chunk 0.
1908
+
1909
+ Most functions will not trigger this split — the default of 100 lines comfortably
1910
+ covers ordinary function and method bodies. The threshold exists for edge cases
1911
+ (large generated files, data-heavy switch statements) where a single chunk would
1912
+ otherwise be too large to be a useful, self-contained search result.
1913
+
1914
+ **Markdown documents (`chunk_type: "doc"`).** `.md` files are chunked by ATX
1915
+ heading, not by AST — one chunk per heading of level ≤ `markdown_heading_depth`
1916
+ (default 2: one chunk per `##` section). Doc chunks get
1917
+ `chunk_fts` rows like any other chunk, but **no graph presence**: no `symbols`
1918
+ rows, no `imports`, no `edges`, and no `identifier_fts` rows (that index feeds
1919
+ `mast_callers` potential_matches, where a doc that merely *mentions* a symbol
1920
+ name is noise, not a call site).
1921
+
1922
+ Rules:
1923
+
1924
+ - `symbol_name` is the heading path — the file name, every ancestor heading,
1925
+ and the section's own heading joined with `" > "`, e.g.
1926
+ `MAST_SPEC.md > Technical Specification > 7. Index Lifecycle`. Skipped
1927
+ heading levels are omitted from the path.
1928
+ - Headings deeper than `markdown_heading_depth` fold into their enclosing
1929
+ section's content.
1930
+ - Content before the first boundary heading becomes a preamble chunk whose
1931
+ `symbol_name` is the file name alone. A file with no headings is one
1932
+ preamble chunk.
1933
+ - `#` lines inside fenced code blocks (``` or ~~~) are not headings. Setext
1934
+ headings (`===`/`---` underlines) are not recognised — this repo's docs use
1935
+ ATX exclusively.
1936
+ - `is_exported` is always `false` — `only_exported: true` searches exclude
1937
+ docs by construction. `parent_symbol` is always `null`; `language` is
1938
+ `markdown`.
1939
+ - The split rule above applies to oversized sections (same window, overlap,
1940
+ and sub-chunk ID scheme as declarations).
1941
+ - No `context_lines` expansion: sections are self-delimiting, and expansion
1942
+ would duplicate neighbouring sections' text into every chunk.
1943
+ - Doc chunks carry no stability hashes, so the incremental "unchanged file"
1944
+ fast path conservatively rewrites a markdown file whose mtime changed
1945
+ (same treatment as files containing `block` chunks).
1946
+
1947
+ ### 10.2 Signature Extraction
1948
+
1949
+ For `mast_signature` and the signature field in `mast_exports`:
1950
+
1951
+ 1. Locate the declaration node by symbol name via tree-sitter query.
1952
+ 2. Extract node text up to (not including) the `statement_block` child (`{ ... }` body).
1953
+ 3. For interfaces and type aliases: the full declaration is the signature — no body
1954
+ exists to strip.
1955
+ 4. Walk backwards from the declaration's start byte to find the immediately preceding
1956
+ `comment` node. Accept `/** ... */` (TSDoc) or `// ...` (line comment). Include
1957
+ as `doc`.
1958
+
1959
+ ### 10.3 Knowledge Graph Population
1960
+
1961
+ After Phase 1 chunking, mast populates `graph.db` for each indexed file:
1962
+
1963
+ - **`imports`**: parse each `import_statement`. Extract module specifier and named
1964
+ imports. Resolve relative paths against `project_root` for local modules. The
1965
+ resolved imports double as the input to the **local type environment** (see
1966
+ §10.3.1) used by the `POTENTIAL_CALL` resolver.
1967
+ - **`symbols` rows**: insert a row for each top-level declaration AND each
1968
+ `method_definition` inside a class. Method symbols carry qualified names
1969
+ (`ClassName.methodName`) and a `PARENT_OF` edge from the class symbol — see §10.1
1970
+ class decomposition.
1971
+ - **`POTENTIAL_CALL` edges**: within function and method bodies, find `call_expression`
1972
+ nodes. Run the local heuristic resolver (§10.3.1) against the receiver to identify
1973
+ the callee symbol. Only insert an edge when the resolver returns a single known
1974
+ indexed symbol; skip external library calls and unresolved receivers (those are
1975
+ surfaced as `potential_matches` in `mast_callers` via `identifier_fts`, not as
1976
+ edges).
1977
+ - **`IMPLEMENTS` / `EXTENDS` edges**: from `class_declaration` nodes with
1978
+ `implements_clause` or `extends_clause`, resolve the named type and insert the edge.
1979
+ - **`identifier_fts` rows**: extract every identifier token in the chunk (function
1980
+ names, method names, type names, variable references) via tree-sitter `identifier`
1981
+ node enumeration. Deduplicate per chunk. Insert one row per chunk with
1982
+ whitespace-joined identifiers — this is what `mast_callers`'s `potential_matches`
1983
+ query hits. **F5 (schema 1.3.0):** after the bare-token bag, append QUALIFIED
1984
+ compound strings (`"Class.method"`) — a method chunk's own qualified name, plus
1985
+ the resolved `Type.method` string for every call site the local type environment
1986
+ (§10.3.1) linked to a receiver type, appended to the CALLING chunk's row. This is
1987
+ what makes the phrase-quoted `"Class.method"` query `mast_callers`/
1988
+ `mast_rename_impact` issue (§9) actually hit a row: identifier_fts' `unicode61`
1989
+ tokenizer treats `.` as a separator, so the literal compound string tokenizes into
1990
+ two ADJACENT tokens, matching the phrase directly. The bare bag alone essentially
1991
+ never has that adjacency for a method name. A receiver the resolver could not
1992
+ statically link (§10.3.1's "does NOT catch" list) contributes no compound.
1993
+
1994
+ All inserts for a file are wrapped in a single transaction. On incremental reindex,
1995
+ existing rows for the file are deleted before reinsertion (delete-and-replace, not
1996
+ upsert), which keeps the graph consistent with renames and deletions.
1997
+
1998
+ ### 10.3.1 Local Type Environment (POTENTIAL_CALL Resolver)
1999
+
2000
+ The `POTENTIAL_CALL` edge type is named for what it actually is: a name-resolved
2001
+ reference whose receiver was statically linkable to a known symbol. There is no full
2002
+ TypeScript type-checker in mast; the resolver is a deliberately scoped set of
2003
+ heuristics that catches the high-frequency cases without bringing `tsserver` into
2004
+ the indexer.
2005
+
2006
+ **What the resolver catches (will produce a `POTENTIAL_CALL` edge):**
2007
+
2008
+ 1. **Top-level named imports.** `import { handleLogin } from './handler'; handleLogin(req, reply)`
2009
+ → resolves `handleLogin` via the `imports` table to a known symbol.
2010
+ 2. **Class field types** (annotated). `private userRepo: UserRepository` followed
2011
+ somewhere in the class body by `this.userRepo.findByEmail(email)` → resolves
2012
+ `findByEmail` to `UserRepository.findByEmail`. The field's type annotation must
2013
+ be a named type the resolver can find via `imports` or same-file `symbols`.
2014
+ 3. **Constructor parameter properties.** `constructor(private readonly users: UserRepository)`
2015
+ creates an implicit field; `this.users.create(input)` resolves the same way as (2).
2016
+ 4. **Annotated parameters in any function/method.** `function foo(repo: UserRepository) { repo.findById(id) }`
2017
+ → resolves `findById` to `UserRepository.findById`.
2018
+ 5. **`new` expressions.** `const repo = new UserRepository(); repo.findById(id)` →
2019
+ the resolver tracks `repo`'s inferred type as `UserRepository` for the rest of the
2020
+ block (or until shadowed) and resolves the chained call.
2021
+ 6. **Same-file function calls.** A function calling another function in the same file
2022
+ resolves directly via the local `symbols` table.
2023
+
2024
+ **What the resolver does NOT catch (will NOT produce a `POTENTIAL_CALL` edge — but
2025
+ the identifier match still lands in `identifier_fts` and surfaces as
2026
+ `potential_matches`):**
2027
+
2028
+ - **Factory return types without annotation.** `const repo = makeRepository(); repo.findById(id)`
2029
+ — `repo`'s type is inferred and the resolver does not run inference.
2030
+ - **DI container lookups.** `container.get(UserRepository).findById(id)` — the
2031
+ generic erases at the resolver level.
2032
+ - **Chained calls without intermediate binding.** `getUserService().findById(id)`.
2033
+ - **Dynamic dispatch.** `repos[name].findById(id)`.
2034
+ - **Re-exported types not yet resolved through the `re_export_files` chain at edge
2035
+ time.** Resolution can be deferred: edges are inserted on a second pass after all
2036
+ symbols are populated (see "Two-pass edge insertion" below).
2037
+ - **Generic type parameters.** `class Repo<T> { find(id: ID): T }` — the resolver
2038
+ treats `T` as opaque.
2039
+
2040
+ **F5 honesty note on "still lands in `identifier_fts`" above.** That claim was
2041
+ historically true only for BARE identifier tokens (e.g. `findById`), which
2042
+ `mast_callers`/`mast_rename_impact` never actually query — they query the
2043
+ QUALIFIED `"Class.method"` form (§9), which (pre-F5) essentially never phrase-
2044
+ matched anything, silently emptying `potential_matches` for methods regardless
2045
+ of this list. Post-F5, the distinction that matters is whether
2046
+ `LocalTypeEnvironment.resolveCall` (the SAME resolution `POTENTIAL_CALL`
2047
+ edges use) succeeded, not whether the edge survived to `graph.db`:
2048
+ - **Re-exported types not yet resolved through the `re_export_files` chain**
2049
+ — extraction still resolves the receiver's type name and emits the
2050
+ qualified compound; only the DB-layer edge is dropped. This case IS now
2051
+ healed: the mention surfaces in `potential_matches`.
2052
+ - **Factory return types, DI container lookups, chained calls without
2053
+ intermediate binding, dynamic dispatch, generic type parameters** — the
2054
+ receiver is unresolvable at EXTRACTION time (`resolveCall` itself returns
2055
+ null), so no qualified compound is ever produced. These remain a genuine
2056
+ coverage gap for qualified queries specifically — closing it (if ever) is
2057
+ the opt-in `mast index --checker` pass's (§10.3.2) territory, not this
2058
+ fix's; F5 deliberately does not attempt to guess a qualified name for an
2059
+ unresolvable receiver.
2060
+
2061
+ **Coverage characterisation.** In a Fastify + DI service codebase, the resolver
2062
+ catches roughly the field/parameter/import cases — typically 60–80% of real call
2063
+ sites depending on how heavily the codebase uses factories and containers. The
2064
+ intentional design choice is: when in doubt, do NOT produce a `POTENTIAL_CALL` edge,
2065
+ and rely on `identifier_fts` + the `mast_callers` `potential_matches` set to catch
2066
+ the rest. False negatives in the verified set are acceptable; false positives would
2067
+ poison the contract.
2068
+
2069
+ **Two-pass edge insertion.** Because cross-file references depend on all symbols
2070
+ being in the table first, edge insertion runs as a second pass after Phase 1's
2071
+ symbol-population pass completes for the entire indexed file set. This means a full
2072
+ reindex has the ordering: walk all files → insert all chunks + symbols → re-walk
2073
+ all files → insert all edges. On incremental reindex, only the affected files are
2074
+ re-walked in pass two, but all of `graph.db`'s `symbols` table is queryable so cross-
2075
+ file references resolve correctly.
2076
+
2077
+ **Method calls on `super` and `this` without receiver.** `this.foo()` resolves to
2078
+ the enclosing class's `foo` method via the qualified `symbols` row. `super.foo()`
2079
+ resolves to the parent class via the `EXTENDS` edge. Implemented as ordinary
2080
+ receiver bindings (F4, Stage 3): `emitClassEdges` seeds `this` → the enclosing
2081
+ class name and, only when an `extends` clause names a parent, `super` → that
2082
+ parent's name, riding the same `LocalTypeEnvironment` receiver-binding path
2083
+ `field_type` etc. use — no parallel resolution mechanism. The resulting
2084
+ `resolution` values are `this_method` and `super_method` respectively (see the
2085
+ `edges` table comment above and §9.0's confidence-signals table); a class with
2086
+ no `extends` clause never produces a `super_method` edge — an unresolvable
2087
+ `super.foo()` call falls through to the `identifier_fts` potential set instead
2088
+ of guessing. `this.foo()` inside a nested non-arrow function/method/generator
2089
+ body is NOT the class instance and is excluded before it ever reaches the
2090
+ resolver — arrow functions inherit the enclosing `this` and are not excluded.
2091
+
2092
+ (Non-normative, D7/Stage 4) `extractEdges`/`emitCallEdges` also accept an optional
2093
+ `onCallSite` diagnostics callback, invoked once per visited call site with a closed
2094
+ outcome union; it exists solely as a test seam for `call-oracle.test.ts`'s corpus
2095
+ accounting invariant and is not configurable, persisted, or surfaced in any tool
2096
+ response.
2097
+
2098
+ **Name resolution is file-scoped, not name-only.** When two files export a
2099
+ same-named symbol, `insertEdges`' name→id resolution (§10.3) uses the
2100
+ resolution rule's own file evidence to pick the target: `same_file` is scoped
2101
+ to the calling file itself; `import` is scoped to the import's own
2102
+ `resolved_path` (following the `re_export_files`/`RE_EXPORTS` chain into a
2103
+ barrel when the resolved file doesn't declare the symbol directly, per §6.3);
2104
+ `field_type`/`parameter_type`/`new_expression` are scoped the same way via
2105
+ the receiver's type name, when that type name is itself traceable to an
2106
+ import or a same-file declaration. Only when a rule has no such evidence
2107
+ (e.g. a default/namespace import, which is not tracked as a named import) does
2108
+ resolution fall back to a global name match — a known, narrow coverage gap,
2109
+ not the general case. Prior to 2026-07-15 every rule fell back to the global
2110
+ match unconditionally, so a same-named symbol in an earlier-indexed file could
2111
+ silently win a `verified_callers` edge that belonged to a different file
2112
+ (IMPLEMENTATION_PLAN_VEXP.md §P, "Shipped-resolver finding").
2113
+
2114
+ The same file-scoping applies to `RE_EXPORTS` edges (`export { x } from './y'`,
2115
+ §6.3): the extractor resolves the re-export's own `from`-clause module
2116
+ specifier to a real file (`EdgeRecord.toResolvedPath`) at parse time, and
2117
+ `insertEdges` uses that path — not a bare name match across the whole graph —
2118
+ to pick the target when two files export a same-named symbol. An unresolved
2119
+ module (external, or a relative specifier that doesn't probe to a real file)
2120
+ produces no edge rather than a name-only guess. This closed the sibling of the
2121
+ same false-green class for `mast_rename_impact`'s `barrel_exports`
2122
+ (IMPLEMENTATION_PLAN_VEXP.md §P, "Sibling false-green").
2123
+
2124
+ ### 10.3.2 TypeScript-Checker Enrichment Pass (`mast index --checker`)
2125
+
2126
+ An **opt-in** CLI pass (`src/graph/checker-resolver.ts`) that uses the real
2127
+ TypeScript compiler to upgrade `potential_matches` the §10.3.1 heuristic
2128
+ resolver could not statically link. Reshaped from an originally-planned
2129
+ always-on background worker by a spike measurement: holding every workspace
2130
+ project's `ts.Program` alive at once peaked at 2.45 GB RSS, over a 2 GB gate
2131
+ (IMPLEMENTATION_PLAN_VEXP.md Stage 1.1, `eval/spikes/checker-edges/REPORT.md`).
2132
+ The default `mast index` path is behaviourally untouched — this only runs with
2133
+ the `--checker` flag.
2134
+
2135
+ **What it guarantees:**
2136
+
2137
+ - Holds exactly **one** `ts.Program` at a time, disposed before the next
2138
+ tsconfig project loads (the spike's cautionary tale: holding all programs
2139
+ alive made a "warm" re-check *slower* than cold, via GC pressure).
2140
+ - Every candidate is one of the shipped `potential_matches` pool
2141
+ (`collectPotentialMatchCandidates`, shared with `mast_callers` — never a
2142
+ second definition of "what counts as a potential match").
2143
+ - `getAliasedSymbol` alias-chain following (bounded at 8 hops) on every
2144
+ resolution — without it, resolution collapses from ~38% to ~2% (spike
2145
+ finding; an import binding otherwise resolves to itself, not its target).
2146
+ - **False-green gate:** a checker edge is written ONLY when the resolved
2147
+ declaration's `(file, line)` matches the queried symbol's own recorded
2148
+ `(file, line)` (±3 lines, for decorator/JSDoc/overload offsets). A same-name
2149
+ collision (two unrelated declarations sharing a method name, an
2150
+ interface-typed receiver with multiple implementors, a shadowed import)
2151
+ resolves to a DIFFERENT declaration and is classified `resolves_to_different`
2152
+ — never written as a `checker` edge. A wrong "verified" edge is worse than no
2153
+ edge (adversarial fixtures: `src/graph/__tests__/checker-resolver.test.ts`).
2154
+
2155
+ **What it does to each candidate:**
2156
+
2157
+ | Classification | Effect |
2158
+ |---|---|
2159
+ | Resolves to the queried declaration | A `POTENTIAL_CALL` edge is written with `resolution: 'checker'` — joins `verified_callers` exactly like a heuristic edge, and dedupes with one on the same `(from_id, to_id, edge_type)` triple. |
2160
+ | Resolves to a DIFFERENT declaration | Recorded in `checker_verdicts`; drops out of `potential_matches` and is counted in `summary.checker_classified_different_declaration`. |
2161
+ | Not a call site (comment, string, type position) | Recorded in `checker_verdicts`; drops out of `potential_matches` and is counted in `summary.checker_classified_non_call_site`. This residue was 30–44% of the sampled potential pool in the Stage 1.1 spike — classifying it away is itself a major token win, independent of edge upgrades. |
2162
+ | Unresolvable (dynamic dispatch, DI lookup, etc.) | No edge, no verdict — stays a genuine `potential_match`, exactly as before `--checker` ran. |
2163
+
2164
+ **What stays potential — the checker pass does not close every gap:**
2165
+
2166
+ - Files outside every discovered tsconfig project (roughly 22% of the
2167
+ monorepo sample) are left completely untouched; the CLI summary reports the
2168
+ count (`outside_ts_scope`), never a silent cap.
2169
+ `discoverTsConfigProjects` finds every `tsconfig.json` under the project
2170
+ root whose `parseJsonConfigFileContent` resolves at least one file — a base
2171
+ config meant to be `extends`-ed (no own `"include"`) is skipped as
2172
+ `no_include_base_config`, and an unparseable config is skipped with the
2173
+ parser's own error text. Generic and project-shape-agnostic (unlike the
2174
+ Stage 1.1 spike's hardcoded 25-project list for this monorepo specifically).
2175
+ - A call site with no enclosing declared symbol (e.g. top-level script code,
2176
+ a `block` chunk) still gets classified — so it never re-surfaces as review
2177
+ noise — but no `checker` edge is written, since there is no valid `from_id`
2178
+ to attach one to.
2179
+ - Cross-package calls where the target resolves into another workspace
2180
+ package's compiled `.d.ts` output (not the `.ts` source `mast` indexed) fail
2181
+ the `(file, line)` match safely — classified `resolves_to_different`, not a
2182
+ false positive, just a missed upgrade.
2183
+
2184
+ **Verdict staleness — the severity-zero invariant.** A verdict must not
2185
+ outlive the file content it was computed against: a stale verdict silently
2186
+ suppressing a genuinely new call site is worse than never having run the pass.
2187
+ `checker_verdicts.call_site_file_id REFERENCES files(id) ON DELETE CASCADE`
2188
+ ties a verdict's lifetime to the file row it was computed against —
2189
+ `populateFile`'s delete-and-replace on ANY content change (both full
2190
+ `mast index` and the JIT re-parse triggered by staleness detection, §9.0)
2191
+ cascades away every verdict for that file automatically, exactly like
2192
+ `symbols`/`edges`/`imports` already do. `checker_verdicts.call_site_mtime` is
2193
+ checked again at read time (`queryCheckerVerdicts`) as a second, independent
2194
+ guard. Proven directly against the real Phase 1 pipeline (edit a fixture file,
2195
+ reindex, assert the verdict no longer applies) in
2196
+ `src/graph/__tests__/checker-resolver.test.ts`.
2197
+
2198
+ **Persistence.** `checker_verdicts` is a brand-new, additive table (§7.4 — no
2199
+ `CURRENT_SCHEMA_VERSION` bump). Writes are flushed one tsconfig project at a
2200
+ time under `structure.lock` (§7.6), kept as a short batch strictly separate
2201
+ from the compiler-heavy classification loop (which holds no lock). A JIT
2202
+ re-parse from a concurrent read tool is never starved behind either phase —
2203
+ it no longer acquires `structure.lock` at all (§7.6, §9.0), so this
2204
+ separation now matters only for coarse-writer-vs-coarse-writer contention
2205
+ (e.g. a concurrent `mast_reindex`).
2206
+
2207
+ **Consumption.** `mast_callers` and `mast_rename_impact` (via the shared
2208
+ `collectPotentialMatches`) filter `non_call_site`/`resolves_to_different`
2209
+ candidates out of `potential_matches` and report honest counts in
2210
+ `summary.checker_classified_non_call_site` /
2211
+ `summary.checker_classified_different_declaration` (§9) — both `0` until
2212
+ `--checker` has run.
2213
+
2214
+ ---
2215
+
2216
+ ## 11. Hook Architecture
2217
+
2218
+ ### 11.1 Primary Hook — `mast serve` Startup
2219
+
2220
+ Defined in full in §7.4. Summary: a four-step ladder that brings the whole index
2221
+ (graph + FTS) online in 2–4 seconds via a Docker-baked seed index (§13.8), with all
2222
+ 11 tools registered and ready to serve as soon as Step 3 completes — there is no
2223
+ reduced-capability warm-up window. Step 4 then catches up any files changed since
2224
+ the seed was built, in the background.
2225
+
2226
+ This is the **only hook required for the SDD pipeline**.
2227
+
2228
+ ### 11.2 Mid-Task Hook — `mast_reindex` (agent-controlled)
2229
+
2230
+ The agent calls this explicitly after writes. JIT staleness handling (§9.0) already
2231
+ keeps already-indexed files correct on read; `mast_reindex` is what makes a
2232
+ **brand-new** file or symbol discoverable by `mast_search`/`mast_callers`/etc. before
2233
+ the next scheduled or `--watch` reindex reaches it. The implement prompt instructs:
2234
+
2235
+ > After writing or editing files, call `mast_reindex` before any search query that
2236
+ > depends on symbols you just created. This is the only way to guarantee the index
2237
+ > reflects your changes within this task.
2238
+
2239
+ ### 11.3 Optional Developer Hooks — Git
2240
+
2241
+ Installed by `mast install-hooks`:
2242
+
2243
+ **`.git/hooks/post-commit`**
2244
+ ```bash
2245
+ #!/bin/sh
2246
+ mast index "$(git rev-parse --show-toplevel)" --incremental
2247
+ ```
2248
+
2249
+ **`.git/hooks/post-checkout`**
2250
+ ```bash
2251
+ #!/bin/sh
2252
+ mast index "$(git rev-parse --show-toplevel)" --incremental
2253
+ ```
2254
+
2255
+ Not required for the automated SDD pipeline — the startup hook covers the same
2256
+ scenario (files changed since last index).
2257
+
2258
+ ### 11.4 Optional Interactive Hook — `mast serve --watch`
2259
+
2260
+ Local interactive development has no equivalent of the container's startup
2261
+ ladder: git hooks are opt-in and fire only on commit/checkout, so a long-lived
2262
+ interactive session can leave newly-created files and symbols undiscoverable
2263
+ between explicit `mast_reindex` calls, even though JIT re-parse (§9.0) keeps
2264
+ every already-indexed file correct on read. `--watch` closes that gap as an
2265
+ **opt-in** flag — it is a discovery-freshness optimization, never a correctness
2266
+ mechanism, and it does not reopen the §3 no-daemon non-goal (it lives and dies
2267
+ with the serve process).
2268
+
2269
+ Behaviour:
2270
+
2271
+ - A chokidar watcher covers `file_extensions` under the project root,
2272
+ respecting `exclude_patterns` **and the state directory itself** — watching
2273
+ the state dir would self-trigger on every index write.
2274
+ - Events are debounced (~500ms) and coalesced: rapid saves of one file collapse
2275
+ to a single entry; distinct files within the window share one batch.
2276
+ - Each batch runs the existing **incremental indexer** (§7.1) (acquiring
2277
+ `structure.lock` exactly as `mast_reindex` does — deleted files are cleaned
2278
+ up by the manifest diff).
2279
+ - **Single-flight:** events arriving during an in-flight run queue a follow-up
2280
+ run; runs never overlap.
2281
+ - **Lock contention:** a failed run (e.g. `structure.lock` held by
2282
+ `mast_reindex`) is logged and the batch requeued for the next debounce tick;
2283
+ after 3 consecutive failures the batch is dropped **with a warning** (JIT
2284
+ keeps existing-file reads correct, so a drop only delays discovery of new
2285
+ files/symbols).
2286
+ - **Degradation:** watcher construction failure (EMFILE, permissions) or
2287
+ runtime watcher errors log a warning and the server keeps serving without
2288
+ watch. `--watch` can never take down MCP serving.
2289
+ - Shutdown: the watcher closes on stdin close, SIGTERM, and SIGINT.
2290
+
2291
+ ---
2292
+
2293
+ ## 12. SDD Pipeline Integration
2294
+
2295
+ ### 12.1 One-Time Setup
2296
+
2297
+ Add to `kluster init` or run manually once per project:
2298
+
2299
+ ```bash
2300
+ mast init /path/to/app --state-dir .kluster/.mast
2301
+ ```
2302
+
2303
+ ### 12.2 `implement-task.md` Prompt Changes
2304
+
2305
+ Replace the existing `codemogger` MANDATORY FIRST ACTION block with:
2306
+
2307
+ ```markdown
2308
+ ## MANDATORY FIRST ACTION — NO EXCEPTIONS
2309
+
2310
+ Before writing any code, before opening any file:
2311
+
2312
+ 1. Get a project overview:
2313
+ mast_project_skeleton — see all files and their exported symbols in one call
2314
+
2315
+ 2. Run at least 2 targeted searches:
2316
+ mast_search: { "query": "<relevant symbols or concepts>", "only_exported": true }
2317
+ mast_search: { "query": "existing patterns conventions types" }
2318
+
2319
+ 3. Use mast_exports to inspect a file's API before opening it:
2320
+ mast_exports: { "file_path": "src/services/auth/index.ts" }
2321
+
2322
+ 4. Use mast_signature to get a function's contract (includes parameter type shapes):
2323
+ mast_signature: { "symbol": "handleLogin" }
2324
+
2325
+ After writing or editing files, call mast_reindex before searching for symbols
2326
+ you just created:
2327
+ mast_reindex: {}
2328
+ ```
2329
+
2330
+ ### 12.3 What the BT Pipeline Does Not Need to Change
2331
+
2332
+ - No new `execute-command` nodes in any YAML.
2333
+ - No changes to `run-task.yaml` or `implement.yaml`.
2334
+ - The only addition is the `mast` entry in the claude-runner `.mcp.json`.
2335
+
2336
+ ---
2337
+
2338
+ ## 13. Implementation Notes
2339
+
2340
+ ### 13.1 Language
2341
+
2342
+ TypeScript (Node.js LTS). Rationale:
2343
+
2344
+ - Fits the existing monorepo stack — no second language in the container.
2345
+ - The MCP TypeScript SDK (`@modelcontextprotocol/sdk`) is Anthropic's primary SDK
2346
+ and has the best type safety and first-class support.
2347
+ - `better-sqlite3`'s synchronous API removes async complexity from graph queries;
2348
+ the recursive CTEs are blocking operations anyway.
2349
+
2350
+ ### 13.2 Dependency Map
2351
+
2352
+ | Concern | Package | Notes |
2353
+ |---|---|---|
2354
+ | MCP server | `@modelcontextprotocol/sdk` | Official TS SDK, stdio transport |
2355
+ | BM25 | SQLite FTS5 (built-in) | Replaces external BM25 dependency entirely; `trigram` tokenizer for code identifiers |
2356
+ | Knowledge graph | `better-sqlite3` + `@types/better-sqlite3` | Synchronous API, WAL mode, recursive CTEs |
2357
+ | Query builder | `kysely` | Typed SQL query builder over the `better-sqlite3` connection |
2358
+ | AST parsing | `tree-sitter` + `tree-sitter-typescript` | Official Node.js bindings + TypeScript grammar |
2359
+ | Path resolution | `tsconfig-paths` | Resolves tsconfig `paths` aliases at index time |
2360
+ | Validation | `zod` | MCP tool inputs cross a trust boundary; validate `symbol`, `file_path`, `max_depth`, etc. before hitting the DB |
2361
+ | Locking | `proper-lockfile` | PID-based advisory lock; set `stale: 10000` (10s) to handle abrupt container exits |
2362
+ | CLI | `commander` | Standard TS CLI |
2363
+ | File walking | `fast-glob` | Glob pattern support for `exclude_patterns` |
2364
+ | File watching | `chokidar` | Powers `mast serve --watch` (§11.4) |
2365
+ | Token counting | `@anthropic-ai/tokenizer` | Counts `tokens_returned`/`tokens_full_file_upper_bound` for `_stats` (§14.5) |
2366
+ | Identifiers | `uuid` | Per-`mast serve`-session `session_id` for metrics attribution |
2367
+
2368
+ ### 13.4 Project Structure
2369
+
2370
+ ```
2371
+ packages/mast/
2372
+ ├── src/
2373
+ │ ├── cli/
2374
+ │ │ ├── index.ts # commander entry point — mast <command>
2375
+ │ │ ├── init.ts
2376
+ │ │ ├── index-cmd.ts # `mast index` (avoids conflict with src/index.ts)
2377
+ │ │ ├── serve.ts
2378
+ │ │ ├── status.ts
2379
+ │ │ ├── metrics-cmd.ts # `mast metrics --since|--rollup|--vacuum` (§14)
2380
+ │ │ └── install-hooks.ts
2381
+ │ ├── mcp/
2382
+ │ │ ├── server.ts # MCP server setup, tool registration (§7.4 Steps 3-4)
2383
+ │ │ ├── startup.ts # bootstrap + schema-version guard + orphan-state cleanup (§7.4 Steps 1-2)
2384
+ │ │ ├── staleness.ts # stat-and-sync wrapper for all read tools (§9.0)
2385
+ │ │ ├── context.ts # AppContext (db, chunkStore, config, sessionId) shared by every tool
2386
+ │ │ └── tools/
2387
+ │ │ ├── search.ts # §9 mast_search — fused BM25 + ranker D
2388
+ │ │ ├── project-skeleton.ts
2389
+ │ │ ├── exports.ts
2390
+ │ │ ├── signature.ts
2391
+ │ │ ├── callers.ts # verified + potential partition (§9 mast_callers)
2392
+ │ │ ├── dependencies.ts
2393
+ │ │ ├── implementors.ts
2394
+ │ │ ├── rename-impact.ts # §9 mast_rename_impact
2395
+ │ │ ├── reindex.ts
2396
+ │ │ ├── status.ts
2397
+ │ │ └── efficiency.ts # mast_efficiency telemetry tool (§9, §14)
2398
+ │ ├── indexer/
2399
+ │ │ ├── index.ts # orchestrates the single indexing pass (§7.1)
2400
+ │ │ ├── walker.ts # file discovery, exclude pattern matching, manifest diff
2401
+ │ │ ├── watcher.ts # chokidar-backed `mast serve --watch` (§11.4)
2402
+ │ │ └── import-resolver.ts # tsconfig paths + pnpm workspace resolution (§13.7)
2403
+ │ ├── graph/
2404
+ │ │ ├── db.ts # better-sqlite3 + Kysely connection, schema init
2405
+ │ │ ├── populate.ts # AST → graph.db inserts (two-pass edge insertion, §10.3)
2406
+ │ │ ├── queries.ts # callers, implementors, dependencies, type-context
2407
+ │ │ ├── local-type-env.ts # POTENTIAL_CALL resolver heuristics (§10.3.1)
2408
+ │ │ └── checker-resolver.ts # opt-in `mast index --checker` pass (§10.3.2)
2409
+ │ ├── ast/
2410
+ │ │ ├── parser.ts # tree-sitter setup, parse file → AST
2411
+ │ │ ├── extractor.ts # LanguageExtractor contract + FileExtraction types
2412
+ │ │ ├── extract.ts # extension dispatch → per-language extractor
2413
+ │ │ ├── extractors/
2414
+ │ │ │ ├── typescript.ts # TS/JS: class-shell synth, method walk, hashes, splitting (§10.1)
2415
+ │ │ │ └── markdown.ts # heading-based doc chunking (§10.1)
2416
+ │ │ └── types.ts # Chunk, Export, SignatureResult, config, MCP I/O shared types
2417
+ │ ├── store/
2418
+ │ │ ├── sqliteChunkStore.ts # ChunkStore: chunk CRUD against graph.db's `chunks` table
2419
+ │ │ ├── config.ts # config resolution, index.json read/write
2420
+ │ │ ├── lock.ts # structure.lock manager (§7.6)
2421
+ │ │ └── lockMetrics.ts # JSONL lock-hold telemetry sink
2422
+ │ ├── search/
2423
+ │ │ ├── fused.ts # RRF fusion of BM25 + ranker D (§7.3); D-fire telemetry
2424
+ │ │ ├── declex.ts # ranker D — declaration-exact match (§7.3)
2425
+ │ │ ├── fts.ts # FTS5 queries (chunk_fts BM25 + identifier_fts exact match)
2426
+ │ │ └── potential-matches.ts # shared candidate collection for mast_callers/mast_rename_impact
2427
+ │ └── telemetry/
2428
+ │ ├── metrics.ts # metrics table writes, _stats meta builder (§14)
2429
+ │ └── tokenizer.ts # @anthropic-ai/tokenizer wrapper for counterfactuals
2430
+ ├── package.json
2431
+ └── tsconfig.json
2432
+ ```
2433
+
2434
+ **Language extensibility pattern:** `extract.ts` dispatches by file extension to a
2435
+ per-language extractor module in `ast/extractors/`. Each extractor implements the
2436
+ `LanguageExtractor` contract (defined in `ast/extractor.ts`) and owns its **full**
2437
+ extraction story — parsing strategy included:
2438
+
2439
+ ```typescript
2440
+ interface LanguageExtractor {
2441
+ language: Language; // "typescript" | "markdown" | ...
2442
+ extensions: readonly string[]; // [".ts", ".tsx"]
2443
+ extract(src: string, filePath: string, fileMtime: number,
2444
+ options: ExtractorOptions): FileExtraction;
2445
+ }
2446
+
2447
+ interface FileExtraction {
2448
+ language: Language; // concrete language of THIS file
2449
+ chunks: readonly Chunk[];
2450
+ symbols: readonly SymbolRecord[]; // empty for graph-less languages
2451
+ imports: readonly ImportRecord[];
2452
+ edges: readonly EdgeRecord[];
2453
+ identifierRows: readonly IdentifierRow[]; // identifier_fts rows per chunk
2454
+ }
2455
+ ```
2456
+
2457
+ The contract is deliberately parser-agnostic: the TypeScript extractor parses with
2458
+ tree-sitter internally (and keeps `declarationHash`/`bodyHash` as its own methods),
2459
+ while the markdown extractor line-scans — the pipeline never sees a `Tree` and
2460
+ never branches on language. `identifierRows` are produced by the extractor rather
2461
+ than the graph layer because what counts as an "identifier" is a language-level
2462
+ judgment: markdown contributes none, since `identifier_fts` feeds `mast_callers`
2463
+ potential_matches and prose mentions are not call sites.
2464
+
2465
+ Two extractors ship today: `typescript.ts` (`.ts`, `.tsx`, `.js`, `.jsx`) and
2466
+ `markdown.ts` (`.md`, §10.1 doc chunking). Adding Go or Python means adding a new
2467
+ extractor module and a `tree-sitter-<lang>` package — no changes to core indexer
2468
+ logic. Do not use tree-sitter `.scm` query files; the extractor function approach
2469
+ is sufficient and keeps the build simple.
2470
+
2471
+ ### 13.7 Path Resolution
2472
+
2473
+ Monorepo imports use two alias systems that both need resolving to physical file paths
2474
+ for the knowledge graph edges and `type_context` lookups to work:
2475
+
2476
+ **0. NodeNext `.js` specifier substitution** (e.g. `./repo.js` → `./repo.ts`)
2477
+
2478
+ TypeScript ESM/NodeNext code writes the *compiled* extension in relative specifiers
2479
+ (`import { Repo } from './repo.js'`) even though the on-disk source is `./repo.ts`.
2480
+ When a relative specifier carries a JS-family extension, the resolver looks up the
2481
+ TypeScript source first and only falls back to the literal file, matching tsc's
2482
+ "file extension substitution" lookup order:
2483
+
2484
+ | Specifier ext | Lookup order |
2485
+ | ------------- | ----------------------------------- |
2486
+ | `.js` | `.ts`, then `.tsx`, then `.js` |
2487
+ | `.jsx` | `.tsx`, then `.jsx` |
2488
+ | `.mjs` | `.mts`, then `.mjs` |
2489
+ | `.cjs` | `.cts`, then `.cjs` |
2490
+
2491
+ The source-first precedence means that when both `x.ts` and a real `x.js` exist,
2492
+ `./x.js` resolves to `x.ts` (the `.js` names the *output*). A genuine `.js` file with
2493
+ no TypeScript source still resolves to itself. Declaration files (`.d.ts`) are out of
2494
+ scope — MAST indexes implementation files. Without this rule, ESM `.js` specifiers left
2495
+ `resolved_path` NULL and star re-export barrels written with `.js` produced no
2496
+ `re_export_files` rows. See the TypeScript Modules Reference, "File extension
2497
+ substitution".
2498
+
2499
+ **1. tsconfig `paths` aliases** (e.g. `@api/types` → `./src/types/index.ts`)
2500
+
2501
+ Read the nearest `tsconfig.json` at `mast init` time using `tsconfig-paths`. Build a
2502
+ resolver function `resolveAlias(alias: string, fromFile: string): string | null` that
2503
+ is passed into Phase 1 and used wherever `resolved_path` is written to the `imports`
2504
+ table.
2505
+
2506
+ **2. pnpm workspace package names** (e.g. `@kluster-kinetic-01/shared`)
2507
+
2508
+ Walk the workspace root `pnpm-workspace.yaml` (or `workspaces` field in root
2509
+ `package.json`) at startup. For each matched package directory, read its
2510
+ `package.json` `name` field. Build a map `{ packageName → packageDir }`. When an
2511
+ import module matches a package name, resolve to `<packageDir>/src/index.ts` (or the
2512
+ `main`/`exports` field in that package's `package.json`).
2513
+
2514
+ **pnpm symlink handling:** pnpm links workspace packages into `node_modules` as
2515
+ symlinks. A naive resolver may return a path under `node_modules/@pkg/shared` (the
2516
+ symlink) rather than `packages/shared/src/index.ts` (the real file). The `files`
2517
+ table indexes real paths, so a symlink path would produce a graph edge pointing to a
2518
+ path that does not exist in the index. Always call `fs.realpathSync()` on the resolved
2519
+ path before writing it to `resolved_path`. This collapses symlinks to their canonical
2520
+ source paths, ensuring graph edges connect correctly.
2521
+
2522
+ Both resolvers are composed: alias resolution runs first, workspace resolution second,
2523
+ `realpathSync` applied last. External modules (no match in either resolver) leave
2524
+ `resolved_path = NULL` in the `imports` table.
2525
+
2526
+ This resolver is initialised once at `mast serve` startup and at the start of each
2527
+ `mast index` run.
2528
+
2529
+ ### 13.8 Dockerfile Pre-Warming
2530
+
2531
+ The index is pre-warmed during `docker build` to eliminate cold-start dead time on
2532
+ the first container run. Without this, the first task can stall for minutes while
2533
+ the index builds from scratch — a UX risk identified in the design review
2534
+ (Failure 4).
2535
+
2536
+ #### 13.8.1 Seed Index Pre-Warm
2537
+
2538
+ For a 5K-file class-heavy repo, indexing from scratch takes on the order of a
2539
+ minute or more. During that window the agent has a cold, empty index. The seed
2540
+ index moves this work into the Docker build, so the runtime container starts with
2541
+ a fully-warmed index for the build-time commit.
2542
+
2543
+ ```dockerfile
2544
+ # After the application source is copied into the image and dependencies installed:
2545
+ RUN mast init /workspace --state-dir /opt/mast-seed --no-index \
2546
+ && mast index /workspace --state-dir /opt/mast-seed
2547
+ ```
2548
+
2549
+ Two important properties of the seed:
2550
+
2551
+ 1. **The index runs fully at build time.** The seed contains a fully-populated
2552
+ `graph.db` (chunks, symbols, edges, `chunk_fts`, `identifier_fts`). The runtime
2553
+ container is ready to serve at full capability immediately (Step 3 of §7.4) — no
2554
+ warm-up window.
2555
+ 2. **Frozen at build commit.** The seed reflects whatever code was in the image at
2556
+ `docker build` time. Files modified since the build commit are picked up by
2557
+ §7.4 Step 4's filesystem scan (a few seconds for typical incremental staleness),
2558
+ and uncommitted-tree edits trigger JIT re-parse on first query (§9.0).
2559
+
2560
+ **Runtime copy.** The container entrypoint runs:
2561
+
2562
+ ```bash
2563
+ #!/bin/sh
2564
+ if [ -z "$(ls -A /workspace/.kluster/.mast 2>/dev/null)" ]; then
2565
+ cp -r /opt/mast-seed/. /workspace/.kluster/.mast/
2566
+ fi
2567
+ exec mast serve --state-dir /workspace/.kluster/.mast
2568
+ ```
2569
+
2570
+ The copy is conditional on the workspace state being empty. Subsequent container
2571
+ starts on the same mounted volume reuse the existing state (which may have
2572
+ already-applied incremental updates from prior tasks).
2573
+
2574
+ **Seed commit field on `mast_status`.** When the seed is copied, `index.json`
2575
+ records `seed_commit: "<git-rev>"` so `mast_status` can report:
2576
+
2577
+ ```
2578
+ seed_commit: abc1234 (built 2026-05-10T12:00:00Z)
2579
+ last_indexed: 2026-05-13T14:22:00Z
2580
+ files_since_seed: 47
2581
+ ```
2582
+
2583
+ This makes "why does the index look stale?" debuggable without opening
2584
+ `file_manifest.json` by hand.
2585
+
2586
+ **Schema-bump invalidation.** On startup (§7.4 Step 2), if
2587
+ `index.json.schema_version != CURRENT_SCHEMA_VERSION`, both the runtime state
2588
+ AND the seed are considered invalid. The state is wiped and a full reindex runs
2589
+ in the background (Step 4). The seed in `/opt/mast-seed` is not re-built —
2590
+ that's the next image build's job — but it is ignored on this run.
2591
+
2592
+ ### 13.10 What to Keep from mcp-vector-search (as reference only)
2593
+
2594
+ mcp-vector-search is a Python codebase — nothing is reused directly. It serves as a
2595
+ reference for:
2596
+
2597
+ - RRF fusion logic
2598
+ - Incremental indexing strategy (mtime-based staleness)
2599
+
2600
+ ### 13.11 Distribution
2601
+
2602
+ ```
2603
+ pip install # not applicable
2604
+ npm install -g mast-search # installs CLI + MCP server
2605
+ ```
2606
+
2607
+ For the SDD pipeline, mast is installed into the claude-runner Docker image. The
2608
+ Docker-baked seed index (§13.8) is pre-built into the image layer to avoid a
2609
+ from-scratch index build inside a task container.
2610
+
2611
+ For external developers:
2612
+
2613
+ ```bash
2614
+ npm install -g mast-search
2615
+ mast init . # initialise index
2616
+ claude mcp add mast -- mast serve # wire into Claude Code
2617
+ ```
2618
+
2619
+ There is only one install shape — lexical + declaration-exact search is the whole
2620
+ product, so no `--no-embeddings`-style variant is needed.
2621
+
2622
+ ---
2623
+
2624
+ ## 14. Telemetry & Measurement
2625
+
2626
+ The MAST thesis is **chunks not files → fewer tokens per task**. If that claim is
2627
+ not measurable, the index and the Docker layers that support it are a complexity
2628
+ budget the project cannot defend. This section
2629
+ specifies the instrumentation that makes the savings legible to humans (`mast metrics`),
2630
+ visible to the agent (`mast_efficiency`), and persistent across sessions (the SQLite
2631
+ `metrics` table).
2632
+
2633
+ ### 14.1 Goals
2634
+
2635
+ - **Defensible savings number.** When asked "is MAST worth it?", produce a chart-backed
2636
+ answer with a documented methodology, not a vibe.
2637
+ - **In-loop agent feedback.** The agent can call `mast_efficiency` to see whether its
2638
+ own tool usage was efficient, and self-correct on subsequent tasks.
2639
+ - **Per-task attribution.** The SDD pipeline (§12) captures one `mast metrics --session`
2640
+ snapshot per task so savings can be analysed by task type.
2641
+ - **Negligible overhead.** Instrumentation must add < 1ms per tool call; write-heavy,
2642
+ read-light. The one documented exception is `tokens_full_file_upper_bound` for
2643
+ whole-project-scale callers (`mast_project_skeleton`, §14.2): exact tokenization
2644
+ alone already exceeds 1ms per file, so this counterfactual is bounded instead by a
2645
+ per-call tokenization budget (F8, §14.2) — worst case ~0.7-0.8s on a first call
2646
+ against an uncached project, converging toward the < 1ms goal as the cache warms and
2647
+ degenerating to true negligible overhead (cache hits only) once fully warm.
2648
+
2649
+ ### 14.2 The `_stats` Meta Field
2650
+
2651
+ Every read tool (`mast_search`, `mast_signature`, `mast_exports`,
2652
+ `mast_project_skeleton`, `mast_callers`, `mast_dependencies`, `mast_implementors`)
2653
+ attaches a `_stats` object to its response:
2654
+
2655
+ ```json
2656
+ {
2657
+ "results": [ /* ... tool-specific shape ... */ ],
2658
+ "_stats": {
2659
+ "tool": "mast_search",
2660
+ "tokens_returned": 412,
2661
+ "tokens_full_file_upper_bound": 3187,
2662
+ "files_referenced": ["api/services/auth/src/handler.ts", "api/services/auth/src/repository.ts"],
2663
+ "efficiency_ratio": 0.871,
2664
+ "duration_ms": 38
2665
+ }
2666
+ }
2667
+ ```
2668
+
2669
+ - `tokens_returned` — token count of the serialised response body using
2670
+ `@anthropic-ai/tokenizer` (see §14.5).
2671
+ - `tokens_full_file_upper_bound` — sum of `@anthropic-ai/tokenizer` counts over the
2672
+ full contents of every unique file referenced in the results. This is the "what
2673
+ a naive `Read` of every result file would have cost" counterfactual.
2674
+ - `efficiency_ratio` — `1 - (tokens_returned / tokens_full_file_upper_bound)`. Always
2675
+ in `[0, 1]`. Higher is better.
2676
+ - `duration_ms` — wall-clock time for the tool call, including JIT staleness
2677
+ re-parse if triggered.
2678
+
2679
+ **Honest framing.** `tokens_full_file_upper_bound` is explicitly labelled as an upper
2680
+ bound. A smart agent using `Grep -A 10 -B 10` would have used fewer tokens than the
2681
+ upper bound but more than MAST — the real saving sits between zero and `efficiency_ratio`.
2682
+ Reporting the upper bound is defensible because the methodology is documented and
2683
+ the label is honest. Reporting "X% savings" with no upper-bound qualifier would not
2684
+ survive scrutiny.
2685
+
2686
+ **Implementation status.** `tokens_full_file_upper_bound` is computed — for each
2687
+ unique file in `files_referenced`, `estimateFullFileBound` (`telemetry/tokenizer.ts`)
2688
+ reads the file's full contents from the project root and sums `countTokens` over
2689
+ them, with an mtime-keyed cache so repeated calls against an unchanged file don't
2690
+ re-tokenize. It previously shipped as an unimplemented stub that always returned 0,
2691
+ which made `efficiency_ratio` a constant 0 across every recorded row (see the
2692
+ Promotion Log, 2026-07-15) — that regression is what this fixes.
2693
+
2694
+ **Per-call work cap (F8, 2026-08-07).** Reading and tokenizing every referenced file
2695
+ does not scale to a caller like `mast_project_skeleton`, which references every file
2696
+ in the project — `estimateFullFileBound` measured ~28s/call on a 1,334-file project,
2697
+ 99% of it here. Beyond a per-call budget (`FULL_FILE_TOKENIZE_BUDGET_PER_CALL = 32`
2698
+ exact reads per call, cache hits excluded), further cache-miss files are **not** read
2699
+ or tokenized; they are size-estimated instead, at
2700
+ `Math.ceil(sizeBytes / BYTES_PER_TOKEN_ESTIMATE)` bytes-per-token
2701
+ (`BYTES_PER_TOKEN_ESTIMATE = 4`, the standard heuristic for source text) and are not
2702
+ cached — an estimate must never masquerade as an exact cached count. Successive calls
2703
+ over the same file set progressively convert estimates to exact, cached counts as the
2704
+ budget reaches further into the set, converging to fully-exact after enough calls with
2705
+ zero cache thrash (`FULL_FILE_BOUND_CACHE_LIMIT` raised from 200 to 8192 alongside the
2706
+ budget — see `telemetry/tokenizer.ts` doc comments). The upper-bound counterfactual was
2707
+ already explicitly approximate (see "Honest framing" above and §14.5); the size
2708
+ estimate for budget-exceeding files is an additional, documented layer of the same
2709
+ approximation, not a departure from it.
2710
+
2711
+ ### 14.3 The `metrics` Table
2712
+
2713
+ Telemetry persists in `graph.db` (same SQLite database as the knowledge graph; one
2714
+ fewer connection to manage):
2715
+
2716
+ ```sql
2717
+ CREATE TABLE IF NOT EXISTS metrics (
2718
+ id INTEGER PRIMARY KEY,
2719
+ tool_name TEXT NOT NULL,
2720
+ call_timestamp REAL NOT NULL, -- unix epoch seconds (REAL for sub-second precision)
2721
+ tokens_returned INTEGER NOT NULL,
2722
+ tokens_full_file_upper_bound INTEGER NOT NULL,
2723
+ duration_ms INTEGER NOT NULL,
2724
+ mode TEXT, -- historical: pre-2026-08-06 rows only ("hybrid"|"lexical"); new rows NULL
2725
+ session_id TEXT NOT NULL, -- uuid set at mast serve startup
2726
+ status TEXT NOT NULL, -- "ok" | "stale_returned" | "error"
2727
+ args_json TEXT, -- salient tool arguments, capped at 1,000 chars
2728
+ results_json TEXT, -- {file_path, symbol_name} identity pairs, capped at 20 entries
2729
+ declex_json TEXT -- ranker D fire telemetry (mast_search only), NULL when D silent
2730
+ );
2731
+
2732
+ CREATE INDEX IF NOT EXISTS idx_metrics_timestamp ON metrics(call_timestamp);
2733
+ CREATE INDEX IF NOT EXISTS idx_metrics_tool ON metrics(tool_name);
2734
+ CREATE INDEX IF NOT EXISTS idx_metrics_session ON metrics(session_id);
2735
+
2736
+ CREATE TABLE IF NOT EXISTS metrics_daily (
2737
+ -- Pre-aggregated rollup; populated by `mast metrics --rollup`.
2738
+ day TEXT PRIMARY KEY, -- ISO date, e.g. "2026-05-13"
2739
+ tool_name TEXT NOT NULL,
2740
+ calls INTEGER NOT NULL,
2741
+ tokens_returned_total INTEGER NOT NULL,
2742
+ tokens_full_file_total INTEGER NOT NULL,
2743
+ avg_duration_ms REAL NOT NULL,
2744
+ PRIMARY KEY (day, tool_name)
2745
+ );
2746
+ ```
2747
+
2748
+ **Writes are non-blocking.** `metrics.ts` enqueues writes on a per-tick batch
2749
+ (flushed every 1s or every 100 rows, whichever comes first) so the metrics path
2750
+ never blocks a tool response. Worst-case data loss on abrupt container exit is one
2751
+ flush window's worth of rows; acceptable for a savings metric.
2752
+
2753
+ **Argument/result identity columns (`args_json`, `results_json`).** Added additively
2754
+ (`ALTER TABLE metrics ADD COLUMN`, same precedent as `edges.resolution`/`call_line`/
2755
+ `context` — no `CURRENT_SCHEMA_VERSION` bump) to make the "linked chain" measurable:
2756
+ did a later `mast_signature`/`mast_exports`/`mast_callers` call target a symbol or
2757
+ file that an earlier `mast_search` returned in the same session? This was the missing
2758
+ instrumentation the `mast_capsule` v2 hold identified (Promotion Log, 2026-07-15) —
2759
+ without it, a capsule chain-rate measurement can only be an argument-blind upper
2760
+ bound. `args_json` carries the salient tool arguments (query + filters for search;
2761
+ symbol and/or file_path for signature/exports/callers), capped at 1,000 characters;
2762
+ `results_json` carries the tool's returned `{file_path, symbol_name}` identity pairs
2763
+ in rank order, capped at 20 entries. Both caps are stated honestly in the payload
2764
+ when hit (`_truncated`) rather than silently cut. Wired for `mast_search`,
2765
+ `mast_signature`, `mast_exports`, and `mast_callers` — the chain-analysis tools the
2766
+ capsule decision depends on; both columns are `NULL` for every other tool and for
2767
+ rows recorded before this migration.
2768
+
2769
+ **`declex_json` (F18 D-fire telemetry — M2 decision memo condition 3).** Added
2770
+ via the same additive `ALTER TABLE` precedent (no schema bump). Populated only
2771
+ on `mast_search` calls where ranker D (§7.3) actually fired; `NULL` when D was
2772
+ silent, when `declaration_exact_ranker` is off, for every other tool, and for
2773
+ pre-migration rows. Shape: `{fired: true, top_match_channel: "full"|"segment",
2774
+ candidate_count, window_effects: [{chunk_id, symbol_name, rank_with_d,
2775
+ rank_without_d}], _truncated?}`. `window_effects` is a dual-fusion diff computed
2776
+ in-memory per call — the fused (pre-dedup) rank of each affected chunk with D's
2777
+ list included vs excluded from RRF; ranks are the actual positions in each list
2778
+ (`null` only when the chunk is absent from that list entirely, e.g. a D-only
2779
+ anchor has `rank_without_d: null`), capped at 10 entries with a top-level
2780
+ `_truncated` count. This column is the input signal for the F18 kill-switch and
2781
+ the M2 re-entry criteria: fire rate on real queries, and whether D demotes
2782
+ in-window targets, are both answerable from it without re-instrumenting.
2783
+
2784
+ ### 14.4 Rotation Policy
2785
+
2786
+ A 5K-file repo with ~50 tool calls per task × ~30 tasks per day produces ~1,500
2787
+ metrics rows per day. The table needs rotation to keep `SUM()` queries fast.
2788
+
2789
+ - `mast metrics --rollup` (run weekly via a cron-style hook, or manually): aggregates
2790
+ rows older than 30 days into the `metrics_daily` table, then deletes the raw rows.
2791
+ - `mast metrics --vacuum`: runs `VACUUM` on `graph.db` after a rollup to reclaim
2792
+ space.
2793
+ - The rollup is non-destructive at the daily level: 90-day or 1-year historical
2794
+ analysis still works against `metrics_daily`.
2795
+
2796
+ ### 14.5 Tokenizer Choice
2797
+
2798
+ `@anthropic-ai/tokenizer` (`^0.0.4`) counts `tokens_returned` and
2799
+ `tokens_full_file_upper_bound` — but it is **approximate, not ground truth**.
2800
+ The package implements the Claude 2-era tokenizer, and Anthropic never
2801
+ published the Claude 3+ vocabularies, so absolute counts drift for every model
2802
+ that actually consumes MAST output today.
2803
+
2804
+ ```typescript
2805
+ import { countTokens } from "@anthropic-ai/tokenizer";
2806
+ const n = countTokens(responseBodyAsString);
2807
+ ```
2808
+
2809
+ **Why this is still the right mechanism.** §14.2's headline number is the
2810
+ savings *ratio*, and both its numerator and denominator are counted with the
2811
+ same tokenizer — the per-count error mostly cancels, so the ratio is robust
2812
+ even though the absolute counts are not. The same honesty rule that governs
2813
+ the upper-bound counterfactual (§14.2) applies here: report the limitation,
2814
+ don't paper over it.
2815
+
2816
+ **The label.** The active tokenizer is reported verbatim in `mast_efficiency`'s
2817
+ `tokenizer` field and in `mast metrics`'s footer as:
2818
+
2819
+ ```
2820
+ @anthropic-ai/tokenizer (claude-2 era, approximate for current models)
2821
+ ```
2822
+
2823
+ The string has one definition (`TOKENIZER_LABEL` in `src/telemetry/tokenizer.ts`)
2824
+ that every consumer reads, so the wording cannot drift between surfaces.
2825
+
2826
+ **Future seams (not implemented).** An exact mode via the Anthropic API's
2827
+ `count_tokens` endpoint (opt-in, requires an API key) and a `tiktoken
2828
+ cl100k_base` fallback for non-Claude consumers are both documented options;
2829
+ neither ships today, and if either is added the reported label must change to
2830
+ match the active counter.
2831
+
2832
+ ### 14.6 CLI: `mast metrics`
2833
+
2834
+ ```
2835
+ Usage: mast metrics [options]
2836
+
2837
+ Options:
2838
+ --since <duration> e.g. "1h", "24h", "7d", "30d" (default: 7d)
2839
+ --by-tool Break down by tool_name (default: true)
2840
+ --rollup Aggregate rows older than --keep-days into metrics_daily, then delete raw rows
2841
+ --vacuum Delete daily roll-up rows older than --keep-days
2842
+ --keep-days <n> Retention days for --rollup / --vacuum (default: 7 for rollup, 90 for vacuum)
2843
+ --locks Summarize structure-lock hold/wait timing by caller (D6) — reads
2844
+ store/lockMetrics.ts's `<state_dir>/lock-metrics.jsonl`; does not
2845
+ touch graph.db. Prints "No lock metrics recorded." (exit 0) when
2846
+ the file is missing or empty; malformed lines are skipped and
2847
+ counted, never fatal.
2848
+ --json Machine-readable output
2849
+ --state-dir <dir> State directory override
2850
+ ```
2851
+
2852
+ Sample output (`mast metrics --since 7d --by-tool`):
2853
+
2854
+ ```
2855
+ Tool Calls Tokens Avg ms p50 ms p95 ms Efficiency
2856
+ ──────────────────────────────────────────────────────────────────────────────
2857
+ mast_search 1,847 742,103 41.2 32 89 87.4%
2858
+ mast_signature 912 91,820 18.7 15 47 95.6%
2859
+ mast_exports 408 33,041 12.4 10 31 96.3%
2860
+ mast_project_skeleton 127 8,209 9.1 8 22 98.0%
2861
+ mast_callers 321 42,118 22.5 19 58 95.7%
2862
+ mast_dependencies 156 12,047 14.0 11 35 97.6%
2863
+ mast_implementors 88 6,213 10.8 9 26 96.9%
2864
+
2865
+ Tokenizer: @anthropic-ai/tokenizer (claude-2 era, approximate for current models)
2866
+ ```
2867
+
2868
+ `p50`/`p95` (D6 — `adr/009-2026-08-10-measurement-harness.md`) are nearest-rank percentiles (sort
2869
+ ascending, take the value at rank `ceil(P/100 * N)`, no interpolation) computed in JS
2870
+ over the window's raw `duration_ms` values — `--by-tool`'s existing `Avg ms` column
2871
+ alone hides tail latency (this is the column that would have caught F8's 28 s
2872
+ outlier). `--json` emits the same rows as an array, each carrying
2873
+ `p50_duration_ms`/`p95_duration_ms` alongside the existing `avg_duration_ms`.
2874
+
2875
+ Sample output (`mast metrics --locks`):
2876
+
2877
+ ```
2878
+ Caller Count Hold p50 Hold p95 Hold max Wait p50 Wait p95 Wait max Failed
2879
+ ────────────────────────────────────────────────────────────────────────────────────────
2880
+ index-run 42 18 47 61 2 6 9 0
2881
+ jit-staleness 301 1 4 9 0 1 3 2
2882
+
2883
+ Warning: skipped 1 malformed line(s) in lock-metrics.jsonl.
2884
+ ```
2885
+
2886
+ `Hold *` comes from `LockEvent`'s `released.holdMs`; `Wait *` from `acquired.waitMs`;
2887
+ `Failed` counts `failed` events (acquisition attempts that exhausted retries). The
2888
+ malformed-line warning only appears when at least one JSONL line failed to parse or
2889
+ didn't match the `LockEvent` shape — those lines are skipped, never fatal.
2890
+
2891
+ ### 14.7 SDD Pipeline Integration
2892
+
2893
+ The SDD task pipeline (§12) adds one step to its task wrap-up:
2894
+
2895
+ ```yaml
2896
+ # packages/workbench/sdd/flows/strategies/task-pipeline/run-task.yaml (excerpt)
2897
+ - execute-command:
2898
+ name: capture_mast_metrics
2899
+ cmd: mast metrics --session --json > {task_history_dir}/mast-metrics.json
2900
+ run_after: [implement_task, review_task]
2901
+ ```
2902
+
2903
+ This produces one `mast-metrics.json` per task in the per-task history directory.
2904
+ Retrospective analysis ("does MAST save more on bug fixes than on feature additions?")
2905
+ becomes a `jq` over those files, not a re-run of the entire pipeline.
2906
+
2907
+ ### 14.8 Agent Feedback Loop
2908
+
2909
+ The `mast_efficiency` MCP tool (§9) exposes `_stats` aggregates back to the agent
2910
+ within the conversation. The implement-task prompt (§12) instructs:
2911
+
2912
+ > Near the end of your task, call `mast_efficiency { "scope": "session" }`. If
2913
+ > `efficiency_ratio < 0.30`, you have been reading more than you should — prefer
2914
+ > `mast_search` over `Read` for the next task. If `efficiency_ratio > 0.85`, you
2915
+ > are using MAST well; keep going.
2916
+
2917
+ This is the load-bearing reason for `mast_efficiency`'s existence as an MCP tool
2918
+ rather than a CLI-only command: **the instrumentation has to have at least one
2919
+ consumer inside the agent loop**, or it rots from disuse.
2920
+
2921
+ ### 14.9 What Is Deliberately Not Measured
2922
+
2923
+ - **Latency p99.** Tool call duration is captured per-row, but no SLA is asserted
2924
+ on it. Latency optimisation comes after the savings thesis is validated.
2925
+ - **Per-user / per-agent attribution.** v1 has one agent per `mast serve` session;
2926
+ multi-tenancy is out of scope.