llmnav 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (68) hide show
  1. package/CHANGELOG.md +113 -0
  2. package/LICENSE +21 -0
  3. package/README.md +294 -0
  4. package/ROADMAP.md +71 -0
  5. package/bin/llmnav.js +16 -0
  6. package/docs/agent-integration.md +114 -0
  7. package/docs/api.md +290 -0
  8. package/docs/architecture.md +286 -0
  9. package/docs/benchmarking.md +164 -0
  10. package/docs/ci.md +196 -0
  11. package/docs/cli.md +233 -0
  12. package/docs/configuration.md +117 -0
  13. package/docs/editor-integration.md +29 -0
  14. package/docs/faq.md +59 -0
  15. package/docs/graph.md +92 -0
  16. package/docs/language-examples.md +130 -0
  17. package/docs/migration.md +130 -0
  18. package/docs/performance-v0.2.md +42 -0
  19. package/docs/provider-neutral-integration.md +66 -0
  20. package/docs/publishing.md +86 -0
  21. package/docs/quickstart.md +139 -0
  22. package/docs/research.md +31 -0
  23. package/docs/spec.md +424 -0
  24. package/examples/provider-neutral-host.d.mts +17 -0
  25. package/examples/provider-neutral-host.mjs +40 -0
  26. package/package.json +79 -0
  27. package/schema/config.schema.json +296 -0
  28. package/src/agent-protocol.js +117 -0
  29. package/src/agent-tools.js +61 -0
  30. package/src/agents.js +127 -0
  31. package/src/boundaries.js +50 -0
  32. package/src/changes.js +168 -0
  33. package/src/cli.js +459 -0
  34. package/src/config.js +305 -0
  35. package/src/contracts.js +70 -0
  36. package/src/declaration.js +334 -0
  37. package/src/doctor.js +124 -0
  38. package/src/editor.js +107 -0
  39. package/src/evaluation.js +67 -0
  40. package/src/files.js +81 -0
  41. package/src/formatter.js +23 -0
  42. package/src/generator.js +528 -0
  43. package/src/graph-input.js +157 -0
  44. package/src/graph.js +403 -0
  45. package/src/incremental.js +262 -0
  46. package/src/index.d.ts +673 -0
  47. package/src/index.js +115 -0
  48. package/src/initializer.js +137 -0
  49. package/src/inverted-index.js +350 -0
  50. package/src/parser.js +449 -0
  51. package/src/project.js +65 -0
  52. package/src/prompt-bundle.js +108 -0
  53. package/src/registry.js +107 -0
  54. package/src/sarif.js +70 -0
  55. package/src/search-shards.js +75 -0
  56. package/src/search.js +636 -0
  57. package/src/spec.d.ts +27 -0
  58. package/src/spec.js +237 -0
  59. package/src/tokenizer.js +37 -0
  60. package/src/transaction.js +557 -0
  61. package/src/util.js +256 -0
  62. package/src/validator.js +635 -0
  63. package/templates/file-card.txt +8 -0
  64. package/templates/lexicon.json +7 -0
  65. package/templates/line-card.txt +9 -0
  66. package/templates/module-card.txt +9 -0
  67. package/templates/queries.jsonl +1 -0
  68. package/templates/symbol-card.txt +10 -0
package/docs/api.md ADDED
@@ -0,0 +1,290 @@
1
+ # Programmatic API
2
+
3
+ The npm package exposes the deterministic operations used by the CLI. It is ESM-only, requires Node.js 22 or newer, and ships TypeScript declarations.
4
+
5
+ ```js
6
+ import {
7
+ generateProject,
8
+ parseLlmnavBlocks,
9
+ queryProject,
10
+ scanProject,
11
+ validateProject,
12
+ } from "llmnav";
13
+ ```
14
+
15
+ ## Parse one source string
16
+
17
+ ```js
18
+ import { parseLlmnavBlocks } from "llmnav";
19
+
20
+ const blocks = parseLlmnavBlocks(source, "src/session.ts");
21
+ for (const block of blocks) {
22
+ console.log(block.card.id, block.startLine);
23
+ }
24
+ ```
25
+
26
+ Parsing does not perform project-level semantic validation. Read `syntaxErrors` on each block or validate a scanned project.
27
+
28
+ ## Canonicalize one source string safely
29
+
30
+ ```js
31
+ import { canonicalizeSource } from "llmnav";
32
+
33
+ const result = canonicalizeSource(source, "src/session.ts");
34
+ if (result.errors.length > 0) {
35
+ throw new Error(result.errors.map((item) => item.message).join("\n"));
36
+ }
37
+ console.log(result.source);
38
+ ```
39
+
40
+ Canonicalization refuses to erase unknown fields, malformed lines, or duplicate scalar fields. Unsafe blocks remain byte-for-byte unchanged.
41
+
42
+ ## Scan and validate a repository
43
+
44
+ ```js
45
+ import { scanProject, validateProject } from "llmnav";
46
+
47
+ const project = await scanProject(process.cwd());
48
+ const diagnostics = validateProject(project);
49
+ const errors = diagnostics.filter((item) => item.severity === "error");
50
+ ```
51
+
52
+ `scanProject` performs a full source read. Use `scanProjectIncremental` when building a persistent tool that can reuse `.llmnav/cache/file-state.json`.
53
+
54
+ ```js
55
+ import { scanProjectIncremental } from "llmnav";
56
+
57
+ const { project, fileState, stats } = await scanProjectIncremental(process.cwd());
58
+ console.log(stats.parsedFiles, stats.reusedFiles);
59
+ ```
60
+
61
+ Stat hints are an optimization, not deterministic output. The returned `fileState` contains only repository-relative generated data.
62
+
63
+ Attached symbol declarations expose generated `language`, `exported`, `visibility`, optional Go `receiver`, `endOffset`, and declaration `bodyHash` fields. Indexed cards also expose sorted route, event, schema, migration, and command `boundaries` with confidence and evidence.
64
+
65
+ ## Generate incrementally and transactionally
66
+
67
+ ```js
68
+ import { generateProject } from "llmnav";
69
+
70
+ const result = await generateProject(process.cwd());
71
+ if (!result.ok) {
72
+ console.error(result.diagnostics);
73
+ process.exitCode = 1;
74
+ }
75
+
76
+ console.log(result.changedCards);
77
+ console.log(result.affectedBoundaries);
78
+ console.log(result.affectedCatalogs);
79
+ console.log(result.incremental.files);
80
+ console.log(result.incremental.cards);
81
+ console.log(result.incremental.graph);
82
+ console.log(result.transaction);
83
+ ```
84
+
85
+ `result.index.contractFingerprints` contains deterministic `exportedApi` and `configuration` SHA-256 values. A changed fingerprint produces a non-failing `LNV009` diagnostic so callers can require explicit contract review without treating every body edit as an API change.
86
+
87
+ Each `affectedBoundaries` record identifies the changed semantic ID, changed hash dimensions, module, generated structural boundaries, outbound semantic relation targets, and cards that depend on the changed ID. Records and nested IDs use deterministic ordering.
88
+
89
+ Default generation reuses file and card state and commits cache changes through a recoverable directory transaction.
90
+
91
+ A read-only drift check is explicit:
92
+
93
+ ```js
94
+ const result = await generateProject(process.cwd(), { check: true });
95
+ ```
96
+
97
+ A forced full rebuild is available for verification and benchmarks:
98
+
99
+ ```js
100
+ const result = await generateProject(process.cwd(), { incremental: false });
101
+ ```
102
+
103
+ `incremental: false` reparses every source file and retokenizes every card. It must produce the same deterministic cache bytes as incremental generation.
104
+
105
+ Failure injection options exist for the repository test suite and are not a normal application interface.
106
+
107
+ ## Build and update an inverted index
108
+
109
+ ```js
110
+ import { buildInvertedIndex } from "llmnav";
111
+
112
+ const initial = buildInvertedIndex(index);
113
+ const updated = buildInvertedIndex(nextIndex, initial.searchIndex);
114
+
115
+ console.log(updated.stats.indexedCards);
116
+ console.log(updated.stats.reusedCards);
117
+ ```
118
+
119
+ The previous index is reused only when schema version, tokenizer version, field order, and repository ID match. Incremental and full builds serialize identically for the same primary index.
120
+
121
+ Useful lower-level exports include:
122
+
123
+ ```js
124
+ import {
125
+ buildSearchDocument,
126
+ isCompatibleSearchIndex,
127
+ searchCardSetHash,
128
+ searchDocumentHash,
129
+ verifySearchIndex,
130
+ } from "llmnav";
131
+ ```
132
+
133
+ ## Query a generated project
134
+
135
+ ```js
136
+ import { queryProject } from "llmnav";
137
+
138
+ const results = await queryProject(process.cwd(), "replayed refresh token", {
139
+ top: 5,
140
+ });
141
+ ```
142
+
143
+ `queryProject` recovers an interrupted transaction, loads `index.json`, loads or validates `search-index.json`, and applies repository aliases. Result limits are bounded from 1 to 100.
144
+
145
+ ## Query in-memory indexes
146
+
147
+ ```js
148
+ import { queryPreparedIndex } from "llmnav";
149
+
150
+ const metrics = {};
151
+ const results = queryPreparedIndex(index, searchIndex, "reserve credits", {
152
+ top: 5,
153
+ lexicon: { aliases: {} },
154
+ metrics,
155
+ });
156
+
157
+ console.log(metrics.documentTokenizations); // 0
158
+ ```
159
+
160
+ `queryIndex(index, query)` builds and weakly caches an in-memory inverted index when one is not supplied.
161
+
162
+ `queryIndexLegacy(index, query)` retains the v0.1-compatible per-query retokenization path for regression tests and migration measurement. New integrations should not use it in production.
163
+
164
+ ## Inspect changed cards and catalogs
165
+
166
+ ```js
167
+ import { compareCardIndexes, describeAffectedCatalogs } from "llmnav";
168
+
169
+ const changes = compareCardIndexes(previousIndex, currentIndex);
170
+ const catalogs = describeAffectedCatalogs(
171
+ changedFiles,
172
+ config.generation.cacheDirectory,
173
+ config,
174
+ previousIndex,
175
+ currentIndex,
176
+ );
177
+ ```
178
+
179
+ Changed-card records distinguish semantic, structure, and body dimensions. Catalog records include only repository, module, agent-context, and prompt-prefix artifacts.
180
+
181
+ ## Recover or commit a cache transaction
182
+
183
+ Most callers should use `generateProject`. Lower-level transaction functions are exported for integration testing and specialized hosts.
184
+
185
+ ```js
186
+ import { recoverGenerationTransaction } from "llmnav";
187
+
188
+ const recovery = await recoverGenerationTransaction(root, {
189
+ cacheDirectory: ".llmnav/cache",
190
+ });
191
+ ```
192
+
193
+ `commitGeneratedCache` expects a complete artifact map and verifies the staged manifest before replacing the live cache. Its path and failure-injection options are deliberately strict.
194
+
195
+ ## Resolve and build context
196
+
197
+ ```js
198
+ import { buildContext, showProjectCard } from "llmnav";
199
+
200
+ const shown = await showProjectCard(root, "auth.session.renew");
201
+ const context = await buildContext(root, "auth.session.renew", {
202
+ depth: 1,
203
+ budget: 2500,
204
+ });
205
+ ```
206
+
207
+ Redirected IDs resolve through `.llmnav/ids.jsonl` before source cards are selected.
208
+
209
+ ## Constants
210
+
211
+ The package exports versioned generated-format constants:
212
+
213
+ ```js
214
+ import {
215
+ CONTRACT_FINGERPRINT_SCHEMA_VERSION,
216
+ FILE_STATE_SCHEMA_VERSION,
217
+ SEARCH_INDEX_ENCODING,
218
+ SEARCH_INDEX_SCHEMA_VERSION,
219
+ SEARCH_SHARD_ENCODING,
220
+ SEARCH_SHARD_SCHEMA_VERSION,
221
+ SOURCE_INDEXER_VERSION,
222
+ TOKENIZER_VERSION,
223
+ TRANSACTION_SCHEMA_VERSION,
224
+ SARIF_VERSION,
225
+ } from "llmnav";
226
+ ```
227
+
228
+ `diagnosticsToSarif(diagnostics)` maps existing diagnostics to a deterministic SARIF 2.1.0 object without discovering or mutating diagnostics.
229
+
230
+ `diagnosticsToEditor(diagnostics)` groups repository-relative diagnostics into schemaVersion 1 documents with zero-based ranges and stable severity mappings. `renderEditorDiagnostics` serializes the report deterministically. `getEditorIntegration("vscode")` returns a VS Code task and custom problem matcher without modifying editor files.
231
+
232
+ `buildSearchShards(index, searchIndex, shardSize)` returns a deterministic manifest and map of card-range shard contents. A size of zero or a card count at or below the limit returns no shard artifacts. The function slices compact postings and documents rather than rebuilding search documents.
233
+
234
+ `normalizeGraphInput(value, file, contentHash)` validates and normalizes one schemaVersion 1 definition/reference index. `loadGraphInputs(root, config)` reads configured repository-local inputs and returns normalized indexes plus deterministic `LNV014` diagnostics without executing project code.
235
+
236
+ `buildRepositoryGraph(project, index)` returns the schemaVersion 1 qualified node and edge graph. `renderRepositoryGraph(graph)` serializes it deterministically. Successful generation also exposes the graph as `result.graph` and writes it to `.llmnav/cache/graph.json`.
237
+
238
+ `buildRepositoryGraphIncremental(project, index, previousState)` returns `{ graph, state, stats }`. Compatible content-addressed partitions are reused; incompatible state is ignored. `compatibleGraphState` validates the disposable state boundary, and `renderGraphState` serializes `.llmnav/cache/graph-state.json`. `result.incremental.graph` reports total, reused, rebuilt, and removed partition counts.
239
+
240
+ `queryPreparedIndex` accepts an optional `graph`. Graph bonuses preserve lexical seeds and scale by edge confidence and direction. `buildContext` accepts `maxEdges` in addition to `depth` and `budget`, and returns the stable IDs of packed edges as `includedEdges`.
241
+
242
+ `resolveGraphNode(graph, id, localRepositoryId)` returns `resolved`, `ambiguous`, or `missing`. `renderGraphNode(node)` emits compact external definition context. `showProjectCard` returns either a local `card` or an external graph `node`.
243
+
244
+ The normative source vocabulary remains available from `llmnav/spec`.
245
+
246
+ ```js
247
+ import { KEY_ORDER, EFFECT_KINDS, RISK_KINDS } from "llmnav/spec";
248
+ ```
249
+
250
+ ## Agent operation protocol
251
+
252
+ `getAgentToolDefinitions()` returns defensive copies of four schemaVersion 1 tool definitions in fixed order. Their JSON Schema inputs reject unknown fields and deliberately omit the repository root.
253
+
254
+ ```js
255
+ import { executeAgentOperation, getAgentToolDefinitions } from "llmnav";
256
+
257
+ const tools = getAgentToolDefinitions();
258
+ const result = await executeAgentOperation(
259
+ process.cwd(),
260
+ "llmnav_query",
261
+ { task: "rotate a replayed refresh token", top: 5 },
262
+ );
263
+ ```
264
+
265
+ The trusted wrapper binds `root`; the model supplies only the validated operation input. Results use one schemaVersion 1 envelope containing `operation`, `ok`, `data`, and `error`. Input errors use `LNVAP002`, missing IDs use `LNVAP404`, and unexpected operation failures use `LNVAP500`.
266
+
267
+ Long-lived hosts can load one explicit snapshot for repeated navigation calls:
268
+
269
+ ```js
270
+ import { createProjectSession, executeAgentOperation } from "llmnav";
271
+
272
+ const session = await createProjectSession(process.cwd());
273
+ const result = await executeAgentOperation(
274
+ process.cwd(),
275
+ "llmnav_query",
276
+ { task: "rotate a replayed refresh token" },
277
+ { session },
278
+ );
279
+ await session.refresh(); // after generation or checkout changes
280
+ ```
281
+
282
+ `query`, `show`, and `context` reuse the loaded index, postings, graph, lexicon, and registry. `refresh()` replaces the complete snapshot; it never mutates one layer in place. The `check` operation still scans current source and does not use session data.
283
+
284
+ The typed `llmnav/examples/provider-neutral-host.mjs` export composes these APIs into a trusted-root closure. It exposes tool definitions, base and module-selected prompt partitions, one snapshot-backed operation executor, and an explicit refresh method without importing a model SDK.
285
+
286
+ `buildPromptPrefixBundle(input)` constructs ordered package, repository, and module partitions with normalized newlines, SHA-256 content hashes, estimated token counts, and explicit cache-boundary hints. `renderPromptPrefixBundle` serializes it deterministically. `loadPromptPrefixBundle(root)` accepts only a schema-compatible artifact whose exact bytes match `manifest.json`.
287
+
288
+ ## Compatibility boundary
289
+
290
+ The public API follows package semantic versioning. `index.json` schemaVersion 1 and `llmnav/1` source syntax remain compatible. Contract fingerprints are optional additive index fields. `search-index.json`, `file-state.json`, `graph-state.json`, transaction journals, and performance metrics retain their own schema or implementation versions.
@@ -0,0 +1,286 @@
1
+ # Architecture and cache design
2
+
3
+ ## Design target
4
+
5
+ LLMNav minimizes repository text an agent must inspect before reaching the correct declaration. It does this without turning comments into a stale copy of the program and without rebuilding every derived structure on every command.
6
+
7
+ ```text
8
+ stable source cards
9
+ +
10
+ generated declaration and import structure
11
+ +
12
+ repository aliases and ID registry
13
+
14
+ file-level parsed state
15
+
16
+ compatible card index + deterministic inverted index + compact catalogs
17
+
18
+ query → show → bounded context → selected source bodies
19
+ ```
20
+
21
+ ## Three information layers
22
+
23
+ | Layer | Contents | Persistence |
24
+ | --- | --- | --- |
25
+ | Stable meaning | ID, role, search phrases, ownership, invariants, effects, risks, semantic relations, stability | source comments |
26
+ | Generated structure | path, declaration, language, visibility, boundaries, imports, semantic/structure/body hashes | `.llmnav/cache` |
27
+ | Volatile execution hints | file size, mtime, ctime used only to avoid reads | `.llmnav/state`, ignored |
28
+
29
+ Generated paths, signatures, callers, references, timestamps, and hashes are never written into source comments. The generated layer may be discarded and rebuilt without changing source semantics.
30
+
31
+ ## Compatible primary index
32
+
33
+ `.llmnav/cache/index.json` remains the primary interoperability artifact.
34
+
35
+ ```json
36
+ {
37
+ "schemaVersion": 1,
38
+ "specVersion": "1",
39
+ "generatedBy": "llmnav@0.5.1",
40
+ "repositoryId": "example",
41
+ "contractFingerprints": {
42
+ "schemaVersion": 1,
43
+ "exportedApi": { "count": 0, "sha256": "..." },
44
+ "configuration": { "sha256": "..." }
45
+ },
46
+ "sourceHash": "...",
47
+ "cards": []
48
+ }
49
+ ```
50
+
51
+ The primary index preserves schemaVersion 1 and the v0.1 card fields. Contract fingerprints remain optional additive fields, so earlier consumers can continue reading the existing fields. Graph, graph-state, prompt-bundle, agent-operation, and editor-diagnostic artifacts have independent schema versions.
52
+
53
+ The exported API fingerprint covers annotated declarations that are public according to deterministic language conventions. The configuration fingerprint covers the effective validated configuration except the local `$schema` path. Body-only edits do not change either fingerprint. Generation emits warning `LNV009` when an existing fingerprint changes, but the warning does not block an intentional regeneration.
54
+
55
+ Each card carries three independent hashes.
56
+
57
+ | Hash | Input | Typical invalidation |
58
+ | --- | --- | --- |
59
+ | `semantic` | canonical stable card fields | role, invariant, effect, risk, semantic relation change |
60
+ | `structure` | path, scope, declaration, signature, imports | move, rename, signature or import change |
61
+ | `body` | attached declaration bytes for symbol cards; containing file bytes otherwise | body change inside the selected declaration or containing file |
62
+
63
+ ## Local structure enrichment
64
+
65
+ Symbol attachment records the detected language, public/exported status, visibility, Go receiver when present, declaration span, and a declaration-level body hash. TypeScript and Go use dedicated deterministic declaration patterns; JavaScript, Rust, Python, and generic C-like declarations retain the compatible fallback patterns.
66
+
67
+ Generated cards may contain a sorted `boundaries` array. LLMNav detects `route`, `event`, `schema`, `migration`, and `command` boundaries from repository-relative paths and controlled semantic effects or risks. Each record includes `confidence` and explicit evidence such as `path`, `effect`, or `risk`. These hints are generated navigation data and are never copied into source comments.
68
+
69
+ Generation compares the previous and current primary indexes to emit `affectedBoundaries`. The report preserves the card change and hash dimensions while adding module IDs, structural boundary records, outbound relation targets, and reverse semantic dependents. It is returned through the API and `generate --json`; it is not stored in source cards.
70
+
71
+ ## Deterministic inverted index
72
+
73
+ `.llmnav/cache/search-index.json` schemaVersion 2 with `compact-v1` encoding stores a sorted card-ID table, normalized phrase documents, a sorted token dictionary, and posting lists.
74
+
75
+ ### Optional search shards
76
+
77
+ When `generation.searchShardSize` is positive and the repository exceeds that card count, generation emits `search-shards.json` schemaVersion 1 with `card-range-v1` encoding plus zero-padded shard files below `search-shards/`. Each manifest record contains the first and last semantic ID, card count, card-set hash, and artifact SHA-256.
78
+
79
+ Shards slice the already-built compact documents and postings, adjust local card ordinals, and never retokenize cards. Every shard independently satisfies the compact search-index compatibility checks for its card subset. The full `search-index.json` remains authoritative for built-in queries, preserving global document frequency and v0.2 ranking behavior. Cache transactions publish or remove the complete shard set atomically.
80
+
81
+ ## Prompt-prefix bundle
82
+
83
+ `.llmnav/cache/prompt-prefix.json` schemaVersion 1 packages the exact stable prefix material without applying any provider-specific cache API. Its ordered partitions are:
84
+
85
+ 1. provider-neutral tool definitions with package cache scope
86
+ 2. the managed agent protocol with package cache scope
87
+ 3. the repository core catalog with repository cache scope
88
+ 4. module catalogs in semantic-ID order with module cache scope
89
+
90
+ Every partition contains normalized content, SHA-256 content identity, an estimated token count, and an explicit boundary-after hint. The bundle records base and selectable module IDs separately and states that volatile context belongs after them. Its aggregate hash covers ordered partition IDs and content hashes.
91
+
92
+ The artifact contains no timestamp, branch, diff, user task, absolute path, or provider setting. It is covered by `manifest.json` and published in the same cache transaction as its source catalogs. A consumer must preserve base order, choose only relevant module partitions, and add volatile context afterward.
93
+
94
+ ## Repository graph
95
+
96
+ `.llmnav/cache/graph.json` schemaVersion 1 is a deterministic derived artifact. It combines source-card semantic relations, resolvable relative imports, and configured generated definition/reference indexes. Nodes and edges use qualified repository keys. Edges retain confidence and provenance rather than flattening explicit and inferred evidence into one unqualified relation.
97
+
98
+ Unresolved targets are preserved as placeholder nodes. This allows later workspace resolution without inventing a local definition or silently dropping an imported reference. `manifest.json` covers the graph bytes and cache transactions publish it atomically with the primary and search indexes.
99
+
100
+ Search treats lexical results as seeds and applies only a confidence-scaled one-hop graph bonus. Context traversal is breadth-first and bounded by depth, token budget, and edge count. Invalid or manifest-mismatched graph data is not used for ranking; search falls back to the compatible lexical and source-relation behavior.
101
+
102
+ The graph is also the explicit workspace resolution surface. Qualified IDs resolve by exact node key. Unqualified IDs prefer the local repository and resolve externally only when the semantic ID is unique across imported repositories. No directory discovery or network lookup occurs during resolution.
103
+
104
+ `.llmnav/cache/graph-state.json` is a disposable schemaVersion 1 acceleration artifact. Content-addressed partitions isolate local cards and imported indexes. Partition keys and hashes include every graph-relevant dimension, and local path resolution changes invalidate all affected local-import decisions. Incompatible or malformed state is never partially trusted: generation rebuilds it from the current primary index and validated graph inputs.
105
+
106
+ Graph state contains no timestamps, absolute paths, or filesystem identity. Incremental and forced-full builds must produce byte-identical graph and state bytes, and the transaction publishes both with the manifest.
107
+
108
+ Each card entry stores a hash of its searchable fields and its normalized phrase fields. The global token dictionary is sorted once. Every dictionary entry points to a posting list encoded as sorted card ordinals and sparse field-frequency vectors. The ordinals resolve through the sorted `cardIds` table.
109
+
110
+ Field order and field weights are versioned constants. Tokens, card IDs, object keys, and posting entries use locale-independent UTF-16 lexical comparison.
111
+
112
+ The complete search index is deterministic across card traversal order. It contains no timestamp, absolute path, random ID, platform separator, or filesystem metadata.
113
+
114
+ During a query, LLMNav tokenizes only the task text. It reads posting lists for those tokens and does not tokenize every card. If the generated search index is missing, malformed, or incompatible with the primary index, LLMNav builds a compatible in-memory index from `index.json` instead of refusing v0.1 repositories.
115
+
116
+ ## File-level incremental indexing
117
+
118
+ `.llmnav/cache/file-state.json` contains deterministic parsed state for each source file:
119
+
120
+ * repository-relative POSIX path
121
+ * SHA-256 content hash
122
+ * source and semantic byte counts
123
+ * generated import strings
124
+ * parsed LLMNav blocks
125
+ * attached declaration records
126
+
127
+ It contains no mtime, ctime, inode, device ID, absolute path, or OS-specific separator.
128
+
129
+ `.llmnav/state/stat-hints.json` is a volatile optimization. It records file size, mtime nanoseconds, and ctime nanoseconds. It is ignored by Git and never enters deterministic manifests.
130
+
131
+ Generation classifies each file as follows.
132
+
133
+ ```text
134
+ matching stat hint
135
+ → reuse file-state entry without reading source
136
+
137
+ changed or missing stat hint, matching content hash
138
+ → read source once, reuse parsed file-state entry
139
+
140
+ changed content hash
141
+ → parse source and rebuild only that file's records
142
+
143
+ missing current file
144
+ → delete its records
145
+ ```
146
+
147
+ A source file can contain multiple cards. Parsing is file-granular because comment syntax and declaration attachment require surrounding file text. Search-document rebuilding is card-granular because each card has an independent search hash.
148
+
149
+ ## Card-level inverted-index updates
150
+
151
+ The previous `search-index.json` is usable only when its schema, tokenizer version, field order, and repository ID match.
152
+
153
+ For every current card:
154
+
155
+ ```text
156
+ same search-document hash
157
+ → reuse stored document and posting entries
158
+
159
+ different hash
160
+ → remove old posting entries, tokenize the card once, add new entries
161
+
162
+ removed card
163
+ → remove its document and posting entries
164
+ ```
165
+
166
+ The incremental result is serialized from sorted maps and must be byte-identical to a full rebuild. Regression tests compare the complete object and generated bytes after add, modify, and remove operations.
167
+
168
+ ## Stable catalog order
169
+
170
+ Alphabetically re-sorting a large catalog inserts new cards into the middle and shifts every later prompt token. LLMNav records first-seen semantic IDs in `.llmnav/order.lock`.
171
+
172
+ Normal generation preserves existing lines and appends new IDs in deterministic order. A deliberate full reorder is a cache-epoch change and should be reviewed separately from feature work.
173
+
174
+ ## Catalog tiers
175
+
176
+ `repo-core.txt` contains cards whose stability matches `repositoryCatalogStabilities`, which defaults to `architecture`.
177
+
178
+ `modules/<id>.txt` contains cards grouped by configured semantic-ID depth. The default module depth is two and includes `architecture` plus `contract` cards.
179
+
180
+ `index.json`, `cards.jsonl`, `search-index.json`, and `file-state.json` retain every card, including `implementation` cards.
181
+
182
+ Repository and module catalogs contain semantic fields only. Current paths, lines, declarations, imports, and signatures remain in generated indexes and dynamic query output. A body edit, symbol rename, or file move therefore does not invalidate a semantic catalog whose meaning stayed intact.
183
+
184
+ ## Transactional cache generation
185
+
186
+ Generation never mutates the live cache file by file. It builds a complete replacement under `.llmnav/.transactions/<id>/stage`.
187
+
188
+ The complete source scan and cache commit are serialized by `.llmnav/generation.lock`. The lock records a process and opaque owner ID. A reader waits for a live owner to finish and only then evaluates recovery, while an abandoned lock from a dead process can be removed without granting another process authority over an active journal.
189
+
190
+ ```text
191
+ write every staged artifact
192
+ verify exact staged bytes
193
+ verify manifest hashes and primary index schema
194
+ write transaction journal: prepared
195
+ rename live cache to backup
196
+ write journal: old-moved
197
+ rename stage to live cache
198
+ write journal: new-installed
199
+ verify installed cache
200
+ write journal: committed
201
+ remove backup, transaction directory, and journal
202
+ ```
203
+
204
+ The journal is `.llmnav/generation-transaction.json`. It records the writer owner ID, and transaction paths are repository-relative and validated to remain below `.llmnav/.transactions`.
205
+
206
+ When generation discovers new semantic IDs, the staged transaction also owns `.llmnav/ids.jsonl` and `.llmnav/order.lock`. Their previous bytes are backed up beside the cache, their replacements are verified by content hash, and recovery restores or finalizes all three surfaces together.
207
+
208
+ If generation throws before commit, rollback restores the backup. If the process is killed, the next `query`, `generate`, or `doctor` recovers from the journal before reading cache data.
209
+
210
+ An installed cache is considered authoritative only after the journal reaches `committed`. A crash after installing a new cache but before commit restores the previous cache. A crash after recording `committed` keeps the new verified cache and completes cleanup.
211
+
212
+ ## Windows rename behavior
213
+
214
+ Directory replacement is expressed as two renames rather than relying on replacing a non-empty destination directory. Transient rename and removal errors with these codes are retried with bounded delays:
215
+
216
+ ```text
217
+ EACCES
218
+ EBUSY
219
+ EEXIST
220
+ ENOTEMPTY
221
+ EPERM
222
+ ```
223
+
224
+ The failure-injection suite covers process exits after moving the old cache and after installing an uncommitted cache. The CI matrix executes the same suite on `windows-latest` and Ubuntu with Node.js 22 and 24.
225
+
226
+ ## Machine-readable impact records
227
+
228
+ `generate --json` compares the previous and current primary indexes.
229
+
230
+ A changed-card record reports:
231
+
232
+ * stable ID
233
+ * `added`, `modified`, or `removed`
234
+ * changed hash dimensions: `semantic`, `structure`, `body`
235
+ * compact previous and current snapshots
236
+
237
+ An affected-catalog record reports repository, module, or agent-context artifacts whose bytes changed. Output arrays and object serialization are deterministic, making them safe for CI, agent orchestration, and release tooling.
238
+
239
+ ## Search pipeline
240
+
241
+ The v0.2 local ranker performs:
242
+
243
+ 1. Unicode NFKC normalization.
244
+ 2. Exact semantic ID and ID-substring matching.
245
+ 3. Alias resolution from `.llmnav/lexicon.json`.
246
+ 4. Query tokenization with word tokens and CJK bigram/trigram expansion.
247
+ 5. Field-weighted posting-list scoring with inverse document frequency.
248
+ 6. Exact phrase bonuses from pre-normalized card phrases.
249
+ 7. One-hop semantic relation expansion from the highest-ranked cards.
250
+ 8. Deterministic score and semantic-ID sorting.
251
+
252
+ The score model remains compatible with v0.1. Large synthetic tests compare complete ranked ID and score sequences against the v0.1-compatible legacy implementation.
253
+
254
+ ## Prompt-cache placement
255
+
256
+ A model-provider harness can use this stable order:
257
+
258
+ ```text
259
+ stable tool definitions
260
+ stable agent protocol
261
+ .llmnav/cache/repo-core.txt
262
+ cache breakpoint
263
+ selected .llmnav/cache/modules/<module>.txt
264
+ cache breakpoint
265
+ user task
266
+ branch and diff state
267
+ query results
268
+ selected source bodies
269
+ test and tool output
270
+ ```
271
+
272
+ The CLI emits deterministic semantic catalogs but does not call a model API or force provider-specific cache controls.
273
+
274
+ ## Declaration attachment
275
+
276
+ The local parser recognizes common declarations in TypeScript, JavaScript, Go, Rust, Python, and a generic class/function profile. It skips ordinary documentation comments, decorators, and attributes after a card.
277
+
278
+ This parser is a conservative heuristic rather than a complete language AST. A card that cannot attach is an error instead of being silently indexed against the wrong code. Generated language-aware enrichers may improve structure in later releases but must not write derived data into source cards.
279
+
280
+ ## Security model
281
+
282
+ LLMNav reads repository source and writes only `.llmnav`, selected managed instruction files during explicit initialization, and source comments during explicit formatting.
283
+
284
+ It performs no network requests, executes no repository code, loads no plugins, and has no install script. Configured source, evaluation, and cache paths must remain relative to the repository. Existing symbolic links in control and agent-instruction paths are rejected before writes.
285
+
286
+ The local index is not a security boundary. It may contain source paths, current signatures, imports, and semantic descriptions. Apply the same access controls to `.llmnav/cache` as to the source repository.