llmnav 0.5.1

package/docs/publishing.md

@@ -0,0 +1,86 @@
+# Publishing `llmnav` to npm
+
+The repository is release-ready except for owner-specific metadata and npm account configuration.
+
+## 1. Claim the names
+
+Create the GitHub repository named `llmnav`, then check the npm registry immediately before the first release:
+
+```sh
+npm view llmnav
+```
+
+An `E404` means no public package is visible through the registry you queried at that moment. It is not a reservation. The name remains claimable by someone else until publication succeeds.
+
+## 2. Replace release metadata
+
+Update only the owner-specific URLs in `package.json` and `.github/ISSUE_TEMPLATE/config.yml`.
+Do not run a repository-wide replacement: the release checker and doctor intentionally keep the literal `OWNER` sentinel in source code.
+
+```sh
+npm pkg set repository.url="git+https://github.com/YOUR_GITHUB_OWNER/llmnav.git"
+npm pkg set bugs.url="https://github.com/YOUR_GITHUB_OWNER/llmnav/issues"
+npm pkg set homepage="https://github.com/YOUR_GITHUB_OWNER/llmnav#readme"
+```
+
+Then replace `OWNER` only in `.github/ISSUE_TEMPLATE/config.yml` and run:
+
+```sh
+npm run release:check
+```
+
+## 3. Configure npm authentication
+
+The supplied release workflow uses npm provenance and GitHub's OIDC token. In npm package settings, configure the GitHub repository and `.github/workflows/release.yml` as a trusted publisher.
+
+The workflow references a GitHub environment named `npm`. Create that environment for release protection, or remove the `environment` line when no environment gate is desired.
+
+A classic or granular access token can be used for a manual first publication. Do not commit `.npmrc` credentials or an npm token.
+
+## 4. Validate the exact package payload
+
+```sh
+npm ci
+npm run check
+npm run release:check
+npm run smoke:pack
+npm pack --dry-run
+```
+
+Inspect the tarball list. The package intentionally includes the CLI, source API, type declarations, schema, templates, documentation, the typed provider-neutral host example, README, changelog, roadmap, and license. Tests, benchmark harnesses, and development scripts remain in GitHub but are not installed into consumer projects. `npm run smoke:pack` verifies the exact tarball and example export in a clean temporary project.
+
+## 5. Publish the first release
+
+Manual publication:
+
+```sh
+npm login
+npm publish --provenance --access public
+```
+
+Automated publication:
+
+```sh
+git tag v0.5.1
+git push origin v0.5.1
+```
+
+The release workflow rejects a tag that does not match `package.json`.
+
+## 6. Verify from a clean directory
+
+```sh
+mkdir llmnav-smoke
+cd llmnav-smoke
+npm init -y
+npm install --save-dev llmnav
+npx llmnav --version
+npx llmnav init --agents all --package-scripts
+npx llmnav doctor
+```
+
+## Versioning policy
+
+The npm package follows semantic versioning. The comment protocol is versioned separately in the header.
+
+A CLI or API breaking change increments the npm major version. A breaking source syntax change creates a new header such as `llmnav/2`; existing `llmnav/1` parsing should remain available through an explicit migration window.

package/docs/quickstart.md

@@ -0,0 +1,139 @@
+# Quick start
+
+## Install and initialize
+
+```sh
+npm install --save-dev llmnav
+npx llmnav init --agents all --package-scripts
+```
+
+This creates the `.llmnav` control directory, a JSON schema, the semantic ID registry, the stable order lock, deterministic generated indexes and catalogs, volatile state ignore rules, and managed instruction files for coding agents.
+
+Review `.llmnav/config.json` before annotating code. The default scans the repository root while excluding build outputs, dependency directories, generated bundles, and common caches.
+
+Source roots and evaluation files must remain inside the repository. The generated cache directory must remain below `.llmnav/`; parent traversal and symbolic-link control directories are rejected.
+
+## Annotate a module boundary
+
+Choose a boundary that an agent is likely to search for by behavior rather than by current symbol name.
+
+```ts
+/* llmnav/1 module
+id=auth.session
+role=Own refresh-token issuance, rotation, replay detection, and revocation.
+owns=refresh-token family|session revocation
+excludes=access-token signing|user profile storage
+search=session lifecycle|token family|session revocation
+invariant=One token family has at most one live refresh token.
+stability=architecture
+*/
+```
+
+Do not begin by annotating every function. One clear module card is more valuable than dozens of generic helper cards.
+
+## Annotate a behavioral boundary
+
+```ts
+/* llmnav/1 symbol
+id=auth.session.rotate
+role=Rotate one refresh-token family atomically and reject replayed tokens.
+search=refresh token|token rotation|token family|replay detection
+invariant=At most one live refresh token exists per family.
+invariant=Replay revokes the entire token family.
+effect=db.write(session_tokens)|event.emit(auth.session.revoked)
+risk=auth|concurrency
+rel=policy>auth.session.lifecycle
+rel=test>auth.session.rotate.contract
+stability=contract
+*/
+export async function rotateSession() {}
+```
+
+The referenced policy and test IDs must exist as source cards or registry records.
+
+## Format and validate
+
+```sh
+npx llmnav format
+npx llmnav check
+```
+
+`check` exits with status 1 for errors. Warnings do not fail the command.
+
+GitHub annotation output is available for CI:
+
+```sh
+npx llmnav check --format github
+```
+
+## Generate deterministic indexes and catalogs
+
+```sh
+npx llmnav generate
+```
+
+The first run parses every source file and builds `index.json`, `search-index.json`, `file-state.json`, semantic catalogs, and a manifest. Later runs reuse unchanged file parses and card search documents.
+
+Commit `.llmnav/cache`, `.llmnav/ids.jsonl`, and `.llmnav/order.lock`. Do not commit `.llmnav/state` or transaction work files.
+
+Inspect machine-readable impact data when integrating with CI or an agent:
+
+```sh
+npx llmnav generate --json
+```
+
+`changedCards` distinguishes semantic, structure, and body changes. `affectedCatalogs` identifies only repository, module, and agent-context catalogs whose bytes changed.
+
+CI should run:
+
+```sh
+npx llmnav generate --full --check
+```
+
+That command fails when generated files, the registry, or the stable order lock differ from current source. It does not replace the live cache. Normal generation stages and verifies a complete replacement before committing it.
+
+## Search before opening source
+
+```sh
+npx llmnav query "where do replayed refresh tokens revoke their family" --top 5
+npx llmnav show auth.session.rotate
+npx llmnav context auth.session.rotate --depth 1 --budget 2500
+```
+
+Queries use the persistent inverted index and do not retokenize every card. If a previous generation was interrupted, the query command restores the last committed cache before reading it.
+
+The expected agent workflow is:
+
+1. Query from the task language.
+2. Inspect a few cards and generated signatures.
+3. Open the selected declarations.
+4. Expand one semantic hop only when policy, test, fallback, or workflow context is needed.
+5. Use broad grep only after LLMNav fails to return a credible candidate.
+
+## Add task-language aliases
+
+```json
+{
+  "version": 1,
+  "aliases": {
+    "session renewal": "auth.session.rotate",
+    "token replay attack": "auth.session.rotate",
+    "credit reservation": "billing.credit.reserve"
+  }
+}
+```
+
+Aliases are evaluated before lexical ranking and receive a large deterministic score boost.
+
+## Add search regression cases
+
+```jsonl
+{"query":"revoke every session in the family when a refresh token is replayed","expected":["auth.session.rotate","auth.session.revoke-family"]}
+{"query":"reserve credits before starting an external generation job","expected":["billing.credit.reserve"]}
+```
+
+```sh
+npx llmnav eval
+```
+
+Do not lower the gates simply because a new card displaced an old result. Fix ambiguous roles, overloaded search phrases, aliases, or missing semantic relations first.

package/docs/research.md

@@ -0,0 +1,31 @@
+# Research basis
+
+LLMNav is an engineering proposal built from a recurring result in repository-level code intelligence: retrieval quality depends more on selecting the right structure and context than on sending more source text to a model.
+
+## Repository-level retrieval
+
+[RepoCoder](https://arxiv.org/abs/2303.12570) uses an iterative retrieval and generation loop instead of treating repository context as one static prompt. That supports LLMNav's `query → show → context → source` workflow.
+
+[Repoformer](https://arxiv.org/abs/2403.10059) studies selective retrieval and reports that retrieval can be unnecessary or harmful for some completions. LLMNav therefore keeps the first result set small and does not force a repository dump into every request.
+
+[GraphCoder](https://arxiv.org/abs/2406.07003) combines code context with graph structure. [CodexGraph](https://arxiv.org/abs/2408.03910) exposes repository graphs to agents. These works support the separation between hand-written semantic cards and generated structural edges.
+
+[Aider's repository map](https://aider.chat/docs/repomap.html) is a practical example of presenting selected declarations and signatures under a token budget rather than copying every file.
+
+## What the research does not prove
+
+None of these projects validates the exact `llmnav/1` syntax, field weights, byte limits, or CI thresholds in this repository. Those are testable design choices, not established constants.
+
+LLMNav also makes no universal claim that comments increase model accuracy. Unbounded comments can add stale facts, duplicate code, and distort lexical retrieval. The protocol deliberately limits cards to scarce semantic facts and measures retrieval with repository-specific queries.
+
+## Falsifiable claims
+
+A useful LLMNav installation should demonstrate all of the following against its own baseline:
+
+1. The correct semantic ID appears more often in the first five results.
+2. Agents open fewer irrelevant files before reaching the target declaration.
+3. Uncached input tokens decrease when stable catalogs are reused.
+4. Final task success does not regress.
+5. Stale source metadata remains at zero because volatile structure is generated.
+
+Use [benchmarking.md](benchmarking.md) to measure those claims. Do not advertise token or latency reductions measured on another repository as expected results for yours.

package/docs/spec.md

@@ -0,0 +1,424 @@
+# LLMNav/1 specification
+
+Status: experimental normative specification
+
+Package implementation: `llmnav` 0.2.x
+
+## Purpose
+
+LLMNav/1 defines a compact semantic card embedded next to selected code boundaries. A conforming implementation extracts cards, validates stable meaning, generates volatile structure, and exposes the result to coding agents without requiring broad repository reads.
+
+The source card is not a complete documentation record. It contains only information that is expensive to infer from code and expected to remain valid across ordinary implementation changes.
+
+The terms MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are normative.
+
+## Encoding and line model
+
+A card MUST be UTF-8 text.
+
+Each metadata line MUST contain one `key=value` pair. The first `=` separates the key and value. Multiline values are not supported.
+
+Tabs are not meaningful. A formatter MUST emit spaces and deterministic line endings matching the containing source file.
+
+Unknown keys are errors. Implementations MUST NOT silently index them, and formatters MUST NOT silently delete them.
+
+## Headers and scopes
+
+Every card starts with:
+
+```text
+llmnav/1 <scope>
+```
+
+The valid scopes are `file`, `module`, and `symbol`.
+
+A `file` card describes the responsibility of one source file. It SHOULD appear before imports after any shebang, license notice, or language-required package declaration.
+
+A `module` card describes a domain, package, namespace, or architectural boundary. It SHOULD be used when one responsibility spans multiple declarations or files.
+
+A `symbol` card attaches to the next declaration after whitespace, ordinary documentation comments, decorators, or attributes. A conforming checker MUST report a card that cannot be attached to a declaration.
+
+## Comment encodings
+
+### Block comments
+
+```text
+/* llmnav/1 symbol
+id=domain.capability.action
+role=Produce one observable result.
+stability=contract
+*/
+```
+
+### HTML comments
+
+```text
+<!-- llmnav/1 file
+id=ui.checkout.page
+role=Render the checkout workflow and submit one payment confirmation.
+stability=contract
+-->
+```
+
+### Line comments
+
+Line-comment cards MUST use one prefix consistently and MUST end with an explicit `/llmnav` line.
+
+```text
+# llmnav/1 symbol
+# id=domain.capability.action
+# role=Produce one observable result.
+# stability=contract
+# /llmnav
+```
+
+The supported prefixes in the reference implementation are `//`, `#`, and `--`.
+
+## Canonical key order
+
+A formatter MUST emit keys in this order:
+
+```text
+id
+role
+owns
+excludes
+search
+invariant
+effect
+risk
+rel
+stability
+```
+
+Stable key order is part of the format. It makes generated catalogs deterministic and avoids semantically meaningless prompt-prefix changes.
+
+## Field cardinality
+
+| Key | Cardinality | Value form |
+| --- | --- | --- |
+| `id` | exactly one | scalar |
+| `role` | exactly one | scalar sentence |
+| `owns` | zero or one line | pipe-separated list |
+| `excludes` | zero or one line | pipe-separated list |
+| `search` | zero or one line | pipe-separated list |
+| `invariant` | repeated, zero to four | one assertion per line |
+| `effect` | zero or one line | pipe-separated controlled values |
+| `risk` | zero or one line | pipe-separated controlled values |
+| `rel` | repeated, zero to six | one relation per line |
+| `stability` | exactly one | controlled scalar |
+
+Empty values are invalid in every field. Empty items inside a pipe-separated list are invalid. Optional fields MUST be omitted when they have no value. Duplicate values in list or repeatable fields are invalid after Unicode normalization and case folding.
+
+## `id`
+
+`id` is a durable semantic identity. It MUST describe a capability, contract, policy, or workflow rather than a current file or symbol name.
+
+The grammar is:
+
+```text
+id = segment "." segment *("." segment)
+segment = lowercase-letter *(lowercase-letter / digit / "-")
+```
+
+The reference validation expression is:
+
+```regex
+^[a-z][a-z0-9]*(?:\.[a-z][a-z0-9-]*){1,5}$
+```
+
+Examples:
+
+```text
+auth.session.rotate
+billing.credit.reserve
+privacy.export.prepare
+game.arena.collapse-sequence
+```
+
+An ID MUST remain unchanged when a file moves, a declaration is renamed, or an implementation is replaced without changing the semantic capability.
+
+Deleted IDs MUST NOT be reused. `.llmnav/ids.jsonl` records active, redirected, replaced, or retired identities.
+
+```jsonl
+{"id":"auth.session.rotate","state":"active"}
+{"id":"auth.session.renew","state":"redirect","to":"auth.session.rotate"}
+{"id":"billing.credit.charge","state":"replaced","by":["billing.credit.reserve","billing.credit.capture"]}
+```
+
+## `role`
+
+`role` states the observable result produced by the file, module, or symbol.
+
+It MUST NOT merely restate a name or use an empty abstraction such as “handle data”, “manage sessions”, “service utility”, or “process logic”.
+
+It MUST be no longer than 180 characters in the reference profile.
+
+Good:
+
+```text
+role=Reserve user credits before an external generation job starts.
+```
+
+Bad:
+
+```text
+role=Handle billing data.
+```
+
+## `owns` and `excludes`
+
+`owns` lists responsibilities for which the card is authoritative.
+
+`excludes` names adjacent responsibilities that appear related but belong elsewhere. It reduces false-positive routing in repositories with dense domain vocabulary.
+
+```text
+owns=refresh-token family|session revocation
+excludes=access-token signing|user profile storage
+```
+
+These fields SHOULD appear primarily on `file` and `module` cards.
+
+## `search`
+
+`search` bridges task language and code language. It contains phrases that a developer or product owner is likely to use but that may not appear in current identifiers.
+
+When present, the reference profile requires two to six phrases.
+
+```text
+search=refresh token|token rotation|token family|replay detection
+```
+
+Generic phrases such as `service`, `manager`, `handler`, `helper`, `utility`, `data`, `process`, and `logic` are forbidden by default.
+
+A repository SHOULD use one source-card language. Translations, abbreviations, product terminology, and historical names SHOULD be stored in `.llmnav/lexicon.json` rather than duplicated across source comments.
+
+## `invariant`
+
+Each `invariant` is a condition whose violation represents a bug, security failure, accounting failure, or broken contract.
+
+```text
+invariant=At most one live refresh token exists per family.
+invariant=Replay revokes the entire token family.
+```
+
+Descriptions, goals, implementation notes, and temporary assumptions MUST NOT be presented as invariants.
+
+The reference profile permits at most four invariants per card. More usually indicates that the annotated boundary is too broad or that policy belongs in a separate card.
+
+## `effect`
+
+`effect` uses a controlled vocabulary for externally observable or nondeterministic behavior.
+
+The base vocabulary is:
+
+```text
+db.read(name)
+db.write(name)
+cache.read(name)
+cache.write(name)
+event.emit(name)
+event.consume(name)
+net.call(name)
+fs.read
+fs.write
+process.spawn
+clock.read
+random.read
+lock.acquire(name)
+cookie.write(name)
+auth.check(name)
+```
+
+The base vocabulary has fixed argument arity. `db.*`, `cache.*`, `event.*`, `net.call`, `lock.acquire`, `cookie.write`, and `auth.check` require one stable target argument. `fs.read`, `fs.write`, `process.spawn`, `clock.read`, and `random.read` accept no argument. Resource arguments identify a semantic resource rather than a path, host, timestamp, or current implementation detail.
+
+A repository MAY extend the vocabulary in `.llmnav/config.json`. An extension declares only a lower-case effect kind such as `queue.publish`; the reference implementation accepts that custom kind with or without one target argument. Extensions MUST remain machine-parseable and SHOULD use a stable namespace.
+
+Effects do not replace static analysis. They express semantic importance and provide retrieval signals while future enrichers compare declarations against actual sinks.
+
+## `risk`
+
+The base risk vocabulary is:
+
+```text
+auth
+money
+privacy
+concurrency
+migration
+availability
+performance
+```
+
+Cards containing `auth`, `money`, or `privacy` MUST contain at least one invariant and at least one `test>` relation under the reference profile.
+
+A repository MAY add controlled risk values in configuration.
+
+## `rel`
+
+`rel` records a semantic relation that cannot be reliably derived from imports or call syntax.
+
+The grammar is:
+
+```text
+rel=<type>><target-id>
+```
+
+The base relation types are:
+
+```text
+policy
+workflow
+fallback
+mirror
+migration
+test
+replaces
+deprecated-by
+cross-repo
+```
+
+Examples:
+
+```text
+rel=policy>auth.session.lifecycle
+rel=workflow>auth.session.revoke-family
+rel=fallback>auth.session.reauthenticate
+rel=test>auth.session.rotate.contract
+rel=cross-repo>zdp-core-auth/auth.session.rotate
+```
+
+The following structural relations MUST NOT be maintained by hand:
+
+```text
+calls
+imports
+references
+implements
+exports
+overrides
+```
+
+They belong in generated structure indexes because source changes can invalidate them immediately.
+
+A non-cross-repository target MUST resolve to a source card or an ID registry record.
+
+## `stability`
+
+`stability` controls catalog placement and cache strategy.
+
+`architecture` describes ownership, boundaries, and responsibilities that normally survive file moves and implementation changes. Architecture cards enter the repository core catalog by default.
+
+`contract` describes externally relevant behavior, invariants, effects, and semantic workflow links. Contract cards enter module catalogs by default.
+
+`implementation` describes a current algorithm or optimization that may change frequently. Implementation cards remain searchable but are excluded from shared stable catalogs by default.
+
+## Forbidden volatile data
+
+Source cards MUST NOT contain fields for:
+
+```text
+path
+line
+span
+commit
+updated_at
+owner
+callers
+callees
+imports
+references
+implementation_count
+test_status
+current_signature
+```
+
+Equivalent path, line, commit, and timestamp data hidden inside `role`, `search`, or `invariant` values is also invalid.
+
+Current locations, signatures, imports, hashes, and future call-graph edges belong in generated files.
+
+## Size profile
+
+The reference implementation applies these default byte limits:
+
+| Scope | Maximum bytes |
+| --- | ---: |
+| `file` | 400 |
+| `module` | 1,200 |
+| `symbol` | 900 |
+
+For repositories with at least 50,000 scanned source bytes, LLMNav comments SHOULD remain below 1.5% of source bytes. The checker reports a warning when the configured ratio is exceeded.
+
+These are anti-bloat limits, not targets.
+
+## Canonicalization
+
+A canonical formatter MUST refuse to erase malformed lines, unknown fields, overlapping blocks, or duplicate scalar values. Unsafe cards remain unchanged until the checker-reported issue is fixed.
+
+For valid cards, a canonical formatter MUST:
+
+* emit the standard header and terminator for the original comment style
+* emit keys in canonical order
+* emit list fields once with `|` separators
+* emit one line per `invariant` and `rel`
+* remove empty optional fields
+* preserve the containing file's newline convention
+* preserve source indentation
+
+A checker MAY reject non-canonical formatting. The reference implementation does so by default.
+
+## Generated index
+
+The reference implementation emits `.llmnav/cache/index.json` with:
+
+* stable card fields
+* generated path and source line
+* attached declaration name, kind, line, and signature when recognized
+* generated import strings
+* separate semantic, structure, and body SHA-256 hashes
+
+`semantic` hashes change only when card meaning changes.
+
+`structure` hashes change when path, declaration, signature, or import structure changes.
+
+`body` hashes change when the containing source file changes.
+
+Generated files MUST be deterministic. Wall-clock timestamps, absolute paths, platform-specific separators, random transaction IDs, and filesystem stat values MUST NOT be embedded in deterministic catalogs.
+
+The reference implementation preserves `index.json` schemaVersion 1 for v0.1 consumers and emits additive generated accelerators:
+
+* `search-index.json` contains a versioned deterministic token dictionary, normalized phrase documents, and compact posting lists
+* `file-state.json` contains versioned deterministic parsed-file state
+* `manifest.json` hashes every deterministic cache artifact except itself
+
+An implementation MAY use volatile filesystem stat hints to avoid reading unchanged files, but those hints MUST remain outside deterministic cache output and MUST NOT affect generated bytes.
+
+Incremental generation MUST produce the same deterministic artifacts as a full generation for identical source, configuration, registry, aliases, and stable order.
+
+A generated search index MUST be treated as an accelerator rather than source truth. Implementations MUST be able to reject an incompatible accelerator and rebuild it from the primary card index.
+
+Transactional work directories and recovery journals are not deterministic artifacts. They MUST NOT be copied into source comments or committed as semantic metadata.
+
+## Stable order
+
+`.llmnav/order.lock` records semantic IDs in append-only catalog order. Implementations SHOULD preserve existing lines and append new active IDs.
+
+A deleted ID MAY remain in the lock. Removing or globally reordering entries is an explicit cache-epoch operation, not routine formatting.
+
+## Conformance
+
+A source parser conforms to LLMNav/1 when it recognizes all required comment encodings, field syntax, and canonical keys.
+
+A checker conforms when it enforces required fields, controlled values, ID syntax, forbidden volatile data, relation resolution, registry state validity, and declaration attachment.
+
+An indexer conforms when it keeps source semantics separate from generated location and emits deterministic output for identical input.
+
+A coding-agent integration conforms when it directs the agent to query compact cards before broad repository exploration and does not treat generated paths as stable source metadata.
+
+## Versioning
+
+The npm package version and source specification version are independent.
+
+Backward-compatible parser, CLI, ranking, or diagnostic improvements use normal semantic package versioning.
+
+A breaking source grammar change requires a new header such as `llmnav/2`. Implementations MUST NOT reinterpret a `llmnav/1` card under incompatible rules.

package/examples/provider-neutral-host.d.mts

@@ -0,0 +1,17 @@
+import type {
+  AgentOperationResult,
+  AgentToolDefinition,
+  PromptPrefixBundle,
+  PromptPrefixPartition,
+} from "llmnav";
+
+export interface LlmnavHost {
+  toolDefinitions: AgentToolDefinition[];
+  basePromptPartitions: PromptPrefixPartition[];
+  selectPromptPartitions(moduleIds?: string[]): PromptPrefixPartition[];
+  execute(call: { name: string; input?: Record<string, unknown> }): Promise<AgentOperationResult>;
+  refresh(): Promise<LlmnavHost>;
+}
+
+export function createLlmnavHost(root: string): Promise<LlmnavHost>;
+export function selectPromptPartitions(bundle: PromptPrefixBundle, moduleIds?: string[]): PromptPrefixPartition[];