scip-query 0.19.6 → 0.19.10

package/docs/DURABILITY.md

@@ -15,10 +15,12 @@ directory entry survive power loss.
 
 A **crash-durable replacement** is a visibility-atomic replacement that also
 flushes the complete staging file before rename and flushes the containing
-directory after rename. Those ordered flushes are what make acknowledged file
-contents and the name that reaches them recoverable after an operating-system
-or machine crash, subject to the host filesystem and device honoring their
-flush contract.
+directory after rename. When the writer creates a missing directory path, it
+also creates each component from the nearest existing ancestor outward and
+flushes the directory that names each new component. Those ordered flushes are
+what make acknowledged file contents and the complete path that reaches them
+recoverable after an operating-system or machine crash, subject to the host
+filesystem and device honoring their flush contract.
 
 An **authoritative record** is file-backed state whose loss can change which
 generation, owner, policy, or accepted decision the program treats as current.
@@ -35,16 +37,28 @@ from a filename to its file—after the file itself has been flushed. Flushing
 only file contents is insufficient because a crash can otherwise lose the
 rename that made those contents current.
 
+An **achieved durability result** is the guarantee established by the
+operations that actually completed, not the guarantee the caller requested.
+`visibility` means only old-or-new complete process visibility;
+`file-flushed` means the final file bytes were flushed but at least one
+directory namespace could not be synchronized; and `directory-durable` means
+the file plus every newly created ancestor and final directory entry crossed
+the supported persistence frontier.
+
 ## APIs
 
-| API | Staging identity | File flush | Rename | Parent-directory flush |
-| --- | --- | --- | --- | --- |
-| `writeJsonAtomic` / `replaceFileAtomic(..., { durability: "visibility" })` | Exclusive random token | No | Yes | No |
-| `writeJsonDurable` / `replaceFileAtomic(..., { durability: "durable" })` | Exclusive random token | Yes | Yes | Yes where supported |
+| API                                                                        | Staging identity                                | File flush            | Rename/link                       | New-ancestor flush  | Final-directory flush |
+| -------------------------------------------------------------------------- | ----------------------------------------------- | --------------------- | --------------------------------- | ------------------- | --------------------- |
+| `writeJsonAtomic` / `replaceFileAtomic(..., { durability: "visibility" })` | Exclusive random token                          | No                    | Yes                               | No                  | No                    |
+| `writeJsonDurable` / `replaceFileAtomic(..., { durability: "durable" })`   | Exclusive random token                          | Yes                   | Yes                               | Yes where supported | Yes where supported   |
+| `createFileAtomicExclusive(..., { durability: "durable" })`                | Exclusive random token                          | Yes                   | Exclusive hard link               | Yes where supported | Yes where supported   |
+| `cloneFileDurable`                                                         | Final target inside an unpublished staging tree | Yes, after final mode | Caller publishes the staging tree | Yes where supported | Yes where supported   |
 
 `writeJsonAtomic` retains its original `void` return contract for compatibility.
-`writeJsonDurable` and `replaceFileAtomic` return the achieved directory-sync
-status. Verified binary installation owns an equivalent platform-local
+`writeJsonDurable`, `replaceFileAtomic`, and `createFileAtomicExclusive`
+return `requestedDurability`, `achievedDurability`, and `directorySync`.
+Callers must use the achieved fields when logging or forwarding a durability
+claim. Verified binary installation owns an equivalent platform-local
 flush/rename sequence because the enforced architecture forbids dependencies
 between the sibling `platform` and `storage` boundaries.
 
@@ -53,18 +67,25 @@ Known Windows "directory handles unsupported" errors produce
 `directorySync: "unsupported"` after the staged file itself has been flushed
 and renamed. That result means complete visibility plus flushed file contents,
 not the full POSIX directory-entry durability guarantee. Other directory-sync
-errors remain failures.
+errors remain failures. An API that returns only a path or value makes no
+machine-crash claim; protocol APIs that acknowledge work expose the achieved
+result explicitly.
 
 ## Failure Outcomes
 
-| Failure point | Target visible after return/throw | Owned staging file |
-| --- | --- | --- |
-| Exclusive create | Previous target | No owned file was created |
-| Write or short/invalid progress | Previous target | Closed and removed |
-| File flush | Previous target | Closed and removed |
-| Rename | Previous target | Removed by the writer |
-| POSIX directory flush | New complete target; durability unconfirmed and the call throws | Already renamed; no staging path |
-| Unsupported Windows directory flush | New complete target; result reports the limitation | Already renamed; no staging path |
+| Failure point                       | Target visible after return/throw                               | Owned staging file               |
+| ----------------------------------- | --------------------------------------------------------------- | -------------------------------- |
+| Exclusive create                    | Previous target                                                 | No owned file was created        |
+| Write or short/invalid progress     | Previous target                                                 | Closed and removed               |
+| File flush                          | Previous target                                                 | Closed and removed               |
+| Rename                              | Previous target                                                 | Removed by the writer            |
+| POSIX directory flush               | New complete target; durability unconfirmed and the call throws | Already renamed; no staging path |
+| Unsupported Windows directory flush | New complete target; result reports the limitation              | Already renamed; no staging path |
+
+When a parent chain is new, a failure while creating or flushing any ancestor
+also throws before a durable result is returned. The incomplete path can be
+visible to the running process, but recovery may discard every component that
+has not yet been named by a flushed parent.
 
 The post-rename failure row is deliberately explicit. Once rename succeeds,
 rolling back would be another publication with its own crash window. The
@@ -73,25 +94,28 @@ could not confirm the stronger durability guarantee.
 
 ## Call-Site Classification
 
-| Record | Contract | Reason |
-| --- | --- | --- |
-| Reindex `meta.json` | Durable | Names the accepted index status, fingerprint, and generation metadata |
-| SQLite generation manifest and `state.json` | Durable | Flushes one complete immutable artifact set, then atomically selects it for new readers |
-| Shared-generation manifest | Durable file within staging | Authenticates immutable artifacts; the later generation-directory publication remains a separate generation-store operation |
-| Worktree lease and local cache pointer | Durable | Protect generations from collection and bind a worktree to repository cache identity; generation changes, liveness touches, and cleanup serialize through the repository-cache lock |
-| Watch service state | Durable | Publishes the process instance and index generation accepted as current |
-| Rust semantic session `server.json` | Durable | Publishes the live server process and mailbox identity |
-| Project `.scipquery.json` | Durable | Controls indexing, watch, architecture, and detector policy; versioned reads reject unsupported meaning before use, and authorized writes migrate legacy bytes without dropping unknown fields |
-| Codex/Claude hook JSON | Durable | Controls whether and when agent hooks execute |
-| Structured suppression file | Durable | Records an accepted finding and its reason |
-| Health baseline | Durable | Acts as a committed regression policy |
-| Verified binary cache promotion | Durable | Makes checksum-accepted executable or tool bytes current |
-| TypeScript/Rust request and response mailboxes | Visibility-atomic | A timeout or retry reconstructs the ephemeral message |
-| Watch activity | Visibility-atomic | A newer timestamp supersedes an older idle-lifetime observation; it carries no durable intent |
-| Watch refresh request, claim, and completion records | Durable immutable admission and acknowledgement | Accepted intent survives activity replacement and owner crashes; exclusive claims may be recovered only by the next lock owner |
-| TypeScript fragment/overlay manifests | Visibility-atomic | Content-addressed cache artifacts are validated and rebuildable |
-| Repository GC state | Visibility-atomic | Sweep history can be reconstructed conservatively |
-| Affected-set shadow latest record | Visibility-atomic | Calibration telemetry does not control publication |
+| Record                                               | Contract                                                           | Reason                                                                                                                                                                                         |
+| ---------------------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Reindex `meta.json`                                  | Durable                                                            | Names the accepted index status, fingerprint, and generation metadata                                                                                                                          |
+| SQLite generation manifest and `state.json`          | Directory-durable where supported                                  | Durably establishes the generation root, flushes final read-only artifact bytes and metadata, publishes the immutable directory, then selects it for new readers                               |
+| Shared generation                                    | Directory-durable where supported                                  | Flushes every read-only artifact and manifest, the staging namespace, the final generation rename, and the shared `generations/` namespace                                                     |
+| Worktree lease and local cache pointer               | Durable                                                            | Protect generations from collection and bind a worktree to repository cache identity; generation changes, liveness touches, and cleanup serialize through the repository-cache lock            |
+| Watch service state                                  | Durable                                                            | Publishes the process instance and index generation accepted as current                                                                                                                        |
+| Rust semantic session `server.json`                  | Durable                                                            | Publishes the live server process and mailbox identity                                                                                                                                         |
+| Project `.scipquery.json`                            | Durable                                                            | Controls indexing, watch, architecture, and detector policy; versioned reads reject unsupported meaning before use, and authorized writes migrate legacy bytes without dropping unknown fields |
+| Codex/Claude hook JSON                               | Durable                                                            | Controls whether and when agent hooks execute                                                                                                                                                  |
+| Structured suppression file                          | Durable                                                            | Records an accepted finding and its reason                                                                                                                                                     |
+| Health baseline                                      | Durable                                                            | Acts as a committed regression policy                                                                                                                                                          |
+| Verified binary cache promotion                      | Durable                                                            | Makes checksum-accepted executable or tool bytes current                                                                                                                                       |
+| TypeScript/Rust request and response mailboxes       | Directory-durable where supported                                  | Admission is acknowledged only after the pending name is flushed; a new owner directory is durably named before a request moves from pending to inflight; responses precede claim release      |
+| Watch activity                                       | Visibility-atomic                                                  | A newer timestamp supersedes an older idle-lifetime observation; it carries no durable intent                                                                                                  |
+| Watch refresh request, claim, and completion records | Durable immutable admission and acknowledgement                    | Accepted intent survives activity replacement and owner crashes; exclusive claims may be recovered only by the next lock owner                                                                 |
+| Cache ownership credential                           | Directory-durable where supported                                  | A complete file is flushed privately, hardening is revalidated, the public credential is linked exclusively, and its directory is flushed before detailed publication success                  |
+| npm release state                                    | Directory-durable where supported; file-flushed on bounded hosts   | The registry is freshly reconciled and remains external authority; the local coordinate-pair recovery record reports its exact achieved guarantee                                              |
+| Outcome event                                        | File publication locally; repository-durable only after Git commit | The event becomes shared history through the committed repository record, not merely because a checkout-local file exists                                                                      |
+| TypeScript fragment/overlay manifests                | Visibility-atomic                                                  | Content-addressed cache artifacts are validated and rebuildable                                                                                                                                |
+| Repository GC state                                  | Visibility-atomic                                                  | Sweep history can be reconstructed conservatively                                                                                                                                              |
+| Affected-set shadow latest record                    | Visibility-atomic                                                  | Calibration telemetry does not control publication                                                                                                                                             |
 
 Process locks use exclusive descriptor creation rather than replacement. Their
 durable token ownership, malformed-creation grace, guarded recovery, and
@@ -101,3 +125,29 @@ stable compatibility mirrors, and their crash ordering are defined in
 [Local Index Generations](INDEX_GENERATIONS.md). Durable watch demand,
 idempotency, claim recovery, and acknowledgement ordering are defined in
 [Watch Refresh Requests](WATCH_REFRESH_REQUESTS.md).
+
+## Executable crash model
+
+`tests/helpers/persistence-frontier.ts` models the two states a normal
+filesystem test otherwise conflates. Writes, links, renames, and removals first
+change process-visible state. File synchronization advances persisted inode
+bytes and mode; directory synchronization advances persisted names. A modeled
+power loss discards everything beyond those frontiers.
+
+`tests/storage/atomic-file-crash.test.ts` replays replacement, exclusive
+publication, new ancestor creation, and durable artifact cloning after every
+relevant phase. Boundary-specific tests then hold the higher-level protocol
+order:
+
+- local generation stage and same-size corruption tests;
+- shared-generation artifact, manifest, staging, rename, and final-directory
+  stage tests;
+- watch admission, claim, completion, and retry tests;
+- mailbox owner-directory, claim-transfer, response, and recovery tests;
+- ownership short-write, unsupported-sync, and failed-sync tests; and
+- release-state synchronized, unsupported, and failed-write tests.
+
+The deterministic model proves the filesystem primitive’s persisted-state
+semantics. The boundary tests prove each protocol invokes those primitives in
+the required order. Neither an exception callback alone nor a successful live
+filesystem read is described as a simulated power loss.

package/docs/INDEX_GENERATIONS.md

@@ -23,6 +23,10 @@ The worktree cache keeps this internal layout:
   meta.json                        compatibility mirror
   .scipquery-generations/
     state.json                     atomic current-generation pointer
+    gc.json                        last bounded-collection observation
+    gc.lock                        collection/reader-admission coordinator
+    readers/
+      <pid>-<token>.json           live process-generation leases
     <generation-sha256>/
       manifest.json
       index.db
@@ -30,11 +34,19 @@ The worktree cache keeps this internal layout:
       meta.json
 ```
 
-The files inside a named generation directory are immutable. The SHA-256
+The files inside a named generation directory are immutable. Their final
+read-only mode is applied before their final file synchronization, so the
+persisted inode contains both the accepted bytes and the protection metadata.
+The SHA-256
 identity incorporates the database bytes, SCIP bytes when present, and the
 generation-bearing metadata fields. `manifest.json` records the exact size and
 digest of each stored artifact. `state.json` names the accepted directory and
 is the only publication decision read by internal database consumers.
+Opening a named generation authenticates the database, SCIP, and metadata
+bytes against those digests; matching size alone is never sufficient. A
+bounded process-local validation cache is keyed by manifest identity plus
+stable device, inode, size, modification-time, and change-time evidence, so a
+same-size mutation invalidates the cached proof and is rehashed.
 
 The top-level files remain for the SCIP CLI, indexers, older scip-query
 versions, and external inspection. They are derived mirrors, not the internal
@@ -48,18 +60,26 @@ Publication proceeds in this order:
 
 1. Retain the previously published stable artifacts as an immutable
    generation when upgrading a legacy cache.
-2. Copy-on-write clone or copy every accepted candidate artifact into a
-   private staging directory.
-3. Flush the artifact files and manifest, rename the complete staging
-   directory into its content-derived identity, and flush the generation
+2. Durably establish `.scipquery-generations/` from its nearest existing
+   ancestor when this is the cache's first generation.
+3. Copy-on-write clone or copy every accepted candidate artifact into a
+   private staging directory, apply its final read-only mode, and flush the
+   resulting inode.
+4. Flush the manifest and staging namespace, rename the complete staging
+   directory into its content-derived identity, and flush the generation-root
    directory entry.
-4. Durably replace `state.json` so new internal readers select the complete
+5. Establish the versioned query layout after augmentation: add the
+   definition-only covering index, remove known redundant indexes only when
+   schema inspection proves a retained equivalent, and run `ANALYZE` so the
+   planner sees the final row distribution. Query-plan tests hold those
+   proofs.
+6. Durably replace `state.json` so new internal readers select the complete
    new directory.
-5. Replace the three compatibility mirrors and durably record their file
+7. Replace the three compatibility mirrors and durably record their file
    identities for drift diagnostics.
 
-A crash before step 4 leaves the prior pointer authoritative. A crash after
-step 4 leaves the new immutable generation authoritative even if one or more
+A crash before step 6 leaves the prior pointer authoritative. A crash after
+the pointer step leaves the new immutable generation authoritative even if one or more
 compatibility mirrors are old. `scip-query status` and freshness inspection
 report that mirror drift and a later refresh repairs it; database-backed
 queries remain generation-consistent throughout.
@@ -70,6 +90,14 @@ and TypeScript semantic mailbox requests carry the handle identity, so a
 continuation or numeric symbol identifier from an older generation is rejected
 when a service has moved to a newer one.
 
+Current metadata also records `sqliteLayoutVersion`. A current-version
+metadata record that predates this field remains readable and can reuse its
+language SCIP shards, but it cannot take the whole-SQLite unchanged fast path.
+The next refresh applies query-layout maintenance to the stable mirror,
+publishes a new immutable generation and current layout metadata, and does not
+rerun unchanged language indexers. Later unchanged refreshes resume the
+ordinary whole-generation reuse path.
+
 Mailbox protocol version 3 additionally binds that generation identity into a
 content-derived logical-operation key. The deterministic request ID lets a
 retry join an already pending, inflight, or retained completed operation,
@@ -86,18 +114,36 @@ generation implementation, which did not name an immutable artifact set, also
 uses this compatibility path. New publications upgrade either layout without
 deleting the stable files.
 
-Published local generation directories are retained conservatively. Automatic
-collection is intentionally disabled until a cross-process reader lease can
-prove that no surviving handle can later need the retained SCIP companion.
-This can temporarily consume more cache space, but it preserves the stronger
-rule that storage reclamation may never change or remove evidence owned by a
-live query.
+Published local generation directories are retained by a bounded,
+reader-aware collector. Opening an immutable `ScipDatabase` first acquires the
+generation GC lock, resolves the current pointer, and visibility-atomically
+publishes a lease containing its random token, generation identity, PID, and
+operating-system process-start identity when available. Closing the database
+removes that exact lease idempotently.
+
+Publication and collection use the same token-owned process lock. Collection
+always protects the current generation, the named recovery generation, and
+every live reader generation. It removes only oldest unprotected generations
+until both default limits hold: at most eight retained generations and at most
+2 GiB of logical artifact bytes. A lease whose recorded process instance is
+provably dead is reclaimed. A malformed lease protects every generation and
+defers deletion; uncertainty cannot authorize reclamation.
+
+`gc.json` is a visibility-atomic diagnostic containing retained counts, logical
+bytes, protected/readers counts, removed counts, limits, state, time, and any
+reason. `scip-query status` reports that observation together with the current
+retained set and oldest generation age. Collection failure does not invalidate
+the accepted index; it remains explicit bounded-storage debt for the operator.
 
 The repository-wide shared generation store is a separate cache layer. It
-warms a worktree by copying a complete generation into the worktree's private
-cache; the local publisher then creates the local immutable directory and
-pointer described here. Later worktree writes cannot mutate either the shared
-source or a retained local reader.
+warms a worktree from a complete content-addressed generation. Publication
+flushes each final read-only artifact, the manifest, the staging namespace, the
+rename into `generations/<identity>`, and `generations/` itself before
+returning `directory-durable` on supported hosts. On a host without directory
+sync, it returns `file-flushed`; readers still rehash every artifact and reject
+an incomplete generation. The local publisher then creates the private
+immutable directory and pointer described here. Later worktree writes cannot
+mutate either the shared source or a retained local reader.
 
 ## Shared worktree lease invariant
 

package/docs/MAILBOX_LIFECYCLE.md

@@ -122,7 +122,12 @@ boundaries.
 A claim is a time-bounded service ownership record made real by renaming one
 pending file into an owner-specific inflight directory. Rename is the
 ownership compare-and-set: only the process whose rename succeeds owns that
-file. The directory's owner record binds the random owner ID to a PID and,
+file. Before a first claim can remove the pending name, the service durably
+establishes the fixed mailbox skeleton, creates the owner directory, and
+flushes `inflight/` so the complete destination path survives. It then writes
+the owner record, renames the request, flushes the source directory, and
+flushes the owner directory in that order. The directory's owner record binds
+the random owner ID to a PID and,
 when the operating system exposes it, a process-start identity. A process-start
 identity is the operating-system fact that distinguishes successive
 executions occupying the same numeric PID slot.
@@ -147,6 +152,14 @@ answer-affecting operations are read-only or rebuildable, the response
 identity is stable, and exclusive completion remains a final defense against
 two observable answers.
 
+Accepted admission and successful claims expose the achieved directory-sync
+result. On POSIX hosts that complete every directory flush this is
+`directory-durable`. When Node reports directory handles unsupported on
+Windows, it is `file-flushed`: complete file bytes and process-crash recovery
+remain, but the protocol does not upgrade that bounded result into a
+machine-crash namespace claim. Any other directory synchronization error
+throws instead of acknowledging the transition.
+
 ## Fairness and maintenance
 
 Pending work is ordered by `enqueuedAtMs`, then stable request identity.
@@ -158,6 +171,19 @@ at most 64 removals or reclaims. Those caps are what let the watch and Rust
 server loops regain control to update heartbeats, observe stop signals, and
 run unrelated maintenance even when a mailbox was flooded.
 
+The TypeScript watch service initializes its two mailbox layouts once after
+acquiring service ownership. An empty poll reads the already-established
+directories; it does not recreate or chmod the full layout. Consecutive empty
+polls back off from 50 ms through 100 ms and 200 ms to a 250 ms ceiling. A poll
+that processes work returns to a 10 ms drain interval. The Rust service uses
+the same 10–250 ms adaptive range.
+
+Deadline authority is re-evaluated per claimed request. A service reads its
+clock immediately before starting each handler and again after the handler
+returns. Therefore a request that expires while an earlier request runs is
+rejected before execution, and an answer completed after its authoritative
+deadline is not published as success.
+
 Maintenance:
 
 - reclaims expired inflight ownership only after the recorded process instance
@@ -190,6 +216,7 @@ window. Clock-domain rules for these records are documented in
 Focused contract coverage lives in:
 
 - `tests/storage/bounded-mailbox.test.ts`;
+- `tests/storage/atomic-file-crash.test.ts`;
 - `tests/semantic/typescript/typescript-session-mailbox.test.ts`;
 - `tests/reindex/typescript-index-mailbox.test.ts`;
 - `tests/semantic/rust/durable-session-protocol.test.ts`;

package/docs/REINDEX_METADATA_COMPATIBILITY.md

@@ -7,7 +7,7 @@ record rather than an authority by itself: a consumer still verifies the
 database, SCIP companion, immutable-generation state, or current source
 fingerprint required by its operation.
 
-`src/domain/reindex-metadata.ts` is the dependency-free decoding boundary. It
+`src/domain/reindex-metadata.ts` is the domain decoding boundary. It
 classifies an input as:
 
 - `legacy`: a structurally valid version 2 record;
@@ -38,12 +38,20 @@ Future records are never rewritten.
 
 Recognized versions validate `status`, optional timestamp,
 requested/indexed language sets, skipped-language rows, SCIP companion state,
-and v3 shard maps. A fingerprint is an opaque JSON identity for evidence and
-stable-generation compatibility, preserving the pre-decoder v2/v3 contract;
-freshness and publication additionally require it to be an object. Language
-lists contain unique names from the same
+the optional positive `sqliteLayoutVersion`, and v3 shard maps. A fingerprint
+is an opaque JSON identity for evidence and stable-generation compatibility,
+preserving the pre-decoder v2/v3 contract; freshness and publication
+additionally require it to be an object. Language lists contain unique names
+from the same
 `SUPPORTED_LANGUAGES` catalog used by configuration.
 
+The SQLite layout field is an additive v3 upgrade boundary rather than a wire
+version migration. A v3 record without it remains queryable, evidence-usable,
+and eligible for language-shard reuse. It is not eligible for whole-SQLite
+unchanged reuse. The next refresh can therefore reuse unchanged language SCIP
+shards, establish the current post-conversion indexes and planner statistics,
+and publish a new generation without rerunning those indexers.
+
 ## Capability matrix
 
 A capability is a named permission to interpret a decoded record for one
@@ -62,14 +70,16 @@ the same version while intentionally requiring different completeness.
 Consumer-specific comparisons happen only after this matrix accepts the
 record:
 
-- freshness and unchanged-index reuse compare a publishable fingerprint and
-  sorted indexed languages with current inputs;
+- freshness compares a publishable fingerprint and sorted indexed languages
+  with current inputs; whole-SQLite unchanged reuse additionally requires the
+  current `sqliteLayoutVersion`;
 - shared-generation publication additionally verifies the immutable artifact
   set, project root, and database integrity;
 - evidence and TypeScript semantic cache keys accept complete or partial
   records and include status, so partial and complete results cannot collide;
 - SQLite and TypeScript service generation identities use the same canonical
-  projection and reject records without stable-identity capability;
+  projection, including `sqliteLayoutVersion` when present, and reject records
+  without stable-identity capability;
 - per-language and per-project reuse compare the decoded v3 shard
   fingerprints with current fingerprints and require the corresponding shard
   file to exist.

package/docs/SECURITY_MODEL.md

@@ -0,0 +1,146 @@
+# Security model
+
+`scip-query` treats a checked-out repository and every index or cache artifact
+derived from it as untrusted input. An untrusted checkout is a directory of
+files supplied by a project rather than by the operator, distinguished from
+ordinary input by being able to name source paths, tools, configuration, and
+persisted records that the CLI may otherwise be tempted to trust.
+
+The default analysis boundary may read repository files and managed,
+rebuildable cache artifacts. It may not execute a repository-selected program,
+adopt an arbitrary cache directory, escape the canonical project root through
+an indexed path, or install a mutable tool identity merely because the
+checkout requests it.
+
+## Explicit authority
+
+An authority grant is an operator action that permits one otherwise-forbidden
+effect, distinguished from project configuration by coming from the current
+CLI invocation:
+
+- `--trust-project-tools` permits reviewed repository-local indexer
+  executables for that invocation.
+- `--tla-tools <jar>` selects a reviewed TLA jar explicitly. The default
+  downloaded TLA artifact remains version- and SHA-256-pinned.
+- `--install-missing` permits installation of the exact immutable tool
+  identities listed by the command. Without it, missing tools produce setup
+  instructions instead of a mutation.
+- environment overrides such as `SCIP_QUERY_CACHE_DIR` are operator-owned
+  storage choices. Tracked `.scipquery.json` cannot grant destructive authority
+  over an arbitrary external cache.
+
+A managed cache is rebuildable local state rooted under a scip-query-owned
+directory, distinguished from a caller-selected directory by an ownership
+record and canonical containment checks. Destructive cache operations require
+both properties.
+
+## Input budgets
+
+An input budget is a maximum amount of untrusted data one operation will
+materialize or ask a parser to compile, distinguished from silent truncation by
+failing explicitly with the input kind, observed amount, and accepted limit.
+
+| Input class                                            |                    Limit | Representative users                                                     |
+| ------------------------------------------------------ | -----------------------: | ------------------------------------------------------------------------ |
+| Config, manifest, lock, lease, and other small records |                    8 MiB | `.scipquery.json`, package manifests, generation metadata, hook settings |
+| One source or per-document fragment                    |                   64 MiB | indexed source reads, TypeScript/Vue snapshots, mailbox result payloads  |
+| One SCIP index artifact                                |                  512 MiB | merge, sanitize, Rust occurrence fallback, shared-generation hydration   |
+| Profile or retained JSONL artifact                     |                  256 MiB | profiling audits, legacy event ledgers, rotating diagnostic segments     |
+| Generated TLA trace                                    | 16 MiB and 100,000 steps | `tla instrument` recorder                                                |
+| Repository-supplied regular-expression pattern         |         4,096 characters | entry-root patterns and TLA statement bindings                           |
+| Agent hook standard input                              |                    8 MiB | stop-hook request payload                                                |
+
+Regular files are opened first, checked through that descriptor, read, and
+checked again so replacement or growth cannot bypass the pre-read limit.
+Streams and pseudo-files are counted while reading. Large file fingerprints
+are computed in fixed-size chunks rather than by retaining the complete file.
+
+An oversized artifact is not partially interpreted. Operations that require
+the complete artifact fail and name the limit; query-level detectors that
+intentionally sample or cap logical rows retain their existing machine-readable
+coverage metadata and `--full` remediation.
+
+## Complete command output
+
+Output pagination is a resumable transport for one rendered command result,
+distinguished from a result limit by preserving every rendered character.
+Human output above 12,000 characters automatically includes the exact command
+for the next page. Large default JSON remains byte-compatible and writes the
+exact opt-in paging command to stderr before and after the JSON stream.
+
+Every command accepts:
+
+```text
+--output-page-size <characters>
+--output-cursor <cursor>
+```
+
+The opaque cursor binds the command, working directory, non-pagination
+arguments, immutable page number and size, private output snapshot, and
+complete output hash. The first invocation records UTF-8 byte ranges and a
+hash for each page while retaining only one page in memory. A continuation
+reads and verifies only its own range instead of re-running the command or
+rescanning the complete output.
+
+One snapshot is limited to 32 million rendered characters, 64 MiB, and 32,768
+pages. The current user's snapshot pool is limited to 32 snapshots and 256 MiB
+under a process-identity-aware lock. Abandoned writers are reclaimed without
+removing a live writer; complete snapshots expire after one hour and are
+removed after their final page.
+
+Run commands normally without selecting a page size. If human output is
+oversized, the readable multiline page prints one `Continue exactly:` command;
+run it unchanged until the transport-complete marker. That marker proves every
+rendered character was retrieved; command coverage remains a separate claim.
+Explicit paged `--json`
+returns the versioned object envelope and must be followed until
+`page.complete` is `true`. Do not pipe output through `head`, `tail`, or a
+line-range `sed`; those programs discard data without creating a resumable
+position. See [CLI JSON output](CLI_JSON_OUTPUT.md) and the [output-page
+schema](schemas/cli-output-page.schema.json).
+
+## Diff-gate process containment
+
+A diff-gate lease is one live process identity permitted to evaluate a
+project's current diff, distinguished from an ordinary lock file by recording
+and validating the owner's PID, process start, and project identity. CLI and
+Stop-hook gates share this lease, so an overlapping request reports the live
+owner instead of multiplying detector work.
+
+The public gate owns an isolated child process. Its default deadline is 60
+seconds, or 180 seconds with `--full`. On timeout, failure, or interruption the
+parent terminates and reaps that child before releasing the lease. An operator
+may set `SCIP_QUERY_DIFF_GATE_TIMEOUT_MS` to a positive millisecond value; the
+runtime caps it at 10 minutes.
+
+The parent creates a unique, private progress path and one-run token while it
+owns the lease. The child visibility-atomically records the current gate phase
+and detector immediately before and after each detector. A timed-out parent
+accepts only the record carrying its token, reports the active detector or
+phase plus the last completed detector, and removes the record before releasing
+the lease. The progress record is diagnostic evidence, not authority: failure
+to write it cannot turn a passing gate into a failure, and a stale or malformed
+record cannot alter the gate result.
+
+Outcome reconciliation rechecks at most one historical comparison base per
+gate. Deferred bases remain open and the result reports their exact base and
+finding counts, so accumulated event history cannot silently multiply one
+foreground gate into an unbounded replay batch.
+
+## Terminal and JSON output
+
+Human terminal output is presentation text written to an interactive terminal,
+distinguished from JSON by being able to activate terminal control sequences.
+Repository-controlled human fields therefore have control bytes rendered
+inert. JSON values retain their printable data and are protected by JSON
+encoding rather than human-output rewriting.
+
+Registry and release diagnostics redact URL user information and
+credential-bearing query fields before rendering nested errors.
+
+## Reporting a security issue
+
+Do not include a live credential, private repository, or destructive
+proof-of-concept target in a public report. Describe the affected command,
+platform, trust flags, and smallest disposable reproduction needed to
+distinguish the boundary failure.

package/docs/TELEMETRY_RETENTION.md

@@ -27,6 +27,11 @@ segments. That bounded-history deletion is different from concurrency loss:
 which segment is pruned is decided while the append/rotation lock excludes
 other writers.
 
+The retained-set reader also has a 256 MiB byte budget across the files it
+opens. It validates each segment as the same regular file before and after the
+read. Exceeding that operational budget is an explicit read failure; it does
+not silently return a prefix and present it as complete telemetry.
+
 ## Serialization and crash recovery
 
 The rotation lock is `<history>.rotation.lock`. It is a process-instance lock:
@@ -53,10 +58,17 @@ mailbox-response, and release records use their own durable protocols.
 
 Lock timeout and ownership-changed release are typed failures at the shared
 helper boundary. The reindex-activity and affected-shadow production writers
-remain best effort: they catch telemetry failure so an observation cannot
-change the authoritative reindex result. Callers that invoke the shared helper
-directly can distinguish lock timeout, count repaired tail bytes, and count
-partial bytes ignored during reads.
+remain best effort: telemetry failure cannot change the authoritative reindex
+result. A reindex-activity writer returns either `recorded` or `failed` with a
+reason; reindex, foreground watch, and daemon watch surface that failure as a
+warning or watch-state error rather than silently losing the observation.
+
+A retained-set summary carries an activity-confidence state. `complete` means
+every selected non-empty record decoded; `partial` means a valid subset was
+used while exact invalid/skipped/read-error or incomplete-tail counts are
+reported; `unavailable` means no reliable summary could be formed. The summary
+also exposes records read and ignored partial-tail bytes, so consumers can
+distinguish zero activity from missing evidence.
 
 The regression suite forces: