scip-query 0.19.5 → 0.19.8

package/docs/MAILBOX_LIFECYCLE.md

@@ -0,0 +1,197 @@
+# Filesystem mailbox lifecycle
+
+scip-query uses three local filesystem mailboxes to carry work between a
+synchronous CLI process and a reusable service process:
+
+- TypeScript semantic queries sent to the watch service;
+- TypeScript document-emission requests sent to the watch service; and
+- Rust semantic queries sent to the durable rust-analyzer helper.
+
+A filesystem mailbox is a process-coordination queue whose units are complete
+files and whose essential safety property is that each accepted logical
+operation has one durable identity and one inspectable lifecycle across
+process failure. It differs from an arbitrary request directory because
+admission, ownership transfer, completion, expiry, and retention are explicit
+states rather than consequences of whichever process happens to delete a
+file.
+
+A logical operation is a requested computation identified by the SHA-256 of
+its answer-affecting protocol payload. That stable content identity is what
+makes two independently attempted requests units of the same operation:
+retries converge on one pending request, one inflight claim, or one retained
+completion instead of executing under unrelated random IDs. `clientId`
+identifies the attempt that first published the operation; it does not change
+the operation's meaning.
+
+## Layout and states
+
+Each mailbox root has this layout:
+
+```text
+<mailbox>/
+  pending/
+  inflight/<encoded-owner>/
+    .owner.json             # process-instance evidence for claim recovery
+  responses/
+  dead-letter/
+  requests/                 # legacy v2 overlap reads only
+  .admission.lock           # complete, exclusively published quota coordinator
+```
+
+| State     | Real file state                                                                  | Authority and transition                                                                     |
+| --------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
+| Pending   | `pending/<operation-id>.json`                                                    | The immutable request was admitted but no service owns it.                                   |
+| Inflight  | `inflight/<owner>/<request>.<claim-expiry>.claim` plus `.owner.json`             | Atomic rename transferred this request to one recorded process instance for a bounded lease. |
+| Completed | `responses/<operation-id>.json`                                                  | A response was durably and exclusively published before claim release.                       |
+| Rejected  | Error response, with a bounded record under `dead-letter/` when capacity permits | The service explicitly refused malformed, expired, oversized, mismatched, or failed work.    |
+| Expired   | Deadline has passed; the next bounded service batch rejects and collects it      | Expiry prevents abandoned requests from authorizing expensive work indefinitely.             |
+
+The `requests/` directory is not a second current queue. It exists so a new
+service can drain flat requests left by protocol v2. Current writers publish
+only to `pending/`; no migration renames or deletes legacy data pre-emptively.
+
+## Identity and protocol fields
+
+Every current request contains:
+
+- `mailboxVersion`, which versions the shared lifecycle fields;
+- the domain protocol version;
+- `operationKey`, the full SHA-256 logical-operation identity;
+- `id`, deterministically derived as `op-<operationKey>`;
+- `clientId`, the publishing attempt identity;
+- `enqueuedAtMs` and `deadlineAtMs`; and
+- the domain request plus its generation/session identity.
+
+The TypeScript semantic and TypeScript index protocols are version 3. The Rust
+durable-session protocol is also version 3. Current readers recompute the
+operation key, require the ID derived from it, and retain the existing
+generation/base-generation response checks. TypeScript services accept the
+supported flat version-2 request shape during the overlap window. The Rust
+service accepts its former unversioned `{id, request}` shape and the
+immediately prior v3 envelope without a mailbox-session identity; an
+explicitly versioned unknown Rust envelope is not treated as legacy.
+
+Responses repeat the domain protocol, request ID, operation key, completion
+time, authoritative request deadline, and retention expiry. The first
+response published for an operation is authoritative. An old owner that
+finishes after its lease was reclaimed cannot replace a newer response.
+Duplicate admission returns the deadline from that authoritative pending,
+inflight, or completed record, so a retry cannot relabel retained work with a
+new time identity. Rust responses additionally echo the mailbox-session
+identity and are accepted only when protocol, request, operation, session, and
+deadline all match. See
+[Durable Rust session protocol](RUST_DURABLE_SESSION_PROTOCOL.md).
+
+## Admission and bounds
+
+Admission is the act of adding one immutable operation after proving the
+mailbox remains within its retained-resource budget. A short-lived,
+token-checked admission file serializes the count-and-publish decision, which
+prevents two processes from both observing the last free slot and
+oversubscribing it. Its complete owner record is flushed under a private name
+and hard-linked exclusively into the public name; an ownerless or malformed
+public record fails closed and is never deleted from civil-clock age alone.
+
+Default bounds are:
+
+| Bound                              |                                                     Default |
+| ---------------------------------- | ----------------------------------------------------------: |
+| Retained files per mailbox         |                                                       1,024 |
+| Retained bytes per mailbox         |                                                     512 MiB |
+| One request or response            |                                                      64 MiB |
+| Work claimed per service-loop pass |                                                          16 |
+| Claim lease                        | 5 minutes, extended through request deadline plus 5 seconds |
+| Response/idempotency retention     |                                                  10 minutes |
+| Dead-letter retention              |                                                    24 hours |
+| Orphan temporary-file retention    |                                                    1 minute |
+| Maintenance actions per pass       |                                                          64 |
+
+`MailboxBackpressureError` is the typed overload result. Its code distinguishes
+`item-too-large`, `item-capacity`, `byte-capacity`, and `admission-busy`, and
+its status reports the observed retained state and configured limits. A
+duplicate logical operation is checked before capacity rejection, so a retry
+can join work that already owns the last slot.
+
+TypeScript semantic callers retain their established correctness fallback:
+service failure or backpressure selects the in-process provider. Index and
+Rust requester errors propagate to their existing higher-level fallback
+boundaries.
+
+## Ownership, crash recovery, and replay
+
+A claim is a time-bounded service ownership record made real by renaming one
+pending file into an owner-specific inflight directory. Rename is the
+ownership compare-and-set: only the process whose rename succeeds owns that
+file. The directory's owner record binds the random owner ID to a PID and,
+when the operating system exposes it, a process-start identity. A process-start
+identity is the operating-system fact that distinguishes successive
+executions occupying the same numeric PID slot.
+
+The recovery rules are:
+
+| Failure point                                         | Surviving evidence                               | Recovery                                                                                                               |
+| ----------------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
+| Client exits after admission                          | Pending file                                     | A later service batch claims it or explicitly expires it. The client does not delete shared work in `finally`.         |
+| Service exits before claim                            | Pending file                                     | Another owner can claim immediately.                                                                                   |
+| Service exits after claim, before response            | Inflight file, expiry, and dead process identity | After lease expiry and proof that the recorded process instance is gone, maintenance atomically returns it to pending. |
+| Service publishes response, then exits before release | Response plus inflight file                      | Maintenance treats the response as completion and removes the stale claim without re-execution.                        |
+| Old owner finishes after reclamation                  | Competing exclusive response publication         | The first response remains authoritative; the late owner cannot replace it.                                            |
+| Same logical request is retried                       | Same deterministic operation ID                  | The caller joins pending/inflight/completed state.                                                                     |
+| Malformed, expired, or oversized file                 | Claimed input                                    | The service emits an explicit error response and a bounded rejection record.                                           |
+
+These are crash-recoverable claim semantics with first-completion
+idempotency. Civil-clock lease expiry is only a durable recovery hint: it does
+not authorize reclamation while the recorded process instance is live or
+unverifiable. Work can be retried after both lease expiry and owner death. The
+answer-affecting operations are read-only or rebuildable, the response
+identity is stable, and exclusive completion remains a final defense against
+two observable answers.
+
+## Fairness and maintenance
+
+Pending work is ordered by `enqueuedAtMs`, then stable request identity.
+Legacy and malformed files fall back to filesystem modification time. Random
+UUID filename order no longer determines service priority.
+
+Each process call claims at most 16 files, and each maintenance call performs
+at most 64 removals or reclaims. Those caps are what let the watch and Rust
+server loops regain control to update heartbeats, observe stop signals, and
+run unrelated maintenance even when a mailbox was flooded.
+
+Maintenance:
+
+- reclaims expired inflight ownership only after the recorded process instance
+  is dead or its PID has been reused by a different process instance;
+- removes a claim whose response proves it already completed;
+- removes expired responses and dead-letter records;
+- removes abandoned atomic-write staging files after their retention window;
+- preserves non-expired pending and inflight work; and
+- never depends on a client being alive to finish cleanup.
+
+## Telemetry and diagnosis
+
+TypeScript watch state exposes a `mailbox` snapshot under both
+`typescriptSemantic` and `typescriptIndex`. Rust helper state exposes the same
+snapshot. It contains:
+
+- `pending`, `inflight`, `responses`, and `deadLetters`;
+- `invalid`;
+- `totalItems` and `totalBytes`; and
+- `oldestPendingAt` when pending work exists.
+
+This snapshot identifies current retained pressure; it is not a cumulative
+success counter. A rising pending count with a live heartbeat indicates
+service throughput pressure. Inflight work older than its valid lease is
+reclaimed on the next loop only when the owner record proves the process
+instance is gone. Responses are expected during the ten-minute idempotency
+window. Clock-domain rules for these records are documented in
+[Time Semantics](TIME_SEMANTICS.md).
+
+Focused contract coverage lives in:
+
+- `tests/storage/bounded-mailbox.test.ts`;
+- `tests/semantic/typescript/typescript-session-mailbox.test.ts`;
+- `tests/reindex/typescript-index-mailbox.test.ts`;
+- `tests/semantic/rust/durable-session-protocol.test.ts`;
+- `tests/semantic/rust/rust-durable-session.test.ts`; and
+- `tests/platform/watch-service-state.test.ts`.

package/docs/REINDEX_METADATA_COMPATIBILITY.md

@@ -0,0 +1,84 @@
+# Reindex Metadata Compatibility
+
+Reindex metadata is the persisted description in `meta.json` that identifies
+the source-input fingerprint, indexed languages, completeness, and optional
+reuse capabilities of one accepted index generation. It is a compatibility
+record rather than an authority by itself: a consumer still verifies the
+database, SCIP companion, immutable-generation state, or current source
+fingerprint required by its operation.
+
+`src/domain/reindex-metadata.ts` is the dependency-free decoding boundary. It
+classifies an input as:
+
+- `legacy`: a structurally valid version 2 record;
+- `supported`: a structurally valid current version 3 record;
+- `unsupported`: an integer version outside the readable range, with older or
+  future direction; or
+- `malformed`: invalid JSON, a non-object, a missing/non-integer version, or an
+  invalid field in a recognized version.
+
+The decoder shares only dependency-free object-record, timestamp,
+scalar-number, and string-or-null-record predicates in
+`src/domain/record-validation.ts`; moving those generic primitives out of
+individual decoders does not change any version or capability decision.
+
+No consumer may cast an unsupported or malformed record to the current model.
+The decoder returns the original accepted v2/v3 object, so an authorized
+best-effort update such as `lastRefresh` preserves unknown additive fields.
+Future records are never rewritten.
+
+## Version policy
+
+| Wire version | Decoder result | Read policy                                                | Write policy                                                 |
+| ------------ | -------------- | ---------------------------------------------------------- | ------------------------------------------------------------ |
+| 2            | `legacy`       | Supported by common capabilities; no v3 shard capabilities | Read-only compatibility; a rebuilt generation writes v3      |
+| 3            | `supported`    | Current model                                              | All newly published metadata                                 |
+| 4            | `unsupported`  | Reserved migration boundary; reported explicitly           | Never written until its migration and matrix are implemented |
+| Other        | `unsupported`  | Fail closed for reuse/publication                          | Never rewritten                                              |
+
+Recognized versions validate `status`, optional timestamp,
+requested/indexed language sets, skipped-language rows, SCIP companion state,
+and v3 shard maps. A fingerprint is an opaque JSON identity for evidence and
+stable-generation compatibility, preserving the pre-decoder v2/v3 contract;
+freshness and publication additionally require it to be an object. Language
+lists contain unique names from the same
+`SUPPORTED_LANGUAGES` catalog used by configuration.
+
+## Capability matrix
+
+A capability is a named permission to interpret a decoded record for one
+operation. It differs from version support because two consumers can accept
+the same version while intentionally requiring different completeness.
+
+| Capability                    | Required facts                                                                    | Partial status                            | v2  | v3  |
+| ----------------------------- | --------------------------------------------------------------------------------- | ----------------------------------------- | --- | --- |
+| `usableForQuery`              | Valid indexed-language set                                                        | Yes                                       | Yes | Yes |
+| `usableForEvidenceCache`      | Fingerprint value                                                                 | Yes; status is part of the cache key      | Yes | Yes |
+| `publishableGeneration`       | Complete status, object fingerprint, indexed-language set                         | No                                        | Yes | Yes |
+| `stableGenerationIdentity`    | Complete status, fingerprint, valid `updatedAt`; languages projected when present | No                                        | Yes | Yes |
+| `languageShardReuse`          | Valid v3 `languageFingerprints` map                                               | Yes, for individually successful shards   | No  | Yes |
+| `typescriptProjectShardReuse` | Valid v3 `typescriptProjectShards` map                                            | Yes, for individually successful projects | No  | Yes |
+
+Consumer-specific comparisons happen only after this matrix accepts the
+record:
+
+- freshness and unchanged-index reuse compare a publishable fingerprint and
+  sorted indexed languages with current inputs;
+- shared-generation publication additionally verifies the immutable artifact
+  set, project root, and database integrity;
+- evidence and TypeScript semantic cache keys accept complete or partial
+  records and include status, so partial and complete results cannot collide;
+- SQLite and TypeScript service generation identities use the same canonical
+  projection and reject records without stable-identity capability;
+- per-language and per-project reuse compare the decoded v3 shard
+  fingerprints with current fingerprints and require the corresponding shard
+  file to exist.
+
+## Evolution procedure
+
+Adding version 4 requires one reviewed change to the decoder and capability
+matrix, explicit migration into a typed model, fixtures for every
+version/status/capability row, and boundary tests for freshness, evidence,
+generation identity, incremental reuse, semantic sessions, and shared
+publication. A future version must remain visible as unsupported until those
+conditions are met; silently treating it as v3 is a compatibility failure.

package/docs/RUST_DURABLE_SESSION_PROTOCOL.md

@@ -0,0 +1,126 @@
+# Durable Rust session protocol
+
+The durable Rust session protocol is the versioned filesystem-message contract
+between a synchronous scip-query process and the reusable process that owns
+rust-analyzer. Its essential safety property is correlation: a result becomes
+usable only after it proves that it belongs to the same logical operation,
+mailbox namespace, and absolute processing interval that the caller admitted.
+
+A mailbox-session identity is a SHA-256 value derived from the absolute
+durable-session directory and protocol version. The directory is already
+separated by canonical project root and helper-binary fingerprint; the
+explicit identity makes that separation part of every message rather than an
+assumption inherited from its pathname. A response copied from another
+project or helper directory therefore cannot satisfy the reader.
+
+An operation identity is the SHA-256 of the answer-affecting Rust request. It
+is wider than one client attempt: an exact retry joins the same pending,
+inflight, or retained completion. The authoritative deadline is the absolute
+deadline on the first admitted copy of that operation. Later duplicate
+attempts read that retained deadline instead of substituting a new one, which
+keeps the response's time identity stable.
+
+## Current v3 request
+
+Every newly written request contains:
+
+- `mailboxVersion: 1`;
+- `protocolVersion: 3`;
+- `operationKey` and the derived `id: op-<operationKey>`;
+- a non-empty `clientId`;
+- finite `enqueuedAtMs` and `deadlineAtMs`, where their difference equals the
+  domain request's positive `timeoutMs`;
+- `sessionIdentity`; and
+- one strictly decoded `semantic` or `import-definitions` request.
+
+The server recomputes the operation key after validating the domain request.
+It validates all required definition or import-position fields and every
+optional timeout, concurrency, boolean, and worker-environment field before
+calling rust-analyzer. Unknown additive object members remain allowed; a
+wrong discriminant or wrong field type does not.
+
+The civil timestamps are shared-record facts, not wait-loop clocks. Before
+work, the server requires `now <= deadlineAtMs`. After rust-analyzer returns,
+it checks the absolute deadline again and publishes an `expired-request`
+rejection instead of a success if work crossed it. Process-local waits and
+readiness bounds continue to use monotonic time as documented in
+[Time Semantics](TIME_SEMANTICS.md).
+
+## Current v3 response
+
+The bounded mailbox supplies `mailboxVersion`, `operationKey`, `clientId`,
+`completedAtMs`, `expiresAtMs`, and the authoritative `deadlineAtMs`. The Rust
+server adds:
+
+- `protocolVersion: 3`;
+- the request `id`;
+- the request's `sessionIdentity`;
+- either `ok: true`, the session disposition, and the kind-specific response;
+  or
+- `ok: false`, a typed `errorCode`, and a diagnostic message.
+
+The client accepts a success or rejection only when protocol version, mailbox
+version, request ID, operation key, session identity, and authoritative
+deadline exactly match its admitted operation. It also requires both the
+helper's completion time and the observation time to be no later than that
+deadline. A semantic request cannot consume an import-definition response,
+and the reverse is also rejected.
+
+Typed rejection codes are:
+
+| Code                   | Meaning                                                                 |
+| ---------------------- | ----------------------------------------------------------------------- |
+| `unsupported-protocol` | The outer lifecycle was correlatable, but its domain protocol is newer. |
+| `malformed-request`    | Identity, lifecycle, kind, or domain payload validation failed.         |
+| `expired-request`      | Work was already expired or crossed its deadline.                       |
+| `handler-error`        | A validated request reached the Rust host and the host failed.          |
+
+When malformed bytes do not contain enough trustworthy identity to correlate
+a response, the server still retains the rejection and dead-letter evidence,
+but a current client will not accept that uncorrelated file as the answer to a
+request.
+
+## Compatibility matrix
+
+| Writer or peer                        | Current reader/server behavior                                                                                                                                     |
+| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Unversioned v2 `{id, request}` client | Accepted during the bounded overlap window after strict request-kind validation. The server synthesizes lifecycle/session fields and writes an additive response.  |
+| Prior v3 client without session field | Accepted as `prior-v3` when every other v3 lifecycle field and the recomputed operation identity are valid. The server supplies and echoes the namespace identity. |
+| Current v3 client                     | Fully correlated current path.                                                                                                                                     |
+| Future explicitly versioned client    | Never treated as legacy. A safely correlated envelope receives `unsupported-protocol`; an uncorrelatable envelope receives a generic malformed rejection.          |
+| Prior v3 server response              | Rejected by the current client as an incompatible uncorrelated response because it lacks session/deadline proof.                                                   |
+| Current v3 server to prior v3 client  | Additive: the prior reader's established root fields remain in place, and it ignores the new correlation/error metadata.                                           |
+| Future server response                | Rejected before response data is exposed.                                                                                                                          |
+
+The server executable's content fingerprint is part of the mailbox directory,
+so normal upgrades select a new namespace rather than pairing a current client
+with a prior helper. The explicit compatibility behavior still matters for
+retained files, rollback, test fixtures, and manual recovery.
+
+## Replay, retry, and recovery
+
+An exact retry keeps its operation key and request ID. The mailbox returns the
+deadline of the authoritative pending, inflight, or completed record, so the
+retry validates the original response rather than relabeling it with the new
+attempt's deadline. A retained completion observed after that deadline is
+rejected even if the retention window has not yet collected its bytes.
+
+A response with the right request ID but a different operation key is a
+different operation. A response with both identities but a different
+mailbox-session identity is a cross-session replay. A response whose echoed
+deadline differs is a different lifecycle. All three fail before payload
+decoding.
+
+If the current client reports an incompatible prior-server response, remove
+only the specific durable session directory after confirming no current
+scip-query process owns it, then retry so the content-addressed current helper
+creates a fresh namespace. Do not delete a broad cache root.
+
+Executable coverage lives in:
+
+- `tests/semantic/rust/durable-session-protocol.test.ts`;
+- `tests/semantic/rust/rust-durable-session.test.ts`; and
+- `tests/storage/bounded-mailbox.test.ts`.
+
+The shared pending/inflight/completion state machine remains documented in
+[Filesystem mailbox lifecycle](MAILBOX_LIFECYCLE.md).

package/docs/SECURITY_MODEL.md

@@ -0,0 +1,129 @@
+# Security model
+
+`scip-query` treats a checked-out repository and every index or cache artifact
+derived from it as untrusted input. An untrusted checkout is a directory of
+files supplied by a project rather than by the operator, distinguished from
+ordinary input by being able to name source paths, tools, configuration, and
+persisted records that the CLI may otherwise be tempted to trust.
+
+The default analysis boundary may read repository files and managed,
+rebuildable cache artifacts. It may not execute a repository-selected program,
+adopt an arbitrary cache directory, escape the canonical project root through
+an indexed path, or install a mutable tool identity merely because the
+checkout requests it.
+
+## Explicit authority
+
+An authority grant is an operator action that permits one otherwise-forbidden
+effect, distinguished from project configuration by coming from the current
+CLI invocation:
+
+- `--trust-project-tools` permits reviewed repository-local indexer
+  executables for that invocation.
+- `--tla-tools <jar>` selects a reviewed TLA jar explicitly. The default
+  downloaded TLA artifact remains version- and SHA-256-pinned.
+- `--install-missing` permits installation of the exact immutable tool
+  identities listed by the command. Without it, missing tools produce setup
+  instructions instead of a mutation.
+- environment overrides such as `SCIP_QUERY_CACHE_DIR` are operator-owned
+  storage choices. Tracked `.scipquery.json` cannot grant destructive authority
+  over an arbitrary external cache.
+
+A managed cache is rebuildable local state rooted under a scip-query-owned
+directory, distinguished from a caller-selected directory by an ownership
+record and canonical containment checks. Destructive cache operations require
+both properties.
+
+## Input budgets
+
+An input budget is a maximum amount of untrusted data one operation will
+materialize or ask a parser to compile, distinguished from silent truncation by
+failing explicitly with the input kind, observed amount, and accepted limit.
+
+| Input class                                            |                    Limit | Representative users                                                     |
+| ------------------------------------------------------ | -----------------------: | ------------------------------------------------------------------------ |
+| Config, manifest, lock, lease, and other small records |                    8 MiB | `.scipquery.json`, package manifests, generation metadata, hook settings |
+| One source or per-document fragment                    |                   64 MiB | indexed source reads, TypeScript/Vue snapshots, mailbox result payloads  |
+| One SCIP index artifact                                |                  512 MiB | merge, sanitize, Rust occurrence fallback, shared-generation hydration   |
+| Profile or retained JSONL artifact                     |                  256 MiB | profiling audits, legacy event ledgers, rotating diagnostic segments     |
+| Generated TLA trace                                    | 16 MiB and 100,000 steps | `tla instrument` recorder                                                |
+| Repository-supplied regular-expression pattern         |         4,096 characters | entry-root patterns and TLA statement bindings                           |
+| Agent hook standard input                              |                    8 MiB | stop-hook request payload                                                |
+
+Regular files are opened first, checked through that descriptor, read, and
+checked again so replacement or growth cannot bypass the pre-read limit.
+Streams and pseudo-files are counted while reading. Large file fingerprints
+are computed in fixed-size chunks rather than by retaining the complete file.
+
+An oversized artifact is not partially interpreted. Operations that require
+the complete artifact fail and name the limit; query-level detectors that
+intentionally sample or cap logical rows retain their existing machine-readable
+coverage metadata and `--full` remediation.
+
+## Complete command output
+
+Output pagination is a resumable transport for one rendered command result,
+distinguished from a result limit by preserving every rendered character.
+Human output above 12,000 characters automatically includes the exact command
+for the next page. Large default JSON remains byte-compatible and writes the
+exact opt-in paging command to stderr before and after the JSON stream.
+
+Every command accepts:
+
+```text
+--output-page-size <characters>
+--output-cursor <cursor>
+```
+
+The opaque cursor binds the command, working directory, non-pagination
+arguments, next character offset, private output snapshot, and complete output
+hash. The first invocation streams the output to current-user temporary
+storage with private permissions while retaining only one page in memory.
+Continuations read that immutable snapshot instead of re-running a command
+whose timestamps or durations may change. Snapshots expire after one hour and
+are removed after the final page; an unavailable snapshot produces the exact
+page-one restart command.
+
+Continue until `page.complete` is `true`. Do not pipe output through `head`,
+`tail`, or a line-range `sed`; those programs discard data without creating a
+resumable position. Every incomplete JSON page repeats this obligation in
+`agentInstruction`; an incomplete page must not support a conclusion or
+completion claim. See [CLI JSON output](CLI_JSON_OUTPUT.md) and the
+[output-page schema](schemas/cli-output-page.schema.json).
+
+## Diff-gate process containment
+
+A diff-gate lease is one live process identity permitted to evaluate a
+project's current diff, distinguished from an ordinary lock file by recording
+and validating the owner's PID, process start, and project identity. CLI and
+Stop-hook gates share this lease, so an overlapping request reports the live
+owner instead of multiplying detector work.
+
+The public gate owns an isolated child process. Its default deadline is 60
+seconds, or 180 seconds with `--full`. On timeout, failure, or interruption the
+parent terminates and reaps that child before releasing the lease. An operator
+may set `SCIP_QUERY_DIFF_GATE_TIMEOUT_MS` to a positive millisecond value; the
+runtime caps it at 10 minutes.
+
+Outcome reconciliation rechecks at most one historical comparison base per
+gate. Deferred bases remain open and the result reports their exact base and
+finding counts, so accumulated event history cannot silently multiply one
+foreground gate into an unbounded replay batch.
+
+## Terminal and JSON output
+
+Human terminal output is presentation text written to an interactive terminal,
+distinguished from JSON by being able to activate terminal control sequences.
+Repository-controlled human fields therefore have control bytes rendered
+inert. JSON values retain their printable data and are protected by JSON
+encoding rather than human-output rewriting.
+
+Registry and release diagnostics redact URL user information and
+credential-bearing query fields before rendering nested errors.
+
+## Reporting a security issue
+
+Do not include a live credential, private repository, or destructive
+proof-of-concept target in a public report. Describe the affected command,
+platform, trust flags, and smallest disposable reproduction needed to
+distinguish the boundary failure.

package/docs/TELEMETRY_RETENTION.md

@@ -0,0 +1,77 @@
+# Operational Telemetry Retention
+
+Operational telemetry is observational history produced while scip-query
+works. Its concrete referents here are `reindex-activity.jsonl`, which records
+refresh frequency and estimated logical output, and `affected-shadow.jsonl`,
+which records compact affected-set calibration outcomes. It differs from an
+authoritative record because no index generation, policy decision, ownership
+claim, or completed user operation is reconstructed from it.
+
+## Retained segment set
+
+Each history has two ordered segments:
+
+1. `<history>.previous` is the older retained segment.
+2. `<history>` is the current append segment.
+
+The reindex-activity limit is 1 MiB per segment. The affected-shadow limit is
+8 MiB per segment. A single JSON record larger than its configured limit
+expands that segment's effective limit to the complete record size; a
+successful append is never made immediately ineligible by its own size.
+Rotation removes the former previous segment, renames the complete current
+segment to previous, and then creates the new current segment. Readers scan
+previous before current.
+
+Retention deliberately permits deletion of records older than those two
+segments. That bounded-history deletion is different from concurrency loss:
+which segment is pruned is decided while the append/rotation lock excludes
+other writers.
+
+The retained-set reader also has a 256 MiB byte budget across the files it
+opens. It validates each segment as the same regular file before and after the
+read. Exceeding that operational budget is an explicit read failure; it does
+not silently return a prefix and present it as complete telemetry.
+
+## Serialization and crash recovery
+
+The rotation lock is `<history>.rotation.lock`. It is a process-instance lock:
+its record contains a PID, an operating-system process-start identity when
+available, and a random token, and only that owner can release it. Lock
+acquisition, incomplete-tail repair, retention pruning, rename, append, and
+the default retained-set read all occur beneath that lock. Waits use a
+process-local monotonic two-second budget.
+
+Every append is one newline-terminated JSON value. If a process stops during
+the append, the current segment can end in one incomplete tail. The next
+writer truncates only the bytes after the last newline before it rotates or
+appends. Readers ignore and count incomplete tail bytes. A stop after current
+is renamed but before the new append leaves the old complete segment at
+`.previous`; a later writer creates a new current segment.
+
+These files are process-visible but not crash-durable. They are not fsynced,
+so a kernel or power failure may lose the newest observation even after the
+write call returned. That is acceptable only because the histories are
+operational evidence. Authoritative index, lock, suppression, configuration,
+mailbox-response, and release records use their own durable protocols.
+
+## Failure contract
+
+Lock timeout and ownership-changed release are typed failures at the shared
+helper boundary. The reindex-activity and affected-shadow production writers
+remain best effort: they catch telemetry failure so an observation cannot
+change the authoritative reindex result. Callers that invoke the shared helper
+directly can distinguish lock timeout, count repaired tail bytes, and count
+partial bytes ignored during reads.
+
+The regression suite forces:
+
+- a competing writer at tail repair, prior-segment pruning, current rotation,
+  and append;
+- a stop after rotation and a partial append tail;
+- bounded retention across three rotations;
+- deterministic previous-then-current reads;
+- partial legacy tails; and
+- bounded live-owner contention.
+
+The implementation is `src/reindex/rotating-jsonl.ts`; its direct contract
+tests are `tests/reindex/rotating-jsonl.test.ts`.