scip-query 0.19.4 → 0.19.6

package/docs/DURABILITY.md

@@ -0,0 +1,103 @@
+# Filesystem Publication and Durability
+
+This document classifies scip-query's file-backed state by the guarantee a
+writer must provide. The classification is part of the storage contract: a
+caller must choose a writer whose failure semantics match the record's role.
+
+## Guarantees
+
+A **visibility-atomic replacement** is a filesystem publication operation that
+writes complete bytes to a private staging file and renames that file over the
+target. The rename is the decisive characteristic: concurrent readers can
+observe the old complete file or the new complete file, but never the
+writer's partial staging bytes. It does not promise that the new bytes or
+directory entry survive power loss.
+
+A **crash-durable replacement** is a visibility-atomic replacement that also
+flushes the complete staging file before rename and flushes the containing
+directory after rename. Those ordered flushes are what make acknowledged file
+contents and the name that reaches them recoverable after an operating-system
+or machine crash, subject to the host filesystem and device honoring their
+flush contract.
+
+An **authoritative record** is file-backed state whose loss can change which
+generation, owner, policy, or accepted decision the program treats as current.
+Its causal role in choosing later behavior makes silent rollback unsafe, so it
+uses crash-durable replacement.
+
+A **rebuildable record** is file-backed state whose loss can cost time,
+diagnostics, or a retry but cannot make unverified facts authoritative. Because
+another computation or bounded request can reconstruct it, complete
+old-or-new visibility is sufficient.
+
+A **directory flush** is a request to persist a directory entry—the mapping
+from a filename to its file—after the file itself has been flushed. Flushing
+only file contents is insufficient because a crash can otherwise lose the
+rename that made those contents current.
+
+## APIs
+
+| API | Staging identity | File flush | Rename | Parent-directory flush |
+| --- | --- | --- | --- | --- |
+| `writeJsonAtomic` / `replaceFileAtomic(..., { durability: "visibility" })` | Exclusive random token | No | Yes | No |
+| `writeJsonDurable` / `replaceFileAtomic(..., { durability: "durable" })` | Exclusive random token | Yes | Yes | Yes where supported |
+
+`writeJsonAtomic` retains its original `void` return contract for compatibility.
+`writeJsonDurable` and `replaceFileAtomic` return the achieved directory-sync
+status. Verified binary installation owns an equivalent platform-local
+flush/rename sequence because the enforced architecture forbids dependencies
+between the sibling `platform` and `storage` boundaries.
+
+On Windows, Node can reject attempts to open or flush a directory handle.
+Known Windows "directory handles unsupported" errors produce
+`directorySync: "unsupported"` after the staged file itself has been flushed
+and renamed. That result means complete visibility plus flushed file contents,
+not the full POSIX directory-entry durability guarantee. Other directory-sync
+errors remain failures.
+
+## Failure Outcomes
+
+| Failure point | Target visible after return/throw | Owned staging file |
+| --- | --- | --- |
+| Exclusive create | Previous target | No owned file was created |
+| Write or short/invalid progress | Previous target | Closed and removed |
+| File flush | Previous target | Closed and removed |
+| Rename | Previous target | Removed by the writer |
+| POSIX directory flush | New complete target; durability unconfirmed and the call throws | Already renamed; no staging path |
+| Unsupported Windows directory flush | New complete target; result reports the limitation | Already renamed; no staging path |
+
+The post-rename failure row is deliberately explicit. Once rename succeeds,
+rolling back would be another publication with its own crash window. The
+writer therefore leaves the verified new value visible and reports that it
+could not confirm the stronger durability guarantee.
+
+## Call-Site Classification
+
+| Record | Contract | Reason |
+| --- | --- | --- |
+| Reindex `meta.json` | Durable | Names the accepted index status, fingerprint, and generation metadata |
+| SQLite generation manifest and `state.json` | Durable | Flushes one complete immutable artifact set, then atomically selects it for new readers |
+| Shared-generation manifest | Durable file within staging | Authenticates immutable artifacts; the later generation-directory publication remains a separate generation-store operation |
+| Worktree lease and local cache pointer | Durable | Protect generations from collection and bind a worktree to repository cache identity; generation changes, liveness touches, and cleanup serialize through the repository-cache lock |
+| Watch service state | Durable | Publishes the process instance and index generation accepted as current |
+| Rust semantic session `server.json` | Durable | Publishes the live server process and mailbox identity |
+| Project `.scipquery.json` | Durable | Controls indexing, watch, architecture, and detector policy; versioned reads reject unsupported meaning before use, and authorized writes migrate legacy bytes without dropping unknown fields |
+| Codex/Claude hook JSON | Durable | Controls whether and when agent hooks execute |
+| Structured suppression file | Durable | Records an accepted finding and its reason |
+| Health baseline | Durable | Acts as a committed regression policy |
+| Verified binary cache promotion | Durable | Makes checksum-accepted executable or tool bytes current |
+| TypeScript/Rust request and response mailboxes | Visibility-atomic | A timeout or retry reconstructs the ephemeral message |
+| Watch activity | Visibility-atomic | A newer timestamp supersedes an older idle-lifetime observation; it carries no durable intent |
+| Watch refresh request, claim, and completion records | Durable immutable admission and acknowledgement | Accepted intent survives activity replacement and owner crashes; exclusive claims may be recovered only by the next lock owner |
+| TypeScript fragment/overlay manifests | Visibility-atomic | Content-addressed cache artifacts are validated and rebuildable |
+| Repository GC state | Visibility-atomic | Sweep history can be reconstructed conservatively |
+| Affected-set shadow latest record | Visibility-atomic | Calibration telemetry does not control publication |
+
+Process locks use exclusive descriptor creation rather than replacement. Their
+durable token ownership, malformed-creation grace, guarded recovery, and
+legacy compatibility are defined in
+[Process Lock Ownership and Recovery](LOCK_PROTOCOL.md). Generation directories,
+stable compatibility mirrors, and their crash ordering are defined in
+[Local Index Generations](INDEX_GENERATIONS.md). Durable watch demand,
+idempotency, claim recovery, and acknowledgement ordering are defined in
+[Watch Refresh Requests](WATCH_REFRESH_REQUESTS.md).

package/docs/INDEX_GENERATIONS.md

@@ -0,0 +1,121 @@
+# Local Index Generations
+
+`scip-query` treats `index.db`, `index.scip`, and `meta.json` as one local
+index generation. A local index generation is an accepted set of compiler
+artifacts whose database rows, SCIP occurrences, and metadata describe the
+same indexing result. Keeping those files associated is what lets a query
+attach one truthful identity to every row and downstream semantic request.
+
+A generation handle is the storage-owned reference retained by one
+`ScipDatabase`. It differs from a group of cache paths by resolving the
+published pointer once, opening the database beneath that immutable
+generation directory, and retaining the corresponding metadata bytes and SCIP
+path for the connection's entire lifetime.
+
+## Layout
+
+The worktree cache keeps this internal layout:
+
+```text
+<cache>/
+  index.db                         compatibility mirror
+  index.scip                       compatibility mirror
+  meta.json                        compatibility mirror
+  .scipquery-generations/
+    state.json                     atomic current-generation pointer
+    <generation-sha256>/
+      manifest.json
+      index.db
+      index.scip
+      meta.json
+```
+
+The files inside a named generation directory are immutable. The SHA-256
+identity incorporates the database bytes, SCIP bytes when present, and the
+generation-bearing metadata fields. `manifest.json` records the exact size and
+digest of each stored artifact. `state.json` names the accepted directory and
+is the only publication decision read by internal database consumers.
+
+The top-level files remain for the SCIP CLI, indexers, older scip-query
+versions, and external inspection. They are derived mirrors, not the internal
+read authority. A current scip-query process never combines rows from a
+generation directory with metadata or SCIP bytes freshly reread through those
+replaceable paths.
+
+## Publication and crash behavior
+
+Publication proceeds in this order:
+
+1. Retain the previously published stable artifacts as an immutable
+   generation when upgrading a legacy cache.
+2. Copy-on-write clone or copy every accepted candidate artifact into a
+   private staging directory.
+3. Flush the artifact files and manifest, rename the complete staging
+   directory into its content-derived identity, and flush the generation
+   directory entry.
+4. Durably replace `state.json` so new internal readers select the complete
+   new directory.
+5. Replace the three compatibility mirrors and durably record their file
+   identities for drift diagnostics.
+
+A crash before step 4 leaves the prior pointer authoritative. A crash after
+step 4 leaves the new immutable generation authoritative even if one or more
+compatibility mirrors are old. `scip-query status` and freshness inspection
+report that mirror drift and a later refresh repairs it; database-backed
+queries remain generation-consistent throughout.
+
+Metadata-only refreshes create a new immutable generation and switch the same
+pointer. Old database handles retain their old metadata bytes. Result cursors
+and TypeScript semantic mailbox requests carry the handle identity, so a
+continuation or numeric symbol identifier from an older generation is rejected
+when a service has moved to a newer one.
+
+Mailbox protocol version 3 additionally binds that generation identity into a
+content-derived logical-operation key. The deterministic request ID lets a
+retry join an already pending, inflight, or retained completed operation,
+while the service still recomputes the key and rejects a path, payload, or
+generation mismatch. See
+[`MAILBOX_LIFECYCLE.md`](MAILBOX_LIFECYCLE.md) for ownership, expiry, limits,
+and legacy overlap.
+
+## Legacy overlap and retention
+
+A cache without `state.json` remains readable through a bounded legacy
+open-and-file-identity recheck. A state record written by the earlier local
+generation implementation, which did not name an immutable artifact set, also
+uses this compatibility path. New publications upgrade either layout without
+deleting the stable files.
+
+Published local generation directories are retained conservatively. Automatic
+collection is intentionally disabled until a cross-process reader lease can
+prove that no surviving handle can later need the retained SCIP companion.
+This can temporarily consume more cache space, but it preserves the stronger
+rule that storage reclamation may never change or remove evidence owned by a
+live query.
+
+The repository-wide shared generation store is a separate cache layer. It
+warms a worktree by copying a complete generation into the worktree's private
+cache; the local publisher then creates the local immutable directory and
+pointer described here. Later worktree writes cannot mutate either the shared
+source or a retained local reader.
+
+## Shared worktree lease invariant
+
+A worktree lease is a repository-cache reachability record whose generation
+IDs keep an immutable shared generation from collection while that worktree
+uses it. Its essential ownership fields bind the repository, worktree,
+project path, and local cache path through a checksum; `lastSeenAt` is only a
+liveness observation.
+
+Generation attachment, lease liveness touches, and repository cleanup
+serialize through one repository-cache lock. A touch may inspect the local
+pointer before waiting only to identify that lock. Once it owns the lock, it
+rereads the pointer and lease, validates the ownership checksum, current Git
+tree, local metadata fingerprint, generation IDs, and source artifacts, then
+merges only a newer `lastSeenAt`. It never writes a lease assembled from the
+pre-lock observation.
+
+Consequently, a touch waiting behind a new generation attaches to or rejects
+the new lease; it cannot restore the old generation. A deleted lease stays
+deleted, a recreated lease with different ownership stays intact, and a touch
+whose clock is behind another completed touch cannot move liveness backward.

package/docs/LOCK_PROTOCOL.md

@@ -0,0 +1,133 @@
+# Process Lock Ownership and Recovery
+
+scip-query uses one process-lock protocol for watch ownership, reindex
+publication, repository-cache garbage collection, shared-generation builds,
+verified-binary fetches, and the durable Rust semantic server.
+
+A **process lock** is a filesystem ownership record created exclusively at one
+resource path. Exclusive creation makes competing processes observe one
+winner; the recorded random token and operating-system process-start identity
+distinguish that winner from later processes and later lock records that reuse
+the same PID or pathname.
+
+A **process instance** is one execution occupying an operating-system PID
+slot. The PID locates the slot, while the process-start identity distinguishes
+successive executions that occupy it. A PID by itself is therefore accepted
+for conservative legacy inspection but never treated as proof that a live
+process is the original owner.
+
+A **reclaim guard** is a second exclusive, token-owned process lock at
+`<lock>.reclaim`. It serializes recovery attempts so that only one process may
+remove an unchanged abandoned record. The guard does not make an ambiguous
+live owner safe to remove.
+
+## Current Record
+
+New writers emit protocol version 1:
+
+```json
+{
+  "protocol": "scip-query-process-lock",
+  "version": 1,
+  "kind": "reindex",
+  "pid": 41001,
+  "token": "random-owner-token",
+  "processIdentity": {
+    "version": 1,
+    "pid": 41001,
+    "platform": "linux",
+    "startToken": "9147752"
+  },
+  "startedAt": "2026-07-25T19:00:00.000Z",
+  "detail": {
+    "projectRoot": "/repo"
+  }
+}
+```
+
+`kind` identifies the resource protocol using the lock. `detail` carries only
+that protocol's diagnostic metadata; it is not ownership evidence. Ownership
+is the combination of PID, random token, and process identity when the host can
+obtain one.
+
+The creator writes the complete record to a token-unique private candidate,
+flushes and closes that candidate, then publishes the complete inode under the
+public lock name with an exclusive hard link. The link either creates the
+public name or reports that another owner already has it; it never replaces an
+owner. A crash before the link can leave a private candidate, but it cannot
+expose an empty or truncated public lock. Successful publication removes the
+private name and flushes the containing directory where supported.
+
+## Observation and Recovery
+
+| Observed state                                                     | Recovery decision                                                       |
+| ------------------------------------------------------------------ | ----------------------------------------------------------------------- |
+| Current record; process instance is live and matches               | Contended; never remove                                                 |
+| Current record; PID is dead                                        | Reclaim after guarded unchanged recheck                                 |
+| Current record; PID is live but its process-start identity differs | Reclaim the old record without signaling the new PID occupant           |
+| Current record; PID is live but identity cannot be read            | Contended; fail closed                                                  |
+| Supported legacy record; PID is live                               | Contended because the process instance cannot be verified               |
+| Supported legacy record; PID is dead                               | Reclaim after guarded unchanged recheck                                 |
+| Empty, truncated, or malformed public record                       | Contended; fail closed because civil-clock age cannot identify an owner |
+| Record changes before the guarded recheck                          | Do not remove; report contention                                        |
+| Empty or malformed reclaim guard                                   | Do not remove                                                           |
+| Valid reclaim guard whose process instance is dead                 | Recover the guard, then retry once                                      |
+
+The unchanged recheck compares the original bytes, device, inode, size, and
+modification time. Recovery is limited to one retry: persistent ambiguity
+remains contention instead of becoming an unbounded delete loop.
+
+Malformed public records from a legacy writer, external damage, or manual
+editing require operator review. A civil timestamp is not causal ownership
+evidence: moving the system clock forward cannot prove which process created
+the bytes or whether that process still owns the resource.
+
+## Release
+
+Release rereads the current record and removes it only when PID and token still
+match the owner's retained record. When the owner recorded a process identity,
+that identity must match as well. A missing path, malformed record, legacy
+record, token mismatch, or successor record makes release a no-op. Successful
+removal flushes the containing directory where supported so an acknowledged
+release does not resurrect the prior directory entry after a crash.
+
+This protocol does not expire live ownership and therefore does not require a
+monotonically increasing fencing token. A **fencing token** is an ordered
+ownership value checked by the protected resource before it accepts a write;
+it is necessary when an old live owner can outlast a lease and attempt
+publication after a newer owner. scip-query instead refuses to steal a
+verifiably live lock.
+
+## Legacy Compatibility
+
+Readers retain narrow decoders for the prior formats:
+
+- watch and reindex JSON ownership records;
+- generic repository-cache and shared-build `{ "pid": ... }` records;
+- the durable Rust semantic server's numeric PID record.
+
+New writers always emit the current common format. A live legacy owner remains
+contended because its process instance cannot be proven. A dead legacy owner
+can be reclaimed. Unsupported or malformed JSON is never guessed into a
+legacy owner.
+
+## Diagnostics and Manual Recovery
+
+Successful reindex recovery reports that it recovered an abandoned lock. A
+manual refresh that encounters an already-stale
+watcher-owned reindex record retains the existing “preempting watcher refresh”
+diagnostic while stating that the prior owner was already stale.
+
+For other lock users, normal recovery is silent and contention reports the
+resource-specific lock path. If a lock remains contended:
+
+1. inspect the JSON without editing it;
+2. verify whether its PID and process-start identity name the current process;
+3. for a malformed record, establish ownership outside scip-query and move the
+   record aside rather than asking wall-clock age to authorize deletion;
+4. retry the operation so the guarded recovery path can run for attributable
+   dead owners.
+
+Do not manually delete a valid lock whose process instance is live. If host
+permissions prevent process-identity verification, stop the owning process or
+move the record aside only after establishing ownership outside scip-query.

package/docs/MAILBOX_LIFECYCLE.md

@@ -0,0 +1,197 @@
+# Filesystem mailbox lifecycle
+
+scip-query uses three local filesystem mailboxes to carry work between a
+synchronous CLI process and a reusable service process:
+
+- TypeScript semantic queries sent to the watch service;
+- TypeScript document-emission requests sent to the watch service; and
+- Rust semantic queries sent to the durable rust-analyzer helper.
+
+A filesystem mailbox is a process-coordination queue whose units are complete
+files and whose essential safety property is that each accepted logical
+operation has one durable identity and one inspectable lifecycle across
+process failure. It differs from an arbitrary request directory because
+admission, ownership transfer, completion, expiry, and retention are explicit
+states rather than consequences of whichever process happens to delete a
+file.
+
+A logical operation is a requested computation identified by the SHA-256 of
+its answer-affecting protocol payload. That stable content identity is what
+makes two independently attempted requests units of the same operation:
+retries converge on one pending request, one inflight claim, or one retained
+completion instead of executing under unrelated random IDs. `clientId`
+identifies the attempt that first published the operation; it does not change
+the operation's meaning.
+
+## Layout and states
+
+Each mailbox root has this layout:
+
+```text
+<mailbox>/
+  pending/
+  inflight/<encoded-owner>/
+    .owner.json             # process-instance evidence for claim recovery
+  responses/
+  dead-letter/
+  requests/                 # legacy v2 overlap reads only
+  .admission.lock           # complete, exclusively published quota coordinator
+```
+
+| State     | Real file state                                                                  | Authority and transition                                                                     |
+| --------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
+| Pending   | `pending/<operation-id>.json`                                                    | The immutable request was admitted but no service owns it.                                   |
+| Inflight  | `inflight/<owner>/<request>.<claim-expiry>.claim` plus `.owner.json`             | Atomic rename transferred this request to one recorded process instance for a bounded lease. |
+| Completed | `responses/<operation-id>.json`                                                  | A response was durably and exclusively published before claim release.                       |
+| Rejected  | Error response, with a bounded record under `dead-letter/` when capacity permits | The service explicitly refused malformed, expired, oversized, mismatched, or failed work.    |
+| Expired   | Deadline has passed; the next bounded service batch rejects and collects it      | Expiry prevents abandoned requests from authorizing expensive work indefinitely.             |
+
+The `requests/` directory is not a second current queue. It exists so a new
+service can drain flat requests left by protocol v2. Current writers publish
+only to `pending/`; no migration renames or deletes legacy data pre-emptively.
+
+## Identity and protocol fields
+
+Every current request contains:
+
+- `mailboxVersion`, which versions the shared lifecycle fields;
+- the domain protocol version;
+- `operationKey`, the full SHA-256 logical-operation identity;
+- `id`, deterministically derived as `op-<operationKey>`;
+- `clientId`, the publishing attempt identity;
+- `enqueuedAtMs` and `deadlineAtMs`; and
+- the domain request plus its generation/session identity.
+
+The TypeScript semantic and TypeScript index protocols are version 3. The Rust
+durable-session protocol is also version 3. Current readers recompute the
+operation key, require the ID derived from it, and retain the existing
+generation/base-generation response checks. TypeScript services accept the
+supported flat version-2 request shape during the overlap window. The Rust
+service accepts its former unversioned `{id, request}` shape and the
+immediately prior v3 envelope without a mailbox-session identity; an
+explicitly versioned unknown Rust envelope is not treated as legacy.
+
+Responses repeat the domain protocol, request ID, operation key, completion
+time, authoritative request deadline, and retention expiry. The first
+response published for an operation is authoritative. An old owner that
+finishes after its lease was reclaimed cannot replace a newer response.
+Duplicate admission returns the deadline from that authoritative pending,
+inflight, or completed record, so a retry cannot relabel retained work with a
+new time identity. Rust responses additionally echo the mailbox-session
+identity and are accepted only when protocol, request, operation, session, and
+deadline all match. See
+[Durable Rust session protocol](RUST_DURABLE_SESSION_PROTOCOL.md).
+
+## Admission and bounds
+
+Admission is the act of adding one immutable operation after proving the
+mailbox remains within its retained-resource budget. A short-lived,
+token-checked admission file serializes the count-and-publish decision, which
+prevents two processes from both observing the last free slot and
+oversubscribing it. Its complete owner record is flushed under a private name
+and hard-linked exclusively into the public name; an ownerless or malformed
+public record fails closed and is never deleted from civil-clock age alone.
+
+Default bounds are:
+
+| Bound                              |                                                     Default |
+| ---------------------------------- | ----------------------------------------------------------: |
+| Retained files per mailbox         |                                                       1,024 |
+| Retained bytes per mailbox         |                                                     512 MiB |
+| One request or response            |                                                      64 MiB |
+| Work claimed per service-loop pass |                                                          16 |
+| Claim lease                        | 5 minutes, extended through request deadline plus 5 seconds |
+| Response/idempotency retention     |                                                  10 minutes |
+| Dead-letter retention              |                                                    24 hours |
+| Orphan temporary-file retention    |                                                    1 minute |
+| Maintenance actions per pass       |                                                          64 |
+
+`MailboxBackpressureError` is the typed overload result. Its code distinguishes
+`item-too-large`, `item-capacity`, `byte-capacity`, and `admission-busy`, and
+its status reports the observed retained state and configured limits. A
+duplicate logical operation is checked before capacity rejection, so a retry
+can join work that already owns the last slot.
+
+TypeScript semantic callers retain their established correctness fallback:
+service failure or backpressure selects the in-process provider. Index and
+Rust requester errors propagate to their existing higher-level fallback
+boundaries.
+
+## Ownership, crash recovery, and replay
+
+A claim is a time-bounded service ownership record made real by renaming one
+pending file into an owner-specific inflight directory. Rename is the
+ownership compare-and-set: only the process whose rename succeeds owns that
+file. The directory's owner record binds the random owner ID to a PID and,
+when the operating system exposes it, a process-start identity. A process-start
+identity is the operating-system fact that distinguishes successive
+executions occupying the same numeric PID slot.
+
+The recovery rules are:
+
+| Failure point                                         | Surviving evidence                               | Recovery                                                                                                               |
+| ----------------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
+| Client exits after admission                          | Pending file                                     | A later service batch claims it or explicitly expires it. The client does not delete shared work in `finally`.         |
+| Service exits before claim                            | Pending file                                     | Another owner can claim immediately.                                                                                   |
+| Service exits after claim, before response            | Inflight file, expiry, and dead process identity | After lease expiry and proof that the recorded process instance is gone, maintenance atomically returns it to pending. |
+| Service publishes response, then exits before release | Response plus inflight file                      | Maintenance treats the response as completion and removes the stale claim without re-execution.                        |
+| Old owner finishes after reclamation                  | Competing exclusive response publication         | The first response remains authoritative; the late owner cannot replace it.                                            |
+| Same logical request is retried                       | Same deterministic operation ID                  | The caller joins pending/inflight/completed state.                                                                     |
+| Malformed, expired, or oversized file                 | Claimed input                                    | The service emits an explicit error response and a bounded rejection record.                                           |
+
+These are crash-recoverable claim semantics with first-completion
+idempotency. Civil-clock lease expiry is only a durable recovery hint: it does
+not authorize reclamation while the recorded process instance is live or
+unverifiable. Work can be retried after both lease expiry and owner death. The
+answer-affecting operations are read-only or rebuildable, the response
+identity is stable, and exclusive completion remains a final defense against
+two observable answers.
+
+## Fairness and maintenance
+
+Pending work is ordered by `enqueuedAtMs`, then stable request identity.
+Legacy and malformed files fall back to filesystem modification time. Random
+UUID filename order no longer determines service priority.
+
+Each process call claims at most 16 files, and each maintenance call performs
+at most 64 removals or reclaims. Those caps are what let the watch and Rust
+server loops regain control to update heartbeats, observe stop signals, and
+run unrelated maintenance even when a mailbox was flooded.
+
+Maintenance:
+
+- reclaims expired inflight ownership only after the recorded process instance
+  is dead or its PID has been reused by a different process instance;
+- removes a claim whose response proves it already completed;
+- removes expired responses and dead-letter records;
+- removes abandoned atomic-write staging files after their retention window;
+- preserves non-expired pending and inflight work; and
+- never depends on a client being alive to finish cleanup.
+
+## Telemetry and diagnosis
+
+TypeScript watch state exposes a `mailbox` snapshot under both
+`typescriptSemantic` and `typescriptIndex`. Rust helper state exposes the same
+snapshot. It contains:
+
+- `pending`, `inflight`, `responses`, and `deadLetters`;
+- `invalid`;
+- `totalItems` and `totalBytes`; and
+- `oldestPendingAt` when pending work exists.
+
+This snapshot identifies current retained pressure; it is not a cumulative
+success counter. A rising pending count with a live heartbeat indicates
+service throughput pressure. Inflight work older than its valid lease is
+reclaimed on the next loop only when the owner record proves the process
+instance is gone. Responses are expected during the ten-minute idempotency
+window. Clock-domain rules for these records are documented in
+[Time Semantics](TIME_SEMANTICS.md).
+
+Focused contract coverage lives in:
+
+- `tests/storage/bounded-mailbox.test.ts`;
+- `tests/semantic/typescript/typescript-session-mailbox.test.ts`;
+- `tests/reindex/typescript-index-mailbox.test.ts`;
+- `tests/semantic/rust/durable-session-protocol.test.ts`;
+- `tests/semantic/rust/rust-durable-session.test.ts`; and
+- `tests/platform/watch-service-state.test.ts`.

package/docs/REINDEX_METADATA_COMPATIBILITY.md

@@ -0,0 +1,84 @@
+# Reindex Metadata Compatibility
+
+Reindex metadata is the persisted description in `meta.json` that identifies
+the source-input fingerprint, indexed languages, completeness, and optional
+reuse capabilities of one accepted index generation. It is a compatibility
+record rather than an authority by itself: a consumer still verifies the
+database, SCIP companion, immutable-generation state, or current source
+fingerprint required by its operation.
+
+`src/domain/reindex-metadata.ts` is the dependency-free decoding boundary. It
+classifies an input as:
+
+- `legacy`: a structurally valid version 2 record;
+- `supported`: a structurally valid current version 3 record;
+- `unsupported`: an integer version outside the readable range, with older or
+  future direction; or
+- `malformed`: invalid JSON, a non-object, a missing/non-integer version, or an
+  invalid field in a recognized version.
+
+The decoder shares only dependency-free object-record, timestamp,
+scalar-number, and string-or-null-record predicates in
+`src/domain/record-validation.ts`; moving those generic primitives out of
+individual decoders does not change any version or capability decision.
+
+No consumer may cast an unsupported or malformed record to the current model.
+The decoder returns the original accepted v2/v3 object, so an authorized
+best-effort update such as `lastRefresh` preserves unknown additive fields.
+Future records are never rewritten.
+
+## Version policy
+
+| Wire version | Decoder result | Read policy                                                | Write policy                                                 |
+| ------------ | -------------- | ---------------------------------------------------------- | ------------------------------------------------------------ |
+| 2            | `legacy`       | Supported by common capabilities; no v3 shard capabilities | Read-only compatibility; a rebuilt generation writes v3      |
+| 3            | `supported`    | Current model                                              | All newly published metadata                                 |
+| 4            | `unsupported`  | Reserved migration boundary; reported explicitly           | Never written until its migration and matrix are implemented |
+| Other        | `unsupported`  | Fail closed for reuse/publication                          | Never rewritten                                              |
+
+Recognized versions validate `status`, optional timestamp,
+requested/indexed language sets, skipped-language rows, SCIP companion state,
+and v3 shard maps. A fingerprint is an opaque JSON identity for evidence and
+stable-generation compatibility, preserving the pre-decoder v2/v3 contract;
+freshness and publication additionally require it to be an object. Language
+lists contain unique names from the same
+`SUPPORTED_LANGUAGES` catalog used by configuration.
+
+## Capability matrix
+
+A capability is a named permission to interpret a decoded record for one
+operation. It differs from version support because two consumers can accept
+the same version while intentionally requiring different completeness.
+
+| Capability                    | Required facts                                                                    | Partial status                            | v2  | v3  |
+| ----------------------------- | --------------------------------------------------------------------------------- | ----------------------------------------- | --- | --- |
+| `usableForQuery`              | Valid indexed-language set                                                        | Yes                                       | Yes | Yes |
+| `usableForEvidenceCache`      | Fingerprint value                                                                 | Yes; status is part of the cache key      | Yes | Yes |
+| `publishableGeneration`       | Complete status, object fingerprint, indexed-language set                         | No                                        | Yes | Yes |
+| `stableGenerationIdentity`    | Complete status, fingerprint, valid `updatedAt`; languages projected when present | No                                        | Yes | Yes |
+| `languageShardReuse`          | Valid v3 `languageFingerprints` map                                               | Yes, for individually successful shards   | No  | Yes |
+| `typescriptProjectShardReuse` | Valid v3 `typescriptProjectShards` map                                            | Yes, for individually successful projects | No  | Yes |
+
+Consumer-specific comparisons happen only after this matrix accepts the
+record:
+
+- freshness and unchanged-index reuse compare a publishable fingerprint and
+  sorted indexed languages with current inputs;
+- shared-generation publication additionally verifies the immutable artifact
+  set, project root, and database integrity;
+- evidence and TypeScript semantic cache keys accept complete or partial
+  records and include status, so partial and complete results cannot collide;
+- SQLite and TypeScript service generation identities use the same canonical
+  projection and reject records without stable-identity capability;
+- per-language and per-project reuse compare the decoded v3 shard
+  fingerprints with current fingerprints and require the corresponding shard
+  file to exist.
+
+## Evolution procedure
+
+Adding version 4 requires one reviewed change to the decoder and capability
+matrix, explicit migration into a typed model, fixtures for every
+version/status/capability row, and boundary tests for freshness, evidence,
+generation identity, incremental reuse, semantic sessions, and shared
+publication. A future version must remain visible as unsupported until those
+conditions are met; silently treating it as v3 is a compatibility failure.