scip-query 0.19.4 → 0.19.6

package/docs/RUST_DURABLE_SESSION_PROTOCOL.md

@@ -0,0 +1,126 @@
+# Durable Rust session protocol
+
+The durable Rust session protocol is the versioned filesystem-message contract
+between a synchronous scip-query process and the reusable process that owns
+rust-analyzer. Its essential safety property is correlation: a result becomes
+usable only after it proves that it belongs to the same logical operation,
+mailbox namespace, and absolute processing interval that the caller admitted.
+
+A mailbox-session identity is a SHA-256 value derived from the absolute
+durable-session directory and protocol version. The directory is already
+separated by canonical project root and helper-binary fingerprint; the
+explicit identity makes that separation part of every message rather than an
+assumption inherited from its pathname. A response copied from another
+project or helper directory therefore cannot satisfy the reader.
+
+An operation identity is the SHA-256 of the answer-affecting Rust request. It
+is wider than one client attempt: an exact retry joins the same pending,
+inflight, or retained completion. The authoritative deadline is the absolute
+deadline on the first admitted copy of that operation. Later duplicate
+attempts read that retained deadline instead of substituting a new one, which
+keeps the response's time identity stable.
+
+## Current v3 request
+
+Every newly written request contains:
+
+- `mailboxVersion: 1`;
+- `protocolVersion: 3`;
+- `operationKey` and the derived `id: op-<operationKey>`;
+- a non-empty `clientId`;
+- finite `enqueuedAtMs` and `deadlineAtMs`, where their difference equals the
+  domain request's positive `timeoutMs`;
+- `sessionIdentity`; and
+- one strictly decoded `semantic` or `import-definitions` request.
+
+The server recomputes the operation key after validating the domain request.
+It validates all required definition or import-position fields and every
+optional timeout, concurrency, boolean, and worker-environment field before
+calling rust-analyzer. Unknown additive object members remain allowed; a
+wrong discriminant or wrong field type does not.
+
+The civil timestamps are shared-record facts, not wait-loop clocks. Before
+work, the server requires `now <= deadlineAtMs`. After rust-analyzer returns,
+it checks the absolute deadline again and publishes an `expired-request`
+rejection instead of a success if work crossed it. Process-local waits and
+readiness bounds continue to use monotonic time as documented in
+[Time Semantics](TIME_SEMANTICS.md).
+
+## Current v3 response
+
+The bounded mailbox supplies `mailboxVersion`, `operationKey`, `clientId`,
+`completedAtMs`, `expiresAtMs`, and the authoritative `deadlineAtMs`. The Rust
+server adds:
+
+- `protocolVersion: 3`;
+- the request `id`;
+- the request's `sessionIdentity`;
+- either `ok: true`, the session disposition, and the kind-specific response;
+  or
+- `ok: false`, a typed `errorCode`, and a diagnostic message.
+
+The client accepts a success or rejection only when protocol version, mailbox
+version, request ID, operation key, session identity, and authoritative
+deadline exactly match its admitted operation. It also requires both the
+helper's completion time and the observation time to be no later than that
+deadline. A semantic request cannot consume an import-definition response,
+and the reverse is also rejected.
+
+Typed rejection codes are:
+
+| Code                   | Meaning                                                                 |
+| ---------------------- | ----------------------------------------------------------------------- |
+| `unsupported-protocol` | The outer lifecycle was correlatable, but its domain protocol is newer. |
+| `malformed-request`    | Identity, lifecycle, kind, or domain payload validation failed.         |
+| `expired-request`      | Work was already expired or crossed its deadline.                       |
+| `handler-error`        | A validated request reached the Rust host and the host failed.          |
+
+When malformed bytes do not contain enough trustworthy identity to correlate
+a response, the server still retains the rejection and dead-letter evidence,
+but a current client will not accept that uncorrelated file as the answer to a
+request.
+
+## Compatibility matrix
+
+| Writer or peer                        | Current reader/server behavior                                                                                                                                     |
+| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Unversioned v2 `{id, request}` client | Accepted during the bounded overlap window after strict request-kind validation. The server synthesizes lifecycle/session fields and writes an additive response.  |
+| Prior v3 client without session field | Accepted as `prior-v3` when every other v3 lifecycle field and the recomputed operation identity are valid. The server supplies and echoes the namespace identity. |
+| Current v3 client                     | Fully correlated current path.                                                                                                                                     |
+| Future explicitly versioned client    | Never treated as legacy. A safely correlated envelope receives `unsupported-protocol`; an uncorrelatable envelope receives a generic malformed rejection.          |
+| Prior v3 server response              | Rejected by the current client as an incompatible uncorrelated response because it lacks session/deadline proof.                                                   |
+| Current v3 server to prior v3 client  | Additive: the prior reader's established root fields remain in place, and it ignores the new correlation/error metadata.                                           |
+| Future server response                | Rejected before response data is exposed.                                                                                                                          |
+
+The server executable's content fingerprint is part of the mailbox directory,
+so normal upgrades select a new namespace rather than pairing a current client
+with a prior helper. The explicit compatibility behavior still matters for
+retained files, rollback, test fixtures, and manual recovery.
+
+## Replay, retry, and recovery
+
+An exact retry keeps its operation key and request ID. The mailbox returns the
+deadline of the authoritative pending, inflight, or completed record, so the
+retry validates the original response rather than relabeling it with the new
+attempt's deadline. A retained completion observed after that deadline is
+rejected even if the retention window has not yet collected its bytes.
+
+A response with the right request ID but a different operation key is a
+different operation. A response with both identities but a different
+mailbox-session identity is a cross-session replay. A response whose echoed
+deadline differs is a different lifecycle. All three fail before payload
+decoding.
+
+If the current client reports an incompatible prior-server response, remove
+only the specific durable session directory after confirming no current
+scip-query process owns it, then retry so the content-addressed current helper
+creates a fresh namespace. Do not delete a broad cache root.
+
+Executable coverage lives in:
+
+- `tests/semantic/rust/durable-session-protocol.test.ts`;
+- `tests/semantic/rust/rust-durable-session.test.ts`; and
+- `tests/storage/bounded-mailbox.test.ts`.
+
+The shared pending/inflight/completion state machine remains documented in
+[Filesystem mailbox lifecycle](MAILBOX_LIFECYCLE.md).

package/docs/TELEMETRY_RETENTION.md

@@ -0,0 +1,72 @@
+# Operational Telemetry Retention
+
+Operational telemetry is observational history produced while scip-query
+works. Its concrete referents here are `reindex-activity.jsonl`, which records
+refresh frequency and estimated logical output, and `affected-shadow.jsonl`,
+which records compact affected-set calibration outcomes. It differs from an
+authoritative record because no index generation, policy decision, ownership
+claim, or completed user operation is reconstructed from it.
+
+## Retained segment set
+
+Each history has two ordered segments:
+
+1. `<history>.previous` is the older retained segment.
+2. `<history>` is the current append segment.
+
+The reindex-activity limit is 1 MiB per segment. The affected-shadow limit is
+8 MiB per segment. A single JSON record larger than its configured limit
+expands that segment's effective limit to the complete record size; a
+successful append is never made immediately ineligible by its own size.
+Rotation removes the former previous segment, renames the complete current
+segment to previous, and then creates the new current segment. Readers scan
+previous before current.
+
+Retention deliberately permits deletion of records older than those two
+segments. That bounded-history deletion is different from concurrency loss:
+which segment is pruned is decided while the append/rotation lock excludes
+other writers.
+
+## Serialization and crash recovery
+
+The rotation lock is `<history>.rotation.lock`. It is a process-instance lock:
+its record contains a PID, an operating-system process-start identity when
+available, and a random token, and only that owner can release it. Lock
+acquisition, incomplete-tail repair, retention pruning, rename, append, and
+the default retained-set read all occur beneath that lock. Waits use a
+process-local monotonic two-second budget.
+
+Every append is one newline-terminated JSON value. If a process stops during
+the append, the current segment can end in one incomplete tail. The next
+writer truncates only the bytes after the last newline before it rotates or
+appends. Readers ignore and count incomplete tail bytes. A stop after current
+is renamed but before the new append leaves the old complete segment at
+`.previous`; a later writer creates a new current segment.
+
+These files are process-visible but not crash-durable. They are not fsynced,
+so a kernel or power failure may lose the newest observation even after the
+write call returned. That is acceptable only because the histories are
+operational evidence. Authoritative index, lock, suppression, configuration,
+mailbox-response, and release records use their own durable protocols.
+
+## Failure contract
+
+Lock timeout and ownership-changed release are typed failures at the shared
+helper boundary. The reindex-activity and affected-shadow production writers
+remain best effort: they catch telemetry failure so an observation cannot
+change the authoritative reindex result. Callers that invoke the shared helper
+directly can distinguish lock timeout, count repaired tail bytes, and count
+partial bytes ignored during reads.
+
+The regression suite forces:
+
+- a competing writer at tail repair, prior-segment pruning, current rotation,
+  and append;
+- a stop after rotation and a partial append tail;
+- bounded retention across three rotations;
+- deterministic previous-then-current reads;
+- partial legacy tails; and
+- bounded live-owner contention.
+
+The implementation is `src/reindex/rotating-jsonl.ts`; its direct contract
+tests are `tests/reindex/rotating-jsonl.test.ts`.

package/docs/TIME_SEMANTICS.md

@@ -0,0 +1,77 @@
+# Time Semantics
+
+scip-query separates shared timestamps from local elapsed-time control so a
+system-clock correction cannot prolong a wait or impersonate ownership.
+
+A **civil timestamp** is a calendar coordinate produced by the host's system
+clock and written so different processes and later executions can compare
+records. ISO dates, `enqueuedAtMs`, `deadlineAtMs`, heartbeat times, retention
+times, and diagnostic start/completion times refer to this clock. Network time
+synchronization, administrator changes, suspend/resume behavior, virtual
+machine migration, and a reboot can move its observed value forward or
+backward.
+
+A **monotonic clock reading** is a process-local elapsed-time coordinate whose
+ordering does not move backward while that process runs. `performance.now()`
+is the JavaScript implementation used here. A monotonic reading is suitable
+for durations and local deadlines because a civil-clock adjustment cannot add
+time to or remove time from the measured interval. Its numeric value is not a
+portable timestamp and is never persisted as cross-process evidence.
+
+A **deadline** is the boundary that ends an operation when its resource budget
+is consumed. Local request waits, lock waits, service startup and stop waits,
+Rust readiness work, heartbeat throttles, activity polling, cache sweeps,
+idle shutdown, and reported elapsed durations use monotonic readings. Timer
+delivery can be late if the process is not scheduled, but moving the civil
+clock cannot make the deadline later.
+
+A **durable expiry hint** is a persisted civil timestamp that lets another
+process classify old work for bounded cleanup or rejection. It is weaker than
+ownership evidence because timestamp age cannot identify the process that
+created a record. Mailbox request deadlines, response retention, dead-letter
+retention, and heartbeat times remain durable hints. They may reject or clean
+rebuildable retained data according to the documented retention contract, but
+they do not alone authorize signaling a process, replacing a live service, or
+reclaiming an inflight claim.
+
+## Decision table
+
+| Decision                                                                 | Clock/evidence used                                                                                 | Why                                                                                 |
+| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
+| Stop waiting for a lock, response, startup, shutdown, or readiness phase | Process-local monotonic deadline                                                                    | The wait remains bounded across forward and backward civil-clock jumps.             |
+| Report a duration                                                        | Difference between monotonic readings                                                               | A clock correction cannot create a negative or inflated elapsed duration.           |
+| Publish a heartbeat, start time, completion time, or request expiry      | Civil timestamp                                                                                     | Other processes and later executions need an inspectable shared coordinate.         |
+| Accept a watch or mailbox service as the same owner                      | Protocol/project identity, live PID, and matching process-start identity when recorded              | A timestamp cannot distinguish PID reuse or a different executable instance.        |
+| Replace or signal a watch service                                        | Matching process-start identity plus the explicit stop/start protocol                               | An old heartbeat alone never grants process-signal authority.                       |
+| Reclaim an inflight mailbox claim                                        | Lease expired and recorded process instance is dead or replaced                                     | The civil lease is a delay hint; process-instance loss supplies ownership evidence. |
+| Reclaim a process lock                                                   | Dead recorded process or mismatched process-start identity, followed by a guarded unchanged recheck | Valid locks do not expire; malformed public locks fail closed.                      |
+
+## Cross-process handoff
+
+Absolute monotonic readings are not sent between independent processes.
+Portable mailbox requests carry relative timeout budgets and civil diagnostic
+times. The receiving Rust service converts the relative budget into a new
+deadline in its own monotonic clock domain before it hands work to its worker
+thread. TypeScript and Rust clients likewise use their own monotonic deadline
+while polling for a retained response.
+
+Legacy state without a process-start identity remains readable. A live legacy
+PID is treated conservatively: it can support availability where no process
+mutation is attempted, but it cannot authorize a signal or deletion. An
+unparseable public ownership record fails closed and requires operator review.
+
+## Verification
+
+The clock contract is covered by injected-clock tests that independently move
+civil time forward and backward, advance monotonic time, reuse a PID with a
+different process-start identity, and hold a live owner beyond a civil lease.
+Focused suites include:
+
+- `tests/domain/time.test.ts`;
+- `tests/platform/process-file-lock.test.ts`;
+- `tests/runtime/repository-cache-lifecycle.test.ts`;
+- `tests/runtime/revisioned-file.test.ts`;
+- `tests/runtime/watch-service.test.ts`;
+- `tests/storage/bounded-mailbox.test.ts`;
+- the TypeScript semantic and index mailbox suites; and
+- the durable Rust and rust-analyzer readiness suites.

package/docs/WATCH_REFRESH_REQUESTS.md

@@ -0,0 +1,110 @@
+# Watch Refresh Requests
+
+## Contract
+
+A watch refresh request is a durable command-side intent to make one project's
+index current. Its concrete referents are the records created when a command or
+agent hook observes a stale index and asks the live watch service to reindex.
+It is a request record distinguished from ordinary watch activity by remaining
+authoritative until a corresponding refresh attempt completes or its declared
+deadline expires.
+
+Watch activity is a disposable observation that a command recently used the
+service. Its concrete referent is `watch-activity.json`; it is a
+last-writer-wins timestamp whose only effect is extending the daemon's idle
+lifetime. It is not a queue and carries no current refresh authority.
+
+An admission is the successful publication of one complete immutable request
+under `watch-refresh-requests/requests/`. Its defining property is that the
+stable request path exists before the caller is told the request was accepted.
+A claim is an exclusive processing marker for that admitted request. A
+completion is a durable receipt proving that the attempt associated with the
+claim finished successfully, or that the request expired before execution.
+
+These records provide at-least-once execution for an accepted, unexpired
+request. At-least-once execution means a crash after reindex success but before
+the completion receipt may repeat the safe reindex operation; it never means an
+unacknowledged request can disappear.
+
+## Layout
+
+```text
+<project-cache>/
+  watch-activity.json
+  watch-refresh-requests/
+    requests/<request-id>.json
+    claims/<request-id>.json
+    completions/<request-id>.json
+    staging/
+```
+
+- `requests/` is the immutable admission log.
+- `claims/` contains exclusive, temporary ownership markers.
+- `completions/` contains immutable acknowledgements.
+- `staging/` contains same-filesystem temporary files used to publish a
+  complete record through an exclusive hard-link operation.
+
+The request ID is random unless the caller supplies an idempotency key. An
+idempotency key is a caller-chosen logical-operation identity whose hash
+becomes the stable request ID; retries with that key observe the first admitted
+request instead of appending another one. The first request's detail and
+deadline remain authoritative.
+
+## Lifecycle
+
+1. The requester durably writes a complete staging record.
+2. It exclusively links that record into `requests/`. A collision is a
+   duplicate, not an overwrite.
+3. The watch server, while holding the project watch lock, claims pending,
+   unexpired requests through exclusive claim records.
+4. It deliberately coalesces one claimed batch into one `watch-demand`
+   refresh. Requests arriving while the watcher is busy remain pending.
+5. A successful corresponding reindex writes every completion receipt before
+   removing any claim.
+6. A failed reindex removes its claims without writing completion, making the
+   requests pending after a bounded retry delay.
+7. A successor watch server removes predecessor claims only after acquiring
+   the same exclusive watch lock. Completed requests remain completed; all
+   other claimed requests become pending.
+
+The default request deadline is ten minutes. An expired request gets an
+explicit `expired` completion rather than being silently deleted. Completed
+and expired history is retained for seven days; pruning removes only a receipt
+and its already-acknowledged request. Pending and claimed requests are never
+history-pruned. Idempotency is therefore guaranteed for at least the history
+retention window.
+
+## Crash and Concurrency Outcomes
+
+| Boundary | Observable outcome |
+| --- | --- |
+| Crash before exclusive admission | No request was accepted; only staging may remain |
+| Crash after admission before caller sees success | Request remains pending; retrying the same idempotency key returns the admitted request |
+| Two callers use one idempotency key | Exactly one immutable request wins |
+| Two callers use distinct keys | Both requests remain independently visible and may be coalesced deliberately |
+| Crash after claim before reindex | Successor clears the stale claim and retries the request |
+| Reindex failure | Claim is released; request remains pending |
+| Crash after reindex before completion | Request may execute again; it is never lost |
+| Crash after completion before claim removal | Completion prevents replay; successor removes the obsolete claim |
+| Activity writer races any row above | Activity changes only `watch-activity.json` and cannot modify request state |
+
+## Compatibility and Diagnostics
+
+Protocol version 5 writers use the request store. During the overlap release,
+the watch server still recognizes the former
+`refreshRequestedAt`/`refreshDetail` fields if an older client writes them. On
+observation it converts that timestamp to a deduplicated durable request before
+processing it. New activity writers never place refresh intent in the activity
+file.
+
+`scip-query watch --status` reports pending, claimed, completed, expired, and
+invalid record counts. A nonzero invalid count means an on-disk record failed
+strict validation; the store does not treat malformed bytes as an accepted
+request or completion.
+
+This protocol does not promise exactly-once reindex execution. Exactly-once
+execution would require atomically committing the index generation and the
+request receipt across separate storage authorities. Reindex is safe to
+repeat, so the protocol chooses the stronger practical guarantee: accepted
+intent is never silently lost, and successful acknowledgement is never
+inferred from absence.