codebaseiq 0.1.0

package/docs/TELEMETRY_RETENTION.md

@@ -0,0 +1,84 @@
+# Operational Telemetry Retention
+
+Operational telemetry is observational history produced while codebaseiq
+works. Its concrete referents here are `reindex-activity.jsonl`, which records
+refresh frequency and estimated logical output, and `affected-shadow.jsonl`,
+which records compact affected-set calibration outcomes. It differs from an
+authoritative record because no index generation, policy decision, ownership
+claim, or completed user operation is reconstructed from it.
+
+## Retained segment set
+
+Each history has two ordered segments:
+
+1. `<history>.previous` is the older retained segment.
+2. `<history>` is the current append segment.
+
+The reindex-activity limit is 1 MiB per segment. The affected-shadow limit is
+8 MiB per segment. A single JSON record larger than its configured limit
+expands that segment's effective limit to the complete record size; a
+successful append is never made immediately ineligible by its own size.
+Rotation removes the former previous segment, renames the complete current
+segment to previous, and then creates the new current segment. Readers scan
+previous before current.
+
+Retention deliberately permits deletion of records older than those two
+segments. That bounded-history deletion is different from concurrency loss:
+which segment is pruned is decided while the append/rotation lock excludes
+other writers.
+
+The retained-set reader also has a 256 MiB byte budget across the files it
+opens. It validates each segment as the same regular file before and after the
+read. Exceeding that operational budget is an explicit read failure; it does
+not silently return a prefix and present it as complete telemetry.
+
+## Serialization and crash recovery
+
+The rotation lock is `<history>.rotation.lock`. It is a process-instance lock:
+its record contains a PID, an operating-system process-start identity when
+available, and a random token, and only that owner can release it. Lock
+acquisition, incomplete-tail repair, retention pruning, rename, append, and
+the default retained-set read all occur beneath that lock. Waits use a
+process-local monotonic two-second budget.
+
+Every append is one newline-terminated JSON value. If a process stops during
+the append, the current segment can end in one incomplete tail. The next
+writer truncates only the bytes after the last newline before it rotates or
+appends. Readers ignore and count incomplete tail bytes. A stop after current
+is renamed but before the new append leaves the old complete segment at
+`.previous`; a later writer creates a new current segment.
+
+These files are process-visible but not crash-durable. They are not fsynced,
+so a kernel or power failure may lose the newest observation even after the
+write call returned. That is acceptable only because the histories are
+operational evidence. Authoritative index, lock, suppression, configuration,
+mailbox-response, and release records use their own durable protocols.
+
+## Failure contract
+
+Lock timeout and ownership-changed release are typed failures at the shared
+helper boundary. The reindex-activity and affected-shadow production writers
+remain best effort: telemetry failure cannot change the authoritative reindex
+result. A reindex-activity writer returns either `recorded` or `failed` with a
+reason; reindex, foreground watch, and daemon watch surface that failure as a
+warning or watch-state error rather than silently losing the observation.
+
+A retained-set summary carries an activity-confidence state. `complete` means
+every selected non-empty record decoded; `partial` means a valid subset was
+used while exact invalid/skipped/read-error or incomplete-tail counts are
+reported; `unavailable` means no reliable summary could be formed. The summary
+also exposes records read and ignored partial-tail bytes, so consumers can
+distinguish zero activity from missing evidence.
+
+The regression suite forces:
+
+- a competing writer at tail repair, prior-segment pruning, current rotation,
+  and append;
+- a stop after rotation and a partial append tail;
+- bounded retention across three rotations;
+- deterministic previous-then-current reads;
+- partial legacy tails; and
+- bounded live-owner contention.
+
+The implementation is `src/reindex/rotating-jsonl.ts`; its direct contract
+tests are `tests/reindex/rotating-jsonl.test.ts`.

package/docs/TIME_SEMANTICS.md

@@ -0,0 +1,77 @@
+# Time Semantics
+
+codebaseiq separates shared timestamps from local elapsed-time control so a
+system-clock correction cannot prolong a wait or impersonate ownership.
+
+A **civil timestamp** is a calendar coordinate produced by the host's system
+clock and written so different processes and later executions can compare
+records. ISO dates, `enqueuedAtMs`, `deadlineAtMs`, heartbeat times, retention
+times, and diagnostic start/completion times refer to this clock. Network time
+synchronization, administrator changes, suspend/resume behavior, virtual
+machine migration, and a reboot can move its observed value forward or
+backward.
+
+A **monotonic clock reading** is a process-local elapsed-time coordinate whose
+ordering does not move backward while that process runs. `performance.now()`
+is the JavaScript implementation used here. A monotonic reading is suitable
+for durations and local deadlines because a civil-clock adjustment cannot add
+time to or remove time from the measured interval. Its numeric value is not a
+portable timestamp and is never persisted as cross-process evidence.
+
+A **deadline** is the boundary that ends an operation when its resource budget
+is consumed. Local request waits, lock waits, service startup and stop waits,
+Rust readiness work, heartbeat throttles, activity polling, cache sweeps,
+idle shutdown, and reported elapsed durations use monotonic readings. Timer
+delivery can be late if the process is not scheduled, but moving the civil
+clock cannot make the deadline later.
+
+A **durable expiry hint** is a persisted civil timestamp that lets another
+process classify old work for bounded cleanup or rejection. It is weaker than
+ownership evidence because timestamp age cannot identify the process that
+created a record. Mailbox request deadlines, response retention, dead-letter
+retention, and heartbeat times remain durable hints. They may reject or clean
+rebuildable retained data according to the documented retention contract, but
+they do not alone authorize signaling a process, replacing a live service, or
+reclaiming an inflight claim.
+
+## Decision table
+
+| Decision                                                                 | Clock/evidence used                                                                                 | Why                                                                                 |
+| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
+| Stop waiting for a lock, response, startup, shutdown, or readiness phase | Process-local monotonic deadline                                                                    | The wait remains bounded across forward and backward civil-clock jumps.             |
+| Report a duration                                                        | Difference between monotonic readings                                                               | A clock correction cannot create a negative or inflated elapsed duration.           |
+| Publish a heartbeat, start time, completion time, or request expiry      | Civil timestamp                                                                                     | Other processes and later executions need an inspectable shared coordinate.         |
+| Accept a watch or mailbox service as the same owner                      | Protocol/project identity, live PID, and matching process-start identity when recorded              | A timestamp cannot distinguish PID reuse or a different executable instance.        |
+| Replace or signal a watch service                                        | Matching process-start identity plus the explicit stop/start protocol                               | An old heartbeat alone never grants process-signal authority.                       |
+| Reclaim an inflight mailbox claim                                        | Lease expired and recorded process instance is dead or replaced                                     | The civil lease is a delay hint; process-instance loss supplies ownership evidence. |
+| Reclaim a process lock                                                   | Dead recorded process or mismatched process-start identity, followed by a guarded unchanged recheck | Valid locks do not expire; malformed public locks fail closed.                      |
+
+## Cross-process handoff
+
+Absolute monotonic readings are not sent between independent processes.
+Portable mailbox requests carry relative timeout budgets and civil diagnostic
+times. The receiving Rust service converts the relative budget into a new
+deadline in its own monotonic clock domain before it hands work to its worker
+thread. TypeScript and Rust clients likewise use their own monotonic deadline
+while polling for a retained response.
+
+Legacy state without a process-start identity remains readable. A live legacy
+PID is treated conservatively: it can support availability where no process
+mutation is attempted, but it cannot authorize a signal or deletion. An
+unparseable public ownership record fails closed and requires operator review.
+
+## Verification
+
+The clock contract is covered by injected-clock tests that independently move
+civil time forward and backward, advance monotonic time, reuse a PID with a
+different process-start identity, and hold a live owner beyond a civil lease.
+Focused suites include:
+
+- `tests/domain/time.test.ts`;
+- `tests/platform/process-file-lock.test.ts`;
+- `tests/runtime/repository-cache-lifecycle.test.ts`;
+- `tests/runtime/revisioned-file.test.ts`;
+- `tests/runtime/watch-service.test.ts`;
+- `tests/storage/bounded-mailbox.test.ts`;
+- the TypeScript semantic and index mailbox suites; and
+- the durable Rust and rust-analyzer readiness suites.

package/docs/WATCH_REFRESH_REQUESTS.md

@@ -0,0 +1,123 @@
+# Watch Refresh Requests
+
+## Contract
+
+A watch refresh request is a durable command-side intent to make one project's
+index current. Its concrete referents are the records created when a command or
+agent hook observes a stale index and asks the live watch service to reindex.
+It is a request record distinguished from ordinary watch activity by remaining
+authoritative until a corresponding refresh attempt completes or its declared
+deadline expires.
+
+Watch activity is a disposable observation that a command recently used the
+service. Its concrete referent is `watch-activity.json`; it is a
+last-writer-wins timestamp whose only effect is extending the daemon's idle
+lifetime. It is not a queue and carries no current refresh authority.
+
+An admission is the successful publication of one complete immutable request
+under `watch-refresh-requests/requests/`. Its defining property is that the
+stable request path exists before the caller is told the request was accepted.
+A claim is an exclusive processing marker for that admitted request. A
+completion is a durable receipt proving that the attempt associated with the
+claim finished successfully, or that the request expired before execution.
+
+On hosts that support directory synchronization, these records provide
+at-least-once execution across process, operating-system, and machine failure
+for an accepted, unexpired request. At-least-once execution means a crash after
+reindex success but before the completion receipt may repeat the safe reindex
+operation; it never means an acknowledged request can disappear. On Windows
+hosts where Node cannot synchronize directory handles, admission reports
+`achievedDurability: "file-flushed"` and `directorySync: "unsupported"`:
+process-crash recovery remains intact, but the stronger machine-crash
+namespace guarantee is not claimed.
+
+## Layout
+
+```text
+<project-cache>/
+  watch-activity.json
+  watch-refresh-requests/
+    requests/<request-id>.json
+    claims/<request-id>.json
+    completions/<request-id>.json
+    staging/
+```
+
+- `requests/` is the immutable admission log.
+- `claims/` contains exclusive, temporary ownership markers.
+- `completions/` contains immutable acknowledgements.
+- `staging/` contains same-filesystem temporary files used to publish a
+  complete record through an exclusive hard-link operation.
+
+The request ID is random unless the caller supplies an idempotency key. An
+idempotency key is a caller-chosen logical-operation identity whose hash
+becomes the stable request ID; retries with that key observe the first admitted
+request instead of appending another one. The first request's detail and
+deadline remain authoritative.
+
+## Lifecycle
+
+1. The requester durably establishes the protocol root, `staging/`, and the
+   target record directory from the nearest existing ancestor.
+2. It writes and flushes a complete staging record.
+3. It exclusively links that record into `requests/`. A collision is a
+   duplicate, not an overwrite.
+4. It flushes `requests/` before returning accepted and reports the achieved
+   platform guarantee.
+5. The watch server, while holding the project watch lock, claims pending,
+   unexpired requests through exclusive claim records.
+6. It deliberately coalesces one claimed batch into one `watch-demand`
+   refresh. Requests arriving while the watcher is busy remain pending.
+7. A successful corresponding reindex writes every completion receipt before
+   removing any claim.
+8. A failed reindex removes its claims without writing completion, making the
+   requests pending after a bounded retry delay.
+9. A successor watch server removes predecessor claims only after acquiring
+   the same exclusive watch lock. Completed requests remain completed; all
+   other claimed requests become pending.
+
+The default request deadline is ten minutes. An expired request gets an
+explicit `expired` completion rather than being silently deleted. Completed
+and expired history is retained for seven days; pruning removes only a receipt
+and its already-acknowledged request. Pending and claimed requests are never
+history-pruned. Idempotency is therefore guaranteed for at least the history
+retention window.
+
+## Crash and Concurrency Outcomes
+
+| Boundary                                         | Observable outcome                                                                      |
+| ------------------------------------------------ | --------------------------------------------------------------------------------------- |
+| Crash before exclusive admission                 | No request was accepted; only staging may remain                                        |
+| Crash after admission before caller sees success | Request remains pending; retrying the same idempotency key returns the admitted request |
+| Two callers use one idempotency key              | Exactly one immutable request wins                                                      |
+| Two callers use distinct keys                    | Both requests remain independently visible and may be coalesced deliberately            |
+| Crash after claim before reindex                 | Successor clears the stale claim and retries the request                                |
+| Reindex failure                                  | Claim is released; request remains pending                                              |
+| Crash after reindex before completion            | Request may execute again; it is never lost                                             |
+| Crash after completion before claim removal      | Completion prevents replay; successor removes the obsolete claim                        |
+| Activity writer races any row above              | Activity changes only `watch-activity.json` and cannot modify request state             |
+
+The first two rows are machine-crash claims only when admission returned
+`directory-durable`. A `file-flushed` result is explicit evidence that the
+host could not commit the complete namespace frontier.
+
+## Compatibility and Diagnostics
+
+Protocol version 5 writers use the request store. During the overlap release,
+the watch server still recognizes the former
+`refreshRequestedAt`/`refreshDetail` fields if an older client writes them. On
+observation it converts that timestamp to a deduplicated durable request before
+processing it. New activity writers never place refresh intent in the activity
+file.
+
+`codebaseiq watch --status` reports pending, claimed, completed, expired, and
+invalid record counts. A nonzero invalid count means an on-disk record failed
+strict validation; the store does not treat malformed bytes as an accepted
+request or completion.
+
+This protocol does not promise exactly-once reindex execution. Exactly-once
+execution would require atomically committing the index generation and the
+request receipt across separate storage authorities. Reindex is safe to
+repeat, so the protocol chooses the stronger practical guarantee: accepted
+intent is never silently lost, and successful acknowledgement is never
+inferred from absence.