codebaseiq 0.1.0

package/docs/DURABILITY.md

@@ -0,0 +1,159 @@
+# Filesystem Publication and Durability
+
+This document classifies codebaseiq's file-backed state by the guarantee a
+writer must provide. The classification is part of the storage contract: a
+caller must choose a writer whose failure semantics match the record's role.
+
+## Guarantees
+
+A **visibility-atomic replacement** is a filesystem publication operation that
+writes complete bytes to a private staging file and renames that file over the
+target. The rename is the decisive characteristic: concurrent readers can
+observe the old complete file or the new complete file, but never the
+writer's partial staging bytes. It does not promise that the new bytes or
+directory entry survive power loss.
+
+A **crash-durable replacement** is a visibility-atomic replacement that also
+flushes the complete staging file before rename and flushes the containing
+directory after rename. When the writer creates a missing directory path, it
+also creates each component from the nearest existing ancestor outward and
+flushes the directory that names each new component. Those ordered flushes are
+what make acknowledged file contents and the complete path that reaches them
+recoverable after an operating-system or machine crash, subject to the host
+filesystem and device honoring their flush contract.
+
+An **authoritative record** is file-backed state whose loss can change which
+generation, owner, policy, or accepted decision the program treats as current.
+Its causal role in choosing later behavior makes silent rollback unsafe, so it
+uses crash-durable replacement.
+
+A **rebuildable record** is file-backed state whose loss can cost time,
+diagnostics, or a retry but cannot make unverified facts authoritative. Because
+another computation or bounded request can reconstruct it, complete
+old-or-new visibility is sufficient.
+
+A **directory flush** is a request to persist a directory entry—the mapping
+from a filename to its file—after the file itself has been flushed. Flushing
+only file contents is insufficient because a crash can otherwise lose the
+rename that made those contents current.
+
+An **achieved durability result** is the guarantee established by the
+operations that actually completed, not the guarantee the caller requested.
+`visibility` means only old-or-new complete process visibility;
+`file-flushed` means the final file bytes were flushed but at least one
+directory namespace could not be synchronized; and `directory-durable` means
+the file plus every newly created ancestor and final directory entry crossed
+the supported persistence frontier.
+
+## APIs
+
+| API                                                                        | Staging identity                                | File flush            | Rename/link                       | New-ancestor flush  | Final-directory flush |
+| -------------------------------------------------------------------------- | ----------------------------------------------- | --------------------- | --------------------------------- | ------------------- | --------------------- |
+| `writeJsonAtomic` / `replaceFileAtomic(..., { durability: "visibility" })` | Exclusive random token                          | No                    | Yes                               | No                  | No                    |
+| `writeJsonDurable` / `replaceFileAtomic(..., { durability: "durable" })`   | Exclusive random token                          | Yes                   | Yes                               | Yes where supported | Yes where supported   |
+| `createFileAtomicExclusive(..., { durability: "durable" })`                | Exclusive random token                          | Yes                   | Exclusive hard link               | Yes where supported | Yes where supported   |
+| `cloneFileDurable`                                                         | Final target inside an unpublished staging tree | Yes, after final mode | Caller publishes the staging tree | Yes where supported | Yes where supported   |
+
+`writeJsonAtomic` retains its original `void` return contract for compatibility.
+`writeJsonDurable`, `replaceFileAtomic`, and `createFileAtomicExclusive`
+return `requestedDurability`, `achievedDurability`, and `directorySync`.
+Callers must use the achieved fields when logging or forwarding a durability
+claim. Verified binary installation owns an equivalent platform-local
+flush/rename sequence because the enforced architecture forbids dependencies
+between the sibling `platform` and `storage` boundaries.
+
+On Windows, Node can reject attempts to open or flush a directory handle.
+Known Windows "directory handles unsupported" errors produce
+`directorySync: "unsupported"` after the staged file itself has been flushed
+and renamed. That result means complete visibility plus flushed file contents,
+not the full POSIX directory-entry durability guarantee. Other directory-sync
+errors remain failures. An API that returns only a path or value makes no
+machine-crash claim; protocol APIs that acknowledge work expose the achieved
+result explicitly.
+
+## Failure Outcomes
+
+| Failure point                       | Target visible after return/throw                               | Owned staging file               |
+| ----------------------------------- | --------------------------------------------------------------- | -------------------------------- |
+| Exclusive create                    | Previous target                                                 | No owned file was created        |
+| Write or short/invalid progress     | Previous target                                                 | Closed and removed               |
+| File flush                          | Previous target                                                 | Closed and removed               |
+| Rename                              | Previous target                                                 | Removed by the writer            |
+| POSIX directory flush               | New complete target; durability unconfirmed and the call throws | Already renamed; no staging path |
+| Unsupported Windows directory flush | New complete target; result reports the limitation              | Already renamed; no staging path |
+
+When a parent chain is new, a failure while creating or flushing any ancestor
+also throws before a durable result is returned. The incomplete path can be
+visible to the running process, but recovery may discard every component that
+has not yet been named by a flushed parent.
+
+The post-rename failure row is deliberately explicit. Once rename succeeds,
+rolling back would be another publication with its own crash window. The
+writer therefore leaves the verified new value visible and reports that it
+could not confirm the stronger durability guarantee.
+
+## Call-Site Classification
+
+| Record                                               | Contract                                                           | Reason                                                                                                                                                                                         |
+| ---------------------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Reindex `meta.json`                                  | Durable                                                            | Names the accepted index status, fingerprint, and generation metadata                                                                                                                          |
+| SQLite generation manifest and `state.json`          | Directory-durable where supported                                  | Durably establishes the generation root, flushes final read-only artifact bytes and metadata, publishes the immutable directory, then selects it for new readers                               |
+| Shared generation                                    | Directory-durable where supported                                  | Flushes every read-only artifact and manifest, the staging namespace, the final generation rename, and the shared `generations/` namespace                                                     |
+| Worktree lease and local cache pointer               | Durable                                                            | Protect generations from collection and bind a worktree to repository cache identity; generation changes, liveness touches, and cleanup serialize through the repository-cache lock            |
+| Watch service state                                  | Durable                                                            | Publishes the process instance and index generation accepted as current                                                                                                                        |
+| Rust semantic session `server.json`                  | Durable                                                            | Publishes the live server process and mailbox identity                                                                                                                                         |
+| Project `.codebaseiq.json`                            | Durable                                                            | Controls indexing, watch, architecture, and detector policy; versioned reads reject unsupported meaning before use, and authorized writes migrate legacy bytes without dropping unknown fields |
+| Codex/Claude hook JSON                               | Durable                                                            | Controls whether and when agent hooks execute                                                                                                                                                  |
+| Structured suppression file                          | Durable                                                            | Records an accepted finding and its reason                                                                                                                                                     |
+| Health baseline                                      | Durable                                                            | Acts as a committed regression policy                                                                                                                                                          |
+| Verified binary cache promotion                      | Durable                                                            | Makes checksum-accepted executable or tool bytes current                                                                                                                                       |
+| TypeScript/Rust request and response mailboxes       | Directory-durable where supported                                  | Admission is acknowledged only after the pending name is flushed; a new owner directory is durably named before a request moves from pending to inflight; responses precede claim release      |
+| Watch activity                                       | Visibility-atomic                                                  | A newer timestamp supersedes an older idle-lifetime observation; it carries no durable intent                                                                                                  |
+| Watch refresh request, claim, and completion records | Durable immutable admission and acknowledgement                    | Accepted intent survives activity replacement and owner crashes; exclusive claims may be recovered only by the next lock owner                                                                 |
+| Cache ownership credential                           | Directory-durable where supported                                  | A complete file is flushed privately, hardening is revalidated, the public credential is linked exclusively, and its directory is flushed before detailed publication success                  |
+| npm release state                                    | Directory-durable where supported; file-flushed on bounded hosts   | The registry is freshly reconciled and remains external authority; the local coordinate-pair recovery record reports its exact achieved guarantee                                              |
+| TypeScript fragment/overlay manifests                | Visibility-atomic                                                  | Content-addressed cache artifacts are validated and rebuildable                                                                                                                                |
+| Repository GC state                                  | Visibility-atomic                                                  | Sweep history can be reconstructed conservatively                                                                                                                                              |
+| Affected-set shadow latest record                    | Visibility-atomic                                                  | Calibration telemetry does not control publication                                                                                                                                             |
+
+Process locks use exclusive descriptor creation rather than replacement. Their
+durable token ownership, malformed-creation grace, guarded recovery, and
+legacy compatibility are defined in
+[Process Lock Ownership and Recovery](LOCK_PROTOCOL.md). Generation directories,
+stable compatibility mirrors, and their crash ordering are defined in
+[Local Index Generations](INDEX_GENERATIONS.md). Durable watch demand,
+idempotency, claim recovery, and acknowledgement ordering are defined in
+[Watch Refresh Requests](WATCH_REFRESH_REQUESTS.md).
+
+The outcome-event journal was retired in version `0.20.0` together with
+autonomous completion state. Current codebaseiq neither writes nor reads those
+records and therefore makes no current durability claim for them. Git history
+retains the removed repository records; deprecated record-shape types remain
+temporarily available only for consumer-owned historical data. See
+[Workflow API retirement compatibility](api/compatibility/2026-08-09-workflow-retirement.md).
+
+## Executable crash model
+
+`tests/helpers/persistence-frontier.ts` models the two states a normal
+filesystem test otherwise conflates. Writes, links, renames, and removals first
+change process-visible state. File synchronization advances persisted inode
+bytes and mode; directory synchronization advances persisted names. A modeled
+power loss discards everything beyond those frontiers.
+
+`tests/storage/atomic-file-crash.test.ts` replays replacement, exclusive
+publication, new ancestor creation, and durable artifact cloning after every
+relevant phase. Boundary-specific tests then hold the higher-level protocol
+order:
+
+- local generation stage and same-size corruption tests;
+- shared-generation artifact, manifest, staging, rename, and final-directory
+  stage tests;
+- watch admission, claim, completion, and retry tests;
+- mailbox owner-directory, claim-transfer, response, and recovery tests;
+- ownership short-write, unsupported-sync, and failed-sync tests; and
+- release-state synchronized, unsupported, and failed-write tests.
+
+The deterministic model proves the filesystem primitive’s persisted-state
+semantics. The boundary tests prove each protocol invokes those primitives in
+the required order. Neither an exception callback alone nor a successful live
+filesystem read is described as a simulated power loss.

package/docs/INDEX_GENERATIONS.md

@@ -0,0 +1,167 @@
+# Local Index Generations
+
+`codebaseiq` treats `index.db`, `index.scip`, and `meta.json` as one local
+index generation. A local index generation is an accepted set of compiler
+artifacts whose database rows, SCIP occurrences, and metadata describe the
+same indexing result. Keeping those files associated is what lets a query
+attach one truthful identity to every row and downstream semantic request.
+
+A generation handle is the storage-owned reference retained by one
+`ScipDatabase`. It differs from a group of cache paths by resolving the
+published pointer once, opening the database beneath that immutable
+generation directory, and retaining the corresponding metadata bytes and SCIP
+path for the connection's entire lifetime.
+
+## Layout
+
+The worktree cache keeps this internal layout:
+
+```text
+<cache>/
+  index.db                         compatibility mirror
+  index.scip                       compatibility mirror
+  meta.json                        compatibility mirror
+  .codebaseiq-generations/
+    state.json                     atomic current-generation pointer
+    gc.json                        last bounded-collection observation
+    gc.lock                        collection/reader-admission coordinator
+    readers/
+      <pid>-<token>.json           live process-generation leases
+    <generation-sha256>/
+      manifest.json
+      index.db
+      index.scip
+      meta.json
+```
+
+The files inside a named generation directory are immutable. Their final
+read-only mode is applied before their final file synchronization, so the
+persisted inode contains both the accepted bytes and the protection metadata.
+The SHA-256
+identity incorporates the database bytes, SCIP bytes when present, and the
+generation-bearing metadata fields. `manifest.json` records the exact size and
+digest of each stored artifact. `state.json` names the accepted directory and
+is the only publication decision read by internal database consumers.
+Opening a named generation authenticates the database, SCIP, and metadata
+bytes against those digests; matching size alone is never sufficient. A
+bounded process-local validation cache is keyed by manifest identity plus
+stable device, inode, size, modification-time, and change-time evidence, so a
+same-size mutation invalidates the cached proof and is rehashed.
+
+The top-level files remain for the SCIP CLI, indexers, older codebaseiq
+versions, and external inspection. They are derived mirrors, not the internal
+read authority. A current codebaseiq process never combines rows from a
+generation directory with metadata or SCIP bytes freshly reread through those
+replaceable paths.
+
+## Publication and crash behavior
+
+Publication proceeds in this order:
+
+1. Retain the previously published stable artifacts as an immutable
+   generation when upgrading a legacy cache.
+2. Durably establish `.codebaseiq-generations/` from its nearest existing
+   ancestor when this is the cache's first generation.
+3. Copy-on-write clone or copy every accepted candidate artifact into a
+   private staging directory, apply its final read-only mode, and flush the
+   resulting inode.
+4. Flush the manifest and staging namespace, rename the complete staging
+   directory into its content-derived identity, and flush the generation-root
+   directory entry.
+5. Establish the versioned query layout after augmentation: add the
+   definition-only covering index, remove known redundant indexes only when
+   schema inspection proves a retained equivalent, and run `ANALYZE` so the
+   planner sees the final row distribution. Query-plan tests hold those
+   proofs.
+6. Durably replace `state.json` so new internal readers select the complete
+   new directory.
+7. Replace the three compatibility mirrors and durably record their file
+   identities for drift diagnostics.
+
+A crash before step 6 leaves the prior pointer authoritative. A crash after
+the pointer step leaves the new immutable generation authoritative even if one or more
+compatibility mirrors are old. `codebaseiq status` and freshness inspection
+report that mirror drift and a later refresh repairs it; database-backed
+queries remain generation-consistent throughout.
+
+Metadata-only refreshes create a new immutable generation and switch the same
+pointer. Old database handles retain their old metadata bytes. Result cursors
+and TypeScript semantic mailbox requests carry the handle identity, so a
+continuation or numeric symbol identifier from an older generation is rejected
+when a service has moved to a newer one.
+
+Current metadata also records `sqliteLayoutVersion`. A current-version
+metadata record that predates this field remains readable and can reuse its
+language SCIP shards, but it cannot take the whole-SQLite unchanged fast path.
+The next refresh applies query-layout maintenance to the stable mirror,
+publishes a new immutable generation and current layout metadata, and does not
+rerun unchanged language indexers. Later unchanged refreshes resume the
+ordinary whole-generation reuse path.
+
+Mailbox protocol version 3 additionally binds that generation identity into a
+content-derived logical-operation key. The deterministic request ID lets a
+retry join an already pending, inflight, or retained completed operation,
+while the service still recomputes the key and rejects a path, payload, or
+generation mismatch. See
+[`MAILBOX_LIFECYCLE.md`](MAILBOX_LIFECYCLE.md) for ownership, expiry, limits,
+and legacy overlap.
+
+## Legacy overlap and retention
+
+A cache without `state.json` remains readable through a bounded legacy
+open-and-file-identity recheck. A state record written by the earlier local
+generation implementation, which did not name an immutable artifact set, also
+uses this compatibility path. New publications upgrade either layout without
+deleting the stable files.
+
+Published local generation directories are retained by a bounded,
+reader-aware collector. Opening an immutable `ScipDatabase` first acquires the
+generation GC lock, resolves the current pointer, and visibility-atomically
+publishes a lease containing its random token, generation identity, PID, and
+operating-system process-start identity when available. Closing the database
+removes that exact lease idempotently.
+
+Publication and collection use the same token-owned process lock. Collection
+always protects the current generation, the named recovery generation, and
+every live reader generation. It removes only oldest unprotected generations
+until both default limits hold: at most eight retained generations and at most
+2 GiB of logical artifact bytes. A lease whose recorded process instance is
+provably dead is reclaimed. A malformed lease protects every generation and
+defers deletion; uncertainty cannot authorize reclamation.
+
+`gc.json` is a visibility-atomic diagnostic containing retained counts, logical
+bytes, protected/readers counts, removed counts, limits, state, time, and any
+reason. `codebaseiq status` reports that observation together with the current
+retained set and oldest generation age. Collection failure does not invalidate
+the accepted index; it remains explicit bounded-storage debt for the operator.
+
+The repository-wide shared generation store is a separate cache layer. It
+warms a worktree from a complete content-addressed generation. Publication
+flushes each final read-only artifact, the manifest, the staging namespace, the
+rename into `generations/<identity>`, and `generations/` itself before
+returning `directory-durable` on supported hosts. On a host without directory
+sync, it returns `file-flushed`; readers still rehash every artifact and reject
+an incomplete generation. The local publisher then creates the private
+immutable directory and pointer described here. Later worktree writes cannot
+mutate either the shared source or a retained local reader.
+
+## Shared worktree lease invariant
+
+A worktree lease is a repository-cache reachability record whose generation
+IDs keep an immutable shared generation from collection while that worktree
+uses it. Its essential ownership fields bind the repository, worktree,
+project path, and local cache path through a checksum; `lastSeenAt` is only a
+liveness observation.
+
+Generation attachment, lease liveness touches, and repository cleanup
+serialize through one repository-cache lock. A touch may inspect the local
+pointer before waiting only to identify that lock. Once it owns the lock, it
+rereads the pointer and lease, validates the ownership checksum, current Git
+tree, local metadata fingerprint, generation IDs, and source artifacts, then
+merges only a newer `lastSeenAt`. It never writes a lease assembled from the
+pre-lock observation.
+
+Consequently, a touch waiting behind a new generation attaches to or rejects
+the new lease; it cannot restore the old generation. A deleted lease stays
+deleted, a recreated lease with different ownership stays intact, and a touch
+whose clock is behind another completed touch cannot move liveness backward.

package/docs/LOCK_PROTOCOL.md

@@ -0,0 +1,133 @@
+# Process Lock Ownership and Recovery
+
+codebaseiq uses one process-lock protocol for watch ownership, reindex
+publication, repository-cache garbage collection, shared-generation builds,
+verified-binary fetches, and the durable Rust semantic server.
+
+A **process lock** is a filesystem ownership record created exclusively at one
+resource path. Exclusive creation makes competing processes observe one
+winner; the recorded random token and operating-system process-start identity
+distinguish that winner from later processes and later lock records that reuse
+the same PID or pathname.
+
+A **process instance** is one execution occupying an operating-system PID
+slot. The PID locates the slot, while the process-start identity distinguishes
+successive executions that occupy it. A PID by itself is therefore accepted
+for conservative legacy inspection but never treated as proof that a live
+process is the original owner.
+
+A **reclaim guard** is a second exclusive, token-owned process lock at
+`<lock>.reclaim`. It serializes recovery attempts so that only one process may
+remove an unchanged abandoned record. The guard does not make an ambiguous
+live owner safe to remove.
+
+## Current Record
+
+New writers emit protocol version 1:
+
+```json
+{
+  "protocol": "codebaseiq-process-lock",
+  "version": 1,
+  "kind": "reindex",
+  "pid": 41001,
+  "token": "random-owner-token",
+  "processIdentity": {
+    "version": 1,
+    "pid": 41001,
+    "platform": "linux",
+    "startToken": "9147752"
+  },
+  "startedAt": "2026-07-25T19:00:00.000Z",
+  "detail": {
+    "projectRoot": "/repo"
+  }
+}
+```
+
+`kind` identifies the resource protocol using the lock. `detail` carries only
+that protocol's diagnostic metadata; it is not ownership evidence. Ownership
+is the combination of PID, random token, and process identity when the host can
+obtain one.
+
+The creator writes the complete record to a token-unique private candidate,
+flushes and closes that candidate, then publishes the complete inode under the
+public lock name with an exclusive hard link. The link either creates the
+public name or reports that another owner already has it; it never replaces an
+owner. A crash before the link can leave a private candidate, but it cannot
+expose an empty or truncated public lock. Successful publication removes the
+private name and flushes the containing directory where supported.
+
+## Observation and Recovery
+
+| Observed state                                                     | Recovery decision                                                       |
+| ------------------------------------------------------------------ | ----------------------------------------------------------------------- |
+| Current record; process instance is live and matches               | Contended; never remove                                                 |
+| Current record; PID is dead                                        | Reclaim after guarded unchanged recheck                                 |
+| Current record; PID is live but its process-start identity differs | Reclaim the old record without signaling the new PID occupant           |
+| Current record; PID is live but identity cannot be read            | Contended; fail closed                                                  |
+| Supported legacy record; PID is live                               | Contended because the process instance cannot be verified               |
+| Supported legacy record; PID is dead                               | Reclaim after guarded unchanged recheck                                 |
+| Empty, truncated, or malformed public record                       | Contended; fail closed because civil-clock age cannot identify an owner |
+| Record changes before the guarded recheck                          | Do not remove; report contention                                        |
+| Empty or malformed reclaim guard                                   | Do not remove                                                           |
+| Valid reclaim guard whose process instance is dead                 | Recover the guard, then retry once                                      |
+
+The unchanged recheck compares the original bytes, device, inode, size, and
+modification time. Recovery is limited to one retry: persistent ambiguity
+remains contention instead of becoming an unbounded delete loop.
+
+Malformed public records from a legacy writer, external damage, or manual
+editing require operator review. A civil timestamp is not causal ownership
+evidence: moving the system clock forward cannot prove which process created
+the bytes or whether that process still owns the resource.
+
+## Release
+
+Release rereads the current record and removes it only when PID and token still
+match the owner's retained record. When the owner recorded a process identity,
+that identity must match as well. A missing path, malformed record, legacy
+record, token mismatch, or successor record makes release a no-op. Successful
+removal flushes the containing directory where supported so an acknowledged
+release does not resurrect the prior directory entry after a crash.
+
+This protocol does not expire live ownership and therefore does not require a
+monotonically increasing fencing token. A **fencing token** is an ordered
+ownership value checked by the protected resource before it accepts a write;
+it is necessary when an old live owner can outlast a lease and attempt
+publication after a newer owner. codebaseiq instead refuses to steal a
+verifiably live lock.
+
+## Legacy Compatibility
+
+Readers retain narrow decoders for the prior formats:
+
+- watch and reindex JSON ownership records;
+- generic repository-cache and shared-build `{ "pid": ... }` records;
+- the durable Rust semantic server's numeric PID record.
+
+New writers always emit the current common format. A live legacy owner remains
+contended because its process instance cannot be proven. A dead legacy owner
+can be reclaimed. Unsupported or malformed JSON is never guessed into a
+legacy owner.
+
+## Diagnostics and Manual Recovery
+
+Successful reindex recovery reports that it recovered an abandoned lock. A
+manual refresh that encounters an already-stale
+watcher-owned reindex record retains the existing “preempting watcher refresh”
+diagnostic while stating that the prior owner was already stale.
+
+For other lock users, normal recovery is silent and contention reports the
+resource-specific lock path. If a lock remains contended:
+
+1. inspect the JSON without editing it;
+2. verify whether its PID and process-start identity name the current process;
+3. for a malformed record, establish ownership outside codebaseiq and move the
+   record aside rather than asking wall-clock age to authorize deletion;
+4. retry the operation so the guarded recovery path can run for attributable
+   dead owners.
+
+Do not manually delete a valid lock whose process instance is live. If host
+permissions prevent process-identity verification, stop the owning process or
+move the record aside only after establishing ownership outside codebaseiq.