@iowarp/clio-coder 0.3.0

package/docs/scientific-validation.md

@@ -0,0 +1,105 @@
+# Clio Coder Scientific Validation Contracts
+
+> [!TIP]
+> **Interactive Spec Available:** An interactive numerical tolerance calculator and HPC queue execution simulator is located at [docs/html/validation_blueprint.html](html/validation_blueprint.html) (Version: 0.3.0).
+
+Scientific software development cannot treat simple file presence as proof of correctness. A simulation script that crashes on rank 48, or writes out NetCDF arrays filled with `NaN`s, may still successfully write a file to the disk. 
+
+Clio Coder recognizes **scientific validation contract files** as an opt-in signal for a higher evidence bar. In v0.3.0, core Clio does not parse or enforce a scientific contract schema. The presence of `.clio-coder/validation.yaml`, `.clio-coder/validation.yml`, `validation.yaml`, `validation.yml`, or `VALIDATION.md` at the workspace root raises the default rigor level to `high`; the file contents are advisory material for developers, project agents, and external validators.
+
+The convention below is a recommended shape for scientific projects that need to document expected dimensions, attributes, numerical tolerances, scheduler context, and verification commands for scientific artifacts. Developed at the [Gnosis Research Center (GRC)](https://grc.iit.edu) at Illinois Tech as part of the NSF-funded scientific-software context (NSF Award [#2411318](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2411318)), this convention links execution metadata with physical output checks without claiming that the current harness executes those checks automatically.
+
+---
+
+## Validation Contract Convention
+
+A validation contract can be stored as YAML or Markdown. A custom or project-level agent (such as a local `scientific-validator` agent example under `.clio-coder/agents/`) or the developer can draft these files and commit them next to the research code. Clio core currently checks only for the documented filenames at the workspace root.
+
+### Example netCDF / Slurm validation contract:
+```yaml
+version: 1
+task: "Regenerate the regional climate output and confirm grid metadata."
+runtime:
+  kind: slurm
+  nodes: 4
+  ranks: 64
+  walltime: "01:30:00"
+  modules:
+    - "intel/2024"
+    - "openmpi/5.0"
+    - "netcdf-c/4.9"
+artifacts:
+  - path: out/region_west.nc
+    format: NetCDF
+    expected_dimensions:
+      time: 8760
+      lat: 360
+      lon: 720
+    expected_attributes:
+      Conventions: "CF-1.10"
+    numerical_tolerances:
+      relative: 1.0e-6
+    preserve: false
+  - path: ckpt/run-0042.chk
+    format: Checkpoint files
+    preserve: true
+validators:
+  - "ncdump -h out/region_west.nc"
+  - "python tools/check_grid.py out/region_west.nc"
+notes: |
+  The run is submitted with sbatch; queue exit status is not a completion check.
+  Re-run check_grid.py after job completion is observed.
+```
+
+### Suggested Fields:
+1. **`version`:** Set to `1` for project-local compatibility.
+2. **`runtime.kind`:** Document execution mode (`local`, `slurm`, `mpi`, or `other`).
+3. **`artifacts`:** List output files or directories the validation plan should inspect.
+4. **`preserve`:** Project convention for artifacts that should not be deleted by cleanup workflows. Clio's built-in protected-artifact guard is separate and is driven by live `protect_path` effects, not by this YAML field.
+5. **`validators`:** List shell commands or scripts a verifier should run to validate the generated files.
+
+---
+
+## Numerical Tolerances
+
+Comparing floating-point values in scientific computations must accommodate round-offs, hardware differences, and compiler optimizations. Project validators can document any tolerance vocabulary they enforce. A common convention is:
+
+| Tolerance Type | Formula / Check | Purpose |
+| :--- | :--- | :--- |
+| **`relative`** | $\frac{|val - ref|}{|ref|} \le relative$ | Fractional difference check. Crucial for scaling datasets. |
+| **`absolute`** | $|val - ref| \le absolute$ | Additive difference check. Used when reference value is close to `0`. |
+| **`ulp`** | $StepsBetween(val, ref) \le ulp$ | Unit in the Last Place. Measures floating-point representation steps. |
+
+> [!NOTE]
+> Clio core does not currently execute tolerance comparisons and does not apply a default numerical tolerance. Put defaults directly in project validators or contract text.
+
+---
+
+## Common Scientific Artifact Families
+
+The following labels are useful project conventions for validation contracts and reports. They are not a closed, core-enforced enum in v0.3.0:
+
+- **`HDF5` / `NetCDF` / `Zarr`:** Multi-dimensional scientific array files.
+- **`FITS`:** Flexible Image Transport System (used in astrophysics).
+- **`CSV` / `Parquet`:** Structured tabular data and datasets.
+- **`VTK and ParaView`:** Visualizations and mesh outputs.
+- **`Slurm job output`:** Standard logs emitted by Slurm queue managers.
+- **`MPI rank-sensitive tests`:** Diagnostic outputs matching multi-rank jobs.
+- **`Checkpoint files` / `Simulation restart artifacts`:** Stateful binary dumps.
+- **`Plots and generated figures`:** Output graphics (verified via path + checksum metadata).
+
+## HPC Schedulers and Validation Lifecycle
+
+Scheduler-driven runs require distinct validation handling compared to local unit tests:
+- **Queue status is not validation**: Checking if a Slurm command like `sbatch` exits successfully only proves that the Slurm scheduler accepted the job script. A good contract tells the verifier how to check actual simulation artifacts inside `out/` or `ckpt/` after job completion.
+- **Environment module loading**: The `runtime.modules` array can document the exact software stack dependencies (such as `intel/2024`, `openmpi/5.0`) that must be loaded before running the validators.
+- **HPC and Data Integration**: For large-scale allocations such as those at the Argonne Leadership Computing Facility (ALCF), projects can archive verification logs through their own storage or data-transfer workflow. Clio core does not manage Globus transfers.
+- **Validator execution**: In the current alpha version, contract validation is advisory. Quality/verification agents (such as the base `verifier` agent or custom project-level agents) read the contract to guide developers and write out verification receipts. Automated in-harness contract execution is not implemented yet.
+
+### How a Validation Contract Raises Session Rigor
+
+Clio Coder integrates scientific validation contracts directly into its safety model to raise the evidence standard automatically:
+- **Automatic Escalation**: At startup, Clio scans the workspace root. The presence of any validation contract (e.g. `.clio-coder/validation.yaml`, `.clio-coder/validation.yml`, `validation.yaml`, `validation.yml`, or `VALIDATION.md`) automatically escalates the session's rigor level from `normal` to `high`.
+- **High-Rigor Gate Requirements**: Once the rigor is raised to `high`, the finish gate is active. It engages on a settled `turn_end` only when the recent window contains successful workspace mutation evidence and no validation evidence or explicit limitation. The window is entries since the last user message, capped at 80 entries:
+  - Clio issues a `request_continuation` middleware effect to keep the session running.
+  - Clio injects a dynamic warning reminder (`HIGH_RIGOR_REVALIDATION_MESSAGE`) instructing the agent to run a verification command or to declare a limitation before it can conclude the turn.

package/docs/session-lifecycle.md

@@ -0,0 +1,156 @@
+# Session Lifecycle
+
+This document is the authoritative specification for Clio Coder interactive and headless session lifecycles, on-disk ledger structures, tree-based conversation branching, checkpoints, and recovery protocols in `v0.3.0`.
+
+Source implementations: `src/engine/session.ts` and `src/domains/session/`.
+
+---
+
+## 1. On-Disk Session Layout
+
+Sessions are persisted durably on disk under the platform state root (`clioStateDir()`, resolving to `~/.local/state/clio-coder` on Linux, `~/Library/Application Support/clio-coder/state` on macOS, or `%LOCALAPPDATA%\clio-coder\state` on Windows):
+
+```text
+<stateDir>/sessions/<cwdHash>/<sessionId>/
+  meta.json        # ClioSessionMeta JSON document
+  current.jsonl    # Append-only structured session event ledger
+  tree.json        # SessionTreeNode[] conversation tree graph
+```
+
+- `<cwdHash>`: First 16 hexadecimal characters of SHA-256 of canonical workspace root path (`createHash("sha256").update(resolve(cwd)).digest("hex").slice(0, 16)`).
+- `<sessionId>`: UUIDv7 generated at session initialization.
+
+---
+
+## 2. Session Metadata (`meta.json`)
+
+Session metadata is written atomically (`.tmp` + `fsyncSync` + `renameSync`) by `src/engine/session.ts:atomicWrite` and updated without closing via `src/domains/session/manager.ts:persistSessionMeta`.
+
+```typescript
+export interface ClioSessionMeta {
+  id: string;
+  cwd: string;
+  cwdHash: string;
+  createdAt: string;
+  endedAt: string | null;
+  model: string | null;
+  target: string | null;
+  clioVersion: string;
+  piMonoVersion: string;
+  platform: string;
+  nodeVersion: string;
+  sessionFormatVersion?: number; // CURRENT_SESSION_FORMAT_VERSION = 3
+}
+```
+
+Format version `CURRENT_SESSION_FORMAT_VERSION = 3` (`src/engine/session.ts:66`) is stamped on all sessions created in `v0.3.0`. Sessions with missing or earlier format versions trigger schema migrations in `src/domains/session/migrations/` on `/resume`.
+
+---
+
+## 3. Append-Only Context Ledger (`current.jsonl`)
+
+The session ledger `current.jsonl` records all conversation events, model turns, tool executions, and checkpoints in strict append-only order.
+
+### Header Line
+
+The first line of `current.jsonl` is the canonical session header:
+
+```json
+{"type":"session","version":3,"id":"01912a34-b567-7890-abcd-ef0123456789","timestamp":"2026-08-14T12:00:00.000Z","cwd":"/path/to/project"}
+```
+
+### Entry Taxonomy
+
+Subsequent lines represent typed `SessionEntry` objects (`src/domains/session/entries.ts`):
+
+1. **`message`**: User inputs, assistant responses, and tool calls/results.
+   ```typescript
+   export interface MessageEntry {
+     kind: "message";
+     turnId: string;
+     parentTurnId: string | null;
+     timestamp: string;
+     role: "user" | "assistant" | "tool_call" | "tool_result" | "system" | "checkpoint";
+     payload: unknown;
+   }
+   ```
+2. **`label`**: User-defined turn bookmark or tag anchored to `targetTurnId`.
+3. **`sessionInfo`**: Metadata event (such as model switch, target change, or thinking level adjustment).
+4. **`compactionSummary`**: Progressive compaction snapshot retaining historical context up to `firstKeptTurnId`.
+
+### Write Durability & Atomicity
+
+- Appends hold an open `O_APPEND` file descriptor across the writer lifetime (`src/engine/session.ts:openSync`).
+- Each line append is executed via a single `write(2)` call.
+- `fsyncSync` is debounced during high-frequency streaming turns and unconditionally forced on checkpoint (`persistTree`) and session shutdown (`close`).
+- Torn last lines resulting from abrupt system crashes or power losses are tolerated by the ledger reader (`src/engine/session.ts:readSessionFileEntries`), which logs a warning and skips the incomplete trailing record.
+
+---
+
+## 4. Conversation Tree Graph (`tree.json`) & Lineage
+
+Clio Coder tracks all conversation turns as a directed tree graph, enabling non-destructive branching, navigation, and message forking.
+
+### Tree Structure
+
+`tree.json` stores the complete node linkage:
+
+```typescript
+export interface SessionTreeNode {
+  id: string;              // UUIDv7 turn identifier
+  parentId: string | null; // UUIDv7 parent turn identifier
+  at: string;              // ISO-8601 creation timestamp
+  kind: "user" | "assistant" | "tool_call" | "tool_result" | "system" | "checkpoint";
+}
+```
+
+### Active Path Lineage Selection
+
+When an operator branches or switches turns using `/tree` or `Alt+T`, the next append point changes without mutating or deleting historical entries in `current.jsonl`.
+
+The active path filter (`src/domains/session/tree/active-path.ts:filterEntriesToActivePath`) traces ancestry back from the active leaf:
+1. Retains all message entries on the direct ancestral path from leaf to root.
+2. Retains sidecar entries anchored to turns on the active path (`targetTurnId`, `parentTurnId`, or `firstKeptTurnId`).
+3. Retains unanchored global sidecars (`parentTurnId: null`).
+4. Prunes abandoned sibling branches from the context window supplied to the LLM.
+
+### Branch Forking (`/fork`)
+
+The `/fork` command (`src/domains/session/tree/fork.ts:forkFromParentTurn`) initializes an independent session branched from an arbitrary turn:
+1. Closes the current session writer.
+2. Creates a new session directory and metadata inheriting `cwd`, `model`, and `target` from the parent.
+3. Traces ancestry up to `parentTurnId` and copies only the active path entries into the new session ledger.
+4. Stamps `parentSession` and `parentTurnId` in the new session header.
+
+---
+
+## 5. Session Resumption (`/resume`) & Working Directory Fallback
+
+When resuming a session via `/resume <sessionId>` or `CLIO_CODER_RESUME_SESSION_ID`:
+1. `src/domains/session/manager.ts:resumeSessionState` loads `meta.json` and runs migrations.
+2. `src/domains/session/cwd-fallback.ts:resolveSessionCwd` probes the recorded `meta.cwd` against the filesystem.
+3. If the directory is invalid, it returns a typed failure reason:
+   - `no-cwd`: `meta.cwd` is missing or empty.
+   - `missing`: The directory no longer exists on disk.
+   - `not-a-directory`: The path points to a non-directory file or broken symlink.
+4. The interactive layer displays the `cwd-fallback` overlay prompting the operator to choose a valid workspace directory.
+
+---
+
+## 6. Protected-Artifact Write-Ahead Journal
+
+To guarantee that protected artifacts and validation locks survive unexpected crashes between tool execution and ledger commitment, Clio Coder maintains a write-ahead journal (`src/domains/session/protected-artifact-journal.ts`):
+
+- Location: `<stateDir>/protected-artifact-pending/<sessionKey>/<recordId>.json`
+- Lifecycle:
+  1. **Stage**: Before a mutating tool result is returned, the protected artifact registration is staged atomically to the journal directory.
+  2. **Commit**: Once the turn completes and appends to `current.jsonl`, the staged journal file is unlinked.
+  3. **Reconcile**: During session startup or resume, `reconcilePendingProtectedArtifacts` reads any leftover staged records and injects them into the protection engine before accepting user commands.
+
+---
+
+## 7. In-Session Task Board & Usage Accounting
+
+- **Task Board** (`src/domains/session/task-board.ts`): Maintains session task items with states (`todo`, `in_progress`, `done`, `failed`). Emits middleware reminders when uncompleted tasks remain before turn end.
+- **Usage Accounting** (`src/domains/session/usage.ts`): Aggregates session token counts across input, output, cache read, cache write, and reasoning tokens. Anchors against provider-reported totals on settled turns.
+- **Aborted Turn Persistence**: When a turn is interrupted by `Ctrl+C` or a SIGINT signal, partial assistant output and completed tool executions are committed to `current.jsonl` with `interrupted: true` before yielding the prompt.

package/docs/skills-marketplace.md

@@ -0,0 +1,46 @@
+# Skills Marketplace
+
+> [!TIP]
+> **Interactive Spec Available:** An interactive dashboard is located at [docs/html/skills_blueprint.html](html/skills_blueprint.html) (Version: 0.3.0).
+
+The Skills Hub (`/skill`) shows project skills, user skills, and the marketplace. Every marketplace row comes from the same local lookup that `clio-coder skills install <name>` and `/skill:<name>` resolve through, so the hub lists nothing it cannot install.
+
+## Where marketplace rows come from
+
+There is one source, `discoverMarketplaceSkills()` in `src/domains/resources/skills/marketplace.ts`, and it reads two kinds of real local data:
+
+1. **A catalog directory** of actual `SKILL.md` packages: `CLIO_CODER_SKILL_CATALOG_DIR`, or a `skills/` directory in the working tree when it holds packages. Metadata comes from the packages themselves through the skill loader, so a row's version, category, and audit state are the package's own.
+2. **A JSON index** at `CLIO_CODER_SKILL_MARKETPLACE_INDEX` or `<configDir>/skill-marketplace.json`, whose entries name a `sourceUrl` that `clio-coder skills install` accepts. `npm run skills:pin` publishes that file's `version`, `audit`, and `category` fields.
+
+Catalog packages win over index entries on a name collision, because the local files are the thing that installs. Neither source reaches the network; the hub opens on local data and never blocks.
+
+When both sources are absent the hub has nothing to list and says so, naming the remedy:
+
+```
+no skills installed and no local marketplace configured. install one with
+`clio-coder skills install <path|github-url>`, or point CLIO_CODER_SKILL_CATALOG_DIR at a
+skills/ catalog.
+```
+
+The CLI reports the same state as `no local skill marketplace catalog or index configured`. A marketplace source that exists but fails (an unreadable index, a broken catalog package) is a diagnostic row in the hub, not a silent omission.
+
+## Using the hub
+
+| Key | Action |
+|---|---|
+| type | Filter all groups |
+| `Enter` | Insert `/skill:<name> ` into the editor for the task text |
+| `Tab` | Toggle the detail pane (split layout on wide terminals) |
+| `i` | Install the selected marketplace skill into the project scope through the local marketplace resolver |
+| `PgUp`/`PgDn` | Scroll the detail pane |
+
+Invoking an uninstalled marketplace skill with `/skill:<name>` prompts before installing it. `i` runs the same install path eagerly from the hub.
+
+The CLI `clio-coder skills` commands manage local skill discovery, validation, and
+creation. Extension resource roots and share archives are documented in
+[extensions-and-sharing.md](extensions-and-sharing.md); this page owns the TUI
+Hub and marketplace behavior.
+
+## Publishing a skill
+
+Add a directory under `skills/<category>/<name>/` (or `skills/<name>/`) in the repo containing a `SKILL.md` with `name` and `description` frontmatter. The directory name must match `[A-Za-z0-9][A-Za-z0-9._-]*`. Run `npm run skills:pin` to republish `skills/skill-marketplace.json`, which is the index consumers point `CLIO_CODER_SKILL_MARKETPLACE_INDEX` at or copy to `<configDir>/skill-marketplace.json`. Scientific and niche coding domains are the marketplace's focus; see the existing `skills/` tree for the house format.