ultimate-pi 0.1.7 → 0.2.2

package/CONTRIBUTING.md

@@ -34,6 +34,45 @@ npm run check:ts        # typecheck extensions
 
 Pre-commit hooks run `biome check` and `tsc` on staged files automatically.
 
+## Sentrux (architectural quality gate)
+
+[Sentrux](https://github.com/sentrux/sentrux) provides real-time structural quality metrics for AI-agent-written code. It acts as a feedback loop sensor — scanning codebase architecture, detecting degradation, and enforcing rules via MCP.
+
+### Quick start
+
+```bash
+# Install (macOS / Linux / Windows)
+curl -fsSL https://raw.githubusercontent.com/sentrux/sentrux/main/install.sh | sh
+
+# Install all 52 language plugins
+sentrux plugin add-standard
+
+# Run a quality scan
+sentrux check .
+
+# Save baseline before agent session
+sentrux gate --save .
+
+# Compare after — catches degradation
+sentrux gate .
+```
+
+### MCP Integration
+
+The sentrux MCP server is configured in `.pi/mcp.json`. Agents can use tools like `scan`, `session_start`, `session_end`, `check_rules`, `health`, and `evolution` to monitor code quality during development.
+
+### Rules Engine
+
+Create `.sentrux/rules.toml` to define architectural constraints:
+
+```toml
+[constraints]
+max_cycles = 0
+max_coupling = "B"
+max_cc = 25
+no_god_files = true
+```
+
 ## Firecrawl (self-hosted web scraping)
 
 The Firecrawl skill depends on a Firecrawl instance. This repo includes a self-hosted setup powered by Docker.
@@ -89,6 +128,18 @@ Configurable via env vars (set before launching pi):
 - Reloads on `/reload`.
 - Status command: `/env-loader-status`
 
+### Harness governance extensions
+
+These Pi extensions are loaded from `.pi/extensions/` via the root `package.json`
+`pi.extensions` manifest (no extra registration needed):
+
+- `.pi/extensions/policy-gate.ts` — plan-before-mutate + phase enforcement
+- `.pi/extensions/budget-guard.ts` — budget hard-stop and `budget_exhausted` events
+- `.pi/extensions/trace-recorder.ts` — run trace artifacts in `.pi/harness/runs/`
+- `.pi/extensions/review-integrity.ts` — evaluator/adversary session isolation checks
+- `.pi/extensions/test-diff-integrity.ts` — suspicious test diff detection/escalation
+- `.pi/extensions/debate-orchestrator.ts` — headless debate bus + consensus packets
+
 ### PostHog analytics
 
 `@posthog/pi` — wraps the upstream [posthog-pi](https://github.com/PostHog/posthog-pi) extension to capture AI generation spans, tool spans, and traces in [PostHog](https://posthog.com). Install via `pi install @posthog/pi`. See the upstream repo for configuration and env vars.
@@ -103,7 +154,6 @@ Configurable via env vars (set before launching pi):
 | firecrawl (13 skills) | [firecrawl](https://firecrawl.dev) |
 | obsidian/wiki skills (11 skills) | [AgriciDaniel/claude-obsidian](https://github.com/AgriciDaniel/claude-obsidian) |
 | posthog-analyst | bundled (PostHog MCP integration) |
-| defuddle | bundled (web content cleaning) |
 
 ### Firecrawl sub-skills
 

package/README.md

@@ -2,50 +2,294 @@
 
 > The **ultimate AI coding harness** on top of [**pi.dev**](https://pi.dev).
 
-## Goal
+## What this project is
 
-A production-grade coding harness where the agent:
-1. Uses compressed context before raw tool spam,
-2. Stays concise and deterministic in output style,
-3. Is easy to extend with skills and extensions.
+`ultimate-pi` is a production-oriented harness for AI-assisted coding with strict safety and governance built in.
 
-## Getting started
+It gives you:
+
+- A phase-based workflow (`plan -> execute -> evaluate -> adversary -> merge`)
+- Enforcement that blocks unsafe behavior (for example, mutating code before planning)
+- Structured artifacts in `.pi/harness/` for auditability and replay
+- A practical bootstrap command that sets up tools, graph, and runtime integrations
+
+If you are new: start with the **Quick Start** section and run one task through the full pipeline.
+
+## 5-minute quickstart
+
+If you just want to get started fast:
+
+1. Install into your current project:
 
 ```bash
-npm install -g ultimate-pi
+pi install npm:ultimate-pi
+/reload
 ```
 
-From your project folder:
+2. Bootstrap the harness:
 
-```bash
-pi install ultimate-pi -l
+```text
+/harness-setup
+```
+
+3. Run your first task:
+
+```text
+/harness-auto "implement feature X safely"
 ```
 
-Then reload PI:
+That command runs the strict pipeline:
+`plan -> execute -> evaluate -> adversary -> policy decision`.
+
+If it blocks, inspect with:
+
+```text
+/harness-trace-last
+/harness-policy-status
+```
+
+## Table of Contents
+
+- [5-minute quickstart](#5-minute-quickstart)
+- [How the harness works](#how-the-harness-works)
+- [Prerequisites](#prerequisites)
+- [Quick Start (new users)](#quick-start-new-users)
+- [Run your first harness task](#run-your-first-harness-task)
+- [Command reference](#command-reference)
+- [Harness artifacts and file layout](#harness-artifacts-and-file-layout)
+- [Safety and governance defaults](#safety-and-governance-defaults)
+- [Router tuning flow](#router-tuning-flow)
+- [Troubleshooting](#troubleshooting)
+- [Contributing](#contributing)
+
+## How the harness works
+
+The harness enforces a deterministic execution lifecycle:
+
+1. **Plan**  
+   Create a `PlanPacket` before any mutating work.
+2. **Execute**  
+   Implement only within the approved plan scope.
+3. **Evaluate**  
+   Run independent evaluation and produce an `EvalVerdict`.
+4. **Adversary**  
+   Run adversarial review and produce an `AdversaryReport`.
+5. **Policy / Merge decision**  
+   Debate consensus + severity policy decides `pass`, `conditional_pass`, `block`, or `human_required`.
+
+### Why this matters
+
+- You get fewer silent mistakes.
+- Reviews are reproducible, not opinion-only.
+- Incidents and overrides are recorded in structured, machine-readable artifacts.
+
+## Prerequisites
+
+Minimum recommended environment:
+
+- `node >= 18`
+- `npm >= 9`
+- `git`
+- `python >= 3.10` (for Graphify workflow)
+
+Optional but commonly used:
+
+- `gh` CLI for GitHub workflow
+- Docker (only if you want self-hosted Firecrawl)
+
+## Quick Start (new users)
+
+From your project folder:
 
 ```bash
+pi install npm:ultimate-pi
 /reload
 ```
 
-Optional: install from GitHub Packages:
+Run the full bootstrap:
 
-```bash
-npm install -g @aryaniyaps/ultimate-pi --registry=https://npm.pkg.github.com
+```text
+/harness-setup
+```
+
+`/harness-setup` is idempotent and designed as the one-command initializer for:
+
+- Graphify knowledge graph setup
+- CLI tool installation and checks
+- Harness/runtime directory scaffolding
+- Extension package verification
+- Model-router bootstrap configuration
+
+## Run your first harness task
+
+### Fastest path
+
+Use the one-command pipeline:
+
+```text
+/harness-auto "implement feature X safely"
 ```
 
-## Bootstrap
+This runs:
+
+`plan -> execute -> evaluate -> adversary -> policy decision -> commit/PR (no auto-merge)`
 
-Run the harness setup prompt to configure everything (wiki, firecrawl, env vars, extensions):
+### Manual path (recommended for learning)
 
+1. Plan
+
+```text
+/harness-plan "implement feature X safely"
 ```
-/harness-setup
+
+2. Execute with approved plan:
+
+```text
+/harness-run --plan <path-to-plan-packet.json>
 ```
 
-## Wiki
+3. Evaluate:
 
-The harness includes an Obsidian wiki vault as its knowledge layer. Run `/wiki` in your PI session to scaffold it, then open the vault in [Obsidian](https://obsidian.md). Recommended community plugins: Dataview, Graph Analysis, Obsidian Git.
+```text
+/harness-eval --run <run-id>
+/harness-review --run <run-id>
+```
 
+4. Adversarial review:
+
+```text
+/harness-critic --run <run-id>
+```
+
+5. If blocked or ambiguous, record incident:
+
+```text
+/harness-incident --run <run-id> --trigger "<reason>"
+```
+
+6. Trace/debug:
+
+```text
+/harness-trace --run <run-id>
+```
+
+## Command reference
+
+### Core workflow commands
+
+- `/harness-setup` - bootstrap complete environment and harness scaffolding
+- `/harness-auto "<task>"` - run strict end-to-end pipeline
+- `/harness-plan "<task>"` - generate read-only `PlanPacket`
+- `/harness-run --plan <file>` - execute approved scope only
+- `/harness-eval --run <run-id>` - benchmark/evaluation summary
+- `/harness-review --run <run-id>` - independent evaluator verdict
+- `/harness-critic --run <run-id>` - adversarial findings and merge-block signal
+- `/harness-incident --run <run-id> --trigger "<reason>"` - incident record
+- `/harness-trace --run <run-id>` - replay and artifact completeness
+- `/harness-abort [reason]` - reset safely to plan phase and lock mutation until new plan
+
+### Operational/status commands
+
+- `/harness-policy-status`
+- `/harness-budget-status`
+- `/harness-review-integrity-status`
+- `/harness-test-integrity-last`
+- `/harness-trace-last`
+- `/harness-debate-open`
+- `/harness-debate-round`
+- `/harness-debate-consensus`
+
+## Harness artifacts and file layout
+
+Primary harness directories:
+
+- `.pi/harness/specs/` - JSON schemas for core contracts
+- `.pi/harness/runs/` - per-run trace summaries + event indexes
+- `.pi/harness/incidents/` - incident and policy override records
+- `.pi/harness/debates/` - debate rounds, consensus packets, budget events
+- `.pi/harness/router/` - router tuning proposals and apply flow scripts
+
+Core contract schemas in `.pi/harness/specs/`:
+
+- `PlanPacket`
+- `RunTrace`
+- `EvalVerdict`
+- `AdversaryReport`
+- `RoundResult`
+- `ConsensusPacket`
+- `BudgetExhausted`
+- `IncidentRecord`
+- `RouterTuningProposal`
+
+## Safety and governance defaults
+
+The harness intentionally locks in these behaviors:
+
+- **Plan-before-mutate**: write/edit/mutating shell commands blocked outside execute phase
+- **Mandatory adversarial review** in the strict pipeline
+- **Review isolation**: evaluator/adversary cannot share executor session context
+- **Budget hard-stops** with structured `budget_exhausted` events
+- **Test-diff integrity checks** for suspicious test weakening patterns
+- **Severity policy thresholds**:
+  - block if `security >= 0.70` or `correctness >= 0.70`
+  - block if `architecture >= 0.80` or `test_integrity >= 0.80`
+- **Override policy**: single human approver with explicit justification
+- **Never auto-merge**
+
+## Router tuning flow
+
+Router changes are two-step and approval-gated:
+
+1. Propose (no live mutation):
+
+```bash
+node .pi/harness/router/propose-router-tuning.mjs \
+  --evidence /path/to/evidence.json \
+  --candidate /path/to/candidate-router.json \
+  --proposal-out .pi/harness/router/proposals/proposal-001.json
+```
+
+2. Apply (explicit human approval + justification + `--write`):
+
+```bash
+node .pi/harness/router/apply-router-proposal.mjs \
+  --proposal .pi/harness/router/proposals/proposal-001.json \
+  --approve-by "human.name" \
+  --justification "why this is safe" \
+  --write
+```
+
+Blind writes to `.pi/model-router.json` are intentionally disallowed.
+
+## Troubleshooting
+
+### `/harness-setup` fails early
+
+- Check `node --version`, `npm --version`, `git --version`
+- Ensure Node is at least 18
+
+### Graphify not available
+
+- Install Python 3.10+
+- Then install Graphify and build/update graph
+
+### Review/integrity blocks in evaluate/adversary phase
+
+- This means review is not isolated from execute context
+- Fork/switch session, then rerun review commands
+
+### Budget hard-stop triggers
+
+- Use `/harness-budget-status`
+- Reduce scope, split task, or restart with a narrower plan
+
+### Suspicious test diff warning
+
+- Use `/harness-test-integrity-last`
+- Restore or justify test changes; expect adversarial scrutiny
 
 ## Contributing
 
-See [CONTRIBUTING.md](./CONTRIBUTING.md) for local development setup, linting, and extension config details.
+For local dev setup, lint/test commands, Firecrawl notes, extension details, and architectural quality gate workflow, see:
+
+- [CONTRIBUTING.md](./CONTRIBUTING.md)

package/biome.json

@@ -1,12 +1,18 @@
 {
-	"$schema": "https://biomejs.dev/schemas/2.0.6/schema.json",
+	"$schema": "https://biomejs.dev/schemas/2.4.14/schema.json",
 	"vcs": {
 		"enabled": true,
 		"clientKind": "git",
 		"useIgnoreFile": true
 	},
 	"files": {
-		"ignoreUnknown": false
+		"ignoreUnknown": false,
+		"maxSize": 5242880,
+		"includes": [
+			"**/*.{ts,tsx,js,jsx,json,jsonc,css}",
+			"!graphify-out/**/*",
+			"!graphify-books-out/**/*"
+		]
 	},
 	"formatter": {
 		"enabled": true,

package/lefthook.yml

@@ -1,8 +1,9 @@
 pre-commit:
   commands:
     lint:
-      glob: "*.{ts,js,json,md}"
-      run: npx @biomejs/biome check --no-errors-on-unmatched --staged
+      glob: "*.{ts,js,json}"
+      exclude: "graphify-out/**/*"
+      run: npx @biomejs/biome check --no-errors-on-unmatched --write {staged_files}
     typecheck:
       glob: "*.ts"
       run: npm run check:ts

package/node_modules/@sting8k/pi-vcc/README.md

@@ -0,0 +1,200 @@
+# pi-vcc
+
+[![npm](https://img.shields.io/npm/v/@sting8k/pi-vcc)](https://www.npmjs.com/package/@sting8k/pi-vcc)
+
+Algorithmic conversation compactor for [Pi](https://github.com/badlogic/pi-mono). No LLM calls — produces a brief transcript via extraction and formatting.
+
+Inspired by [VCC](https://github.com/lllyasviel/VCC) **(View-oriented Conversation Compiler)**.
+
+## Why pi-vcc
+
+|  | Pi default | pi-vcc |
+|---|---|---|
+| **Method** | LLM-generated summary | Algorithmic extraction, no LLM |
+| **Determinism** | Non-deterministic, can hallucinate | Same input = same output, always |
+| **Token reduction** | Varies | 35-99% on real sessions (higher on longer sessions) |
+| **Compaction latency** | Waits for LLM call | 30-470ms, no API calls |
+| **History after compaction** | Gone — agent only sees summary | Fully searchable via `vcc_recall` |
+| **Repeated compactions** | Each rewrite risks losing more | Sections merge and accumulate |
+| **Cost** | Burns tokens on summarization call | Zero — no API calls |
+| **Structure** | Free-form prose | Brief transcript + 4 semantic sections |
+
+### Real session metrics
+
+Measured on real session JSONLs under `~/.pi/agent/sessions` (chars = rendered message text).
+
+| Session | Messages | Before | After | Reduction | Time |
+|---|---|---|---|---|---|
+| Session A | 2,943 | 997,162 | 7,959 | 99.2% | 64ms |
+| Session B | 1,703 | 428,334 | 7,762 | 98.2% | 29ms |
+| Session C | 1,657 | 424,183 | 9,577 | 97.7% | 54ms |
+| Session D | 1,004 | 2,258,477 | 4,439 | 99.8% | 30ms |
+| Session E | 486 | 295,006 | 11,163 | 96.2% | 30ms |
+| Session F | 46 | 5,234 | 3,364 | 35.7% | 5ms |
+| Session G | 27 | 8,595 | 2,489 | 71.0% | 2ms |
+
+## Features
+
+- **No LLM** — purely algorithmic, zero extra API cost
+- **Brief transcript** — chronological conversation flow, each tool call collapsed to a one-liner with `(#N)` refs, text truncated to keep it compact
+- **5 semantic sections** — session goal, files & changes, commits, outstanding context, user preferences
+- **Bounded merge** — rolling sections re-capped after merge instead of growing unbounded
+- **Lossless recall** — `vcc_recall` reads raw session JSONL, so old history stays searchable across compactions
+- **Regex search** — `vcc_recall` supports regex patterns (`hook|inject`, `fail.*build`) and OR-ranked multi-word queries
+- **Result ranking** — search results ranked by term relevance, rare terms weighted higher than common ones
+- **`/pi-vcc-recall`** — slash command to search history directly, results shown as collapsible message and auto-fed to agent as context
+- **Fallback cut** — still works when Pi core returns nothing to summarize
+- **`/pi-vcc`** — manual compaction on demand
+
+## Install
+
+```bash
+pi install npm:@sting8k/pi-vcc
+```
+
+Or from GitHub:
+
+```bash
+pi install https://github.com/sting8k/pi-vcc
+```
+
+Or try without installing:
+
+```bash
+pi -e https://github.com/sting8k/pi-vcc
+```
+
+## Usage
+
+Once installed, pi-vcc registers a `session_before_compact` hook.
+
+- Run `/pi-vcc` to trigger pi-vcc compaction manually.
+- By default, `/compact` and auto-threshold compactions still go through pi core (LLM-based). Set `overrideDefaultCompaction: true` in the config to let pi-vcc handle all compaction paths.
+- To search older history after compaction, use `vcc_recall`.
+- To search and feed results to agent yourself, run `/pi-vcc-recall <query> [page:N]`.
+  - Tip: type `/recall` and Pi will autocomplete to `/pi-vcc-recall`.
+
+### How compaction works
+
+Pi splits the conversation at the **last user message**. Everything after — the **kept tail** — stays intact and untouched. pi-vcc only summarizes the older portion before that cut point.
+
+### Compacted message structure
+
+```
+[Session Goal]
+- Fix the authentication bug in login flow
+- [Scope change]
+- Also update the session token refresh logic
+
+[Files And Changes]
+- Modified: src/auth/session.ts
+- Created: tests/auth-refresh.test.ts
+
+[Commits]
+- a1b2c3d: fix(auth): refresh token after password reset
+
+[Outstanding Context]
+- lint check still failing on line 42
+
+[User Preferences]
+- Prefer Vietnamese responses
+- Always run tests before committing
+
+[user]
+Fix the auth bug, users can't log in after password reset
+
+[assistant]
+Root cause is a missing token refresh after password reset...
+* bash "bun test tests/auth.test.ts" (#12)
+* edit "src/auth/session.ts" (#14)
+* bash "bun test tests/auth.test.ts" (#16)
+...(28 earlier lines omitted)
+```
+
+Sections appear only when relevant — a session with no git commits won't have `[Commits]`.
+
+**Sections:**
+
+| Section | Description |
+|---|---|
+| `[Session Goal]` | Initial goal + scope changes (regex-based extraction) |
+| `[Files And Changes]` | Modified/created files from tool calls (capped, paths trimmed to common root) |
+| `[Commits]` | Git commits made during the session (last 8, hash + first line) |
+| `[Outstanding Context]` | Unresolved items — errors, pending questions |
+| `[User Preferences]` | Regex-extracted from user messages (`always`, `never`, `prefer`...) |
+| Brief transcript | Chronological conversation flow — rolling window of ~120 recent lines, tool calls collapsed to one-liners with `(#N)` refs |
+
+**Merge policy:**
+- `Session Goal`, `User Preferences`: concise sticky sections
+- `Outstanding Context`: fresh-only (replaced each compaction)
+- `Files And Changes`, `Commits`: unique union across compactions
+- Brief transcript: rolling window, older lines drop off
+
+## Recall (Lossless History)
+
+Pi's default compaction discards old messages permanently. After compaction, the agent only sees the summary.
+
+`vcc_recall` bypasses this by reading the raw session JSONL file directly. It parses every message entry in the file, regardless of how many compactions have happened.
+
+### Search
+
+Queries support **regex** and **multi-word OR logic** ranked by relevance:
+
+```
+vcc_recall({ query: "auth token" })           // OR search, ranked
+vcc_recall({ query: "auth token", page: 2 })  // paginated (5 results/page)
+vcc_recall({ query: "hook|inject" })           // regex pattern
+vcc_recall({ query: "fail.*build" })           // regex pattern
+```
+
+### Browse
+
+Without a query, returns the last 25 entries as brief summaries:
+
+```
+vcc_recall()
+```
+
+### Expand
+
+Returns full untruncated content for specific indices found via search:
+
+```
+vcc_recall({ expand: [41, 42] })
+```
+
+Typical workflow: **search → find relevant entry indices → expand those indices for full content**.
+
+> Some tool results are truncated by Pi core at save time. `expand` returns everything in the JSONL but can't recover what Pi already cut.
+
+## Pipeline
+
+1. **Normalize** — raw Pi messages → uniform blocks (user, assistant, tool_call, tool_result, thinking)
+2. **Filter noise** — strip system messages, empty blocks
+3. **Build sections** — extract goal, file paths, blockers, preferences
+4. **Brief transcript** — chronological conversation flow, tool calls collapsed to one-liners, text truncated
+5. **Format** — render into bracketed sections + transcript
+6. **Merge** — if previous summary exists: sticky sections merge, volatile sections replace, transcript rolls
+
+## Config
+
+Config lives at `~/.pi/agent/pi-vcc-config.json` (auto-scaffolded on first load with safe defaults):
+
+```json
+{
+  "overrideDefaultCompaction": false,
+  "debug": false
+}
+```
+
+- **`overrideDefaultCompaction`** *(default `false`)*: when `false`, pi-vcc only runs for `/pi-vcc`; `/compact` and auto-threshold compactions fall through to pi core. Set `true` to make pi-vcc handle all compaction paths.
+- **`debug`** *(default `false`)*: when `true`, each compaction writes detailed info to `/tmp/pi-vcc-debug.json` — message counts, cut boundary, summary preview, sections.
+
+## Related Work
+
+- [VCC](https://github.com/lllyasviel/VCC) — the original transcript-preserving conversation compiler
+- [Pi](https://github.com/badlogic/pi-mono) — the AI coding agent this extension is built for
+
+## License
+
+MIT

package/node_modules/@sting8k/pi-vcc/index.ts

@@ -0,0 +1,14 @@
+import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
+import { scaffoldSettings } from "./src/core/settings";
+import { registerBeforeCompactHook } from "./src/hooks/before-compact";
+import { registerPiVccCommand } from "./src/commands/pi-vcc";
+import { registerVccRecallCommand } from "./src/commands/vcc-recall";
+import { registerRecallTool } from "./src/tools/recall";
+
+export default (pi: ExtensionAPI) => {
+  scaffoldSettings();
+  registerBeforeCompactHook(pi);
+  registerPiVccCommand(pi);
+  registerVccRecallCommand(pi);
+  registerRecallTool(pi);
+};

package/node_modules/@sting8k/pi-vcc/package.json

@@ -0,0 +1,26 @@
+{
+  "name": "@sting8k/pi-vcc",
+  "version": "0.3.12",
+  "description": "Algorithmic conversation compactor for pi - transcript-preserving structured summaries, no LLM calls",
+  "main": "index.ts",
+  "keywords": [
+    "pi-package",
+    "pi-extension",
+    "vcc",
+    "compact",
+    "compaction"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/sting8k/pi-vcc.git"
+  },
+  "peerDependencies": {
+    "@mariozechner/pi-coding-agent": "*",
+    "@sinclair/typebox": "*"
+  },
+  "pi": {
+    "extensions": [
+      "./index.ts"
+    ]
+  }
+}