ultimate-pi 0.3.1 → 0.4.0

package/.pi/prompts/harness-setup.md

@@ -1,24 +1,42 @@
 ---
-description: Full harness bootstrap — Graphify knowledge graph setup, optional self-hosted firecrawl (Docker), CLI tools install, pi extension packages, and verification. Run once per project.
-argument-hint: "[--skip-graphify] [--skip-tools] [--skip-firecrawl-self] [--force]"
+description: Full harness bootstrap — Graphify knowledge graph setup, Scrapling/harness-web install, CLI tools, pi extension packages, and verification. Run once per project.
+argument-hint: "[--skip-graphify] [--skip-tools] [--non-interactive] [--force]"
 ---
 
 # harness-setup — Full Harness Bootstrap
 
 Bootstraps the complete ultimate-pi agentic harness: Graphify knowledge graph, CLI tools, pi extension packages, configuration files, and verification. Idempotent — safe to re-run, skips what's already installed.
 
+## Agent execution notes (read first)
+
+**Prefer bundled scripts over re-implementing steps.** Each script is idempotent. Do not duplicate Step 2 subsections (2.1–2.8) when `harness-cli-verify.sh` already passed.
+
+| Pitfall | Correct approach |
+|---------|------------------|
+| `UP_PKG="$(pwd)"` in an **external** repo | Wrong — scripts live in the npm package. Resolve via `harness-resolve-up-pkg.mjs` (see Step 0). |
+| Provider detection from `OPENAI_*` / `ANTHROPIC_*` env only | Wrong for pi users — keys live in `~/.pi/agent/auth.json`. Use `harness-generate-model-router.mjs` (Pi `ModelRegistry.getAvailable()`). |
+| Re-running 2.1–2.8 manually after CLI verify | Wasteful — trust `harness-cli-verify.sh` output; only fix reported ✗ lines. |
+| Overwriting `AGENTS.md` after graphify | Graphify appends a section — **merge**, do not replace (Step 4.3). |
+| `sentrux-rules-sync` without project manifest | Use **`harness-sentrux-bootstrap.mjs`** (Step 4.4) — seeds manifest + idempotent rules sync. |
+| Re-running bootstrap with `--force` on unchanged manifest | Wasteful but safe — default bootstrap skips when hash unchanged; `--force` only after manifest edits. |
+| `graph.json` uses `links`, not `edges` | Step 6 stats: `g.get('edges', g.get('links', []))`. |
+| Guessing harness-web / `.env` defaults when `ask_user` is available | **Mandatory `ask_user`** at Step 4.0 unless `--non-interactive`. |
+| `sudo apt-get` without passwordless sudo | Skip — report manual fix; do not block the rest of setup. |
+| `graphify codex install` | **Never run** — it writes `.codex/hooks.json`. Harness targets pi only (`graphify install --platform pi`). |
+| Overwriting `.env` | Use `harness-sync-env.mjs` — never rewrite; append missing keys only. |
+
 ## Parse arguments
 
 Read `$ARGUMENTS` and map flags:
 
 - `--skip-graphify`
 - `--skip-tools`
-- `--skip-firecrawl-self`
+- `--non-interactive`
 - `--force`
 
 If a flag is unknown, stop and return:
 
-`Usage: /harness-setup [--skip-graphify] [--skip-tools] [--skip-firecrawl-self] [--force]`
+`Usage: /harness-setup [--skip-graphify] [--skip-tools] [--non-interactive] [--force]`
 
 ## Step 0 — Pre-flight Environment Check
 
@@ -32,16 +50,29 @@ Block if node < 18, npm < 9, or git missing. Report versions and continue.
 
 Read `.pi/auto-commit.json` for co-author + branch config.
 
-Resolve the installed **ultimate-pi** package root (works in this repo and after `pi install npm:ultimate-pi`):
+Resolve **`UP_PKG`** (ultimate-pi npm package root — **not** the target project cwd):
 
 ```bash
-UP_PKG="$(node -p "require('path').dirname(require.resolve('ultimate-pi/package.json'))")"
+UP_PKG=""
+for _pkg_root in \
+  "$(node -p "try{require('path').dirname(require.resolve('ultimate-pi/package.json'))}catch{''}" 2>/dev/null)" \
+  "$(npm root -g 2>/dev/null)/ultimate-pi" \
+  "$(pwd)"; do
+  [ -n "$_pkg_root" ] || continue
+  [ -f "$_pkg_root/.pi/scripts/harness-resolve-up-pkg.mjs" ] || continue
+  UP_PKG="$(node "$_pkg_root/.pi/scripts/harness-resolve-up-pkg.mjs")"
+  break
+done
+if [ -z "$UP_PKG" ] || [ ! -f "$UP_PKG/.pi/scripts/harness-cli-verify.sh" ]; then
+  echo "✗ ultimate-pi package not found. Install: pi install npm:ultimate-pi"
+  exit 1
+fi
 echo "ultimate-pi package: $UP_PKG"
 ```
 
-For extension package names, read **`$UP_PKG/.pi/settings.example.json`** (shipped template). Merge its `packages` array into the **project** `.pi/settings.json` if missing — do not copy the repo-dev `.pi/settings.json` from the package (it may contain `".."` and is not published).
+**Developing ultimate-pi from its git clone:** `$(pwd)` is tried last; it wins only when the clone contains `.pi/scripts/harness-resolve-up-pkg.mjs`.
 
-If `require.resolve('ultimate-pi/package.json')` fails (bare clone without resolving the package name), run from **this repository root**: `UP_PKG="$(pwd)"`.
+For extension package names, read **`$UP_PKG/.pi/settings.example.json`**. **Merge** its `packages` array into the **project** `.pi/settings.json` (add missing entries; keep user packages). Do not copy the repo-dev `.pi/settings.json` from the package (it may contain `".."` and is not published).
 
 ## Step 0.5 — Graphify (skip if `--skip-graphify`)
 
@@ -74,7 +105,7 @@ bash "$UP_PKG/.pi/scripts/harness-graphify-bootstrap.sh"
 If the bootstrap script is missing, run it from the installed ultimate-pi package (`.pi/scripts/` inside the npm package), or execute equivalent steps manually:
 
 1. Install `graphifyy` (`uv tool install` preferred; else `pip`/`pip3 install --user`)
-2. `graphify install --platform pi` (and `graphify cursor install` if `.cursor/` exists)
+2. `graphify install --platform pi` only. **Do not** run `graphify codex install` or `graphify cursor install`.
 3. `GRAPHIFY_VIZ_NODE_LIMIT=200000 graphify update .` — **required**; exits non-zero on failure
 4. If `GEMINI_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `MOONSHOT_API_KEY` is set: `graphify extract .` for full semantic graph (optional enrichment)
 5. `graphify hook install` only when `.git/` exists
@@ -93,133 +124,22 @@ Read and summarize `graphify-out/GRAPH_REPORT.md` — god nodes and surprising c
 | `graph.json` exists but 0 nodes | Stale/partial output — re-run with `--force` |
 | `graphify extract` fails | No API key — code graph from `update` is still valid; note in report |
 
-## Step 1.5 — Optional Self-Hosted Firecrawl
-
-Ask: "Use self-hosted Firecrawl (local Docker) or cloud (api.firecrawl.dev)? [cloud/self]"
-Default: **cloud**.
-
-If user chooses **self**:
-
-### 1.5.1 — Docker Engine Install
-
-Check if Docker is already available:
-```bash
-if ! command -v docker &>/dev/null; then
-	# Detect OS and install Docker Engine
-	if [ -f /etc/os-release ]; then
-		. /etc/os-release
-		case "$ID" in
-			ubuntu|debian)
-				curl -fsSL https://get.docker.com | sh
-				;;
-				fedora|rhel|centos)
-				curl -fsSL https://get.docker.com | sh
-				;;
-				arch)
-				pacman -S --noconfirm docker
-				;;
-			*)
-				echo "Unsupported distro: $ID. Install Docker manually: https://docs.docker.com/engine/install/"
-				;;
-		esac
-	elif command -v brew &>/dev/null; then
-		# macOS — install Docker Desktop via brew
-		brew install --cask docker
-	else
-		echo "Cannot detect OS. Install Docker manually: https://docs.docker.com/engine/install/"
-	fi
-
-	# Enable and start Docker
-	sudo systemctl enable --now docker 2>/dev/null || true
-
-	# Add current user to docker group (no sudo needed)
-	sudo usermod -aG docker $USER 2>/dev/null || true
-	newgrp docker 2>/dev/null || echo "Docker group added. Restart terminal or run: newgrp docker"
-fi
-```
-
-Verify:
-```bash
-docker --version
-docker compose version
-```
-
-Block if Docker install fails. Show manual install link.
-
-### 1.5.2 — Set Up Self-Hosted Firecrawl Files
-
-The `firecrawl/` directory in the project root contains all self-hosted config:
-
-```
-firecrawl/
-├── docker-compose.yaml   # Multi-service compose (API, Playwright, Redis, RabbitMQ, Postgres, SearXNG)
-├── README.md             # Self-hosted usage docs
-├── .env.template         # Environment variables template
-└── searxng/
-    ├── searxng.env       # SearXNG-specific env
-    └── settings.yml      # SearXNG engine config
-```
-
-Create `.env` from template if missing:
-```bash
-if [ ! -f firecrawl/.env ]; then
-	if [ -f firecrawl/.env.template ]; then
-		cp firecrawl/.env.template firecrawl/.env
-		echo "Created firecrawl/.env from template."
-	else
-		cat > firecrawl/.env << 'EOF'
-# Firecrawl Self-Hosted Configuration
-PORT=3002
-INTERNAL_PORT=3002
-REDIS_URL=redis://redis:6379
-POSTGRES_USER=postgres
-POSTGRES_PASSWORD=postgres
-POSTGRES_DB=postgres
-USE_DB_AUTHENTICATION=false
-NUM_WORKERS_PER_QUEUE=8
-CRAWL_CONCURRENT_REQUESTS=10
-MAX_CONCURRENT_JOBS=5
-BROWSER_POOL_SIZE=5
-BULL_AUTH_KEY=changeme
-SEARXNG_EXTERNAL_PORT=8080
-# Optional AI: uncomment and set
-# OPENAI_API_KEY=
-# OPENAI_BASE_URL=
-# MODEL_NAME=
-# OLLAMA_BASE_URL=
-EOF
-		echo "Created firecrawl/.env with defaults."
-	fi
-fi
-```
+## Step 1.5 — Scrapling / harness-web (web layer)
 
-### 1.5.3 — Start Services
+No Docker stack or API keys. Installed by `harness-cli-verify.sh` in Step 2; optional early install:
 
 ```bash
-docker compose -f firecrawl/docker-compose.yaml up -d
+command -v uv &>/dev/null || curl -LsSf https://astral.sh/uv/install.sh | sh
+export PATH="$HOME/.local/bin:$PATH"
+uv tool install "scrapling[fetchers]"
+scrapling install   # Chromium for default stealth scrape; may need sudo for OS libs on Linux
+mkdir -p .web
+python3 "$UP_PKG/.pi/scripts/harness-web.py" search "ultimate-pi harness" -o .web/smoke-search.json --limit 3
+python3 "$UP_PKG/.pi/scripts/harness-web.py" scrape "https://example.com" -o .web/smoke-page.md --fast
 ```
 
-Wait for health:
-```bash
-echo "Waiting for services to be healthy..."
-for i in $(seq 1 30); do
-	if curl -sf http://localhost:3002/v1/health &>/dev/null; then
-		echo "✓ Firecrawl API is healthy"
-		break
-	fi
-	sleep 2
-done
-```
-
-### 1.5.4 — Verify Self-Hosted Instance
-
-```bash
-curl -sf http://localhost:3002/v1/health && echo "✓ Self-hosted Firecrawl running on :3002" || echo "✗ Firecrawl not healthy yet — check: docker compose -f firecrawl/docker-compose.yaml logs"
-docker compose -f firecrawl/docker-compose.yaml ps
-```
-
-If user chose **cloud**, skip all 1.5.x steps. Just note:
-> "Using cloud Firecrawl. Ensure `FIRECRAWL_API_KEY` is set. Run `firecrawl login` in Step 2.1."
+- **`--skip-tools`:** skip Step 2 (includes Scrapling verify).
+- On Linux/WSL, if stealth scrape fails, install browser libs from `harness-cli-verify.sh` output or use `--fast` for static targets.
 
 ## Step 2 — Install & Verify Global CLI Tools (skip if `--skip-tools`)
 
@@ -231,7 +151,7 @@ bash "$UP_PKG/.pi/scripts/harness-cli-verify.sh"
 # Reinstall everything: bash "$UP_PKG/.pi/scripts/harness-cli-verify.sh" --force
 ```
 
-**Required (script must exit 0):** firecrawl-cli, ctx7, biome, ast-grep (`sg`), sentrux (when harness manifest present).
+**Required (script must exit 0):** scrapling + harness-web smoke, ctx7, biome, ast-grep (`sg`), sentrux (when harness manifest present).
 
 **Warnings allowed:** gh (if not authenticated), agent-browser (if OS libs need manual `sudo apt-get install`), ck (empty corpus on tiny repos).
 
@@ -248,46 +168,25 @@ bash "$UP_PKG/.pi/scripts/harness-cli-verify.sh"
 
 **Do not continue** past Step 2 if `harness-cli-verify.sh` exits non-zero.
 
+**After the script exits 0:** do **not** re-run sections 2.1–2.8 individually unless the script reported a specific ✗ failure. Record auth/warning lines from script output in the final report.
+
 ### Manual reference (if script missing in target repo)
 
 Use `bash "$UP_PKG/.pi/scripts/harness-cli-verify.sh"` (see Step 0 for `UP_PKG`), or install tools individually:
 
-### 2.1 — firecrawl-cli (Web Search + Scrape + Crawl + Interact + Download + Parse)
-
-```bash
-if ! command -v firecrawl &>/dev/null || [ "$FORCE" = "true" ]; then
-	npm install -g firecrawl-cli@latest
-fi
-```
-
-Verify:
-```bash
-firecrawl --status
-```
+### 2.1 — scrapling + harness-web (Web Search + Scrape)
 
-**If self-hosted mode (Step 1.5 was chosen):** skip cloud auth. Point CLI at local instance:
-```bash
-export FIRECRAWL_API_URL=http://localhost:3002
-export FIRECRAWL_API_KEY=""
-```
-Add to shell profile for persistence:
-```bash
-echo 'export FIRECRAWL_API_URL=http://localhost:3002' >> ~/.bashrc 2>/dev/null
-echo 'export FIRECRAWL_API_KEY=""' >> ~/.bashrc 2>/dev/null
-```
+Handled by `harness-cli-verify.sh` (`verify_scrapling`). Manual fallback:
 
-**If cloud mode:** authenticate if not already:
 ```bash
-firecrawl login --browser
-# OR
-firecrawl login --api-key "<key>"
+uv tool install "scrapling[fetchers]"
+scrapling install
+mkdir -p .web
+python3 "$UP_PKG/.pi/scripts/harness-web.py" search "query" -o .web/search.json --limit 5
+python3 "$UP_PKG/.pi/scripts/harness-web.py" scrape "https://example.com" -o .web/page.md --fast
 ```
 
-Quick smoke test (skills ship with `ultimate-pi` via npm — do **not** run `firecrawl setup skills`):
-```bash
-mkdir -p .firecrawl
-firecrawl scrape "https://firecrawl.dev" -o .firecrawl/install-check.md
-```
+See `.agents/skills/scrapling-web/SKILL.md`.
 
 ### 2.2 — ctx7 (Context7 Library Docs + Skills Management)
 
@@ -412,35 +311,12 @@ if ! command -v sentrux &>/dev/null || [ "$FORCE" = "true" ]; then
 fi
 ```
 
-Verify:
-```bash
-sentrux --version && echo "✓ sentrux installed" || echo "✗ sentrux install failed"
-```
-
 Install all 52 language plugins:
 ```bash
 sentrux plugin add-standard 2>/dev/null || echo "Plugins already installed or failed"
 ```
 
-Configure MCP server in `.pi/mcp.json` (see Step 4.3).
-
-Generate architectural rules from the harness manifest (creates/updates `.sentrux/rules.toml`):
-```bash
-node "$UP_PKG/.pi/scripts/sentrux-rules-sync.mjs" --force
-# Or in pi: /harness-sentrux-sync
-```
-
-Edit layers/boundaries in `.pi/harness/sentrux/architecture.manifest.json` when the repo layout changes, then re-run sync. Custom TOML below the `harness:managed` markers is preserved.
-
-Verify rules:
-```bash
-sentrux check . && echo "✓ sentrux rules pass" || echo "✗ sentrux check failed"
-```
-
-Set up structural regression baseline (optional):
-```bash
-sentrux gate --save . 2>/dev/null || echo "Baseline will be saved on first gate run"
-```
+Configure MCP server in `.pi/mcp.json` (see Step 4.2). **Rules.toml bootstrap runs in Step 4.3** (idempotent, merge-safe).
 
 ## Step 3 — Pi Extension Packages
 
@@ -466,170 +342,100 @@ Verify each package:
 |---------|---------|-------|
 | `@posthog/pi` | Analytics event capture | F0 |
 | `pi-lean-ctx` | Context runtime (read/bash/find/grep/MCP bridge) | F0 |
-| `@tintinweb/pi-subagents` | L4 critic sub-agent spawn/control | P16 |
+| `harness-subagents` (bundled extension) | L4 sub-agent spawn, blackboard, package agents | P16 |
 | `@sting8k/pi-vcc` | VCC compaction / conversation memory | Shipped |
 | `pi-model-router` | Vendored (`vendor/`); activates after `.pi/model-router.json` exists | F0 |
 
 ## Step 3.5 — Model Router Configuration (Dynamic)
 
-`.pi/model-router.json` is **user-specific** (differs per user's providers).
-It is gitignored. Generate it dynamically from your `.env`.
+`.pi/model-router.json` is **user-specific** (gitignored). Generate from **Pi authenticated providers** (`~/.pi/agent/auth.json`, OAuth, env) — **not** env-var guessing alone.
 
-The script below:
-1. Detects available AI providers from env vars (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, and `OPENAI_API_BASE` to detect opencode gateway)
-2. Generates a full `model-router.json` with `auto`, `cheap`, and `deep` profiles
-3. Only writes if file doesn't exist yet (safe to re-run, will skip existing)
+Pi API (see `packages/coding-agent` docs / SDK example `02-custom-model.ts`):
 
-```bash
-UP_PKG="$(node -p "require('path').dirname(require.resolve('ultimate-pi/package.json'))")"
+- `AuthStorage.create()` → credentials store
+- `ModelRegistry.create(authStorage)` → registry
+- `await modelRegistry.getAvailable()` → models with working auth (same as interactive pi)
 
+```bash
 # Verify vendored extension source ships with ultimate-pi
 ls "$UP_PKG/vendor/pi-model-router/extensions/index.ts" 2>/dev/null \
   && echo "✓ vendored pi-model-router" \
   || echo "✗ missing vendor/pi-model-router"
 
-# Generate config from detected providers (only if missing)
-if [ -f .pi/model-router.json ]; then
-  echo "✓ .pi/model-router.json already exists — preserving user config"
-else
-  node << 'GENDONE'
-const fs = require('fs');
-const path = '.pi/model-router.json';
-
-// --- Detect providers from env ---
-const hasOpenCode = process.env.OPENAI_API_BASE?.includes('opencode.ai');
-const hasOpenAI = !!process.env.OPENAI_API_KEY;
-const hasAnthropic = !!process.env.ANTHROPIC_API_KEY;
-const hasGoogle = !!process.env.GOOGLE_API_KEY;
-
-// If opencode gateway is detected, prefer opencode-go/ models
-// Otherwise use standard provider prefixes
-const P = hasOpenCode ? 'opencode-go' : 'openai';
-
-function model(prefix, name) { return `${prefix}/${name}`; }
-
-// Best available high-end model per provider
-const highModel = hasOpenCode
-  ? model('opencode-go', 'deepseek-v4-pro')
-  : hasAnthropic
-    ? 'anthropic/claude-sonnet-4-20250514'
-    : hasGoogle
-      ? 'google/gemini-2.5-flash-001'
-      : hasOpenAI
-        ? model('openai', 'gpt-4o')
-        : null;
-
-const mediumModel = hasOpenCode
-  ? model('opencode-go', 'qwen3.6-plus')
-  : hasAnthropic
-    ? 'anthropic/claude-sonnet-4-20250514'
-    : hasGoogle
-      ? 'google/gemini-flash-latest'
-      : hasOpenAI
-        ? model('openai', 'gpt-4o-mini')
-        : null;
-
-const lowModel = hasOpenCode
-  ? model('opencode-go', 'deepseek-v4-flash')
-  : hasAnthropic
-    ? 'anthropic/claude-3-5-haiku-20241022'
-    : hasGoogle
-      ? 'google/gemini-flash-lite-latest'
-      : hasOpenAI
-        ? model('openai', 'gpt-4o-mini')
-        : null;
-
-if (!highModel || !mediumModel || !lowModel) {
-  console.log('✗ No AI provider env detected — skip model-router.json (set OPENAI_API_BASE for opencode, or ANTHROPIC/GOOGLE/OPENAI keys)');
-  process.exit(0);
-}
-
-const fallbacks = [];
-if (hasAnthropic && !highModel.startsWith('anthropic/')) fallbacks.push('anthropic/claude-3-5-sonnet-20241022');
-if (hasGoogle && !highModel.startsWith('google/')) fallbacks.push('google/gemini-flash-latest');
-
-const config = {
-  defaultProfile: 'auto',
-  debug: false,
-  classifierModel: mediumModel,
-  phaseBias: 0.5,
-  maxSessionBudget: 1.0,
-  largeContextThreshold: 100000,
-  rules: [
-    {
-      matches: ['deploy', 'production', 'release'],
-      tier: 'high',
-      reason: 'Safety check for production tasks'
-    },
-    { matches: 'changelog', tier: 'low' }
-  ],
-  profiles: {
-    auto: {
-      high: { model: highModel, thinking: 'high', fallbacks },
-      medium: { model: mediumModel, thinking: 'medium' },
-      low: { model: lowModel, thinking: 'low' }
-    },
-    cheap: {
-      high: { model: mediumModel, thinking: 'low' },
-      medium: { model: lowModel, thinking: 'off' },
-      low: { model: lowModel, thinking: 'off' }
-    },
-    deep: {
-      high: { model: highModel, thinking: 'xhigh', fallbacks },
-      medium: { model: mediumModel, thinking: 'medium' },
-      low: { model: lowModel, thinking: 'low' }
-    }
-  }
-};
-
-fs.mkdirSync('.pi', { recursive: true });
-fs.writeFileSync(path, JSON.stringify(config, null, 2) + '\n');
-console.log('✓ Generated .pi/model-router.json from detected providers:');
-if (hasOpenCode) console.log('  Provider: opencode gateway');
-if (hasOpenAI) console.log('  Detected: OPENAI_API_KEY');
-if (hasAnthropic) console.log('  Detected: ANTHROPIC_API_KEY');
-if (hasGoogle) console.log('  Detected: GOOGLE_API_KEY');
-console.log(`  High tier: ${highModel}`);
-console.log(`  Medium tier: ${mediumModel}`);
-console.log(`  Low tier: ${lowModel}`);
-GENDONE
-fi
+# Generate from Pi registry (skips if .pi/model-router.json exists; --force to regenerate)
+node "$UP_PKG/.pi/scripts/harness-generate-model-router.mjs"
+# Preview only: node "$UP_PKG/.pi/scripts/harness-generate-model-router.mjs" --dry-run
 
 # Merge router defaults after config exists (never adds npm packages — router is vendored)
 node "$UP_PKG/.pi/scripts/harness-sync-model-router.mjs"
 ```
 
-Do NOT block. If generation fails, warn in report and continue (defaults script clears `defaultProvider` if it pointed at `router` while no config file exists).
+If generation prints "No authenticated Pi providers": warn in report — user should run **`/login`** in pi (or `pi login`) then re-run Step 3.5. Do **not** infer providers from `OPENAI_API_KEY` alone; pi sessions often use `opencode-go` via auth.json without those env vars.
+
+Do NOT block setup. If no config is written, `harness-sync-model-router.mjs` clears a premature `defaultProvider: "router"` in `.pi/settings.json`.
 
 **Router onboarding** — The vendored extension starts only after `.pi/model-router.json` appears. Running the script above prepares that file plus optional Pi defaults (**`router` / `auto`**) via `harness-sync-model-router.mjs` when `defaultProvider` was unset—then **`/reload`**.
 
 Manual override: **`/router profile auto`** anytime after reload if they changed defaults.
 
-## Step 3.6 — Seed `.pi/agents` (pi-subagents)
+## Step 3.6 — Harness agents (package-resolved)
 
-`@tintinweb/pi-subagents` reads agent definitions from **this project's** `.pi/agents/`, not from the installed npm tree. Copy packaged agents when missing (preserves user edits):
+`harness-subagents` loads agents from the installed **`ultimate-pi`** package (`$UP_PKG/.pi/agents/**`) with namespaced ids (`harness/planner`, `pi-pi/agent-expert`). **Do not copy** agents into the project unless you want a deliberate override.
+
+Optional per-repo overrides: place `.md` files at the **same relative path** (e.g. `.pi/agents/harness/planner.md` overrides the package planner).
+
+Verify manifest drift after `pi update ultimate-pi`:
 
 ```bash
-UP_PKG="$(node -p "require('path').dirname(require.resolve('ultimate-pi/package.json'))")"
-mkdir -p .pi/agents/harness .pi/agents/pi-pi
-for dir in harness pi-pi; do
-  [ -d "$UP_PKG/.pi/agents/$dir" ] || continue
-  for f in "$UP_PKG/.pi/agents/$dir"/*.md; do
-    [ -f "$f" ] || continue
-    base="$(basename "$f")"
-    [ -f ".pi/agents/$dir/$base" ] || cp "$f" ".pi/agents/$dir/$base"
-  done
-done
-echo "✓ .pi/agents (harness + pi-pi) seeded from package"
+node "$UP_PKG/.pi/scripts/harness-agents-manifest.mjs" --check
 ```
 
 ## Step 4 — Configuration Files
 
+### 4.0 — Project `.env` (non-destructive)
+
+Harness extensions read config from project-root `.env` via `dotenv-loader.ts` on session start. **Never overwrite** an existing `.env`.
+
+```bash
+# If .env exists: append only missing harness keys (preserves all current values)
+node "$UP_PKG/.pi/scripts/harness-sync-env.mjs"
+```
+
+If **no** `.env` at project root:
+
+- Unless `--non-interactive`, **call `ask_user`**:
+
+```json
+{
+  "question": "No .env at project root. Create one from the harness template?",
+  "context": "Non-destructive: only creates if missing; never overwrites existing files.",
+  "options": [
+    { "title": "Create from harness template", "description": "Runs harness-sync-env.mjs --create-missing" },
+    { "title": "Skip for now", "description": "Warn in report; user copies template manually later" }
+  ],
+  "allowFreeform": false
+}
+```
+
+- On **create**: `node "$UP_PKG/.pi/scripts/harness-sync-env.mjs" --create-missing`
+- On **skip** or `--non-interactive`: warn in report (non-interactive skips creation)
+- If `ask_user` cancelled: stop with `needs_clarification`
+
+Rules:
+
+- **Do not** `cp` over an existing `.env`.
+- **Do not** edit or remove keys the user already set.
+- Re-runs only add keys from `$UP_PKG/.pi/harness/env.harness.template` that are absent (managed block at EOF).
+- Ensure `.env` is gitignored (Step 4.1).
+
+Template keys (placeholders — user fills secrets): `HARNESS_TELEMETRY_ENABLED`, `HARNESS_WEB_*`, `PI_VCC_CONFIG_PATH`, plus commented optional PostHog / Graphify vars.
+
 ### 4.1 — .gitignore Entries
 
 Ensure `.gitignore` contains:
 ```
-.firecrawl/
+.env
+.web/
 .raw/
 .vault-meta/
 .pi/harness/critics/
@@ -668,9 +474,43 @@ This gives agents real-time access to structural health metrics:
 - `check_rules` — architectural constraint enforcement
 - `health`, `rescan`, `evolution`, `dsm`, `test_gaps`
 
-### 4.3 — Project AGENTS.md
+### 4.3 — Sentrux rules bootstrap (required)
+
+**Skill:** invoke **harness-sentrux-setup** before hand-editing rules or manifest.
+
+**Optional agent:** spawn `harness/sentrux-bootstrap` if Sentrux setup needs a dedicated pass.
+
+From **project root**, run the bundled bootstrap (seeds manifest when missing, syncs `.sentrux/rules.toml` without clobbering custom TOML):
+
+```bash
+node "$UP_PKG/.pi/scripts/harness-sentrux-bootstrap.mjs"
+# After editing architecture.manifest.json:
+node "$UP_PKG/.pi/scripts/harness-sentrux-bootstrap.mjs" --force
+# In pi: /harness-sentrux-sync  (always --force sync)
+```
+
+| Command | When |
+|---------|------|
+| `harness-sentrux-bootstrap.mjs` (no flags) | `/harness-setup`, first install, re-run safe |
+| `harness-sentrux-bootstrap.mjs --force` | Manifest layers/boundaries/constraints changed |
+| `sentrux-rules-sync.mjs --check` | CI / harness-verify drift only |
+| `/harness-sentrux-sync` | Interactive re-sync from pi |
+
+`harness-seed-project-contracts.mjs` (Step 0.5) may copy `architecture.manifest.json` early; bootstrap still personalizes `project` on first seed and writes `rules.toml`.
+
+Verify rules:
+```bash
+sentrux check . && echo "✓ sentrux rules pass" || echo "✗ sentrux check failed"
+```
+
+Set up structural regression baseline (optional):
+```bash
+sentrux gate --save . 2>/dev/null || echo "Baseline will be saved on first gate run"
+```
+
+### 4.4 — Project AGENTS.md
 
-Create a minimal `AGENTS.md` in the project root for agent onboarding:
+**Do not overwrite** an existing `AGENTS.md` — graphify bootstrap may have appended a `## Graphify` section. If missing, create minimal onboarding content; if present, only add harness subsections that are absent.
 
 ```markdown
 # ultimate-pi: Agentic Harness
@@ -686,7 +526,7 @@ Created: $(date +%Y-%m-%d)
 - .pi/harness/specs/ → Harness contracts and schema docs
 - .pi/harness/incidents/ → Incident and override records
 - `.agents/skills/` (npm package) → Harness skills (no copy into `.pi/skills/` needed)
-- `.pi/agents/` → Specialized agents (seed from package — see Step 3.6)
+- `.pi/agents/` → Optional per-repo agent overrides (package agents load automatically — see Step 3.6)
 
 ## Graphify-First Workflow
 
@@ -758,7 +598,7 @@ ls .pi/model-router.json 2>/dev/null && echo "✓ model-router config" || echo "
 ls -d ./raw 2>/dev/null && echo "✓ ./raw directory exists" || echo "! ./raw directory missing"
 
 # gitignore entries
-grep -q '.firecrawl/' .gitignore 2>/dev/null && echo "✓ .gitignore" || echo "! .gitignore missing entries"
+grep -q '.web/' .gitignore 2>/dev/null && echo "✓ .gitignore" || echo "! .gitignore missing entries"
 ```
 
 ## Step 6 — Graph Knowledge Report Bootstrap
@@ -772,7 +612,7 @@ import json
 with open('graphify-out/graph.json') as f:
     g = json.load(f)
 nodes = g['nodes']
-edges = g['edges']
+edges = g.get('edges', g.get('links', []))
 communities = len(set(n.get('community', 0) for n in nodes))
 god_nodes = sorted(nodes, key=lambda n: n.get('degree', 0), reverse=True)[:5]
 print(f'Nodes: {len(nodes)}  |  Edges: {len(edges)}  |  Communities: {communities}')
@@ -790,30 +630,31 @@ Output summary table:
 |-----------|--------|--------|
 | Knowledge Graph | ✓/✗ | `graphify-out/graph.json` — graph status |
 | Graphify Hooks | ✓/✗ | git post-commit/post-checkout hooks |
-| firecrawl-cli | ✓/✗ | Auth: yes/no |
+| scrapling / harness-web | ✓/✗ | Auth: yes/no |
 | ctx7 | ✓/✗ | Login: yes/no |
 | agent-browser | ✓/✗ | Config: .pi/harness/browser.json |
 | ck-search | ✓/✗ | MCP: registered/CLI-only |
 | biome | ✓/✗ | Project config: found/default |
 | ast-grep | ✓/✗ | AST-aware code search (`sg`)
 | gh CLI | ✓/✗ | Auth: yes/no |
-| sentrux | ✓/✗ | Version + plugins: 52 languages |
+| sentrux | ✓/✗ | CLI + plugins; rules via Step 4.3 bootstrap |
+| Sentrux rules.toml | ✓/✗ | `.sentrux/rules.toml` synced from manifest |
 | pi extensions | ✓/✗ | 4 packages |
 | model router | ✓/✗ | Package + config verified, activation via `/router profile auto` |
+| `.env` | ✓/✗/ask | Created / keys appended / user declined |
 
-| .gitignore | ✓/✗ | 7 entries added |
+| .gitignore | ✓/✗ | entries added (incl. `.env`) |
 | ./raw directory | ✓/✗ | Created for graphify source ingestion |
-| Firecrawl mode | self/cloud | Self-hosted on :3002 / Cloud (api.firecrawl.dev) |
-| Docker Engine | ✓/✗/N/A | Installed / Not needed (cloud mode) |
+| harness-web (Scrapling) | ✓/✗ | search + scrape smoke |
 
 Next steps:
 1. If tools missing: re-run with `--force` or install individually
 2. If graph not built: run `bash "$UP_PKG/.pi/scripts/harness-graphify-bootstrap.sh"` (or `graphify update .` from project root)
 3. If hooks not installed: run `graphify hook install`
 4. If gh not authenticated: `gh auth login`
-5. If self-hosted Firecrawl unhealthy: `docker compose -f firecrawl/docker-compose.yaml logs`
-6. If sentrux plugins missing: `sentrux plugin add-standard`
-7. First harness run: `/harness "your task description"`
+5. If sentrux plugins missing: `sentrux plugin add-standard`
+7. If rules.toml missing or out of date: `node "$UP_PKG/.pi/scripts/harness-sentrux-bootstrap.mjs" --force`
+8. First harness run: `/harness "your task description"`
 
 ## Guard Rails
 
@@ -823,12 +664,11 @@ Next steps:
 - **Graphify bootstrap is mandatory** (unless `--skip-graphify`): Run `bash "$UP_PKG/.pi/scripts/harness-graphify-bootstrap.sh"`. Never use `graphify . --wiki`. Initial setup must run `graphify update .` and verify `graphify-out/graph.json` has nodes.
 - **Python packages (Graphify)**: Before install, detect via PATH, `pip`/`pip3 show graphifyy`, `uv`, or apt. Prefer `uv tool install graphifyy`.
 - **Node.js >= 18 required**: Some pi packages use modern Node APIs.
-- **Docker required for self-hosted**: Step 1.5 needs Docker Engine + Compose. Block if install fails.
-- **Sufficient RAM for self-hosted**: Firecrawl stack needs ~8GB+ free (API: 8G, Playwright: 4G, others).
+- **Scrapling browsers**: `scrapling install` downloads Chromium (~hundreds of MB). Stealth scrape needs OS libs on Linux (see harness-cli-verify).
 - **Idempotent**: All checks skip if already installed. `--force` overrides.
 - **No destructive actions**: Creates files only if missing. Never overwrites existing content.
 - **Partial success**: If some tools fail, report which and continue. User can fix individually.
-- **Rate limits**: ctx7 login is optional. firecrawl auth is required for cloud; none needed for self-hosted.
+- **Rate limits**: ctx7 login is optional. harness-web has no API key; respect SERP/site rate limits.
 
 
 ## Error Handling
@@ -845,18 +685,17 @@ Next steps:
 | Invalid `graphify .` usage | Replace with `graphify update .` — the `.` subcommand does not exist. |
 | graphify-out empty / 0 nodes | Re-run `bash "$UP_PKG/.pi/scripts/harness-graphify-bootstrap.sh" --force` from project root. |
 | graphify hook install fails | Hooks need `.git/` directory. Verify inside git repo. Manual: `git config core.hooksPath .pi/git-hooks` |
-| firecrawl auth failed | Show manual login instructions. Continue with other tools. |
+| harness-web / scrapling failed | `uv tool install "scrapling[fetchers]" && scrapling install`; re-run harness-cli-verify. |
 | gh not installed | Show GitHub CLI install link. Skip label creation. |
 | pi packages install fail | Show error output. Check npm permissions. |
 | graph already exists | Report node count. Refresh with `graphify update .` unless user passed `--force`. |
 | biome.json missing | Create minimal config. |
 | settings.json not writable | Warn. Settings won't persist across sessions. |
 | No internet | Block for tool installs. Continue for graphify-only steps if `--skip-tools`. |
-| Docker not running | Start: `sudo systemctl start docker`. Block if cannot start. |
-| Docker install fails | Show manual link: https://docs.docker.com/engine/install/. Block Step 1.5, continue rest. |
-| Port 3002 already in use | Warn. User must free port or change `PORT` in `firecrawl/.env`. |
-| Self-hosted health check timeout | Show logs: `docker compose -f firecrawl/docker-compose.yaml logs`. Continue — may need more time. |
 | sentrux install fails | Show install script output. Fallback: download from https://github.com/sentrux/sentrux/releases/latest |
+| No model-router.json / "No authenticated Pi providers" | Run `/login` in pi, then `node "$UP_PKG/.pi/scripts/harness-generate-model-router.mjs" --force` |
+| UP_PKG not found | `pi install npm:ultimate-pi` or `npm i -g ultimate-pi`; verify with `node "$UP_PKG/.pi/scripts/harness-resolve-up-pkg.mjs"` |
+| No `.env` at project root | `ask_user` create vs skip; on create: `harness-sync-env.mjs --create-missing` |
 
 ## Flags
 
@@ -864,6 +703,6 @@ Next steps:
 |------|--------|
 | `--skip-graphify` | Skip Step 0.5 (graph build). Only when a valid `graphify-out/graph.json` already exists. |
 | `--skip-tools` | Skip Step 2 (CLI tool installs). Use when tools already set up. |
-| `--skip-firecrawl-self` | Skip Step 1.5 (self-hosted Firecrawl). Always use cloud. |
+| `--non-interactive` | Skip all `ask_user` prompts; skip `.env` creation with warning. CI/automation only. |
 | `--force` | Reinstall all tools even if already present. Overwrite existing files. |