ultimate-pi 0.1.2 → 0.1.4

package/.pi/settings.json

@@ -1,5 +1,15 @@
 {
-  "packages": [
-    ".."
-  ]
-}
+	"defaultProvider": "router",
+	"defaultModel": "auto",
+	"retry": {
+		"enabled": true,
+		"maxRetries": 3
+	},
+	"packages": [
+		"..",
+		"npm:@posthog/pi",
+		"npm:pi-lean-ctx",
+		"npm:@tintinweb/pi-subagents",
+		"npm:@yeliu84/pi-model-router"
+	]
+}

package/.pi/skills/agent-router/SKILL.md

@@ -0,0 +1,174 @@
+---
+name: agent-router
+description: >
+  Dynamic agent discovery, matching, and delegation. Discovers all available agents
+  at runtime, reads their frontmatter to understand capabilities, matches tasks to
+  the right specialist, and routes via the Agent tool. Use when the user needs
+  delegation, asks which agent can help, wants parallel dispatch, or says
+  "delegate this", "find an agent for", "route to", "dispatch", "which agent".
+allowed-tools: Read Glob Find Bash
+---
+
+# agent-router: Dynamic Agent Discovery & Routing
+
+You don't memorize agent lists. You discover them live, read their capabilities,
+and route tasks to the right specialists.
+
+## Why Dynamic Discovery
+
+Agent definitions live in two locations (project and global). They change over
+time. New agents get added. Default agents get overridden. A static list in
+SYSTEM.md rots. Dynamic discovery tells you what's actually available right now.
+
+## Discovery
+
+### Step 1 — List All Available Agents
+
+```bash
+find .pi/agents -name '*.md' -type f 2>/dev/null
+find ~/.pi/agent/agents -name '*.md' -type f 2>/dev/null
+```
+
+Project agents (`.pi/agents/`) override identically-named global ones. Default
+agents (`Explore`, `Plan`, `general-purpose`) are always available even without
+files here.
+
+### Step 2 — Parse Agent Frontmatter
+
+For each candidate agent, read the file and extract:
+
+| Field | Extract this |
+|-------|-------------|
+| `description` | What the agent does, when to use it |
+| `tools` | Available built-in tools |
+| `thinking` | Reasoning depth: `off`, `low`, `medium`, `high`, `xhigh` |
+| `max_turns` | Turn limit (0 = unlimited) |
+| `model` | Model override if specified |
+| `isolation` | `worktree` if isolated git execution |
+
+> [!info] Subdirectory agents
+> Agents in `.pi/agents/<team>/<name>.md` are referenced as `<team>/<name>`.
+> Example: `pi-pi/agent-expert` lives at `.pi/agents/pi-pi/agent-expert.md`.
+
+### Step 3 — Cache Discovery Results
+
+After discovery, summarize available agents for the current session. No need
+to re-read frontmatter for already-profiled agents.
+
+## Matching Tasks to Agents
+
+### Direct Match — Keyword Triggers
+
+| User says | Look for agent with |
+|-----------|-------------------|
+| Explicit agent name | Match directly |
+| "build a skill" / "create skill" | Skill-related keywords in description |
+| "create an agent" | Agent-related keywords in description |
+| "Pi extension" / "Pi theme" | Pi component domain keywords |
+| "rethink" / "first principles" | Prioritization keywords |
+| "ingest to wiki" | Wiki ingestion keywords |
+| "lint the wiki" | Wiki lint keywords |
+
+### Fuzzy Match — Semantic Fit
+
+When no direct keyword match:
+
+1. Read the task description carefully.
+2. Compare against each agent's `description` field.
+3. Pick the agent whose description most closely matches the task domain.
+4. If uncertain, read the agent's full prompt body for more context.
+5. If nothing fits, use `general-purpose`.
+
+### Orchestrator Pattern
+
+Some agents are meta-agents that coordinate teams. When an orchestrator agent
+exists for a domain (e.g., `pi-pi/pi-orchestrator`), route domain tasks to
+the orchestrator — not individual experts. The orchestrator knows how to
+dispatch its team members in parallel.
+
+To identify orchestrators: look for agents whose descriptions mention
+"coordinates", "dispatches", "orchestrates", "meta-agent", or whose tools
+include `Agent`.
+
+## Routing
+
+### Dispatch
+
+Use the `Agent` tool:
+
+```typescript
+Agent({
+  subagent_type: "<team>/<name>",
+  prompt: "<specific, scoped task description>",
+  run_in_background: true,  // for parallel dispatch
+  inherit_context: false,   // fresh context by default
+})
+```
+
+### Parallel Dispatch
+
+When a task touches multiple domains:
+1. Identify all relevant agents.
+2. Dispatch all of them with `run_in_background: true`.
+3. Collect results with `get_subagent_result`.
+4. Synthesize findings.
+
+### Sequential Pipelines
+
+When one agent's output feeds another:
+1. Dispatch first agent (foreground or background).
+2. Collect result.
+3. Feed result into second agent's prompt.
+4. Continue until pipeline completes.
+
+### Collecting Results
+
+```typescript
+get_subagent_result({ agent_id: "<id>", wait: true })
+```
+
+Summarize agent findings for the user. Don't dump raw output — extract
+key decisions, actionable outputs, and surprises.
+
+## Delegation Rules
+
+1. **Agents are for multi-turn, tool-using workflows.** For single-step template
+   expansion (static prompt + argument substitution), use a prompt template instead.
+2. **One job per agent.** If the task needs "and" in the description, split into
+   separate dispatches.
+3. **Match tools + thinking to task difficulty.** A file-lookup task doesn't need
+   `bash` + `medium` thinking. A novel architecture decision needs `high`.
+4. **Parallelize by default.** When multiple domains are relevant, dispatch all
+   agents simultaneously. Serial dispatch wastes wall-clock time.
+5. **Experts research, orchestrators build.** Don't ask a domain expert to write
+   files if an orchestrator exists for that domain.
+6. **Always read frontmatter before dispatching.** Don't assume an agent's
+   capabilities from its name alone.
+7. **If no agent fits, fall back to `general-purpose`.** Then consider whether
+   a new agent should be created for the missing capability.
+
+## Discovery Example
+
+```bash
+# Find all agents
+$ find .pi/agents -name '*.md' -type f
+
+# Sample output:
+# .pi/agents/pi-pi/pi-orchestrator.md
+# .pi/agents/pi-pi/agent-expert.md
+# .pi/agents/pi-pi/skill-expert.md
+# .pi/agents/rethink.md
+# .pi/agents/wiki-ingest.md
+# .pi/agents/wiki-lint.md
+
+# Then read each to understand capabilities
+# Then match user task to the right agent(s)
+```
+
+## Guardrails
+
+- Do not hardcode agent lists. Always discover dynamically.
+- Do not dispatch an agent you haven't read the frontmatter for.
+- Do not use an agent when a prompt template suffices.
+- If discovery returns nothing, use default agents (`Explore`, `Plan`, `general-purpose`).
+- Report routing decisions: which agent(s) dispatched, why, what to expect.

package/.pi/sounds/project-sounds.json

@@ -0,0 +1,25 @@
+{
+    "themeName": "ultimate-pi-custom",
+    "randomizeSounds": true,
+    "sounds": {
+        "success": [
+            "success/1-tada-track.mp3",
+            "success/2-jobs-done-track.mp3",
+            "success/3-yay-track.mp3"
+        ],
+        "error": [
+            "error/1-ksi-wth-track.mp3",
+            "error/2-smash-track.mp3",
+            "error/3-buzzer-track.mp3"
+        ],
+        "alert": [
+            "alert/1-kaching-track.mp3"
+        ],
+        "notification": [
+            "notification/1-soft-notification-track.mp3"
+        ],
+        "reminder": [
+            "reminder/1-soft-notification-track.mp3"
+        ]
+    }
+}

package/CONTRIBUTING.md

@@ -0,0 +1,116 @@
+# Contributing to ultimate-pi
+
+## Local development setup
+
+1. Clone and install dependencies:
+
+   ```bash
+   git clone https://github.com/aryaniyaps/ultimate-pi.git
+   cd ultimate-pi
+   npm install
+   ```
+
+   `npm install` automatically sets up pre-commit hooks via [Lefthook](https://github.com/evilmartians/lefthook).
+
+2. Install the package locally into PI:
+
+   ```bash
+   pi install . -l
+   ```
+
+   Then restart PI or run `/reload`.
+
+## Linting & formatting
+
+Uses [Biome](https://biomejs.dev) for linting, formatting, and import sorting.
+
+```bash
+npm run lint            # check lint + format errors
+npm run lint:fix        # auto-fix lint + format errors
+npm run format          # format all files
+npm run format:check    # check formatting without writing
+npm run check:ts        # typecheck extensions
+```
+
+Pre-commit hooks run `biome check` and `tsc` on staged files automatically.
+
+## Firecrawl (self-hosted web scraping)
+
+The Firecrawl skill depends on a Firecrawl instance. This repo includes a self-hosted setup powered by Docker.
+
+### Quick start
+
+```bash
+cd firecrawl
+cp .env.template .env   # first time only — edit if needed
+docker compose up -d     # pulls pre-built GHCR images automatically
+```
+
+Firecrawl API is now at `http://localhost:3002`. Admin UI at `http://localhost:3002/admin/<BULL_AUTH_KEY>/queues`.
+
+### Services
+
+| Service | Image | Port |
+|---------|-------|------|
+| `api` | `ghcr.io/firecrawl/firecrawl` | 3002 |
+| `playwright-service` | `ghcr.io/firecrawl/playwright-service:latest` | 3000 (internal) |
+| `nuq-postgres` | `ghcr.io/firecrawl/nuq-postgres:latest` | 5432 (internal) |
+| `redis` | `redis:alpine` | 6379 (internal) |
+| `rabbitmq` | `rabbitmq:3-management` | 5672 (internal) |
+| `searxng` | `searxng/searxng:latest` | 8080 |
+
+### Configuration
+
+All options live in `firecrawl/.env`. See `firecrawl/.env.template` for the full reference. Key env vars:
+
+- `PORT` — API port (default: `3002`)
+- `SEARXNG_ENDPOINT` — enables `/search` API (default: `http://searxng:8080`)
+- `OPENAI_API_KEY` — enables AI features (JSON formatting, `/extract` API)
+- `BULL_AUTH_KEY` — admin UI access key (default: `CHANGEME` — change in production)
+
+See `firecrawl/README.md` for detailed docs and SDK usage examples.
+
+## Extensions
+
+### Dotenv loader
+
+`.pi/extensions/dotenv-loader.ts` — loads `.env` files into `process.env` on session start.
+
+Configurable via env vars (set before launching pi):
+
+| Variable | Default | Description |
+|---|---|---|
+| `ENV_LOADER_FILES` | `.env` | Comma-separated list of `.env` file paths (relative to cwd). |
+| `ENV_LOADER_OVERRIDE` | `false` | Set to `true` to overwrite existing env vars. |
+| `ENV_LOADER_SILENT` | `false` | Set to `true` to suppress startup logs. |
+| `ENV_LOADER_ENCODING` | `utf-8` | File encoding for `.env` files. |
+
+- Supports variable expansion (`$VAR` and `${VAR}`).
+- Reloads on `/reload`.
+- Status command: `/env-loader-status`
+
+### PostHog analytics
+
+`@posthog/pi` — wraps the upstream [posthog-pi](https://github.com/PostHog/posthog-pi) extension to capture AI generation spans, tool spans, and traces in [PostHog](https://posthog.com). Install via `pi install @posthog/pi`. See the upstream repo for configuration and env vars.
+
+## Skill sources
+
+| Skill | Upstream |
+|---|---|
+| caveman | [juliusbrussee/caveman](https://github.com/juliusbrussee/caveman) |
+| context7-cli | [upstash/context7](https://github.com/upstash/context7) |
+| find-skills | bundled (context7-compatible discovery) |
+| firecrawl (13 skills) | [firecrawl](https://firecrawl.dev) |
+| obsidian/wiki skills (11 skills) | [AgriciDaniel/claude-obsidian](https://github.com/AgriciDaniel/claude-obsidian) |
+| posthog-analyst | bundled (PostHog MCP integration) |
+| defuddle | bundled (web content cleaning) |
+
+### Firecrawl sub-skills
+
+`firecrawl-search`, `firecrawl-scrape`, `firecrawl-crawl`, `firecrawl-map`, `firecrawl-download`, `firecrawl-parse`, `firecrawl-interact`, `firecrawl-agent`, `firecrawl-build-scrape`, `firecrawl-build-search`, `firecrawl-build-onboarding`, `firecrawl-build-interact`
+
+### Wiki sub-skills
+
+`wiki`, `wiki-save`, `wiki-query`, `wiki-ingest`, `wiki-lint`, `wiki-fold`, `autoresearch`, `canvas`, `obsidian-markdown`, `obsidian-bases`
+
+> `lean-ctx` is installed as a separate pi package (`pi-lean-ctx`) — not bundled as a skill.

package/README.md

@@ -1,58 +1,51 @@
 ![Ultimate PI banner](https://raw.githubusercontent.com/aryaniyaps/ultimate-pi/main/.github/banner-v2.png)
 
-> (Beta) The **ultimate AI coding harness** on top of [**pi.dev**](https://pi.dev).
+> The **ultimate AI coding harness** on top of [**pi.dev**](https://pi.dev).
 
 ## Goal
 
-Ship a production-grade coding harness where the agent:
-1. uses compressed context before raw tool spam,
-2. stays concise and deterministic in output style,
-3. remains easy to extend with skills and extensions.
+A production-grade coding harness where the agent:
+1. Uses compressed context before raw tool spam,
+2. Stays concise and deterministic in output style,
+3. Is easy to extend with skills and extensions.
 
-### Included skills (upstream repositories)
+## Getting started
 
-| Skill | Upstream repo | What it does |
-|---|---|---|
-| caveman | [juliusbrussee/caveman](https://github.com/juliusbrussee/caveman) | Ultra-compressed response style for token-efficient, high-signal communication. |
-| compress | [juliusbrussee/caveman](https://github.com/juliusbrussee/caveman) | Compresses memory/context files into caveman format while preserving technical content. |
-| lean-ctx | [yvgude/lean-ctx](https://github.com/yvgude/lean-ctx) | Context runtime and compression toolkit for reads/search/shell/edit flows. |
-| context7-cli | [upstash/context7](https://github.com/upstash/context7) | Pulls current library docs and manages Context7 skills/config from CLI. |
-| emil-design-eng | [emilkowalski/skill](https://github.com/emilkowalski/skill) | UI/design craftsmanship guidance: interaction polish, motion, and component quality. |
-| scrapling-official | [D4Vinci/Scrapling](https://github.com/D4Vinci/Scrapling) | Official Scrapling skill for web extraction/scraping with adaptive parsing, JS rendering, stealth fetchers, and anti-bot handling. |
+```bash
+npm install -g ultimate-pi
+```
 
-### Runtime guardrail extension
+From your project folder:
 
-- `extensions/lean-ctx-enforce.ts`
-  - Detects `lean-ctx` availability.
-  - Blocks built-ins (`read`, `write`, `edit`, `grep`, `find`, `ls`, raw `bash`) when `lean-ctx` exists.
-  - Allows shell only via `lean-ctx -c <command>`.
-  - Adds status commands:
-    - `/lean-ctx-status`
-    - `/system-prompt-status`
+```bash
+pi install ultimate-pi -l
+```
+
+Then reload PI:
+
+```bash
+/reload
+```
 
-## Design choices (concise)
+Optional: install from GitHub Packages:
 
-1. **Lean-ctx-first execution**  
-   Reason: cut context/token waste and keep tool usage structured.
+```bash
+npm install -g @aryaniyaps/ultimate-pi --registry=https://npm.pkg.github.com
+```
 
-2. **Caveman-by-default response style**  
-   Reason: short, exact, low-noise outputs for coding loops.
+## Bootstrap
 
-3. **Policy in system prompt + enforcement in extension**  
-   Reason: prompt sets intent; extension guarantees behavior.
+Run the harness setup prompt to configure everything (wiki, firecrawl, env vars, extensions):
 
-4. **Skill-based composition**  
-   Reason: easy to swap/upgrade capabilities without rewiring core package.
+```
+/harness-setup
+```
 
-5. **Model-agnostic compression path**  
-   Reason: `compress` skill calls PI CLI, so provider/model can change per environment.
+## Wiki
 
-## Use locally
+The harness includes an Obsidian wiki vault as its knowledge layer. Run `/wiki` in your PI session to scaffold it, then open the vault in [Obsidian](https://obsidian.md). Recommended community plugins: Dataview, Graph Analysis, Obsidian Git.
 
-From repo root:
 
-```bash
-pi install . -l
-```
+## Contributing
 
-Then restart PI or run `/reload`.
+See [CONTRIBUTING.md](./CONTRIBUTING.md) for local development setup, linting, and extension config details.

package/biome.json

@@ -0,0 +1,34 @@
+{
+	"$schema": "https://biomejs.dev/schemas/2.0.6/schema.json",
+	"vcs": {
+		"enabled": true,
+		"clientKind": "git",
+		"useIgnoreFile": true
+	},
+	"files": {
+		"ignoreUnknown": false
+	},
+	"formatter": {
+		"enabled": true,
+		"indentStyle": "tab"
+	},
+	"linter": {
+		"enabled": true,
+		"rules": {
+			"recommended": true
+		}
+	},
+	"javascript": {
+		"formatter": {
+			"quoteStyle": "double"
+		}
+	},
+	"assist": {
+		"enabled": true,
+		"actions": {
+			"source": {
+				"organizeImports": "on"
+			}
+		}
+	}
+}

package/firecrawl/.env.template

@@ -0,0 +1,58 @@
+# ===== Required ENVS ======
+PORT=3002
+HOST=0.0.0.0
+USE_DB_AUTHENTICATION=false
+
+# ===== Optional ENVS ======
+# OpenAI API key for AI features (JSON formatting, /extract API)
+# OPENAI_API_KEY=
+
+# Experimental: Use Ollama
+# OLLAMA_BASE_URL=http://localhost:11434/api
+# MODEL_NAME=deepseek-r1:7b
+# MODEL_EMBEDDING_NAME=nomic-embed-text
+
+# Experimental: Use any OpenAI-compatible API
+# OPENAI_BASE_URL=https://example.com/v1
+# OPENAI_API_KEY=
+
+# ===== Proxy =====
+# PROXY_SERVER can be a full URL (e.g. http://0.1.2.3:1234) or just an IP:port combo
+# PROXY_SERVER=
+# PROXY_USERNAME=
+# PROXY_PASSWORD=
+
+# ===== /search API =====
+# SearXNG instance running alongside Firecrawl in docker-compose
+SEARXNG_ENDPOINT=http://searxng:8080
+# SEARXNG_ENGINES=
+# SEARXNG_CATEGORIES=
+
+# ===== Authentication / Admin =====
+# Supabase not configured (self-hosted doesn't support it yet)
+# SUPABASE_ANON_TOKEN=
+# SUPABASE_URL=
+# SUPABASE_SERVICE_TOKEN=
+
+# Change this for security in production
+BULL_AUTH_KEY=CHANGEME
+
+# ===== PostgreSQL =====
+POSTGRES_USER=firecrawl
+POSTGRES_PASSWORD=firecrawl_password
+POSTGRES_DB=postgres
+
+# ===== Optional: Telemetry / Logging =====
+# POSTHOG_API_KEY=
+# POSTHOG_HOST=
+# SLACK_WEBHOOK_URL=
+
+# ===== Optional: Local webhooks =====
+# ALLOW_LOCAL_WEBHOOKS=true
+
+# ===== Auto-configured by docker-compose (do not change) =====
+# REDIS_URL=redis://redis:6379
+# PLAYWRIGHT_MICROSERVICE_URL=http://playwright-service:3000/scrape
+
+# ===== Optional: Block media (images, video, etc.) during scrape =====
+# BLOCK_MEDIA=true

package/firecrawl/README.md

@@ -0,0 +1,49 @@
+# Firecrawl Self-Hosted
+
+Web scraping infrastructure for the pi harness. Runs entirely in Docker — no local source checkout needed.
+
+## Quick Start
+
+```bash
+# 1. Create your env file
+cp .env.template .env
+# Edit .env if needed (defaults work for local dev)
+
+# 2. Pull pre-built images and start
+docker compose -f firecrawl/docker-compose.yaml up -d
+
+# 3. Verify
+curl http://localhost:3002/v1/health
+```
+
+## Services
+
+| Service | Port | Description |
+|---------|------|-------------|
+| `api` | 3002 | Firecrawl API + Bull queue UI at `/admin/<BULL_AUTH_KEY>/queues` |
+| `playwright-service` | 3000 (internal) | Headless browser rendering |
+| `redis` | 6379 (internal) | Job queue backing store |
+| `rabbitmq` | 5672 (internal) | Message broker for workers |
+| `nuq-postgres` | 5432 (internal) | Crawl state persistence |
+| `searxng` | 8080 | Metasearch engine for `/search` API |
+| `searxng-valkey` | 6379 (internal) | SearXNG cache |
+
+## Configuration
+
+All config lives in `.env`. See `.env.template` for all available options.
+
+### Enabling AI features
+
+Uncomment and set `OPENAI_API_KEY` in `.env`. Also supports Ollama and any OpenAI-compatible endpoint.
+
+### Using local builds instead of pre-built images
+
+Flip the `image:` ↔ `build:` lines in `docker-compose.yaml` for `api`, `playwright-service`, and `nuq-postgres`. You'll need the Firecrawl repo checked out alongside this one at `../firecrawl-selfhosted/apps/`.
+
+## SDK Usage
+
+When using Firecrawl SDKs with a self-hosted instance, pass the base URL — no API key required:
+
+```js
+const client = new Firecrawl({ apiKey: "", baseUrl: "http://localhost:3002" });
+```