@pi-unipi/compactor 0.1.1

package/README.md

@@ -0,0 +1,86 @@
+# @pi-unipi/compactor
+
+Context engine for Pi coding agent. Fuses zero-LLM compaction, session continuity, sandbox execution, FTS5 search, and tool display optimization into a single cohesive package.
+
+## Features
+
+- **Zero-LLM Compaction** — 6-stage pipeline (normalize → filter → build sections → brief → format → merge) achieves 95%+ token reduction with zero API cost
+- **Session Continuity** — XML resume snapshots survive compaction, preserving context across session boundaries
+- **Sandbox Execution** — 11 languages with process isolation, security hardening, and output capping
+- **FTS5 Search** — Full-text search over indexed content with auto-chunking
+- **Tool Display** — Mode-aware rendering for read, grep, find, ls, bash, edit, write tools
+
+## Commands
+
+| Command | Description |
+|---------|-------------|
+| `/unipi:compact` | Manual compaction with stats |
+| `/unipi:compact-recall` | Search session history |
+| `/unipi:compact-stats` | Context savings dashboard |
+| `/unipi:compact-doctor` | Run diagnostics |
+| `/unipi:compact-settings` | TUI settings overlay |
+| `/unipi:compact-preset <name>` | Apply quick preset |
+| `/unipi:compact-index` | Index current project |
+| `/unipi:compact-search` | Search indexed content |
+| `/unipi:compact-purge` | Wipe all indexed content |
+
+## Tools
+
+| Tool | Description |
+|------|-------------|
+| `compact` | Trigger manual compaction |
+| `vcc_recall` | BM25 session history search |
+| `ctx_execute` | Run code, stdout enters context |
+| `ctx_execute_file` | Process file via FILE_CONTENT |
+| `ctx_batch_execute` | Atomic batch of commands + searches |
+| `ctx_index` | Chunk content → FTS5 index |
+| `ctx_search` | Query indexed content |
+| `ctx_fetch_and_index` | Fetch URL → index |
+| `ctx_stats` | Context savings dashboard |
+| `ctx_doctor` | Diagnostics checklist |
+
+## Configuration
+
+Config lives at `~/.unipi/config/compactor/config.json`. Each of the 9 strategies can be toggled on/off and cycled through modes.
+
+### Presets
+
+| Preset | Description |
+|--------|-------------|
+| `opencode` | Maximal context preservation, minimal display |
+| `balanced` | Moderate across all strategies |
+| `verbose` | Everything visible, everything tracked |
+| `minimal` | Only compaction + basic recall |
+
+Apply via `/unipi:compact-preset <name>`.
+
+## Architecture
+
+```
+┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
+│ Compaction Core │  │ Session Engine  │  │ Display Engine  │
+│ (zero-LLM)      │  │ (SQLite + XML)  │  │ (mode-aware)    │
+└────────┬────────┘  └────────┬────────┘  └────────┬────────┘
+         │                    │                    │
+         └────────────────────┼────────────────────┘
+                              ▼
+                    ┌─────────────────────┐
+                    │   Config Manager    │
+                    └─────────────────────┘
+```
+
+## Installation
+
+Included in `@pi-unipi/unipi` metapackage. To use standalone:
+
+```json
+{
+  "pi": {
+    "extensions": ["node_modules/@pi-unipi/compactor/src/index.ts"]
+  }
+}
+```
+
+## License
+
+MIT

package/package.json

@@ -0,0 +1,54 @@
+{
+  "name": "@pi-unipi/compactor",
+  "version": "0.1.1",
+  "description": "Context engine for Pi — zero-LLM compaction, session continuity, sandbox execution, FTS5 search, and tool display optimization",
+  "type": "module",
+  "license": "MIT",
+  "author": "Neuron Mr White",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Neuron-Mr-White/unipi.git",
+    "directory": "packages/compactor"
+  },
+  "keywords": [
+    "pi-package",
+    "pi-extension",
+    "pi-coding-agent",
+    "unipi",
+    "compactor",
+    "context",
+    "sandbox",
+    "fts5"
+  ],
+  "files": [
+    "src/**/*.ts",
+    "skills/**/*",
+    "README.md"
+  ],
+  "pi": {
+    "extensions": [
+      "src/index.ts"
+    ],
+    "skills": [
+      "skills"
+    ]
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "@pi-unipi/core": "*"
+  },
+  "peerDependencies": {
+    "@mariozechner/pi-coding-agent": "*",
+    "@mariozechner/pi-tui": "*",
+    "@sinclair/typebox": "*"
+  },
+  "devDependencies": {
+    "@types/node": "^25.6.0",
+    "typescript": "^6.0.0"
+  },
+  "optionalDependencies": {
+    "better-sqlite3": ">=11.0.0"
+  }
+}

package/skills/compactor/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: compactor
+description: Routing decision tree for compactor — when to compact, search, index, or diagnose.
+---
+
+# Compactor Routing
+
+Use this skill to decide which compactor action to take.
+
+## Decision Tree
+
+```
+User wants to...
+├── Reduce context / free tokens
+│   └── /unipi:compact or `compact` tool
+│
+├── Find something from earlier in the session
+│   └── /unipi:compact-recall <query> or `vcc_recall` tool
+│
+├── Run code safely
+│   └── `ctx_execute` tool (single) or `ctx_batch_execute` (batch)
+│
+├── Index project files for search
+│   └── /unipi:compact-index or `ctx_index` tool
+│
+├── Search indexed content
+│   └── /unipi:compact-search <query> or `ctx_search` tool
+│
+├── Fetch and index a URL
+│   └── `ctx_fetch_and_index` tool
+│
+├── View compactor stats
+│   └── /unipi:compact-stats or `ctx_stats` tool
+│
+├── Diagnose issues
+│   └── /unipi:compact-doctor or `ctx_doctor` tool
+│
+├── Change settings
+│   └── /unipi:compact-settings (TUI overlay)
+│   └── /unipi:compact-preset <name> (quick preset)
+│
+└── Wipe indexed content
+    └── /unipi:compact-purge
+```
+
+## When to Compact
+
+- Context window > 80% full
+- Session has > 100 messages
+- Before starting a complex multi-step task
+- When the agent starts repeating itself
+
+## When to Recall
+
+- User references something from earlier
+- Need to find a file path mentioned before
+- Looking for a previous error or decision
+- Searching for a specific code snippet
+
+## When to Index
+
+- Starting work on a large codebase
+- Need fast search across many files
+- Documentation or reference material
+- Before a research-heavy task
+
+## Presets
+
+| Preset | Best For |
+|--------|----------|
+| `opencode` | Code-heavy work, minimal context waste |
+| `balanced` | General use, good defaults |
+| `verbose` | Maximum context preservation |
+| `minimal` | Maximum token savings |

package/skills/compactor-doctor/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: compactor-doctor
+description: Diagnostics — validate config, DB, FTS5, runtimes, troubleshoot issues.
+---
+
+# Compactor Doctor
+
+Run diagnostics and troubleshoot compactor issues.
+
+## Commands
+
+- `/unipi:compact-doctor` — run all checks
+- `ctx_doctor` tool — agent-callable diagnostics
+
+## Checks Performed
+
+| Check | What It Validates |
+|-------|-------------------|
+| **Config file** | `~/.unipi/config/compactor/config.json` exists and is valid |
+| **Session DB** | SQLite connection works, schema correct |
+| **Content Store** | FTS5 index accessible, tables exist |
+| **Runtime: node** | Node.js available for sandbox |
+| **Runtime: python3** | Python 3 available for sandbox |
+| **Runtime: bash** | Bash available for sandbox |
+
+## Status Icons
+
+- ✅ **pass** — check succeeded
+- ⚠️ **warn** — non-critical issue (e.g., optional runtime missing)
+- ❌ **fail** — critical issue, feature may not work
+
+## Common Issues
+
+### "Config file: Using defaults"
+- Normal on first run
+- Config auto-created on next settings save
+- Fix: `/unipi:compact-settings` → save
+
+### "Session DB: Connection failed"
+- SQLite not available
+- Check if `better-sqlite3` is installed
+- Fix: `npm install better-sqlite3`
+
+### "Content Store: FTS5 error"
+- SQLite FTS5 extension not available
+- Requires SQLite 3.9+ with FTS5
+- Fix: Update system SQLite
+
+### "Runtime: python3 Not found"
+- Python not installed or not in PATH
+- Only needed for Python sandbox execution
+- Fix: Install Python 3 or ignore if not needed
+
+## Manual Diagnostics
+
+### Check config
+```bash
+cat ~/.unipi/config/compactor/config.json
+```
+
+### Check DB files
+```bash
+ls -la ~/.unipi/db/compactor/
+```
+
+### Check SQLite version
+```bash
+sqlite3 --version
+```
+
+### Test FTS5
+```bash
+sqlite3 ':memory:' "CREATE VIRTUAL TABLE t USING fts5(x); SELECT 1;"
+```

package/skills/compactor-ops/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: compactor-ops
+description: Engineering ops orchestration — batch operations, project indexing, maintenance.
+---
+
+# Compactor Ops
+
+Orchestrate batch operations, project indexing, and maintenance tasks.
+
+## Project Indexing Workflow
+
+When starting work on a new codebase:
+
+1. **Index project files** — `/unipi:compact-index` or `ctx_index` for specific files
+2. **Verify index** — `/unipi:compact-stats` to check chunk count
+3. **Test search** — `/unipi:compact-search <term>` to verify quality
+
+## Batch Code Execution
+
+Use `ctx_batch_execute` for atomic multi-step operations:
+
+```json
+{
+  "items": [
+    { "type": "execute", "language": "shell", "code": "ls -la src/" },
+    { "type": "execute", "language": "python", "code": "import ast; print('OK')" },
+    { "type": "search", "query": "authentication module" }
+  ]
+}
+```
+
+## Maintenance Tasks
+
+### Clear stale index
+```
+/unipi:compact-purge
+/unipi:compact-index
+```
+
+### Reset configuration
+```
+/unipi:compact-preset balanced
+```
+
+### Full diagnostics
+```
+/unipi:compact-doctor
+```
+
+## Integration Patterns
+
+### Before complex task
+1. `/unipi:compact` — free up context
+2. `/unipi:compact-index` — index relevant files
+3. Start work with fresh context + indexed search
+
+### After long session
+1. `/unipi:compact-stats` — check savings
+2. `/unipi:compact` — compact if needed
+3. `/unipi:compact-recall <topic>` — verify recall quality
+
+### Research workflow
+1. `ctx_fetch_and_index` — index reference docs
+2. `ctx_search` — find relevant sections
+3. `ctx_execute` — test hypotheses in sandbox

package/skills/compactor-stats/SKILL.md

@@ -0,0 +1,49 @@
+---
+name: compactor-stats
+description: Stats display — context savings, session metrics, index health.
+---
+
+# Compactor Stats
+
+Display and interpret compactor statistics.
+
+## Commands
+
+- `/unipi:compact-stats` — quick dashboard
+- `ctx_stats` tool — detailed stats (agent-callable)
+
+## Metrics Explained
+
+| Metric | Description |
+|--------|-------------|
+| **Session events** | Total events tracked in current session |
+| **Compactions** | Number of times context was compacted |
+| **Tokens saved** | Estimated tokens freed by compaction |
+| **Indexed docs** | Number of files/sources in FTS5 index |
+| **Indexed chunks** | Total searchable chunks |
+| **Sandbox runs** | Code executions in sandbox |
+| **Search queries** | FTS5 searches performed |
+
+## Health Indicators
+
+- **Compactions > 0** on long sessions = working correctly
+- **Indexed chunks > 0** = search available
+- **Sandbox runs** = code execution active
+
+## Reading Stats
+
+```
+📊 Compactor Stats
+Session events: 42
+Compactions: 3
+Tokens saved: 15000
+Indexed docs: 12 (48 chunks)
+Sandbox runs: 7
+Search queries: 15
+```
+
+This means:
+- 42 tool calls/events tracked
+- 3 compactions saved ~15K tokens
+- 12 files indexed into 48 searchable chunks
+- 7 code executions, 15 searches performed

package/skills/compactor-tools/SKILL.md

@@ -0,0 +1,120 @@
+---
+name: compactor-tools
+description: Reference card for all compactor tools — parameters, usage, examples.
+---
+
+# Compactor Tools Reference
+
+## compact
+
+Trigger manual context compaction.
+
+```
+compact()
+```
+
+- Returns compaction stats (summarized, kept, tokens estimated)
+- Actual work done by `session_before_compact` hook
+- Zero LLM calls — pure regex/text processing
+
+## vcc_recall
+
+Search session history using BM25 or regex.
+
+```
+vcc_recall(query: string, mode?: "bm25" | "regex", limit?: number, offset?: number, expand?: boolean)
+```
+
+- **query**: Search terms
+- **mode**: `bm25` (default, ranked) or `regex` (pattern match)
+- **limit**: Max results (default 10)
+- **offset**: Pagination offset
+- **expand**: Return full message content for hits
+
+## ctx_execute
+
+Run code in a sandboxed environment.
+
+```
+ctx_execute(language: string, code: string, timeout?: number)
+```
+
+**Supported languages:** javascript, typescript, python, shell, ruby, go, rust, php, perl, r, elixir
+
+- Only stdout enters context (stderr filtered)
+- 100MB output cap with process kill
+- 30s default timeout
+- Auto-detects Bun for JS/TS
+
+## ctx_execute_file
+
+Execute a file with content injected as `FILE_CONTENT`.
+
+```
+ctx_execute_file(language: string, path: string, timeout?: number)
+```
+
+## ctx_batch_execute
+
+Run multiple commands atomically.
+
+```
+ctx_batch_execute(items: Array<ExecuteItem | SearchItem>)
+```
+
+- Execute items: `{ type: "execute", language, code, timeout? }`
+- Search items: `{ type: "search", query, limit? }`
+
+## ctx_index
+
+Index content into FTS5 for fast search.
+
+```
+ctx_index(label: string, content?: string, filePath?: string, contentType?: "markdown"|"json"|"plain", chunkSize?: number)
+```
+
+- Provide either `content` or `filePath`
+- Auto-chunks by content type
+- Deduplicates by label
+
+## ctx_search
+
+Query indexed content.
+
+```
+ctx_search(query: string, limit?: number, offset?: number)
+```
+
+- Returns ranked results with title, content, source, rank
+- Supports pagination via offset/limit
+
+## ctx_fetch_and_index
+
+Fetch URL → markdown → index.
+
+```
+ctx_fetch_and_index(url: string, label?: string, chunkSize?: number)
+```
+
+- Converts HTML to markdown
+- Auto-indexes for later search
+
+## ctx_stats
+
+Context savings dashboard.
+
+```
+ctx_stats()
+```
+
+Returns: session events, compactions, tokens saved, indexed docs, sandbox runs, search queries.
+
+## ctx_doctor
+
+Diagnostics checklist.
+
+```
+ctx_doctor()
+```
+
+Checks: config file, session DB, content store, runtimes (node, python3, bash).