@pi-unipi/compactor 0.1.6 → 0.2.1

package/README.md

@@ -15,45 +15,71 @@ Context engine for Pi coding agent. Fuses zero-LLM compaction, session continuit
 | Command | Description |
 |---------|-------------|
 | `/unipi:compact` | Manual compaction with stats |
-| `/unipi:compact-recall` | Search session history |
+| `/unipi:session-recall` | Search session history (BM25 or regex) |
+| `/unipi:content-index` | Index current project into FTS5 |
+| `/unipi:content-search` | Search indexed content |
+| `/unipi:content-purge` | Wipe all indexed content |
 | `/unipi:compact-stats` | Context savings dashboard |
 | `/unipi:compact-doctor` | Run diagnostics |
 | `/unipi:compact-settings` | TUI settings overlay |
 | `/unipi:compact-preset <name>` | Apply quick preset |
-| `/unipi:compact-index` | Index current project |
-| `/unipi:compact-search` | Search indexed content |
-| `/unipi:compact-purge` | Wipe all indexed content |
-
-## Tools
-
-| Tool | Description |
-|------|-------------|
-| `compact` | Trigger manual compaction |
-| `vcc_recall` | BM25 session history search |
-| `ctx_execute` | Run code, stdout enters context |
-| `ctx_execute_file` | Process file via FILE_CONTENT |
-| `ctx_batch_execute` | Atomic batch of commands + searches |
-| `ctx_index` | Chunk content → FTS5 index |
-| `ctx_search` | Query indexed content |
-| `ctx_fetch_and_index` | Fetch URL → index |
-| `ctx_stats` | Context savings dashboard |
-| `ctx_doctor` | Diagnostics checklist |
+| `/unipi:compact-help` | Show detailed documentation |
+
+## Agent Tools
+
+| Tool | Family | Description |
+|------|--------|-------------|
+| `compact` | compaction | Trigger manual compaction (dryRun: true to preview) |
+| `session_recall` | session | BM25 session history search |
+| `sandbox` | sandbox | Run code in sandbox (11 languages) |
+| `sandbox_file` | sandbox | Execute file via FILE_CONTENT |
+| `sandbox_batch` | sandbox | Atomic batch of commands + searches |
+| `content_index` | content | Chunk content → FTS5 index |
+| `content_search` | content | Query indexed content |
+| `content_fetch` | content | Fetch URL → index |
+| `compactor_stats` | compactor | Context savings dashboard |
+| `compactor_doctor` | compactor | Diagnostics checklist |
+| `context_budget` | compactor | Estimate remaining context window |
+
+## Two-Tier Skills
+
+- **Tier 1** (`compactor`): ~175 tokens, always loaded. Routing + critical rules + Ralph awareness.
+- **Tier 2** (`compactor-detail`): On-demand. Full tool reference, anti-patterns, sandbox languages, FTS5 modes, workflows.
 
 ## Configuration
 
-Config lives at `~/.unipi/config/compactor/config.json`. Each of the 9 strategies can be toggled on/off and cycled through modes.
+Config lives at `~/.unipi/config/compactor/config.json`. Per-project overrides at `<project>/.unipi/config/compactor.json`.
 
 ### Presets
 
 | Preset | Description |
 |--------|-------------|
-| `opencode` | Maximal context preservation, minimal display |
-| `balanced` | Moderate across all strategies |
-| `verbose` | Everything visible, everything tracked |
-| `minimal` | Only compaction + basic recall |
+| `precise` | Code-heavy, minimal waste — compaction: full, pipeline: 2/6 on |
+| `balanced` | Daily use (default) — all strategies moderate, pipeline: all on |
+| `thorough` | Debug/audit — everything on, full transcript |
+| `lean` | Quick fixes — compaction only, pipeline: all off |
 
 Apply via `/unipi:compact-preset <name>`.
 
+### Pipeline Features
+
+| Feature | Description | Context |
+|---------|-------------|---------|
+| TTL Cache | Cache with time-based expiry | On Compaction |
+| Auto Injection | Inject behavioral state after compaction | On Compaction |
+| MMap Pragma | Use mmap for SQLite I/O | On Compaction |
+| Proximity Reranking | Rerank search results by proximity | On Search |
+| Timeline Sort | Sort session events chronologically | On Search |
+| Progressive Throttling | Slow down indexing for large projects | On Index |
+
+### TUI
+
+Tabbed settings interface (Presets / Strategies / Pipeline):
+- `/` key opens search filter in Strategies tab
+- Preset selection shows 3-line preview
+- Per-project override checkbox (`o` key)
+- Keyboard: `←→` cycle modes, `Space` toggle, `s` save, `Esc` cancel
+
 ## Architecture
 
 ```

package/index.ts

@@ -0,0 +1,7 @@
+/**
+ * @pi-unipi/compactor — Root entry point
+ *
+ * Re-exports from src/index.ts for TypeScript module resolution.
+ */
+
+export { default } from "./src/index.js";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pi-unipi/compactor",
-  "version": "0.1.6",
+  "version": "0.2.1",
   "description": "Context engine for Pi — zero-LLM compaction, session continuity, sandbox execution, FTS5 search, and tool display optimization",
   "type": "module",
   "license": "MIT",
@@ -21,6 +21,7 @@
     "fts5"
   ],
   "files": [
+    "index.ts",
     "src/**/*.ts",
     "skills/**/*",
     "README.md"

package/skills/compactor/SKILL.md

@@ -1,74 +1,30 @@
 ---
 name: compactor
-description: Routing decision tree for compactor — when to compact, search, index, or diagnose.
+description: Context management — compact session, recall history, run code, search content.
 ---
 
-# Compactor Routing
+# Compactor — Context Management
 
-Use this skill to decide which compactor action to take.
+## When Context Is Tight
+- `context_budget` → check % full. `compact` → free tokens (zero-LLM, 98%+ reduction).
+  Compact BEFORE complex work. `compact(dryRun: true)` to preview without compacting.
+- `compactor_stats` → check savings. `compactor_doctor` → diagnose.
 
-## Decision Tree
+## Finding Past Work
+- `session_recall(query)` → search this session (BM25 or regex).
+- `content_search(query)` → search indexed files/docs.
+  → Index first: `content_index` or `content_fetch(url)`.
 
-```
-User wants to...
-├── Reduce context / free tokens
-│   └── /unipi:compact or `compact` tool
-│
-├── Find something from earlier in the session
-│   └── /unipi:compact-recall <query> or `vcc_recall` tool
-│
-├── Run code safely
-│   └── `ctx_execute` tool (single) or `ctx_batch_execute` (batch)
-│
-├── Index project files for search
-│   └── /unipi:compact-index or `ctx_index` tool
-│
-├── Search indexed content
-│   └── /unipi:compact-search <query> or `ctx_search` tool
-│
-├── Fetch and index a URL
-│   └── `ctx_fetch_and_index` tool
-│
-├── View compactor stats
-│   └── /unipi:compact-stats or `ctx_stats` tool
-│
-├── Diagnose issues
-│   └── /unipi:compact-doctor or `ctx_doctor` tool
-│
-├── Change settings
-│   └── /unipi:compact-settings (TUI overlay)
-│   └── /unipi:compact-preset <name> (quick preset)
-│
-└── Wipe indexed content
-    └── /unipi:compact-purge
-```
+## Running Code
+- `sandbox(lang, code)` → single script. `sandbox_batch(items)` → atomic.
+  `sandbox_file(lang, path)` → run file. Only stdout enters context.
 
-## When to Compact
+## Complex Multi-Step Tasks
+⚠ When the task spans many operations, PREFER Ralph loops
+   (`/unipi:work`, `ralph_start`) if available — they manage
+   context pressure better than monolithic runs.
 
-- Context window > 80% full
-- Session has > 100 messages
-- Before starting a complex multi-step task
-- When the agent starts repeating itself
-
-## When to Recall
-
-- User references something from earlier
-- Need to find a file path mentioned before
-- Looking for a previous error or decision
-- Searching for a specific code snippet
-
-## When to Index
-
-- Starting work on a large codebase
-- Need fast search across many files
-- Documentation or reference material
-- Before a research-heavy task
-
-## Presets
-
-| Preset | Best For |
-|--------|----------|
-| `opencode` | Code-heavy work, minimal context waste |
-| `balanced` | General use, good defaults |
-| `verbose` | Maximum context preservation |
-| `minimal` | Maximum token savings |
+## Critical Rules
+- Compact BEFORE starting, not when full.
+- `session_recall` instead of scrolling history.
+- Index project files early if you'll search often.

package/skills/compactor-detail/SKILL.md

@@ -0,0 +1,133 @@
+---
+name: compactor-detail
+description: Full compactor reference — tool parameters, anti-patterns, sandbox languages, FTS5 modes, workflows.
+---
+
+# Compactor — Full Reference
+
+## Tool Parameter Reference
+
+### compact
+```
+compact()
+```
+Trigger manual context compaction. Zero LLM — pure regex/text processing.
+Returns stats after next `session_compact` event.
+
+### session_recall
+```
+session_recall(query: string, mode?: "bm25" | "regex", limit?: number, offset?: number, expand?: boolean)
+```
+Search session history. BM25 is default (TF-IDF ranked). Regex is fallback.
+
+### sandbox
+```
+sandbox(language: string, code: string, timeout?: number)
+```
+Run code in sandboxed env. Only stdout enters context. 100MB output cap. 30s default timeout.
+
+### sandbox_file
+```
+sandbox_file(language: string, path: string, timeout?: number)
+```
+Execute file. Content injected as `FILE_CONTENT` variable.
+
+### sandbox_batch
+```
+sandbox_batch(items: Array<{type: "execute", language, code} | {type: "search", query}>)
+```
+Atomic batch — all items run, results returned together.
+
+### content_index
+```
+content_index(label: string, content?: string, filePath?: string, contentType?: "markdown"|"json"|"plain", chunkSize?: number)
+```
+Index content into FTS5. Provide either `content` or `filePath`. Auto-chunks by type.
+
+### content_search
+```
+content_search(query: string, limit?: number, offset?: number)
+```
+Search FTS5 index. Returns ranked results with title, content, source, rank.
+
+### content_fetch
+```
+content_fetch(url: string, label?: string, chunkSize?: number)
+```
+Fetch URL → markdown → index. Auto-indexes for later search.
+
+### compactor_stats
+```
+compactor_stats()
+```
+Dashboard: session events, compactions, tokens saved, indexed docs, sandbox runs, search queries.
+
+### compactor_doctor
+```
+compactor_doctor()
+```
+Diagnostics: config file, session DB, content store, runtimes (node, python3, bash).
+
+### context_budget
+```
+context_budget()
+```
+Estimate remaining context tokens and % full. Returns guidance on whether to compact.
+
+## Sandbox Language Reference
+
+| Language | Binaries | Timeout | Notes |
+|----------|----------|---------|-------|
+| javascript | Bun/node | 30s | Default: Bun if available |
+| typescript | Bun/node | 30s | Compiled via Bun transform |
+| python | python3 | 30s | - |
+| shell | bash | 30s | Pipelines supported |
+| ruby | ruby | 30s | - |
+| go | go | 30s | go run |
+| rust | rustc+cargo | 30s | cargo script or rustc |
+| php | php | 30s | - |
+| perl | perl | 30s | - |
+| r | Rscript | 30s | - |
+| elixir | elixir | 30s | - |
+
+## FTS5 Search Modes
+
+| Mode | When To Use |
+|------|-------------|
+| **porter** | Exact term matching with stemming |
+| **trigram** | Fuzzy/spelling errors, partial matches |
+| **rrf** | Best overall (Reciprocal Rank Fusion of porter+trigram) |
+| **fuzzy** | Auto-correction of misspellings from vocabulary |
+
+Default: `rrf` (best general-purpose).
+
+## Anti-Patterns
+
+1. **Don't call `compact` in a tight loop.** It triggers the full compaction pipeline. Call once before complex work.
+2. **Don't search without indexing.** `content_search` has nothing to search until you `content_index` or `content_fetch`.
+3. **Don't use `sandbox` for file ops.** Use bash instead. Sandbox is for computation.
+4. **Don't use `session_recall` with empty query.** It needs meaningful search terms.
+5. **Don't index node_modules.** Stick to source files and documentation.
+6. **Don't compact mid-task.** Wait for a natural break point.
+
+## Workflow Patterns
+
+### Research → Index → Search → Test
+1. `content_fetch(url)` — index reference docs
+2. `content_search(query)` — find relevant sections
+3. `sandbox(lang, code)` — test hypotheses
+
+### Diagnose → Fix → Verify
+1. `compactor_doctor` — check system health
+2. Fix issues (install runtimes, rebuild index)
+3. `compactor_stats` — verify metrics
+
+### Before Complex Work
+1. `compact` — free up context
+2. `content_index` — index relevant files
+3. `session_recall("goals")` — load context
+
+### After Long Session
+1. `compactor_stats` — check savings
+2. `compact` — compact if needed
+3. `session_recall(topic)` — verify recall quality