pi-smart-compact 7.5.0

package/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+## [7.5.0] - 2026-05-17
+
+### Added
+
+- **Redundancy-aware pre-pruning**: New `pruning.ts` module collapses redundant message sequences before compaction — duplicate file reads (keep last), collapsed error chains, agent acknowledgment message removal, long tool output truncation (>800 chars). Reduces compaction input by 15-30%.
+- **Topic-level compression budgeting**: `allocateTopicBudgets()` assigns per-topic token allocations based on priority (critical 2x, high 1.5x), error density, recency weighting, and decision count. Assembly prompt includes budget hints per segment.
+- **Project context fingerprint**: New `fingerprint.ts` module stores lightweight per-project metadata (language, framework, key directories, known files) across sessions. Compaction uses this for better file verification and context injection.
+- **Post-compaction damage detection**: New `damage.ts` module monitors agent behavior after compaction for regression signals — re-reads of compacted files, user complaints, re-questions about compacted topics. Logs damage reports for future analysis.
+- **Compaction preview context**: Project fingerprint and pruning stats are shown in notifications before compaction starts.
+
+### Changed
+
+- Single-pass prompt now includes project context from fingerprint.
+- `preProcessSummaries` accepts optional `budgetTokens` parameter for topic-level budget allocation.
+
+## [7.4.0] - 2026-05-17
+
+### Added
+
+- **Decision propagation**: Batch summarization now injects active decisions from previous segments into each batch prompt, preventing cross-batch decision amnesia and reducing semantic drift.
+- **Deterministic patch**: New `patchDeterministic()` function injects verification gaps directly into the relevant summary sections (files → Files Modified, errors → Critical Context, etc.) without any LLM call.
+- **Lazy verification**: Verification scores ≥ 85 skip patching entirely. Scores 75–84 use deterministic patch only. Scores < 75 fall back to LLM patch only if deterministic patch is insufficient.
+- **Session-aware prompts**: `SESSION_TYPE_INSTRUCTIONS` map provides session-type-specific focus instructions (debugging, implementation, review, discussion) injected into single-pass synthesis.
+- **Immutable Context framing**: Assembly prompt now presents deterministic data as "IMMUTABLE CONTEXT (do not modify)" with explicit rules against fabrication and contradiction.
+- **Metrics memory cap**: `_metrics` array capped at 200 entries (pruning to 100 when exceeded) to prevent memory leaks in long-running processes.
+
+### Changed
+
+- **Renamed**: Extension renamed from `semantic-compact` to `pi-smart-compact`. Command changed from `/compact-semantic` to `/smart-compact`. Tool changed from `semantic_compact` to `smart_compact`. Config key changed from `semanticCompact` to `smartCompact`.
+
+- **Token estimation**: Language-aware (Turkish/CE character penalty) and JSON-aware penalty. Per-provider calibration instead of global shared factor to prevent cross-session bleed.
+- **Verification path matching**: Replaced basename-only `string.includes()` with path suffix array matching to reduce false positives (e.g., "index" no longer matches every file containing "index").
+- **Boundary merging**: LLM boundaries no longer completely override heuristic boundaries. Low-confidence LLM boundaries (confidence < 0.4) are filtered. Remaining LLM boundaries are merged with heuristic boundaries that fill gaps.
+- **Constraint mining**: Added Turkish diacritical character variants (önemli, şart, zorunlu, kesinlikle, asla, sakın) and new Turkish pattern categories (prohibition: yapma/kullanma/asla, preference: tercih/isterim/olsun).
+- **Keep-boundary token calc**: Uses content text instead of JSON.stringify(message) to avoid metadata overhead in token estimation.
+
+### Added
+
+- **Adaptive exploration gate**: Simple sessions (≤3 topics, ≤1 unresolved errors, ≤2 decisions, ≤2 directory groups) skip Phase 2 exploration entirely, saving 3-8 LLM calls.
+- **Tool support cache**: Provider tool support results cached for 30 minutes with TTL-based eviction. Prevents repeated probe calls for known-unsupported providers.
+
+### Fixed
+
+- Constraint regex bug: `önemli` (with ö) now correctly matches alongside `onemli`.
+- `calibrateFromResponse` now requires provider parameter to scope calibration per-provider.
+
+## [7.3.1] - 2025-05-16
+
+### Added
+- Full EESV pipeline: Extract → Explore → Synthesize → Verify
+- Deterministic extraction of files, errors, decisions, constraints, topics
+- LLM tool-calling exploration with 6 exploration tools
+- Parallel batch synthesis with provider-aware concurrency
+- Automated verification with coverage checks and hallucination detection
+- Quality score (0-100) and gap patching
+- Incremental compaction cache (1hr TTL, delta extraction)
+- Live progress overlay and detailed result screen
+- Metrics logging (`~/.pi/agent/.cache/compact-metrics.jsonl`)
+- Cache-aware LLM calls with session affinity
+- Three compression profiles: light, balanced, aggressive
+
+### Changed
+- Refactored from single 2200-line file to modular `src/` architecture
+- Improved token estimation with provider-specific ratios and EMA calibration
+- Added `LlmMessage` and `StructuredExtraction` types replacing `any`
+
+### Fixed
+- Tool loop safety (prevents orphaned tool result errors)
+- No-op edit detection
+- Graceful fallback when models don't support tool calling

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Semantic Compact Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,200 @@
+# Smart Compact
+
+> EESV-powered smart compaction extension for the [Pi Coding Agent](https://github.com/earendil-works/pi-coding-agent).
+
+[![Version](https://img.shields.io/badge/version-7.5.0-blue)](./package.json)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
+
+---
+
+## What is it?
+
+**Smart Compact** compresses long conversation contexts by understanding *what happened* instead of blindly truncating. It uses the **EESV architecture**:
+
+| Phase | What it does | LLM calls |
+|-------|-------------|-----------|
+| **Extract** | Deterministically pull files, errors, decisions, constraints, topics | 0 |
+| **Explore** | Use LLM tools to verify boundaries and enrich context (skipped for simple sessions) | 0–8 |
+| **Synthesize** | Parallel batch summarization + assembly | N + 1 |
+| **Verify** | Check coverage, detect hallucinations, patch gaps | 0–1 |
+
+**Result:** Shorter context that preserves the *meaning* and *state* of your session.
+
+---
+
+## Features
+
+- 🔍 **Deterministic extraction** — zero-LLM-call file/error/decision mining
+- 🧭 **Tool-calling exploration** — targeted investigation with `get_message_range`, `search_conversation`, `get_error_chain`
+- ⚡ **Parallel batch synthesis** — provider-aware concurrency (2–5 in flight)
+- ✅ **Automated verification** — coverage checks, hallucination detection, gap patching
+- 📊 **Live metrics** — token savings, cache hit rate, latency per phase
+- 🎛️ **Profiles** — `light` / `balanced` / `aggressive` compression
+- 💾 **Backup & incremental cache** — safe rollback, delta re-compaction
+- 🧠 **Adaptive exploration** — skips Phase 2 for simple sessions, saving 3–8 LLM calls
+- 🔧 **Provider-aware token estimation** — language and JSON-aware with per-provider calibration
+
+---
+
+## Install
+
+```bash
+# Inside your Pi agent extensions directory
+cd ~/.pi/agent/extensions
+git clone https://github.com/YOUR_USERNAME/pi-smart-compact.git
+cd pi-smart-compact
+bun install
+```
+
+Add to your Pi `settings.json`:
+
+```json
+{
+  "extensions": ["pi-smart-compact"]
+}
+```
+
+Or via `package.json` (already configured):
+
+```json
+{
+  "pi": {
+    "extensions": ["./src/index.ts"]
+  }
+}
+```
+
+---
+
+## Usage
+
+```bash
+# TUI — pick model + profile
+/smart-compact
+
+# Direct — specific model + profile
+/smart-compact anthropic/claude-sonnet-4 balanced
+
+# Dry run — preview only
+/smart-compact dry-run
+
+# Verbose — detailed logging
+/smart-compact debug
+
+# Add a steering note
+/smart-compact "focus on auth changes"
+```
+
+### Tool Usage
+
+The extension also registers a tool named `smart_compact`:
+
+```json
+{
+  "name": "smart_compact",
+  "parameters": {
+    "profile": "balanced",
+    "verbose": false,
+    "dry_run": false
+  }
+}
+```
+
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────┐
+│  Conversation (too long)                     │
+└──────────────┬──────────────────────────────┘
+               │
+  ┌────────────▼────────────┐
+  │  Phase 1: EXTRACT       │  ← deterministic (0 LLM calls)
+  │  • files modified/read  │
+  │  • errors + retries     │
+  │  • decisions            │
+  │  • constraints          │
+  └────────────┬────────────┘
+               │
+  ┌────────────▼────────────┐
+  │  Phase 2: EXPLORE       │  ← LLM with tools (0–8 rounds)
+  │  • verify topic bounds  │  ← skipped for simple sessions
+  │  • find cross-references│
+  │  • assess status        │
+  └────────────┬────────────┘
+               │
+  ┌────────────▼────────────┐
+  │  Phase 3: SYNTHESIZE    │  ← parallel batch summarize
+  │  • chunk messages       │
+  │  • summarize batches    │
+  │  • assemble final       │
+  └────────────┬────────────┘
+               │
+  ┌────────────▼────────────┐
+  │  Phase 4: VERIFY        │  ← deterministic checks
+  │  • coverage gaps?       │
+  │  • hallucinated files?  │
+  │  • patch if needed      │
+  └────────────┬────────────┘
+               │
+  ┌────────────▼────────────┐
+  │  Compact context applied│
+  └─────────────────────────┘
+```
+
+---
+
+## Profiles
+
+| Profile | Summary Budget | Keep Recent | Best For |
+|---------|---------------|-------------|----------|
+| **light** | 10K tokens | 30K tokens | Debugging, complex multi-file refactors |
+| **balanced** | 6K tokens | 20K tokens | General development (default) |
+| **aggressive** | 3K tokens | 10K tokens | Quick exploration, prototyping |
+
+---
+
+## Configuration
+
+Create `~/.pi/agent/settings.json`:
+
+```json
+{
+  "smartCompact": {
+    "profile": "balanced",
+    "summaryModel": "anthropic/claude-sonnet-4",
+    "segmentationModel": "anthropic/claude-haiku-3",
+    "autoTrigger": true,
+    "backupEnabled": true,
+    "profiles": {
+      "light": { "summaryBudgetTokens": 10000, "keepRecentTokens": 30000 }
+    }
+  }
+}
+```
+
+---
+
+## Development
+
+```bash
+bun install
+bun test         # runs test suite
+```
+
+---
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b feat/amazing-feature`)
+3. Commit your changes (`git commit -am 'Add amazing feature'`)
+4. Push (`git push origin feat/amazing-feature`)
+5. Open a Pull Request
+
+---
+
+## License
+
+MIT © [Alper](https://github.com/alper)

package/package.json

@@ -0,0 +1,42 @@
+{
+  "name": "pi-smart-compact",
+  "version": "7.5.0",
+  "description": "EESV smart compaction extension for Pi Coding Agent — deterministic extraction, exploration, synthesis, verification with redundancy pruning, project fingerprinting, and damage detection.",
+  "license": "MIT",
+  "author": "Alper Tarhan <alpertarhan@gmail.com>",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/alpertarhan/pi-smart-compact.git"
+  },
+  "homepage": "https://github.com/alpertarhan/pi-smart-compact#readme",
+  "bugs": {
+    "url": "https://github.com/alpertarhan/pi-smart-compact/issues"
+  },
+  "keywords": [
+    "pi",
+    "pi-extension",
+    "pi-coding-agent",
+    "pi-smart-compact",
+    "smart-compaction",
+    "context-compression",
+    "eesv",
+    "conversation-summarization",
+    "llm-context"
+  ],
+  "files": [
+    "src",
+    "README.md",
+    "LICENSE",
+    "CHANGELOG.md"
+  ],
+  "scripts": {
+    "test": "bun test",
+    "typecheck": "bunx tsc --noEmit"
+  },
+  "pi": {
+    "extensions": ["./src/index.ts"]
+  },
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/src/constants.ts

@@ -0,0 +1,140 @@
+/**
+ * Constants, prompts, and profile defaults.
+ */
+
+import type { CompressionProfile, ProfileConfig } from "./types.ts";
+
+export const VERSION = "7.5.0";
+export const CHARS_PER_TOKEN = 3.8;
+
+export const COMPACT_SYSTEM_PREFIX =
+  "You are an expert conversation summarizer for a coding agent. " +
+  "Produce structured markdown summaries. " +
+  "Follow output format exactly. " +
+  "Use EXACT names — never paraphrase code identifiers. " +
+  "Trust deterministic extraction data over intuition.";
+
+export const PROFILES: Record<CompressionProfile, ProfileConfig> = {
+  light: {
+    summaryBudgetTokens: 10000,
+    keepRecentTokens: 30000,
+    minChunkTokens: 800,
+    maxChunkTokens: 12000,
+    singlePassMaxTokens: 40000,
+    batchMaxTokens: 30000,
+  },
+  balanced: {
+    summaryBudgetTokens: 6000,
+    keepRecentTokens: 20000,
+    minChunkTokens: 500,
+    maxChunkTokens: 8000,
+    singlePassMaxTokens: 30000,
+    batchMaxTokens: 24000,
+  },
+  aggressive: {
+    summaryBudgetTokens: 3000,
+    keepRecentTokens: 10000,
+    minChunkTokens: 300,
+    maxChunkTokens: 6000,
+    singlePassMaxTokens: 20000,
+    batchMaxTokens: 18000,
+  },
+};
+
+export const DEFAULT_CONFIG = {
+  profile: "balanced" as CompressionProfile,
+  profiles: PROFILES,
+  summaryModel: null as string | null,
+  segmentationModel: null as string | null,
+  autoTrigger: true,
+  backupEnabled: true,
+  backupDir: "",
+};
+
+export const NO_OP_RE = /applied:\s*0|no changes applied|nothing to (?:do|change)|0 edits? applied/i;
+export const SHIFT_RE = /simdi|peki|bide|bi de|gecelim|bakalim|yapalim|baska|sonra|tamam simdi|now let|also|next|let's|moving on|switch to/i;
+export const CHOICE_RE = /use\s+\S+\s+(?:instead|not|rather)|don't\s+use|avoid\s+|switch\s+to\s+|go\s+with\s+|prefer\s+/i;
+
+// ── Prompt Templates ──
+
+export const SINGLE_PASS_PREFIX =
+  "Summarize this coding agent conversation. Produce ONE structured summary.\n" +
+  "\nRules for Accuracy:\n" +
+  "1. Session Type: read-only tool calls = REVIEW, not implementation\n" +
+  "2. Status: Check for user complaints before marking \"Done\"\n" +
+  "3. Exact Names: Quote specific variable/function/parameter names, don't paraphrase\n" +
+  "4. Files: Use the VERIFIED file lists above (deterministically extracted, zero hallucination risk)\n" +
+  "\nOutput EXACTLY this format:\n\n" +
+  "## Goal\n[What the user is trying to accomplish]\n" +
+  "## Constraints & Preferences\n- [CRITICAL: user requirements, preferences, constraints]\n" +
+  "## Progress\n### Done\n- [x] [Completed tasks with file references]\n### In Progress\n- [ ] [Current work state]\n### Blocked\n- [Issues]\n" +
+  "## Key Decisions\n- **[Decision]**: [Rationale]\n" +
+  "## Files Modified\n- [Verified list from deterministic extraction]\n" +
+  "## Files Read\n- [Verified list from deterministic extraction]\n" +
+  "## Next Steps\n1. [What should happen next]\n" +
+  "## Critical Context\n- [Specific data, patterns, info needed to continue]\n- [Error patterns or gotchas]\n" +
+  "## Topics Covered\n[Chronological bullet list with priority in brackets]\n";
+
+export const SINGLE_PASS_SUFFIX =
+  "\n{PREV_CONTEXT}\n\n{EXTRACTION_CONTEXT}\n\n{EXPLORATION_CONTEXT}\n\n<conversation>\n{CONVERSATION}\n</conversation>";
+
+export const BATCH_PROMPT_PREFIX =
+  "Summarize these conversation segments.\n\nRules for Accuracy:\n" +
+  "1. Use EXACT file paths from extraction data\n" +
+  "2. Status: only mark \"done\" if there's clear evidence (successful test run, user confirmation)\n" +
+  "3. Quote specific values, don't paraphrase code\n\n" +
+  "For EACH segment produce:\n" +
+  "### {TOPIC_NAME}\n" +
+  "**Priority**: [critical|high|normal|low]\n" +
+  "**Summary**: [2-4 sentences: what happened, errors, code changes with paths]\n" +
+  "**Decisions**: [comma-separated, or \"None\"]\n" +
+  "**Modified**: [comma-separated paths, or \"None\"]\n" +
+  "**Read**: [comma-separated paths, or \"None\"]\n";
+
+export const BATCH_PROMPT_SUFFIX = "\n{EXTRACTION_CONTEXT}\n\n<segments>\n{TEXT}\n</segments>";
+
+export const ASSEMBLY_PROMPT_PREFIX =
+  "Merge these topic summaries into ONE coherent summary.\n\n" +
+  "## IMMUTABLE CONTEXT (do not modify or contradict these facts)\n" +
+  "These are deterministically verified from the original conversation. They take priority over ANY summary content below.\n\n" +
+  "Rules:\n" +
+  "1. Preserve ALL critical/high info. Condense normal, minimize low.\n" +
+  "2. Chronological order.\n" +
+  "3. The pre-processed data below is GROUND TRUTH — trust it over individual summaries.\n" +
+  "4. Files Modified list is deterministically verified — if a summary says a file was modified but it's NOT in the list above, omit it.\n" +
+  "5. Key Decisions below are verified — preserve them exactly, do not paraphrase the decision text.\n" +
+  "6. Do NOT fabricate file paths, function names, or error messages not present in the verified data.\n\n" +
+  "Format:\n" +
+  "## Goal\n[Overall objective]\n" +
+  "## Constraints & Preferences\n- [CRITICAL requirements, preferences, constraints]\n" +
+  "## Progress\n### Done\n- [x] [Completed tasks with file refs]\n### In Progress\n- [ ] [Current work state]\n### Blocked\n- [Issues]\n" +
+  "## Key Decisions\n- **[Decision]**: [Rationale]\n" +
+  "## Files Modified\n- [Verified deterministic list]\n" +
+  "## Files Read\n- [Verified deterministic list]\n" +
+  "## Next Steps\n1. [What should happen next]\n" +
+  "## Critical Context\n- [Data, patterns, info needed]\n" +
+  "## Topics Covered\n[Chronological bullets with priority]\n";
+
+export const ASSEMBLY_PROMPT_SUFFIX =
+  "\nIMMUTABLE CONTEXT (verified deterministic data):\n- Key Decisions: {DECISIONS}\n- Files Modified (VERIFIED): {MODIFIED}\n- Files Read (VERIFIED): {READ}\n\n{EXPLORATION_CONTEXT}\n{PREV_CONTEXT}\n\n<summaries>{SUMMARIES}</summaries>";
+
+// ── Session-type-specific prompt instructions ──
+
+export const SESSION_TYPE_INSTRUCTIONS: Record<string, string> = {
+  debugging: "Focus on: error chains, root cause analysis, attempted fixes, resolution status. Prioritize error messages and stack traces. Mark files as Done only if all errors resolved.",
+  implementation: "Focus on: files created/modified, architectural decisions, feature completeness, test coverage. Prioritize code changes with exact paths.",
+  review: "Focus on: files read, issues found, recommendations, approval status. Prioritize findings over changes. Read-only tool calls = REVIEW, not implementation.",
+  discussion: "Focus on: decisions made, trade-offs discussed, consensus reached. Prioritize rationale over implementation details.",
+};
+
+export const EXPLORER_SYSTEM_PROMPT =
+  "You are a conversation analyst. You have deterministic extraction data and can query the raw conversation using tools.\n\n" +
+  "Your job:\n" +
+  "1. Verify/enrich the extracted boundaries (merge, split, or add as needed)\n" +
+  "2. Identify cross-topic relationships\n" +
+  "3. Find implicit constraints (user tone, frustration, urgency)\n" +
+  "4. Assess completion status accurately\n" +
+  "5. Extract the narrative arc\n\n" +
+  "Use tools BEFORE forming conclusions. You may make up to 8 tool calls.\n\n" +
+  "After exploration, output ONLY a JSON object (no markdown):\n" +
+  '{"boundaries":[{"afterIndex":N,"topic":"...","priority":"critical|high|normal|low","confidence":0.0-1.0}],"mainGoal":"...","sessionType":"implementation|review|debugging|discussion","enrichedConstraints":[...],"crossReferences":[...],"statusAssessment":{"done":[...],"inProgress":[...],"blocked":[...]},"criticalContext":[...],"keyDecisions":[...]}';