@jairodri/huckleberry-cli 1.0.1 → 1.1.0

package/skills/claw-compactor/references/README.md

@@ -0,0 +1,44 @@
+# References
+Technical documentation for claw-compactor internals.
+
+## Files
+- **compression-techniques.md** — Deep dive into all 5 compression techniques
+- **benchmarks.md** — Real-world performance measurements
+- **architecture.md** — System architecture and module relationships
+- **testing.md** — Test strategy and coverage goals
+- **compression-prompts.md** — LLM prompt templates for observation compression
+
+## Key Design Decisions
+
+### Dictionary Encoding
+The codebook uses `$XX` codes (uppercase alpha) to avoid conflicts with:
+- Shell variables (`$lower_case`)
+- Markdown formatting (`**bold**`)
+- Natural text (`$100`, `$USD`)
+
+Code length starts at 3 chars (`$AA`) and grows to 4 (`$AAA`) after 676 entries.
+
+### Workspace Path Shorthand
+`$WS` replaces the full workspace path. This is the single highest-value substitution for most workspaces since the path appears in every file reference.
+
+Example codebook:
+```json
+{
+ "$A1": "example_user",
+ "$A2": "10.0.1",
+ "$A3": "workspace"
+}
+```
+
+**Before:** `ssh deploy@10.0.1.2` / `ssh admin@10.0.1.3`
+**After:** `ssh deploy@$A2.2` / `ssh admin@$A2.3`
+
+### Token Estimation
+Two backends:
+1. **tiktoken** (preferred) — exact cl100k_base encoding, same as Claude models
+2. **Heuristic fallback** — CJK-aware chars÷4 approximation, ~90% accurate
+
+### Workspace paths
+- `/home/user/workspace` → `$WS`
+
+All path compression is fully reversible via `decompress_paths()`.

package/skills/claw-compactor/references/architecture.md

@@ -0,0 +1,197 @@
+# Architecture
+
+## System Overview
+claw-compactor is a modular compression pipeline with a single entry point (`mem_compress.py`) that routes to specialized compressors, all sharing a common library layer.
+
+```
+ ┌──────────────────────┐
+ │ mem_compress.py │
+ │ 553 lines │
+ │ │
+ │ • CLI argument parsing │
+ │ • Command routing │
+ │ • Pipeline orchestrator│
+ │ • Progress reporting │
+ └──────────┬─────────────┘
+ │
+ ┌──────────┬───────────┬───┴────┬──────────┬──────────┬─────────┐
+ ▼ ▼ ▼ ▼ ▼ ▼ ▼
+ ┌─────────┐ ┌────────┐ ┌────────┐ ┌──────┐ ┌────────┐ ┌───────┐ ┌──────┐
+ │compress │ │dict_ │ │observ- │ │dedup │ │generate│ │audit │ │estim-│
+ │_memory │ │compress│ │ation_ │ │_mem │ │_summary│ │_memory│ │ate_ │
+ │ │ │ │ │compres-│ │ │ │_tiers │ │ │ │tokens│
+ │ 230 LOC │ │ 170 LOC│ │sor │ │147LOC│ │ 292 LOC│ │216LOC │ │131LOC│
+ │ │ │ │ │ 346 LOC│ │ │ │ │ │ │ │ │
+ └────┬────┘ └───┬────┘ └───┬────┘ └──┬───┘ └───┬────┘ └──┬────┘ └──┬───┘
+ │ │ │ │ │ │ │
+ └──────────┴───────────┴────┬────┴─────────┴─────────┴─────────┘
+ ▼
+ ┌─────────────────┐
+ │ lib/ │
+ │ tokens.py 68 │ Token estimation engine
+ │ markdown.py 312 │ MD parsing & manipulation
+ │ dedup.py 119 │ Shingle-hash dedup
+ │ dictionary.py273│ Codebook compression
+ │ rle.py 165 │ Run-length encoding
+ │ tokenizer_ │
+ │ optimizer 188 │ Format optimization
+ │ config.py 81 │ JSON config loading
+ │ exceptions 24 │ Custom exception types
+ └─────────────────┘
+ Total: 3,602 LOC
+
+## Data Flow: Full Pipeline
+┌──────────────────────┐ ┌──────────────────────┐
+ │ memory/*.md │ │ .openclaw/agents/*/sessions/ │
+ │ MEMORY.md │ │ *.jsonl │
+ │ TOOLS.md, etc. │ │ (raw transcripts) │
+ └──────────┬───────────┘ └──────────┬────────────┘
+ ▼ ▼
+ │ 1. estimate_tokens │ │ 2. observation_ │
+ │ Baseline count │ │ compressor │
+ │ (read-only) │ │ JSONL → XML → MD │
+ └──────────┬───────────┘ │ 97% compression │
+ │ └──────────┬────────────┘
+ ▼ │
+ ┌──────────────────────┐ │
+ │ 3. compress_memory │ │
+ │ Rule engine: │ ▼
+ │ • dedup lines │ ┌──────────────────────┐
+ │ • strip redundancy │ │ memory/observations/ │
+ │ • merge sections │ │ (compressed output) │
+ └──────────┬───────────┘ └────────────────────────┘
+ │ 4. dictionary_ │
+ │ compress │
+ │ Build codebook → │
+ │ Apply $XX codes │
+ └──────────┬───────────┘
+ │ 5. dedup_memory │ │ memory/.codebook.json│
+ │ Cross-file scan │ │ (codebook artifact) │
+ │ Shingle hashing │ └────────────────────────┘
+ │ 6. generate_ │────▶│ memory/MEMORY-L0.md │
+ │ summary_tiers │ │ memory/MEMORY-L1.md │
+ │ L0/L1/L2 budgets │ │ (tier summaries) │
+ └──────────────────────┘ └────────────────────────┘
+
+## Module Responsibilities
+
+### Entry Point
+**`mem_compress.py`** (553 LOC)
+The unified CLI. Parses arguments, routes to the appropriate command handler, and orchestrates the full pipeline. Handles progress reporting, JSON output mode, and error formatting.
+
+### Compressor Modules
+**`compress_memory.py`** (230 LOC)
+Two-phase memory compression. Phase 1: deterministic rule engine (dedup lines, strip markdown filler, merge similar sections). Phase 2: optional LLM prompt generation for semantic compression. Operates on `.md` files in the workspace.
+
+**`dictionary_compress.py`** (170 LOC)
+CLI wrapper around `lib/dictionary.py`. Scans workspace markdown files, builds/loads codebook, applies/reverses compression. Manages the `.codebook.json` artifact.
+
+**`observation_compressor.py`** (346 LOC)
+Parses OpenClaw `.jsonl` session transcripts, extracts tool call interactions, classifies them by type (feature, bugfix, decision, etc.), and generates structured observation summaries. The single biggest source of savings (~97%). Tracks processed sessions in `.observed-sessions.json`.
+
+**`dedup_memory.py`** (147 LOC)
+Cross-file near-duplicate detection. Uses shingle hashing (n-gram fingerprinting) with Jaccard similarity. Reports duplicates or optionally auto-merges them.
+
+**`generate_summary_tiers.py`** (292 LOC)
+Creates L0/L1/L2 summaries from MEMORY.md. Classifies sections by priority (decision > action > config > log > archive), then fills each tier within its token budget, highest-priority sections first.
+
+**`estimate_tokens.py`** (131 LOC)
+Token counting and compression potential scoring. Scans all markdown files, reports per-file and total token usage. Identifies files with high compression potential.
+
+**`audit_memory.py`** (216 LOC)
+Health checker. Reports staleness (files not updated recently), bloat (high token/info ratio), and compression opportunities. Suggests specific actions.
+
+**`compressed_context.py`** (280 LOC)
+Compressed Context Protocol. Three compression levels (ultra/medium/light) for context passing between models. Generates decompression instructions for the receiving model's system prompt.
+
+### Library Layer
+**`lib/tokens.py`** (68 LOC)
+Token estimation. Uses tiktoken's `cl100k_base` encoding when available, falls back to a CJK-aware heuristic (Chinese characters count as 1.5 tokens, others as chars÷4). Single function: `estimate_tokens(text) → int`.
+
+**`lib/markdown.py`** (312 LOC)
+Markdown parsing utilities. Section extraction by header level, section merging, content normalization, Chinese punctuation handling, header classification by priority keywords.
+
+**`lib/dedup.py`** (119 LOC)
+Shingle-hash deduplication engine. Generates n-gram (shingle) sets from text, computes Jaccard similarity between shingle sets, and groups entries by approximate length to reduce comparison space. O(n×k) instead of O(n²).
+
+**`lib/dictionary.py`** (273 LOC)
+The codebook engine. Scans text for n-gram frequencies (1-4 words), scores candidates by `freq × (len(phrase) - len(code)) - codebook_overhead`, builds a codebook of `$XX` codes, and provides `compress_text`/`decompress_text` as perfect inverses.
+
+**`lib/rle.py`** (165 LOC)
+Run-length encoding for structured patterns. Path compression (`$WS` shorthand), IP prefix extraction (`$IP` codes), and enumeration detection. All with roundtrip decompression.
+
+**`lib/tokenizer_optimizer.py`** (188 LOC)
+Encoding-aware format transformations. Converts markdown tables to key:value notation (60-70% savings), normalizes Chinese fullwidth punctuation, strips bold/italic/backtick markers, minimizes whitespace and indentation, compacts bullet lists.
+
+**`lib/config.py`** (81 LOC)
+Configuration loader. Reads `claw-compactor-config.json` from workspace root, merges with sensible defaults. All settings optional.
+
+**`lib/exceptions.py`** (24 LOC)
+Custom exception hierarchy: `MemCompressError` (base), `FileNotFoundError_`, etc.
+
+## Layer 0: cacheRetention (Before Compression)
+Before any compression runs, **prompt caching** (`cacheRetention: "long"`) provides a 90% discount on cached prompt tokens with a 1-hour TTL. This is orthogonal to compression — it reduces cost on whatever tokens remain.
+
+Cost reduction stack:
+ Layer 0: cacheRetention: "long" → 90% cost discount on cached tokens
+ Layer 1: observe (transcripts) → ~97% token reduction
+ Layer 2: compress (rule engine) → 4-8% token reduction
+ Layer 3: dict (codebook) → 4-5% token reduction
+ Layer 4: optimize (tokenizer) → 1-3% token reduction
+
+Layers 1-4 reduce token count. Layer 0 reduces cost-per-token. They multiply.
+
+## Heartbeat Integration Flow
+┌─────────────────────────┐
+ │ Heartbeat fires │
+ │ (every ~30 min) │
+ └────────────┬────────────┘
+ │ Read HEARTBEAT.md │
+ │ → memory maintenance? │
+ │ yes
+ │ Run: benchmark │
+ │ (non-destructive) │
+ ┌────┴────┐
+ │ >5% ? │
+ └────┬────┘
+ yes │ │ no
+ ┌──────────────┐│
+ │ Run: full ││
+ │ pipeline ││
+ └──────────────┘│
+ │◀──┘
+ │ New transcripts? │
+ │ (unprocessed JSONL) │
+ ┌──────────────┐ HEARTBEAT_OK
+ │ Run: observe │
+ └──────────────┘
+
+**Trigger logic:** The agent checks `HEARTBEAT.md` for a memory maintenance entry. If present, it runs `benchmark` first (cheap read-only). Only if savings exceed 5% does it commit to the full pipeline. New unprocessed transcripts always trigger `observe` regardless of benchmark results.
+
+## Design Decisions
+
+### Why shingle hashing for deduplication?
+Naive pairwise comparison is O(n²) — unacceptable for workspaces with hundreds of sections. Shingle hashing (n-gram fingerprinting) gives us:
+- O(n × k) complexity where k is the number of shingles per entry
+- 3-word shingles with MD5 fingerprints provide good collision resistance
+- Jaccard similarity on shingle sets is a well-studied near-duplicate metric
+- Bucketing by approximate length further reduces comparisons
+
+### Why tiktoken with heuristic fallback?
+tiktoken gives exact token counts but requires compilation (Rust dependency). Many environments don't have it installed. The fallback heuristic (chars÷4, CJK-aware) is ~90% accurate — good enough for compression decisions. No hard dependency means the skill works out of the box everywhere.
+
+### Why `$XX` codes instead of longer variable names?
+Two-character codes minimize per-occurrence overhead. The codebook scoring function accounts for this: a phrase is only worth encoding if `freq × (len(phrase) - len(code)) > codebook_overhead`. Short codes win because the overhead term (the codebook entry itself) is amortized across many occurrences.
+
+### Why section-level priority scoring for tiers?
+Not all memory content is equal. A decision about architecture is worth more context tokens than a log of which files were edited. Priority classification (decision > action > config > log > archive) ensures L0 summaries contain the most important information, even at ~200 tokens.
+
+### Why non-destructive by default?
+Agents make mistakes. Every write operation is opt-in: `--dry-run` shows stats, `dedup` reports without modifying, `benchmark` never writes. This is critical for trust — users need to verify before committing to changes.
+
+### Why XML format for observations (inspired by claude-mem)?
+Structured XML (`<observation>`, `<type>`, `<title>`, `<facts>`) is:
+1. Unambiguous to parse (unlike free-form markdown)
+2. Token-efficient (tags are reusable tokens in cl100k_base)
+3. Compatible with claude-mem's proven format
+4. Easy to classify and search programmatically

package/skills/claw-compactor/references/benchmarks.md

@@ -0,0 +1,94 @@
+# Performance Benchmarks
+
+## Methodology
+All benchmarks run on a production workspace with:
+- 30 days of active daily use
+- 15 memory files (MEMORY.md, TOOLS.md, AGENTS.md, SOUL.md, daily notes)
+- 173 session transcripts (.jsonl files)
+- Python 3.12, tiktoken installed
+
+Token counts use tiktoken `cl100k_base` encoding (same as Claude models). All measurements are deterministic — same input produces same output every run.
+
+## Memory File Compression
+Workspace: all `.md` files in root + `memory/`
+
+| Rule Engine | 11,855 | 11,398 | 457 | 3.9% |
+| Dictionary Encoding | 11,398 | 10,891 | 507 | 4.4% |
+| Tokenizer Optimization | 10,891 | 10,766 | 125 | 1.1% |
+| RLE Patterns | 10,766 | 10,710 | 56 | 0.5% |
+| **Total** | **11,855** | **10,710** | **1,145** | **9.7%** |
+
+### Per-File Breakdown
+| TOOLS.md | 3,421 | 2,985 | 12.7% | High repetition (IPs, paths) |
+| MEMORY.md | 4,102 | 3,810 | 7.1% | Mixed content |
+| AGENTS.md | 2,156 | 2,044 | 5.2% | Mostly prose, less compressible |
+| memory/2024-01-15.md | 892 | 831 | 6.8% | Daily notes |
+| memory/2024-01-14.md | 734 | 690 | 6.0% | Daily notes |
+| SOUL.md | 550 | 540 | 1.8% | Short, unique content |
+
+**Observation:** Files with repetitive structured data (TOOLS.md) compress best. Short, unique prose (SOUL.md) compresses least.
+
+## Session Transcript Compression
+173 session transcripts:
+
+- Total transcripts: 173 files
+- Total raw size: ~4.5M tokens
+- After observation compression: ~135K tokens
+- Compression ratio: **97%**
+- Average per session (before): ~26,000 tokens
+
+### By Session Type
+Long coding session (>100 tool calls), Avg Raw=52,000, Avg Compressed=1,200, Ratio=97.7%
+Config/setup session, Avg Raw=18,000, Avg Compressed=520, Ratio=97.1%
+Research/browsing session, Avg Raw=31,000, Avg Compressed=890, Ratio=97.1%
+Short task (<10 tool calls), Avg Raw=4,200, Avg Compressed=280, Ratio=93.3%
+
+## Tiered Summary Savings
+MEMORY.md (4,102 tokens) → tiered summaries:
+
+L0 (Ultra-compact), Token Budget=200, Actual=187, Savings vs Full=95.4%
+L1 (Normal), Token Budget=500, Actual=478, Savings vs Full=88.4%
+L2 (Full), Token Budget=—, Actual=4,102, Savings vs Full=0%
+
+**Impact on sub-agents:** A sub-agent loading L0 instead of full MEMORY.md saves 3,915 tokens per spawn. At 20 sub-agent spawns/day, that's 78,300 tokens/day saved.
+
+## Independent Technique Contribution
+Each technique measured independently (not cumulative):
+
+Rule engine alone, Savings on Memory Files=3.9%, Notes=Dedup + strip + merge
+Dictionary alone, Savings on Memory Files=4.8%, Notes=Before rule engine (slightly higher)
+Tokenizer optimize alone, Savings on Memory Files=1.4%, Notes=Tables → key:value biggest win
+RLE alone, Savings on Memory Files=0.7%, Notes=Path-dependent
+Combined, Savings on Memory Files=9.7%, Notes=Less than sum (some overlap)
+
+## Token Cost Savings Estimate
+Based on Anthropic's Claude pricing (as of 2024):
+
+| Claude Sonnet 4 | $3/M tokens | 15M tokens | $45.00 | $22.50 | **$22.50** |
+| Claude Opus 4 | $15/M tokens | 15M tokens | $225.00 | $112.50 | **$112.50** |
+| Claude Haiku 3.5 | $0.25/M tokens | 15M tokens | $3.75 | $1.88 | **$1.88** |
+
+*Estimate for active daily use: 50 sessions × 10 context loads each × 30K avg tokens per load.
+
+### Breakdown by Source
+| Session transcripts | 4.5M | 97% | 4.365M | $13.10 |
+| Memory file loads | 8.5M* | 10% | 850K | $2.55 |
+| Sub-agent context | 2M* | 88% (L0) | 1.76M | $5.28 |
+| **Total** | **15M** | **46.5%** | **6.975M** | **$20.93** |
+
+*Estimated from session frequency × tokens per load.
+
+## Execution Performance
+Benchmark runtime (Apple Silicon, 64GB RAM):
+
+`estimate`, Time=0.3s, Notes=Token counting only
+`compress` (rule engine), Time=0.8s, Notes=15 files
+`dict` (build + compress), Time=1.2s, Notes=N-gram scanning
+`dedup`, Time=0.5s, Notes=Shingle computation
+`observe` (1 session), Time=0.1s, Notes=Rule-based extraction
+`observe` (173 sessions), Time=8.2s, Notes=Batch processing
+`tiers`, Time=0.4s, Notes=Summary generation
+`full` (complete pipeline), Time=11.5s, Notes=All steps
+`benchmark` (dry-run), Time=2.1s, Notes=Read-only analysis
+
+All operations are I/O-bound, not CPU-bound. The bottleneck is reading/writing markdown files.

package/skills/claw-compactor/references/compression-prompts.md

@@ -0,0 +1,39 @@
+# Compression Prompts
+LLM prompts used by claw-compactor, adapted from claude-mem's observation compression approach.
+
+## Design Rationale
+Claude-mem captures tool observations as structured XML (`<observation>` → type, title, facts, narrative, concepts). Our prompts adapt this principle for flat markdown memory files — extracting and preserving the same categories of information while aggressively removing filler.
+
+Key insight from claude-mem: **facts and decisions are the most token-efficient form of memory**. Narratives add context but cost 5-10× more tokens. Our compression targets facts first.
+
+## Compression Prompt (used by compress_memory.py)
+```
+You are a memory compression specialist. Compress the following memory
+content while preserving ALL factual information, decisions, and action items.
+
+Rules:
+- Remove filler words, redundant explanations, and verbose formatting
+- Merge related items into concise bullet points
+- Preserve dates, names, numbers, and technical details exactly
+- Keep section structure but tighten headers
+- Target: reduce to ~{target_pct}% of original size
+- Output valid markdown
+
+Content to compress:
+---
+{content}
+
+Compressed version:
+
+### Why this prompt works
+- "ALL factual information" prevents lossy compression of key data
+- "dates, names, numbers, technical details exactly" preserves identifiers (IPs, IDs, versions)
+- "section structure" maintains navigability
+- Explicit target percentage gives the model a concrete goal
+
+## Tier Summary Prompts
+Not currently LLM-generated — tiers use algorithmic section selection based on priority scores and token budgets. This is more deterministic and reproducible than LLM-based summarization.
+
+If LLM-based tier generation is desired, use compress_memory.py's prompt with modified targets:
+- Level 0: target_pct=5 with additional instruction "key-value pairs only"
+- Level 1: target_pct=15 with additional instruction "organized sections"

package/skills/claw-compactor/references/compression-techniques.md

@@ -0,0 +1,219 @@
+# Compression Techniques
+claw-compactor applies 5 independent compression techniques in a layered pipeline. Each targets a different source of token waste and can run independently.
+
+---
+
+## 1. Rule-Based Compression
+**Module:** `compress_memory.py` + `lib/markdown.py`
+**Typical savings:** 4-8% on memory files
+**Lossless:** Yes
+
+The rule engine applies deterministic transformations that remove redundancy without losing any information.
+
+### Rules Applied
+Exact dedup, Description=Remove duplicate lines within a section, Typical Impact=1-3%
+Near-dedup merge, Description=Merge sections with >60% Jaccard similarity, Typical Impact=1-2%
+Whitespace strip, Description=Collapse excessive blank lines, trailing spaces, Typical Impact=0.5-1%
+Empty section removal, Description=Remove headers with no body content, Typical Impact=0.5%
+Markdown filler, Description=Strip unnecessary bold/italic/backtick markers, Typical Impact=0.5-1%
+Chinese punctuation, Description=Fullwidth `,.!` → halfwidth `,.!` (saves 1 token each), Typical Impact=0-1%
+
+### Before / After
+**Before:**
+```markdown
+
+## Remote Machines
+
+### Production Server
+- IP: 10.0.2.1, Internal: 10.0.1.2, User: deploy
+
+- Internal IP: 10.0.2.1, IP: 10.0.1.2, SSH user: deploy
+- SSH: `ssh -i ~/.ssh/server_key.pem deploy@10.0.2.1`
+
+## Notes
+```
+
+**After:**
+
+The duplicate "Production Server" section was merged (near-dedup), and the empty "Notes" section was removed.
+
+## 2. Dictionary Encoding
+**Module:** `dictionary_compress.py` + `lib/dictionary.py`
+**Typical savings:** 4-5% on memory files
+**Lossless:** Yes (perfect roundtrip)
+
+### How It Works
+1. **Scan** — Analyze all workspace markdown files for n-gram frequencies (1-4 words)
+2. **Score** — For each candidate phrase: `score = freq × (len(phrase) - len(code)) - codebook_overhead`
+3. **Build** — Select top-scoring phrases, assign `$A1`, `$A2`, ... codes
+4. **Compress** — Replace all occurrences of phrases with their codes
+5. **Store** — Save codebook to `memory/.codebook.json`
+
+### Codebook Format
+```json
+{
+ "version": 1,
+ "entries": {
+ "$A1": "example_user",
+ "$A2": "10.0.1",
+ "$A3": "workspace",
+ "$A4": "server_key.pem",
+ "$A5": "my-secret-token-2024"
+ }
+
+**Before (TOOLS.md excerpt):**
+- user: example_user
+- SSH: ssh -i ~/.ssh/server_key.pem deploy@10.0.1.2
+- IP: 10.0.1.1, Token: my-secret-token-2024, Workspace: /home/user/workspace
+
+- user: $A1
+- SSH: ssh -i ~/.ssh/$A4 deploy@$A2.2
+- IP: $A2.1, Token: $A5, Workspace: /home/$A1/$A3
+
+### Roundtrip Guarantee
+`decompress_text(compress_text(text, codebook), codebook) == text` — always. The compression and decompression functions are perfect inverses. This is verified by 50+ roundtrip tests covering edge cases (overlapping phrases, adjacent codes, Unicode, empty input).
+
+### Collision Avoidance
+Codes use the `$` prefix followed by uppercase alphanumeric characters. The codebook builder checks that no code is a substring of another code and that no code appears naturally in the source text.
+
+## 3. Session Observation Compression
+**Module:** `observation_compressor.py`
+**Typical savings:** ~97% on session transcripts
+**Lossless:** No (facts preserved, verbosity removed)
+
+This is the single largest source of savings. Raw session transcripts contain verbose tool output — file contents, command results, API responses — most of which is never needed again.
+
+### Pipeline
+.jsonl transcript (26,000 tokens)
+ │
+ ▼
+ Parse messages → extract tool calls
+ Classify interactions → [feature|bugfix|decision|discovery|config|...]
+ Rule-based extraction → key facts, errors, decisions
+ Generate LLM prompt (optional) → structured XML
+ Format as markdown observation (~780 tokens)
+
+### Observation XML Format
+```xml
+<observations>
+ <observation>
+ <type>config</type>
+ <title>Network configured for multi-node setup</title>
+ <facts>
+ - Gateway: 10.0.1.1, Remote node: 10.0.1.2, Worker: 10.0.1.3
+ </facts>
+ <narrative>Set up mesh network connecting 3 nodes</narrative>
+ </observation>
+</observations>
+
+**Before (raw session, 847 lines):**
+{"role":"assistant","content":"Let me check the network..."}
+{"role":"tool","name":"exec","content":"network status\n200 OK...\n"}
+{"role":"assistant","content":"Good, the network is active. Let me check peers..."}
+... (800+ more lines of tool output)
+
+**After (observation, 12 lines):**
+
+## 1. [config] Multi-Node Network Setup
+**Facts:**
+- Gateway: 10.0.1.1, Remote node: 10.0.1.2, All peers connected
+
+**Result:** 3-node mesh network operational
+
+## 4. RLE Pattern Compression
+**Module:** `lib/rle.py`
+**Typical savings:** 1-2%
+**Lossless:** Yes (roundtrip supported)
+
+Targets three categories of structured repetitive data:
+
+### Path Compression
+Long workspace paths are replaced with `$WS`:
+
+Before: /home/user/workspace/skills/claw-compactor/scripts/lib/tokens.py
+After: $WS/skills/claw-compactor/scripts/lib/tokens.py
+
+Decompression: `decompress_paths(text, "/home/user/workspace")`
+
+### IP Family Compression
+When multiple IPs share a common prefix (≥2 occurrences), the prefix is extracted:
+
+Before:
+ - 10.0.1.1
+
+After:
+ $IP1=10.0.1.
+ - $IP1.1
+
+### Enumeration Compaction
+Detects comma-separated uppercase lists and compacts them:
+
+Before: The supported types are FEATURE, BUGFIX, DECISION, DISCOVERY, CONFIG, DEPLOYMENT, DATA, INVESTIGATION
+After: Types: [FEATURE,BUGFIX,DECISION,DISCOVERY,CONFIG,DEPLOYMENT,DATA,INVESTIGATION]
+
+## 5. Compressed Context Protocol (CCP)
+**Module:** `compressed_context.py`
+**Typical savings:** 20-60% depending on level
+**Lossless:** No (designed for model consumption)
+
+CCP is designed for a specific use case: compress context on a cheap model, then feed it to an expensive model. The receiving model gets decompression instructions in its system prompt.
+
+### Three Levels
+
+#### Ultra (40-60% compression)
+Aggressive abbreviation + filler removal. The output looks telegraphic:
+
+ John has approximately 15 years of experience in software development,
+ with a focus on infrastructure and cloud architecture. He is the
+ Chief Executive Officer of TechCorp, based in San Francisco.
+
+ John ~15y exp software dev, focus infra+cloud arch. CEO: TechCorp, loc:SF
+
+Decompression instruction:
+ "Compressed notation: key:val=attribute, loc:X=location,
+ Ny+=N+ years, slash-separated=alternatives. Expand naturally."
+
+#### Medium (20-35% compression)
+Moderate abbreviation with key:value notation:
+
+ The application server runs on port 8080 with a maximum of 256
+ concurrent connections. The database connection pool is configured
+ with 20 minimum and 50 maximum connections.
+
+ App server: port 8080, max 256 concurrent conns.
+ DB pool: min 20, max 50 conns.
+
+#### Light (10-20% compression)
+Light condensation only — remains fully human-readable:
+
+ We decided to use PostgreSQL instead of MySQL for the new project
+ because it has better support for JSON columns and more advanced
+ indexing capabilities that we need for our search functionality.
+
+ Decision: PostgreSQL over MySQL — better JSON column support
+ and advanced indexing for search needs.
+
+### Decompression Instructions
+Each level generates a decompression instruction block to prepend to the receiving model's system prompt:
+
+Ultra: "Compressed notation: key:val=attribute, loc:X=location, ..."
+Medium: "Text uses abbreviated notation: key:value pairs, condensed lists, ..."
+Light: "Text is lightly condensed. Read normally."
+
+## Technique Comparison
+| Rule engine | 4-8% | | Zero | Memory files |
+| Dictionary | 4-5% | | Zero | Repetitive workspaces |
+| Observation | ~97% | * | Zero or 1 LLM call | Session transcripts |
+| RLE | 1-2% | | Zero | Path-heavy, IP-heavy docs |
+| CCP | 20-60% | | Zero | Cross-model context passing |
+
+*Observation compression preserves all facts and decisions; only verbose tool output is removed.
+
+## Pipeline Interaction
+The techniques are designed to compose:
+
+1. **Rule engine first** — removes obvious waste before dictionary scoring
+2. **Dictionary second** — works on cleaner text, better phrase detection
+3. **RLE alongside dictionary** — different targets, no interference
+4. **Observation runs independently** — operates on transcripts, not memory files
+5. **CCP runs last or standalone** — can compress already-compressed output further