@jairodri/huckleberry-cli 1.0.1 → 1.1.0

package/skills/claw-compactor/references/testing.md

@@ -0,0 +1,160 @@
+# Testing
+
+## Philosophy
+claw-compactor follows a **trust-through-testing** approach. Every compression technique must prove:
+
+1. **Correctness** — lossless techniques roundtrip perfectly; lossy techniques preserve all facts
+2. **Safety** — edge cases (empty files, Unicode, malformed markdown) never crash
+3. **Non-inflation** — compressed output is never larger than input
+4. **Idempotency** — running compression twice produces the same result
+
+## Test Suite Overview
+**810+ tests** across 30 test files, covering unit tests, integration tests, and real-workspace validation.
+
+```
+tests/
+├── conftest.py # Shared fixtures
+│
+├── # Core module tests
+├── test_compress_memory.py # Rule engine compression
+├── test_compress_memory_comprehensive.py # Extended rule engine tests
+├── test_dictionary.py # Dictionary encoding basics
+├── test_dictionary_comprehensive.py # Codebook edge cases, roundtrip
+├── test_observation_compressor.py # Observation pipeline
+├── test_observation_comprehensive.py # Extended observation tests
+├── test_compressed_context.py # CCP levels
+├── test_dedup_memory.py # Shingle dedup
+├── test_generate_summary_tiers.py # Tier generation
+├── test_estimate_tokens.py # Token estimation
+├── test_audit_memory.py # Audit checks
+├── test_audit_comprehensive.py # Extended audit tests
+├── # Library tests
+├── test_lib_tokens.py # tiktoken + fallback
+├── test_lib_dedup.py # Shingle hashing, Jaccard
+├── test_lib_markdown.py # MD parsing, normalization
+├── test_rle.py # RLE basics
+├── test_rle_comprehensive.py # Path/IP/enum edge cases
+├── test_tokenizer_optimizer.py # Format optimization
+├── test_tokenizer_optimizer_comprehensive.py # Extended optimizer tests
+├── test_config.py # Config loading
+├── test_tokens.py # Token utilities
+├── # Integration & validation
+├── test_main_entry.py # mem_compress.py CLI routing
+├── test_cli_commands.py # Subprocess CLI invocation
+├── test_pipeline.py # Full pipeline integration
+├── test_integration.py # End-to-end scenarios
+├── test_roundtrip.py # Roundtrip guarantees
+├── test_roundtrip_comprehensive.py # Extended roundtrip tests
+├── test_performance.py # Performance regression
+├── test_benchmark.py # Benchmark command
+├── test_tiers_comprehensive.py # Tier edge cases
+├── test_error_handling.py # Error paths
+├── test_new_features.py # Recent feature tests
+├── test_real_workspace.py # Real workspace validation
+├── test_token_economics.py # Cost calculations
+└── test_markdown_advanced.py # Advanced MD scenarios
+
+## Coverage Matrix
+| compress_memory | | | | |
+| dictionary_compress | | | | (50+ cases) |
+| observation_compressor | | | | N/A (lossy) |
+| dedup_memory | | | | N/A |
+| generate_summary_tiers | | | | N/A |
+| estimate_tokens | | | | N/A |
+| audit_memory | | | | N/A |
+| compressed_context | | | | N/A (lossy) |
+| lib/tokens | | — | | N/A |
+| lib/markdown | | — | | N/A |
+| lib/dedup | | — | | N/A |
+| lib/dictionary | | — | | |
+| lib/rle | | — | | |
+| lib/tokenizer_optimizer | | — | | N/A |
+| lib/config | | — | | N/A |
+| mem_compress (CLI) | | | | N/A |
+
+### Edge Cases Tested
+- **Empty files** — all modules handle gracefully
+- **Unicode/CJK** — Chinese headers, mixed en/zh, emoji, accented characters
+- **Large files** — 100K+ characters, 2000+ sections
+- **Malformed markdown** — unclosed code blocks, broken headers, missing spaces
+- **Headers-only files** — no body content
+- **Single-line files** — minimal content
+- **Nonexistent paths** — proper errors and exit codes
+- **Overlapping dictionary codes** — no collisions
+- **Adjacent `$XX` codes** — correct boundary detection
+- **Empty codebooks** — graceful no-op
+
+## Running Tests
+```bash
+cd skills/claw-compactor
+
+# Run all tests
+PYTHONPATH=scripts python3 -m pytest tests/ -v
+
+# Run a specific test file
+PYTHONPATH=scripts python3 -m pytest tests/test_dictionary.py -v
+
+# Run a specific test class
+PYTHONPATH=scripts python3 -m pytest tests/test_roundtrip.py::TestDictionaryRoundtrip -v
+
+# Run with coverage (requires pytest-cov)
+PYTHONPATH=scripts python3 -m pytest tests/ --cov=lib --cov-report=term-missing
+
+# Quick check (no verbose)
+PYTHONPATH=scripts python3 -m pytest tests/ -q
+
+**Expected output:**
+810 passed in 31s
+
+## Fixtures (conftest.py)
+Shared test fixtures provide consistent test environments:
+
+- `tmp_workspace`: Workspace with MEMORY.md + `memory/` containing 2 daily files
+- `empty_file`: Empty `.md` file
+- `unicode_file`: Chinese + Japanese + emoji + accented characters
+- `large_file`: 2000 sections, 100K+ characters
+- `broken_markdown`: Malformed headers, unclosed code blocks
+- `headers_only`: Only header lines, no body text
+- `single_line`: Single line of text
+- `duplicate_content`: Two files with known overlapping sections
+
+## Adding New Tests
+
+### For a new compression technique
+1. Create `tests/test_<technique>.py`
+2. Include at minimum:
+ - **Basic functionality** — happy path
+ - **Empty input** — should return empty/no-op
+ - **Unicode input** — CJK, emoji, mixed scripts
+ - **Roundtrip** (if lossless) — `decompress(compress(x)) == x`
+ - **Non-inflation** — `len(compress(x)) <= len(x)` in tokens
+ - **Idempotency** — `compress(compress(x)) == compress(x)`
+
+3. Add fixture if needed in `conftest.py`
+
+### For a new edge case
+1. Add to the most relevant existing test file
+2. Use the `@pytest.mark.parametrize` decorator for variants
+3. Document what the edge case covers in the test docstring
+
+### Test naming convention
+```python
+class TestModuleName:
+ def test_basic_functionality(self):
+ """Module handles the standard case."""
+
+ def test_empty_input(self):
+ """Module handles empty input gracefully."""
+
+ def test_unicode_content(self):
+ """Module handles CJK and emoji content."""
+
+ def test_roundtrip_guarantee(self):
+ """Compress then decompress returns original."""
+
+## Continuous Validation
+Tests should be run:
+- Before any code change is committed
+- After modifying any `lib/` module (shared dependencies)
+- After updating compression rules or codebook logic
+- As part of the `full` pipeline verification (post-packaging)

package/skills/claw-compactor/scripts/audit_memory.py

@@ -0,0 +1,216 @@
+#!/usr/bin/env python3
+"""Audit workspace memory files for token usage, staleness, and compression opportunities.
+
+Scans all markdown files in a workspace and reports:
+- Total token budget usage
+- File age distribution
+- Stale entries (not updated in N days)
+- Compression suggestions
+
+Usage:
+    python3 audit_memory.py <workspace_path> [--stale-days 14] [--json]
+
+Part of claw-compactor. License: MIT.
+"""
+
+import argparse
+import json
+import logging
+import os
+import sys
+import time
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, List, Any, Optional
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from lib.tokens import estimate_tokens
+from lib.markdown import parse_sections, compress_markdown_table, strip_emoji
+from lib.exceptions import FileNotFoundError_
+
+logger = logging.getLogger(__name__)
+
+# Default memory token budgets
+DEFAULT_BUDGETS = {
+    "MEMORY.md": 2000,
+    "TOOLS.md": 1500,
+    "AGENTS.md": 2000,
+    "daily_total": 5000,
+    "workspace_total": 15000,
+}
+
+
+def _has_tables(text: str) -> bool:
+    """Check if text contains markdown tables."""
+    return '|' in text and '---' in text
+
+
+def _has_emoji(text: str) -> bool:
+    """Check if text contains emoji characters."""
+    from lib.markdown import _EMOJI_RE
+    return bool(_EMOJI_RE.search(text))
+
+
+def _count_empty_sections(text: str) -> int:
+    """Count sections with no meaningful body content."""
+    from lib.markdown import parse_sections
+    sections = parse_sections(text)
+    return sum(1 for h, b, _ in sections if h and not b.strip())
+
+
+def _file_age_days(path: Path) -> float:
+    """Return the age of *path* in days since last modification."""
+    mtime = path.stat().st_mtime
+    return (time.time() - mtime) / 86400
+
+
+def audit_file(
+    path: Path,
+    stale_days: int = 14,
+) -> Dict[str, Any]:
+    """Audit a single markdown file.
+
+    Returns dict with name, tokens, is_stale, suggestions, etc.
+    """
+    text = path.read_text(encoding="utf-8", errors="replace")
+    tokens = estimate_tokens(text)
+    age = _file_age_days(path)
+    is_stale = age > stale_days
+
+    suggestions: List[str] = []
+
+    # Check for tables that could be compressed
+    if '|' in text and '---' in text:
+        compressed = compress_markdown_table(text)
+        if len(compressed) < len(text) * 0.9:
+            suggestions.append("Table detected — compress_markdown_table could save tokens")
+
+    # Check for emoji
+    stripped = strip_emoji(text)
+    if len(stripped) < len(text):
+        suggestions.append("Contains emoji — strip_emoji could save tokens")
+
+    # Check for empty sections
+    sections = parse_sections(text)
+    empty_count = sum(1 for h, b, _ in sections if h and not b.strip())
+    if empty_count > 0:
+        suggestions.append(f"{empty_count} empty section(s) — remove_empty_sections")
+
+    # Check token budget
+    budget = DEFAULT_BUDGETS.get(path.name, DEFAULT_BUDGETS["workspace_total"])
+    if tokens > budget:
+        suggestions.append(f"Over budget: {tokens:,} tokens (budget: {budget:,})")
+
+    if is_stale:
+        suggestions.append(f"Stale: not modified in {age:.0f} days")
+
+    return {
+        "path": str(path),
+        "file": str(path),
+        "name": path.name,
+        "tokens": tokens,
+        "age_days": round(age, 1),
+        "is_stale": is_stale,
+        "suggestions": suggestions,
+        "sections": len(sections),
+    }
+
+
+def audit_workspace(
+    workspace: str,
+    stale_days: int = 14,
+) -> Dict[str, Any]:
+    """Audit all memory files in *workspace*.
+
+    Raises FileNotFoundError_ if workspace doesn't exist.
+    """
+    p = Path(workspace)
+    if not p.exists():
+        raise FileNotFoundError_(f"Workspace not found: {workspace}")
+
+    files: List[Path] = []
+    for f in sorted(p.glob("*.md")):
+        files.append(f)
+    mem_dir = p / "memory"
+    if mem_dir.is_dir():
+        for f in sorted(mem_dir.glob("*.md")):
+            files.append(f)
+
+    if not files:
+        return {
+            "total_files": 0,
+            "total_tokens": 0,
+            "files": [],
+            "age_distribution": {},
+            "suggestions": [],
+        }
+
+    file_results = [audit_file(f, stale_days=stale_days) for f in files]
+    total_tokens = sum(r["tokens"] for r in file_results)
+
+    # Age distribution
+    age_bins = {"<7d": 0, "7-30d": 0, "30-90d": 0, ">90d": 0}
+    for r in file_results:
+        age = r["age_days"]
+        if age < 7:
+            age_bins["<7d"] += 1
+        elif age < 30:
+            age_bins["7-30d"] += 1
+        elif age < 90:
+            age_bins["30-90d"] += 1
+        else:
+            age_bins[">90d"] += 1
+
+    # Aggregate suggestions
+    all_suggestions = []
+    for r in file_results:
+        for s in r["suggestions"]:
+            all_suggestions.append(f"[{r['name']}] {s}")
+
+    return {
+        "total_files": len(file_results),
+        "total_tokens": total_tokens,
+        "files": file_results,
+        "age_distribution": age_bins,
+        "suggestions": all_suggestions,
+    }
+
+
+def format_report(result: Dict[str, Any]) -> str:
+    """Format audit result as a human-readable report."""
+    lines = [
+        "=== Memory Audit Report ===",
+        f"Files: {result['total_files']}",
+        f"Total tokens: {result['total_tokens']:,}",
+        "",
+        "Age distribution:",
+    ]
+    for bucket, count in result.get("age_distribution", {}).items():
+        lines.append(f"  {bucket}: {count} files")
+
+    if result.get("suggestions"):
+        lines.append("\nSuggestions:")
+        for s in result["suggestions"]:
+            lines.append(f"  - {s}")
+    else:
+        lines.append("\nNo suggestions — workspace looks healthy.")
+
+    return '\n'.join(lines)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Audit workspace memory files")
+    parser.add_argument("workspace", help="Workspace directory")
+    parser.add_argument("--stale-days", type=int, default=14, help="Days before stale")
+    parser.add_argument("--json", action="store_true", help="JSON output")
+    args = parser.parse_args()
+
+    result = audit_workspace(args.workspace, stale_days=args.stale_days)
+    if args.json:
+        print(json.dumps(result, indent=2))
+    else:
+        print(format_report(result))
+
+
+if __name__ == "__main__":
+    main()

package/skills/claw-compactor/scripts/compress_memory.py

@@ -0,0 +1,236 @@
+#!/usr/bin/env python3
+"""Compress memory files using rule-based preprocessing + LLM semantic compression.
+
+Two-phase approach:
+1. Rule engine: dedup lines, strip markdown redundancy, merge similar entries
+2. LLM prompt: generate a prompt for semantic compression of remaining content
+
+Usage:
+    python3 compress_memory.py <path> [--dry-run] [--output FILE] [--older-than DAYS] [--no-llm]
+
+Part of claw-compactor. License: MIT.
+"""
+
+import argparse
+import json
+import logging
+import os
+import sys
+import time
+from datetime import datetime, timedelta
+from pathlib import Path
+from typing import List, Dict, Any, Optional
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from lib.tokens import estimate_tokens, using_tiktoken
+from lib.markdown import (
+    strip_markdown_redundancy, remove_duplicate_lines, parse_sections,
+    normalize_chinese_punctuation, strip_emoji, remove_empty_sections,
+    compress_markdown_table, merge_similar_bullets, merge_short_bullets,
+)
+from lib.dedup import find_duplicates, merge_duplicates
+from lib.exceptions import FileNotFoundError_, MemCompressError
+
+logger = logging.getLogger(__name__)
+
+# LLM prompt template for semantic compression
+COMPRESS_PROMPT = """You are a memory compression assistant. Compress the following text to approximately {target_pct}% of its current size while preserving ALL factual information, decisions, configurations, and actionable items.
+
+Rules:
+- Keep all names, IPs, paths, tokens, dates, and technical details EXACTLY
+- Remove filler words, redundant explanations, and verbose phrasing
+- Merge related items
+- Use concise notation (key:value, abbreviations)
+- Preserve markdown structure (headers, bullets)
+- Output ONLY the compressed text, no commentary
+
+Text to compress:
+---
+{content}
+---
+
+Compressed version:"""
+
+
+def _file_age_days(path: Path) -> float:
+    """Return file age in days based on mtime."""
+    return (time.time() - path.stat().st_mtime) / 86400
+
+
+def rule_compress(
+    text: str,
+    enable_emoji_strip: bool = True,
+) -> str:
+    """Apply all rule-based compression passes to *text*.
+
+    Returns the compressed text. Never increases token count.
+    """
+    if not text:
+        return ""
+
+    result = text
+
+    # 1. Normalize Chinese punctuation
+    result = normalize_chinese_punctuation(result)
+
+    # 2. Strip markdown redundancy (excess blanks, trailing whitespace)
+    result = strip_markdown_redundancy(result)
+
+    # 3. Remove duplicate lines
+    result = remove_duplicate_lines(result)
+
+    # 4. Remove empty sections
+    result = remove_empty_sections(result)
+
+    # 5. Compress markdown tables to key:value
+    result = compress_markdown_table(result)
+
+    # 6. Strip emoji if enabled
+    if enable_emoji_strip:
+        result = strip_emoji(result)
+
+    # 7. Merge similar bullets
+    result = merge_similar_bullets(result)
+
+    # 8. Merge short bullets
+    result = merge_short_bullets(result)
+
+    # 9. Final cleanup
+    result = strip_markdown_redundancy(result)
+
+    return result
+
+
+def generate_llm_prompt(content: str, target_pct: int = 50) -> str:
+    """Generate an LLM prompt for semantic compression of *content*."""
+    return COMPRESS_PROMPT.format(content=content, target_pct=target_pct)
+
+
+def _collect_files(
+    target: str,
+    older_than: Optional[int] = None,
+) -> List[Path]:
+    """Collect markdown files from *target* (file or directory).
+
+    If *older_than* is set, only include files older than N days.
+    """
+    path = Path(target)
+    if not path.exists():
+        raise FileNotFoundError_(f"Path not found: {target}")
+
+    if path.is_file():
+        if older_than is not None and _file_age_days(path) < older_than:
+            return []
+        return [path]
+
+    # Directory: collect all .md files recursively, excluding junk dirs
+    _EXCLUDE_DIRS = {"node_modules", ".git", "__pycache__", ".venv", "venv",
+                     "target", "dist", "build", ".tox", ".mypy_cache", ".pytest_cache"}
+    files = []
+    for f in sorted(path.rglob("*.md")):
+        if _EXCLUDE_DIRS.intersection(f.relative_to(path).parts):
+            continue
+        files.append(f)
+    if older_than is not None:
+        files = [f for f in files if _file_age_days(f) >= older_than]
+    return files
+
+
+def compress_file(
+    path: Path,
+    dry_run: bool = False,
+    output: Optional[str] = None,
+    no_llm: bool = False,
+) -> Dict[str, Any]:
+    """Compress a single file using rule-based compression.
+
+    Args:
+        path: File to compress.
+        dry_run: If True, don't write changes.
+        output: Optional output file path.
+        no_llm: If True, skip LLM prompt generation.
+
+    Returns a dict with compression statistics.
+    """
+    path = Path(path)
+    original = path.read_text(encoding="utf-8")
+    original_tokens = estimate_tokens(original)
+
+    compressed = rule_compress(original)
+    compressed_tokens = estimate_tokens(compressed)
+
+    reduction_pct = ((original_tokens - compressed_tokens) / original_tokens * 100) if original_tokens else 0.0
+    result: Dict[str, Any] = {
+        "file": str(path),
+        "original_tokens": original_tokens,
+        "rule_compressed_tokens": compressed_tokens,
+        "rule_reduction_pct": round(reduction_pct, 2),
+        "dry_run": dry_run,
+    }
+
+    if not no_llm and compressed.strip():
+        result["llm_prompt"] = generate_llm_prompt(compressed)
+
+    if not dry_run:
+        target = Path(output) if output else path
+        target.write_text(compressed, encoding="utf-8")
+        result["written_to"] = str(target)
+
+    return result
+
+
+def llm_compress_file(
+    path: Path,
+    target_pct: int = 40,
+) -> Dict[str, Any]:
+    """Generate an LLM compression prompt for a file and write it to a .prompt file.
+
+    Returns stats dict with original_tokens, rule_compressed_tokens, prompt_file, etc.
+    """
+    text = path.read_text(encoding="utf-8", errors="replace")
+    original_tokens = estimate_tokens(text)
+    compressed = rule_compress(text)
+    rule_tokens = estimate_tokens(compressed)
+    prompt = generate_llm_prompt(compressed, target_pct)
+    prompt_tokens = estimate_tokens(prompt)
+
+    prompt_path = path.with_suffix(".prompt.md")
+    prompt_path.write_text(prompt, encoding="utf-8")
+
+    return {
+        "file": str(path),
+        "original_tokens": original_tokens,
+        "rule_compressed_tokens": rule_tokens,
+        "prompt_tokens": prompt_tokens,
+        "prompt_file": str(prompt_path),
+        "target_pct": target_pct,
+        "instruction": f"Feed this prompt to an LLM for further {target_pct}% compression.",
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Compress memory files")
+    parser.add_argument("path", help="File or directory to compress")
+    parser.add_argument("--dry-run", action="store_true")
+    parser.add_argument("--output", help="Output file")
+    parser.add_argument("--older-than", type=int, help="Only files older than N days")
+    parser.add_argument("--no-llm", action="store_true", help="Skip LLM prompt")
+    parser.add_argument("--json", action="store_true")
+    args = parser.parse_args()
+
+    files = _collect_files(args.path, older_than=args.older_than)
+    results = []
+    for f in files:
+        r = compress_file(f, dry_run=args.dry_run, output=args.output, no_llm=args.no_llm)
+        results.append(r)
+
+    if args.json:
+        print(json.dumps(results, indent=2, ensure_ascii=False))
+    else:
+        for r in results:
+            saved = r["original_tokens"] - r["rule_compressed_tokens"]
+            print(f"{r['file']}: {r['original_tokens']} → {r['rule_compressed_tokens']} tokens (saved {saved})")
+
+
+if __name__ == "__main__":
+    main()