@kulapard/pi-caveman 0.1.0 → 0.2.0

package/AGENTS.md

@@ -22,12 +22,12 @@ Project memory for agents working in this repo. Non-obvious conventions only.
   SDK is needed at test time. A value import from `@earendil-works/pi-coding-agent`
   would break the tests — `tests/extension.test.mjs` asserts this invariant.
 - **Verbatim preservation**: caveman-compress never alters code blocks, inline
-  code, URLs, file paths, commands, or exact error strings. `validate.py` enforces
-  this and `compress.py` aborts + restores the original on any validation failure.
-- `caveman-compress` is **model-bound**: `compress.py` `call_claude()` calls the
-  Anthropic SDK (if `ANTHROPIC_API_KEY` is set) or the `claude --print` CLI. The
-  deterministic, unit-tested pieces are `detect.py`, `validate.py`, and the pure
-  helpers; the live model call is never exercised in tests (it is monkeypatched).
+  code, URLs, file paths, commands, or exact error strings. The skill instructs
+  the agent to self-validate these against the original and, on any mismatch it
+  cannot fix, restore from the `.original` backup rather than leave a corrupted file.
+- `caveman-compress` is **prompt-only**: the Pi agent performs the compression
+  with its own model and file tools, driven by `SKILL.md`. There is no Python and
+  no external model CLI; coverage is the doc-guard test `tests/compress-docs.test.mjs`.
 
 ## Tests / validation
 
@@ -36,9 +36,6 @@ Project memory for agents working in this repo. Non-obvious conventions only.
 - The JS test glob (`tests/**/*.test.mjs`) is expanded by the Node `--test` runner,
   not the shell. Directory-recursion (`--test tests/`) does **not** work on the
   current Node — keep the glob.
-- **Python tests are not on PATH.** Create a venv and install pytest:
-  `python3 -m venv .venv && .venv/bin/pip install pytest`, then run
-  `npm run test:py` (which calls `.venv/bin/pytest skills/caveman-compress`).
 - Several tests are **phantom-reference guards** (`tests/stats-docs.test.mjs`,
   `tests/cavecrew-docs.test.mjs`, `tests/compress-docs.test.mjs`): they assert the
   docs do **not** mention a Claude-Code hooks layer, a plugin install path, the

package/CHANGELOG.md

@@ -0,0 +1,46 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.0] - 2026-06-29
+
+### Changed
+
+- `caveman-compress` is now a prompt-only skill: the Pi agent performs the
+  compression itself (its own model and file tools), driven by `SKILL.md`. The
+  `/caveman-compress` command and the compression rules are unchanged.
+
+### Removed
+
+- The Python `caveman-compress` toolkit (`scripts/` + the pytest suite) and the
+  `test:py` script. The package no longer requires Python or an external
+  `claude` CLI / `ANTHROPIC_API_KEY`.
+
+## [0.1.0] - 2026-06-29
+
+Initial release — a [Pi](https://github.com/earendil-works/pi-coding-agent)
+port of [caveman](https://github.com/JuliusBrussee/caveman).
+
+### Added
+
+- Caveman output-mode extension (`extensions/caveman.ts`): six intensity modes
+  (lite, full, ultra, wenyan-lite, wenyan-full, wenyan-ultra; default full),
+  slash commands, natural-language activation/deactivation, and a session
+  statusline indicator.
+- Skills: `caveman`, `caveman-commit`, `caveman-review`, `caveman-help`,
+  `caveman-stats`, `caveman-compress`, and the `cavecrew` subagent suite.
+- npm packaging as `@kulapard/pi-caveman` (scoped, public) with a `files`
+  whitelist, a `prepublishOnly` test gate, and repository metadata.
+- GitHub Actions: `ci.yml` (test on push and pull requests) and `publish.yml`
+  (publish to npm via Trusted Publishing / OIDC on `v[0-9]*` tags — no stored
+  token, automatic provenance, a tag-equals-version guard, and a concurrency
+  group).
+
+[Unreleased]: https://github.com/kulapard/pi-caveman/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/kulapard/pi-caveman/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/kulapard/pi-caveman/releases/tag/v0.1.0

package/README.md

@@ -56,12 +56,6 @@ persistent install over a per-session `pi -e`.
 ```bash
 npm install            # fetch the Pi SDK + TypeScript dev deps
 npm test               # typecheck + extension/manifest/docs unit tests
-
-# Python tests for the caveman-compress toolkit need pytest in a local venv
-# (pytest is not on PATH on a fresh checkout):
-python3 -m venv .venv
-.venv/bin/pip install pytest
-npm run test:py        # runs: .venv/bin/pytest skills/caveman-compress
 ```
 
 ## Modes
@@ -122,16 +116,12 @@ tool descriptions. pi-caveman does **not** bundle it: that proxy works at the
 MCP-client layer, independent of Pi, and upstream itself ships it as a separate
 package — so it does not belong in this extension-plus-skills package.
 
-The Pi-side equivalent is the Python `caveman-compress` toolkit, invoked via the
-`/caveman-compress` skill/command, which compresses prose memory files in place
-(writing a `FILE.original.md` backup) while preserving code, URLs, and paths
-verbatim. Note plainly: `caveman-compress` is **itself Claude-bound** — it
-performs compression via a live model call (the Anthropic SDK if
-`ANTHROPIC_API_KEY` is set, otherwise the `claude --print` CLI). So "the Pi
-equivalent" means *invoked via a Pi skill/command*, **not** *model-independent*.
-This is the same nature as upstream's MCP shrink (also a model-mediated
-transform); only the integration mechanism differs (a Pi skill here vs. MCP
-middleware upstream).
+The Pi-side equivalent is the `caveman-compress` skill, invoked via the
+`/caveman-compress` command. It is prompt-only: the Pi agent itself compresses a
+prose memory file in place (writing a `FILE.original.md` backup) using its own
+model and file tools, preserving code, URLs, and paths verbatim. No Python and no
+external Claude CLI are involved — compression is performed by the host Pi agent,
+the same way the other skills work.
 
 ## Attribution & license
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kulapard/pi-caveman",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Caveman for Pi: ultra-compressed agent output that preserves technical substance. Six intensity modes, slash commands, natural-language activation, and a session statusline.",
   "type": "module",
   "license": "MIT",
@@ -20,9 +20,7 @@
     "skills",
     "agents",
     "AGENTS.md",
-    "!**/__pycache__",
-    "!**/*.pyc",
-    "!skills/**/tests"
+    "CHANGELOG.md"
   ],
   "engines": {
     "node": ">=18"
@@ -47,7 +45,6 @@
     "pretest": "npm run typecheck",
     "test": "node --experimental-strip-types --test tests/**/*.test.mjs",
     "typecheck": "tsc --noEmit",
-    "test:py": ".venv/bin/pytest skills/caveman-compress",
     "prepublishOnly": "npm test"
   },
   "devDependencies": {

package/skills/caveman-compress/README.md

@@ -40,7 +40,7 @@ Real results on real project files:
 | `mixed-with-code.md` | 888 | 560 | **36.9%** |
 | **Average** | **898** | **481** | **46%** |
 
-All validations passed ✅ — headings, code blocks, URLs, file paths preserved exactly.
+Headings, code blocks, URLs, and file paths preserved exactly.
 
 ## Before / After
 
@@ -65,23 +65,14 @@ All validations passed ✅ — headings, code blocks, URLs, file paths preserved
 
 **Same instructions. ~60% fewer tokens in this example (46% average across the files above). Every. Single. Session.**
 
-## Security
-
-`caveman-compress` is flagged as Snyk High Risk due to subprocess and file I/O patterns detected by static analysis. This is a false positive — see [SECURITY.md](./SECURITY.md) for a full explanation of what the skill does and does not do.
-
 ## Install
 
 This skill ships inside the pi-caveman package. Load the package (see the
 [root README](../../README.md) for the `pi -e … --skill …` / `pi install`
 mechanism), then use `/caveman-compress` in a Pi session.
 
-The compress toolkit lives at `skills/caveman-compress/` within the package; the
-skill instructions run the Python CLI under `scripts/`.
-
-**Requires:** Python 3.10+. Compression calls a model (the Anthropic SDK if
-`ANTHROPIC_API_KEY` is set, otherwise the `claude --print` CLI), so one of those
-must be available — see [Security](#security) and the root README's
-"Compression vs. upstream MCP shrink" note for why this step is model-bound.
+No extra runtime is required: the Pi agent performs the compression itself with
+its own model and file tools — there is no separate tool or language to install.
 
 ## Usage
 
@@ -111,23 +102,20 @@ Examples:
 ```
 /caveman-compress AGENTS.md
         ↓
-detect file type        (no tokens)
+agent detects file type      (prose? else skip)
         ↓
-toolkit calls a model to compress   (tokens — one call)
+agent backs up original  →  AGENTS.original.md   (verbatim, never overwritten)
         ↓
-validate output         (no tokens)
-  checks: headings, code blocks, URLs, file paths, bullets
+agent rewrites prose to caveman, code/URLs/paths left exact
         ↓
-if errors: model fixes cherry-picked issues only    (tokens — targeted fix)
-  does NOT recompress — only patches broken parts
+agent self-validates: protected tokens byte-identical to original
         ↓
-retry up to 2 times
+if a protected token changed: fix it, or restore from backup and report
         ↓
-write compressed → AGENTS.md
-write original   → AGENTS.original.md
+write compressed  →  AGENTS.md
 ```
 
-Only two things use tokens: initial compression + targeted fix if validation fails. Everything else is local Python.
+The agent does this with its own model and file tools — no external CLI, no separate runtime.
 
 ## What Is Preserved
 
@@ -149,15 +137,6 @@ A memory file (`AGENTS.md` / `CLAUDE.md`) loads on **every session start**. A 10
 
 Caveman cut that by ~46% on average. Same instructions. Same accuracy. Less waste.
 
-```
-┌────────────────────────────────────────────┐
-│  TOKEN SAVINGS PER FILE    █████       46% │
-│  SESSIONS THAT BENEFIT     ██████████ 100% │
-│  INFORMATION PRESERVED     ██████████ 100% │
-│  SETUP TIME                █            1x │
-└────────────────────────────────────────────┘
-```
-
 ## Part of Caveman
 
 This skill is part of the [caveman](https://github.com/JuliusBrussee/caveman) toolkit — making the agent use fewer tokens without losing accuracy. pi-caveman is the [Pi](https://github.com/earendil-works/pi-coding-agent) port.

package/skills/caveman-compress/SKILL.md

@@ -19,21 +19,17 @@ Compress natural language files (`AGENTS.md`, `CLAUDE.md`, todos, preferences) i
 
 ## Process
 
-1. The compression scripts live in `scripts/` (adjacent to this SKILL.md). If the path is not immediately available, search for `scripts/__main__.py` next to this SKILL.md.
+You (the Pi agent) perform the compression directly — there is no separate tool to run. Given `/caveman-compress <filepath>`:
 
-2. From the directory containing this SKILL.md, run:
+1. **Skip backups.** If the path ends in `.original.<ext>` (e.g. `AGENTS.original.md`), stop — never compress a backup file.
+2. **Check it is compressible** per **Boundaries** below: prose files (`.md`, `.txt`, `.rst`, `.typ`, `.typst`, `.tex`, or extensionless natural language). If it is code/config (`.py`, `.js`, `.ts`, `.json`, `.yaml`, …) or larger than ~500 KB, report it is out of scope and stop.
+3. **Read** the file's full contents.
+4. **Back up the original.** Write a verbatim copy to `<filename>.original.<ext>` (e.g. `AGENTS.md` → `AGENTS.original.md`), **only if that backup does not already exist** — never overwrite an existing `.original` backup.
+5. **Rewrite** the file in place, applying the **Compression Rules** below. Treat code blocks, inline code, URLs, paths, commands, headings, and table structure as read-only regions.
+6. **Self-validate** against the contents you read in step 3: every protected token — fenced and inline code, URLs, file paths, heading text, table structure, dates/version numbers — must be byte-for-byte identical. If any changed, fix just that region; if you cannot make it identical, restore the file from the `.original` backup and report the failure rather than leave a corrupted file.
+7. **Report** the result: bytes before/after and the approximate reduction.
 
-python3 -m scripts <absolute_filepath>
-
-3. The CLI will:
-- detect file type (no tokens)
-- call a model to compress (Anthropic SDK if `ANTHROPIC_API_KEY` is set, else the `claude --print` CLI)
-- validate output (no tokens)
-- if errors: cherry-pick fix via the same model call (targeted fixes only, no recompression)
-- retry up to 2 times
-- if still failing after 2 retries: report error to user, leave original file untouched
-
-4. Return result to user
+Only the rewrite needs the model (you); detection, backup, and validation are mechanical.
 
 ## Compression Rules
 
@@ -103,8 +99,9 @@ Compressed:
 
 ## Boundaries
 
-- ONLY compress natural language files (.md, .txt, .typ, .typst, .tex, extensionless)
+- ONLY compress natural language files (.md, .txt, .rst, .typ, .typst, .tex, extensionless)
 - NEVER modify: .py, .js, .ts, .json, .yaml, .yml, .toml, .env, .lock, .css, .html, .xml, .sql, .sh
+- Skip files larger than ~500 KB (too big to rewrite safely in one pass)
 - If file has mixed content (prose + code), compress ONLY the prose sections
 - If unsure whether something is code or prose, leave it unchanged
 - Original file is backed up as FILE.original.md before overwriting

package/skills/caveman-compress/SECURITY.md

@@ -1,31 +0,0 @@
-# Security
-
-## Snyk High Risk Rating
-
-`caveman-compress` receives a Snyk High Risk rating due to static analysis heuristics. This document explains what the skill does and does not do.
-
-### What triggers the rating
-
-1. **subprocess usage**: The skill calls the `claude` CLI via `subprocess.run()` as a fallback when `ANTHROPIC_API_KEY` is not set. The subprocess call uses a fixed argument list — no shell interpolation occurs. User file content is passed via stdin, not as a shell argument.
-
-2. **File read/write**: The skill reads the file the user explicitly points it at, compresses it, and writes the result back to the same path. A `.original.md` backup is saved alongside it. No files outside the user-specified path are read or written.
-
-### What the skill does NOT do
-
-- Does not execute user file content as code
-- Does not make network requests except to Anthropic's API (via SDK or CLI)
-- Does not access files outside the path the user provides
-- Does not use shell=True or string interpolation in subprocess calls
-- Does not collect or transmit any data beyond the file being compressed
-
-### Auth behavior
-
-If `ANTHROPIC_API_KEY` is set, the skill uses the Anthropic Python SDK directly (no subprocess). If not set, it falls back to the `claude` CLI, which uses the user's existing Claude desktop authentication.
-
-### File size limit
-
-Files larger than 500KB are rejected before any API call is made.
-
-### Reporting a vulnerability
-
-If you believe you've found a genuine security issue, please open a GitHub issue with the label `security`.

package/skills/caveman-compress/scripts/__init__.py

@@ -1,9 +0,0 @@
-"""Caveman compress scripts.
-
-This package provides tools to compress natural language markdown files
-into caveman format to save input tokens.
-"""
-
-__all__ = ["cli", "compress", "detect", "validate"]
-
-__version__ = "1.0.0"

package/skills/caveman-compress/scripts/__main__.py

@@ -1,3 +0,0 @@
-from .cli import main
-
-main()

package/skills/caveman-compress/scripts/benchmark.py

@@ -1,80 +0,0 @@
-#!/usr/bin/env python3
-from pathlib import Path
-import sys
-
-# Support both direct execution and module import
-try:
-    from .validate import validate
-except ImportError:
-    sys.path.insert(0, str(Path(__file__).parent))
-    from validate import validate
-
-try:
-    import tiktoken
-    _enc = tiktoken.get_encoding("o200k_base")
-except ImportError:
-    _enc = None
-
-
-def count_tokens(text):
-    if _enc is None:
-        return len(text.split())  # fallback: word count
-    return len(_enc.encode(text))
-
-
-def benchmark_pair(orig_path: Path, comp_path: Path):
-    orig_text = orig_path.read_text()
-    comp_text = comp_path.read_text()
-
-    orig_tokens = count_tokens(orig_text)
-    comp_tokens = count_tokens(comp_text)
-    saved = 100 * (orig_tokens - comp_tokens) / orig_tokens if orig_tokens > 0 else 0.0
-    result = validate(orig_path, comp_path)
-
-    return (comp_path.name, orig_tokens, comp_tokens, saved, result.is_valid)
-
-
-def print_table(rows):
-    print("\n| File | Original | Compressed | Saved % | Valid |")
-    print("|------|----------|------------|---------|-------|")
-    for r in rows:
-        print(f"| {r[0]} | {r[1]} | {r[2]} | {r[3]:.1f}% | {'✅' if r[4] else '❌'} |")
-
-
-def main():
-    # Direct file pair: python3 benchmark.py original.md compressed.md
-    if len(sys.argv) == 3:
-        orig = Path(sys.argv[1]).resolve()
-        comp = Path(sys.argv[2]).resolve()
-        if not orig.exists():
-            print(f"❌ Not found: {orig}")
-            sys.exit(1)
-        if not comp.exists():
-            print(f"❌ Not found: {comp}")
-            sys.exit(1)
-        print_table([benchmark_pair(orig, comp)])
-        return
-
-    # Glob mode: repo_root/tests/caveman-compress/
-    # __file__ lives at <repo_root>/skills/caveman-compress/scripts/benchmark.py
-    # Walk up four dirs: scripts → caveman-compress → skills → repo_root.
-    tests_dir = Path(__file__).resolve().parents[3] / "tests" / "caveman-compress"
-    if not tests_dir.exists():
-        print(f"❌ Tests dir not found: {tests_dir}")
-        sys.exit(1)
-
-    rows = []
-    for orig in sorted(tests_dir.glob("*.original.md")):
-        comp = orig.with_name(orig.stem.removesuffix(".original") + ".md")
-        if comp.exists():
-            rows.append(benchmark_pair(orig, comp))
-
-    if not rows:
-        print("No compressed file pairs found.")
-        return
-
-    print_table(rows)
-
-
-if __name__ == "__main__":
-    main()

package/skills/caveman-compress/scripts/cli.py

@@ -1,85 +0,0 @@
-#!/usr/bin/env python3
-"""
-Caveman Compress CLI
-
-Usage:
-    caveman <filepath>
-"""
-
-import sys
-
-# Force UTF-8 on stdout/stderr before any code can print. Windows consoles
-# default to cp1252 and crash on the ❌ glyphs in error/validation branches,
-# masking the real error and leaving the user with a half-compressed file.
-for _stream in (sys.stdout, sys.stderr):
-    reconfigure = getattr(_stream, "reconfigure", None)
-    if callable(reconfigure):
-        try:
-            reconfigure(encoding="utf-8", errors="replace")
-        except Exception:
-            pass
-
-from pathlib import Path
-
-from .compress import backup_dir_for, compress_file
-from .detect import detect_file_type, should_compress
-
-
-def print_usage():
-    print("Usage: caveman <filepath>")
-
-
-def main():
-    if len(sys.argv) != 2:
-        print_usage()
-        sys.exit(1)
-
-    filepath = Path(sys.argv[1])
-
-    # Check file exists
-    if not filepath.exists():
-        print(f"❌ File not found: {filepath}")
-        sys.exit(1)
-
-    if not filepath.is_file():
-        print(f"❌ Not a file: {filepath}")
-        sys.exit(1)
-
-    filepath = filepath.resolve()
-
-    # Detect file type
-    file_type = detect_file_type(filepath)
-
-    print(f"Detected: {file_type}")
-
-    # Check if compressible
-    if not should_compress(filepath):
-        print("Skipping: file is not natural language (code/config)")
-        sys.exit(0)
-
-    print("Starting caveman compression...\n")
-
-    try:
-        success = compress_file(filepath)
-
-        if success:
-            print("\nCompression completed successfully")
-            backup_path = backup_dir_for(filepath) / (filepath.stem + ".original.md")
-            print(f"Compressed: {filepath}")
-            print(f"Original:   {backup_path}")
-            sys.exit(0)
-        else:
-            print("\n❌ Compression failed after retries")
-            sys.exit(2)
-
-    except KeyboardInterrupt:
-        print("\nInterrupted by user")
-        sys.exit(130)
-
-    except Exception as e:
-        print(f"\n❌ Error: {e}")
-        sys.exit(1)
-
-
-if __name__ == "__main__":
-    main()

package/skills/caveman-compress/scripts/compress.py

@@ -1,341 +0,0 @@
-#!/usr/bin/env python3
-"""
-Caveman Memory Compression Orchestrator
-
-Usage:
-    python scripts/compress.py <filepath>
-"""
-
-import os
-import re
-import shutil
-import subprocess
-import sys
-from pathlib import Path
-
-OUTER_FENCE_REGEX = re.compile(
-    r"\A\s*(`{3,}|~{3,})[^\n]*\n(.*)\n\1\s*\Z", re.DOTALL
-)
-
-# YAML frontmatter: starts at file start with --- on its own line, ends with --- on its own line.
-# Captures the entire block (including delimiters and trailing newline) and the body after.
-FRONTMATTER_REGEX = re.compile(
-    r"\A(---\r?\n.*?\r?\n---\r?\n)(.*)", re.DOTALL
-)
-
-
-def split_frontmatter(text: str):
-    """Split YAML frontmatter from body. Returns (frontmatter, body).
-
-    Memory files (and many other markdown docs) start with a YAML frontmatter
-    block delimited by `---` lines. The compression LLM has a habit of stripping
-    or rewriting these despite preserve-structure rules in the prompt — so we
-    surgically remove the frontmatter before compression and prepend it back
-    verbatim to the output. Files without frontmatter pass through unchanged.
-    """
-    m = FRONTMATTER_REGEX.match(text)
-    if m:
-        return m.group(1), m.group(2)
-    return "", text
-
-# Filenames and paths that almost certainly hold secrets or PII. Compressing
-# them ships raw bytes to the Anthropic API — a third-party data boundary that
-# developers on sensitive codebases cannot cross. detect.py already skips .env
-# by extension, but credentials.md / secrets.txt / ~/.aws/credentials would
-# slip through the natural-language filter. This is a hard refuse before read.
-SENSITIVE_BASENAME_REGEX = re.compile(
-    r"(?ix)^("
-    r"\.env(\..+)?"
-    r"|\.netrc"
-    r"|credentials(\..+)?"
-    r"|secrets?(\..+)?"
-    r"|passwords?(\..+)?"
-    r"|id_(rsa|dsa|ecdsa|ed25519)(\.pub)?"
-    r"|authorized_keys"
-    r"|known_hosts"
-    r"|.*\.(pem|key|p12|pfx|crt|cer|jks|keystore|asc|gpg)"
-    r")$"
-)
-
-SENSITIVE_PATH_COMPONENTS = frozenset({".ssh", ".aws", ".gnupg", ".kube", ".docker"})
-
-SENSITIVE_NAME_TOKENS = (
-    "secret", "credential", "password", "passwd",
-    "apikey", "accesskey", "token", "privatekey",
-)
-
-
-def backup_dir_for(filepath: Path) -> Path:
-    """Resolve the out-of-tree backup directory for a given source file.
-
-    Backups must live OUTSIDE the source directory so skill auto-loaders
-    (Claude Code rules/, opencode instructions/, etc.) stop re-ingesting the
-    `.original.md` copies as live files. Base dir is platform-aware:
-      - Windows: %LOCALAPPDATA%\\caveman-compress\\backups
-      - else:    $XDG_DATA_HOME/caveman-compress/backups if set,
-                 else ~/.local/share/caveman-compress/backups
-
-    The source file's parent-dir name is mirrored under the base to reduce
-    cross-project collisions (e.g. two `task.md` files in different repos).
-    """
-    if os.name == "nt" or sys.platform == "win32":
-        local_appdata = os.environ.get("LOCALAPPDATA")
-        base = Path(local_appdata) if local_appdata else Path.home() / "AppData" / "Local"
-        base = base / "caveman-compress" / "backups"
-    else:
-        xdg = os.environ.get("XDG_DATA_HOME")
-        base = Path(xdg) if xdg else Path.home() / ".local" / "share"
-        base = base / "caveman-compress" / "backups"
-    return base / filepath.parent.name
-
-
-def is_sensitive_path(filepath: Path) -> bool:
-    """Heuristic denylist for files that must never be shipped to a third-party API."""
-    name = filepath.name
-    if SENSITIVE_BASENAME_REGEX.match(name):
-        return True
-    lowered_parts = {p.lower() for p in filepath.parts}
-    if lowered_parts & SENSITIVE_PATH_COMPONENTS:
-        return True
-    # Normalize separators so "api-key" and "api_key" both match "apikey".
-    lower = re.sub(r"[_\-\s.]", "", name.lower())
-    return any(tok in lower for tok in SENSITIVE_NAME_TOKENS)
-
-
-def strip_llm_wrapper(text: str) -> str:
-    """Strip outer ```markdown ... ``` fence when it wraps the entire output."""
-    m = OUTER_FENCE_REGEX.match(text)
-    if m:
-        return m.group(2)
-    return text
-
-from .detect import should_compress
-from .validate import validate
-
-MAX_RETRIES = 2
-
-
-# ---------- Claude Calls ----------
-
-
-def call_claude(prompt: str) -> str:
-    """Send a prompt to Claude.
-
-    Prefers the Anthropic SDK when ANTHROPIC_API_KEY is set; otherwise falls
-    back to the ``claude --print`` CLI (which handles desktop auth).
-
-    On Windows the CLI subprocess decoding defaults to the system codepage
-    (cp1251 / cp1252) and crashes on UTF-8 output — see issue #152. Pinning
-    ``encoding="utf-8"`` with ``errors="replace"`` matches the CLI's actual
-    native I/O and prevents the UnicodeDecodeError before validation can
-    report. Windows users with non-ASCII content can also set
-    ``ANTHROPIC_API_KEY`` to route through the SDK and skip the subprocess.
-    """
-    api_key = os.environ.get("ANTHROPIC_API_KEY")
-    if api_key:
-        try:
-            import anthropic
-
-            client = anthropic.Anthropic(api_key=api_key)
-            msg = client.messages.create(
-                model=os.environ.get("CAVEMAN_MODEL", "claude-sonnet-4-5"),
-                max_tokens=8192,
-                messages=[{"role": "user", "content": prompt}],
-            )
-            return strip_llm_wrapper(msg.content[0].text.strip())
-        except ImportError:
-            pass  # anthropic not installed, fall back to CLI
-    # Fallback: use claude CLI (handles desktop auth).
-    # Resolve binary via shutil.which so Windows .cmd/.bat shims (e.g.
-    # %APPDATA%\npm\claude.CMD) work without shell=True. On POSIX,
-    # shutil.which returns the same absolute path as the implicit lookup,
-    # so this is a no-op there. Falls back to bare "claude" if not found
-    # on PATH so subprocess raises a clear FileNotFoundError.
-    claude_bin = shutil.which("claude") or "claude"
-    try:
-        result = subprocess.run(
-            [claude_bin, "--print"],
-            input=prompt,
-            text=True,
-            capture_output=True,
-            check=True,
-            encoding="utf-8",
-            errors="replace",
-        )
-        return strip_llm_wrapper(result.stdout.strip())
-    except subprocess.CalledProcessError as e:
-        raise RuntimeError(f"Claude call failed:\n{e.stderr}")
-
-
-def build_compress_prompt(original: str) -> str:
-    return f"""
-Compress this markdown into caveman format.
-
-STRICT RULES:
-- Do NOT modify anything inside ``` code blocks
-- Do NOT modify anything inside inline backticks
-- Preserve ALL URLs exactly
-- Preserve ALL headings exactly
-- Preserve file paths and commands
-- Return ONLY the compressed markdown body — do NOT wrap the entire output in a ```markdown fence or any other fence. Inner code blocks from the original stay as-is; do not add a new outer fence around the whole file.
-
-Only compress natural language.
-
-TEXT:
-{original}
-"""
-
-
-def build_fix_prompt(original: str, compressed: str, errors: list[str]) -> str:
-    errors_str = "\n".join(f"- {e}" for e in errors)
-    return f"""You are fixing a caveman-compressed markdown file. Specific validation errors were found.
-
-CRITICAL RULES:
-- DO NOT recompress or rephrase the file
-- ONLY fix the listed errors — leave everything else exactly as-is
-- The ORIGINAL is provided as reference only (to restore missing content)
-- Preserve caveman style in all untouched sections
-
-ERRORS TO FIX:
-{errors_str}
-
-HOW TO FIX:
-- Missing URL: find it in ORIGINAL, restore it exactly where it belongs in COMPRESSED
-- Code block mismatch: find the exact code block in ORIGINAL, restore it in COMPRESSED
-- Heading mismatch: restore the exact heading text from ORIGINAL into COMPRESSED
-- Do not touch any section not mentioned in the errors
-
-ORIGINAL (reference only):
-{original}
-
-COMPRESSED (fix this):
-{compressed}
-
-Return ONLY the fixed compressed file. No explanation.
-"""
-
-
-# ---------- Core Logic ----------
-
-
-def compress_file(filepath: Path) -> bool:
-    # Resolve and validate path
-    filepath = filepath.resolve()
-    MAX_FILE_SIZE = 500_000  # 500KB
-    if not filepath.exists():
-        raise FileNotFoundError(f"File not found: {filepath}")
-    if filepath.stat().st_size > MAX_FILE_SIZE:
-        raise ValueError(f"File too large to compress safely (max 500KB): {filepath}")
-
-    # Refuse files that look like they contain secrets or PII. Compressing ships
-    # the raw bytes to the Anthropic API — a third-party boundary — so we fail
-    # loudly rather than silently exfiltrate credentials or keys. Override is
-    # intentional: the user must rename the file if the heuristic is wrong.
-    if is_sensitive_path(filepath):
-        raise ValueError(
-            f"Refusing to compress {filepath}: filename looks sensitive "
-            "(credentials, keys, secrets, or known private paths). "
-            "Compression sends file contents to the Anthropic API. "
-            "Rename the file if this is a false positive."
-        )
-
-    print(f"Processing: {filepath}")
-
-    if not should_compress(filepath):
-        print("Skipping (not natural language)")
-        return False
-
-    original_text = filepath.read_text(errors="ignore")
-    # Store backup outside the source directory so skill auto-loaders don't
-    # re-ingest the `.original.md` copy as a live file. Mirror the source's
-    # parent-dir name + stem under a platform-aware base to reduce collisions.
-    backup_dir = backup_dir_for(filepath)
-    backup_dir.mkdir(parents=True, exist_ok=True)
-    backup_path = backup_dir / (filepath.stem + ".original.md")
-
-    if not original_text.strip():
-        print("❌ Refusing to compress: file is empty or whitespace-only.")
-        return False
-
-    # Check if backup already exists to prevent accidental overwriting
-    if backup_path.exists():
-        print(f"⚠️ Backup file already exists: {backup_path}")
-        print("The original backup may contain important content.")
-        print("Aborting to prevent data loss. Please remove or rename the backup file if you want to proceed.")
-        return False
-
-    # Split YAML frontmatter off before compression. Claude tends to strip or
-    # rewrite frontmatter despite preserve-structure rules; we keep it verbatim
-    # by removing it from the input and re-prepending it to the output.
-    frontmatter, body = split_frontmatter(original_text)
-    if frontmatter:
-        print(f"Detected YAML frontmatter ({len(frontmatter)} chars) — preserving verbatim")
-
-    if not body.strip():
-        print("❌ Refusing to compress: body is empty after frontmatter removal.")
-        return False
-
-    # Step 1: Compress (body only, frontmatter excluded)
-    print("Compressing with Claude...")
-    compressed_body = call_claude(build_compress_prompt(body))
-
-    if compressed_body is None or not compressed_body.strip():
-        print("❌ Compression aborted: Claude returned an empty response.")
-        print("   Original file is untouched (no backup created).")
-        return False
-
-    # Compare the BODY (not the whole file) — frontmatter is preserved verbatim
-    # and would never change, so identity must be judged on the compressible part.
-    if compressed_body.strip() == body.strip():
-        print("❌ Compression aborted: output is identical to input.")
-        print("   Likely causes: Claude refused, returned the prompt verbatim, or the file is")
-        print("   already in caveman form. Original file is untouched (no backup created).")
-        return False
-
-    # Reassemble: frontmatter (verbatim) + compressed body
-    compressed = frontmatter + compressed_body
-
-    # Save original as backup, then verify the backup readback before
-    # touching the input file. If the filesystem dropped bytes (encoding,
-    # antivirus, disk full), unlink the bad backup and abort instead of
-    # leaving the user with a corrupt backup + compressed primary.
-    backup_path.write_text(original_text)
-    backup_readback = backup_path.read_text(errors="ignore")
-    if backup_readback != original_text:
-        print(f"❌ Backup write verification failed: {backup_path}")
-        print("   In-memory original differs from on-disk backup. Aborting before touching the input file.")
-        try:
-            backup_path.unlink()
-        except OSError:
-            pass
-        return False
-    filepath.write_text(compressed)
-
-    # Step 2: Validate + Retry
-    for attempt in range(MAX_RETRIES):
-        print(f"\nValidation attempt {attempt + 1}")
-
-        result = validate(backup_path, filepath)
-
-        if result.is_valid:
-            print("Validation passed")
-            break
-
-        print("❌ Validation failed:")
-        for err in result.errors:
-            print(f"   - {err}")
-
-        if attempt == MAX_RETRIES - 1:
-            # Restore original on failure
-            filepath.write_text(original_text)
-            backup_path.unlink(missing_ok=True)
-            print("❌ Failed after retries — original restored")
-            return False
-
-        print("Fixing with Claude...")
-        compressed = call_claude(
-            build_fix_prompt(original_text, compressed, result.errors)
-        )
-        filepath.write_text(compressed)
-
-    return True

package/skills/caveman-compress/scripts/detect.py

@@ -1,169 +0,0 @@
-#!/usr/bin/env python3
-"""Detect whether a file is natural language (compressible) or code/config (skip)."""
-
-import json
-import re
-from pathlib import Path
-
-# Extensions that are natural language and compressible
-COMPRESSIBLE_EXTENSIONS = {".md", ".txt", ".markdown", ".rst", ".typ", ".typst", ".tex"}
-
-# Extensions that are code/config and should be skipped
-SKIP_EXTENSIONS = {
-    ".py", ".js", ".ts", ".tsx", ".jsx", ".json", ".yaml", ".yml",
-    ".toml", ".env", ".lock", ".css", ".scss", ".html", ".xml",
-    ".sql", ".sh", ".bash", ".zsh", ".go", ".rs", ".java", ".c",
-    ".cpp", ".h", ".hpp", ".rb", ".php", ".swift", ".kt", ".lua",
-    ".dockerfile", ".makefile", ".csv", ".ini", ".cfg",
-}
-
-# The subset of SKIP_EXTENSIONS that is configuration (not source code). Used to
-# decide whether a skipped file reports as "config" vs "code". Must stay a subset
-# of SKIP_EXTENSIONS (asserted below) so the two sets cannot silently drift.
-CONFIG_EXTENSIONS = {
-    ".json", ".yaml", ".yml", ".toml", ".ini", ".cfg", ".env",
-}
-assert CONFIG_EXTENSIONS <= SKIP_EXTENSIONS, (
-    "CONFIG_EXTENSIONS must be a subset of SKIP_EXTENSIONS: "
-    f"{CONFIG_EXTENSIONS - SKIP_EXTENSIONS} not in SKIP_EXTENSIONS"
-)
-
-# Real-world extensionless config/build filenames whose classification is NOT
-# already covered by the SKIP_EXTENSIONS fallback below. These have an empty
-# `Path.suffix`, so the SKIP_EXTENSIONS check never matches them and they would
-# otherwise be content-sniffed as natural language and offered up for
-# compression to a third-party API. Map the lowercased full filename to its
-# classification so a bare `Dockerfile`/`Makefile`/`.gitignore` is handled like
-# its dotted-extension cousins would be.
-#
-# Names that are themselves SKIP_EXTENSIONS entries (e.g. ".env") are
-# deliberately omitted here: the `name in SKIP_EXTENSIONS` fallback in
-# detect_file_type already classifies them identically, so listing them in both
-# places would be redundant and require hand-syncing.
-SKIP_FILENAMES = {
-    "dockerfile": "code",
-    "makefile": "code",
-    "gnumakefile": "code",
-    ".gitignore": "config",
-    ".gitattributes": "config",
-    ".dockerignore": "config",
-    ".editorconfig": "config",
-    ".npmrc": "config",
-    ".prettierrc": "config",
-    ".eslintrc": "config",
-    ".babelrc": "config",
-}
-
-# Patterns that indicate a line is code
-CODE_PATTERNS = [
-    re.compile(r"^\s*(import |from .+ import |require\(|const |let |var )"),
-    re.compile(r"^\s*(def |class |function |async function |export )"),
-    re.compile(r"^\s*(if\s*\(|for\s*\(|while\s*\(|switch\s*\(|try\s*\{)"),
-    re.compile(r"^\s*[\}\]\);]+\s*$"),  # closing braces/brackets
-    re.compile(r"^\s*@\w+"),  # decorators/annotations
-    re.compile(r'^\s*"[^"]+"\s*:\s*'),  # JSON-like key-value
-    re.compile(r"^\s*\w+\s*=\s*[{\[\(\"']"),  # assignment with literal
-]
-
-
-def _is_code_line(line: str) -> bool:
-    """Check if a line looks like code."""
-    return any(p.match(line) for p in CODE_PATTERNS)
-
-
-def _is_json_content(text: str) -> bool:
-    """Check if content is valid JSON."""
-    try:
-        json.loads(text)
-        return True
-    except (json.JSONDecodeError, ValueError):
-        return False
-
-
-def _is_yaml_content(lines: list[str]) -> bool:
-    """Heuristic: check if content looks like YAML."""
-    yaml_indicators = 0
-    for line in lines[:30]:
-        stripped = line.strip()
-        if stripped.startswith("---"):
-            yaml_indicators += 1
-        elif re.match(r"^\w[\w\s]*:\s", stripped):
-            yaml_indicators += 1
-        elif stripped.startswith("- ") and ":" in stripped:
-            yaml_indicators += 1
-    # If most non-empty lines look like YAML
-    non_empty = sum(1 for line in lines[:30] if line.strip())
-    return non_empty > 0 and yaml_indicators / non_empty > 0.6
-
-
-def detect_file_type(filepath: Path) -> str:
-    """Classify a file as 'natural_language', 'code', 'config', or 'unknown'.
-
-    Returns:
-        One of: 'natural_language', 'code', 'config', 'unknown'
-    """
-    ext = filepath.suffix.lower()
-
-    # Extension-based classification
-    if ext in COMPRESSIBLE_EXTENSIONS:
-        return "natural_language"
-    if ext in SKIP_EXTENSIONS:
-        return "config" if ext in CONFIG_EXTENSIONS else "code"
-
-    # Extensionless files (like CLAUDE.md, TODO) — check content.
-    if not ext:
-        # Leading-dot / build files (".env", "Dockerfile", "Makefile",
-        # ".gitignore") have an empty suffix, so the SKIP_EXTENSIONS check above
-        # never matched them and they would be content-sniffed as natural
-        # language. Classify them by full filename first so they are never
-        # offered up for compression to a third-party API.
-        name = filepath.name.lower()
-        if name in SKIP_FILENAMES:
-            return SKIP_FILENAMES[name]
-        if name in SKIP_EXTENSIONS:
-            return "config" if name in CONFIG_EXTENSIONS else "code"
-
-        try:
-            text = filepath.read_text(errors="ignore")
-        except (OSError, PermissionError):
-            return "unknown"
-
-        lines = text.splitlines()[:50]
-
-        if _is_json_content(text[:10000]):
-            return "config"
-        if _is_yaml_content(lines):
-            return "config"
-
-        code_lines = sum(1 for line in lines if line.strip() and _is_code_line(line))
-        non_empty = sum(1 for line in lines if line.strip())
-        if non_empty > 0 and code_lines / non_empty > 0.4:
-            return "code"
-
-        return "natural_language"
-
-    return "unknown"
-
-
-def should_compress(filepath: Path) -> bool:
-    """Return True if the file is natural language and should be compressed."""
-    if not filepath.is_file():
-        return False
-    # Skip backup files
-    if filepath.name.endswith(".original.md"):
-        return False
-    return detect_file_type(filepath) == "natural_language"
-
-
-if __name__ == "__main__":
-    import sys
-
-    if len(sys.argv) < 2:
-        print("Usage: python detect.py <file1> [file2] ...")
-        sys.exit(1)
-
-    for path_str in sys.argv[1:]:
-        p = Path(path_str).resolve()
-        file_type = detect_file_type(p)
-        compress = should_compress(p)
-        print(f"  {p.name:30s} type={file_type:20s} compress={compress}")

package/skills/caveman-compress/scripts/validate.py

@@ -1,213 +0,0 @@
-#!/usr/bin/env python3
-import re
-from collections import Counter
-from pathlib import Path
-
-URL_REGEX = re.compile(r"https?://[^\s)]+")
-FENCE_OPEN_REGEX = re.compile(r"^(\s{0,3})(`{3,}|~{3,})(.*)$")
-HEADING_REGEX = re.compile(r"^(#{1,6})\s+(.*)", re.MULTILINE)
-BULLET_REGEX = re.compile(r"^\s*[-*+]\s+", re.MULTILINE)
-
-# crude but effective path detection
-# Requires either a path prefix (./ ../ / or drive letter) or a slash/backslash within the match
-PATH_REGEX = re.compile(r"(?:\./|\.\./|/|[A-Za-z]:\\)[\w\-/\\\.]+|[\w\-\.]+[/\\][\w\-/\\\.]+")
-
-
-class ValidationResult:
-    def __init__(self):
-        self.is_valid = True
-        self.errors = []
-        self.warnings = []
-
-    def add_error(self, msg):
-        self.is_valid = False
-        self.errors.append(msg)
-
-    def add_warning(self, msg):
-        self.warnings.append(msg)
-
-
-def read_file(path: Path) -> str:
-    return path.read_text(errors="ignore")
-
-
-# ---------- Extractors ----------
-
-
-def extract_headings(text):
-    return [(level, title.strip()) for level, title in HEADING_REGEX.findall(text)]
-
-
-def extract_code_blocks(text):
-    """Line-based fenced code block extractor.
-
-    Handles ``` and ~~~ fences with variable length (CommonMark: closing
-    fence must use same char and be at least as long as opening). Supports
-    nested fences (e.g. an outer 4-backtick block wrapping inner 3-backtick
-    content).
-    """
-    blocks = []
-    lines = text.split("\n")
-    i = 0
-    n = len(lines)
-    while i < n:
-        m = FENCE_OPEN_REGEX.match(lines[i])
-        if not m:
-            i += 1
-            continue
-        fence_char = m.group(2)[0]
-        fence_len = len(m.group(2))
-        open_line = lines[i]
-        block_lines = [open_line]
-        i += 1
-        closed = False
-        while i < n:
-            close_m = FENCE_OPEN_REGEX.match(lines[i])
-            if (
-                close_m
-                and close_m.group(2)[0] == fence_char
-                and len(close_m.group(2)) >= fence_len
-                and close_m.group(3).strip() == ""
-            ):
-                block_lines.append(lines[i])
-                closed = True
-                i += 1
-                break
-            block_lines.append(lines[i])
-            i += 1
-        if closed:
-            blocks.append("\n".join(block_lines))
-        # Unclosed fences are silently skipped — they indicate malformed markdown
-        # and including them would cause false-positive validation failures.
-    return blocks
-
-
-def extract_urls(text):
-    return set(URL_REGEX.findall(text))
-
-
-def extract_paths(text):
-    return set(PATH_REGEX.findall(text))
-
-
-def count_bullets(text):
-    return len(BULLET_REGEX.findall(text))
-
-
-def extract_inline_codes(text):
-    text_without_fences = re.sub(r"^```[\s\S]*?^```", "", text, flags=re.MULTILINE)
-    text_without_fences = re.sub(r"^~~~[\s\S]*?^~~~", "", text_without_fences, flags=re.MULTILINE)
-    return re.findall(r"`([^`]+)`", text_without_fences)
-
-
-# ---------- Validators ----------
-
-
-def validate_headings(orig, comp, result):
-    h1 = extract_headings(orig)
-    h2 = extract_headings(comp)
-
-    if len(h1) != len(h2):
-        result.add_error(f"Heading count mismatch: {len(h1)} vs {len(h2)}")
-
-    if h1 != h2:
-        result.add_warning("Heading text/order changed")
-
-
-def validate_code_blocks(orig, comp, result):
-    c1 = extract_code_blocks(orig)
-    c2 = extract_code_blocks(comp)
-
-    if c1 != c2:
-        result.add_error("Code blocks not preserved exactly")
-
-
-def validate_urls(orig, comp, result):
-    u1 = extract_urls(orig)
-    u2 = extract_urls(comp)
-
-    if u1 != u2:
-        result.add_error(f"URL mismatch: lost={u1 - u2}, added={u2 - u1}")
-
-
-def validate_paths(orig, comp, result):
-    p1 = extract_paths(orig)
-    p2 = extract_paths(comp)
-
-    if p1 != p2:
-        result.add_warning(f"Path mismatch: lost={p1 - p2}, added={p2 - p1}")
-
-
-def validate_bullets(orig, comp, result):
-    b1 = count_bullets(orig)
-    b2 = count_bullets(comp)
-
-    if b1 == 0:
-        return
-
-    diff = abs(b1 - b2) / b1
-
-    if diff > 0.15:
-        result.add_warning(f"Bullet count changed too much: {b1} -> {b2}")
-
-
-def validate_inline_codes(orig, comp, result):
-    c1 = Counter(extract_inline_codes(orig))
-    c2 = Counter(extract_inline_codes(comp))
-
-    if c1 != c2:
-        lost = set(c1.keys()) - set(c2.keys())
-        added = set(c2.keys()) - set(c1.keys())
-        for code, count in c1.items():
-            if code in c2 and c2[code] < count:
-                lost.add(f"{code} (lost {count - c2[code]} of {count} occurrences)")
-        if lost:
-            result.add_error(f"Inline code lost: {lost}")
-        if added:
-            result.add_warning(f"Inline code added: {added}")
-
-
-# ---------- Main ----------
-
-
-def validate(original_path: Path, compressed_path: Path) -> ValidationResult:
-    result = ValidationResult()
-
-    orig = read_file(original_path)
-    comp = read_file(compressed_path)
-
-    validate_headings(orig, comp, result)
-    validate_code_blocks(orig, comp, result)
-    validate_urls(orig, comp, result)
-    validate_paths(orig, comp, result)
-    validate_bullets(orig, comp, result)
-    validate_inline_codes(orig, comp, result)
-
-    return result
-
-
-# ---------- CLI ----------
-
-if __name__ == "__main__":
-    import sys
-
-    if len(sys.argv) != 3:
-        print("Usage: python validate.py <original> <compressed>")
-        sys.exit(1)
-
-    orig = Path(sys.argv[1]).resolve()
-    comp = Path(sys.argv[2]).resolve()
-
-    res = validate(orig, comp)
-
-    print(f"\nValid: {res.is_valid}")
-
-    if res.errors:
-        print("\nErrors:")
-        for e in res.errors:
-            print(f"  - {e}")
-
-    if res.warnings:
-        print("\nWarnings:")
-        for w in res.warnings:
-            print(f"  - {w}")