compact-code 0.1.0__tar.gz

compact_code-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+__pycache__/
+*.egg-info/
+dist/
+.venv/
+.pytest_cache/
+.compact-code.json

compact_code-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Emmanuel Freund
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

compact_code-0.1.0/PKG-INFO

@@ -0,0 +1,76 @@
+Metadata-Version: 2.4
+Name: compact-code
+Version: 0.1.0
+Summary: Dense Python for AI agents, readable view for humans. Same logic, ~30-55% fewer tokens.
+Author-email: Emmanuel Freund <freund.emmanuel@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: agents,ast,claude,llm,minify,tokens
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Code Generators
+Requires-Python: >=3.9
+Provides-Extra: exact
+Requires-Dist: tiktoken>=0.5; extra == 'exact'
+Description-Content-Type: text/markdown
+
+# compact-code
+
+**Dense Python for AI agents, readable view for humans. Same logic, fewer tokens.**
+
+AI coding agents (Claude Code, Cursor, Codex...) read your code far more often than they write it. Every `Read`, every context refill re-pays the token weight of your code. But your code is written for humans: comments, docstrings, type annotations, blank lines, 4-space indentation — none of which the agent needs to understand the logic.
+
+`compact-code` removes all of it, **provably without changing the program** (the AST is preserved), and gives you back a readable view on demand. Think *minified JS + source maps, applied to the AI workflow*.
+
+```
+pip install compact-code
+cd your-project
+compact-code compact        # 3 seconds, AST-safe, reversible via git
+pytest                      # your tests are the safety net
+compact-code stats          # see what you now save on every read
+```
+
+## What it does
+
+- removes comments, docstrings and type annotations (keeps functional class-field annotations: dataclasses, NamedTuple, pydantic)
+- strips blank lines, re-indents to 1 space
+- never touches string contents, public names, parameters or logic
+- verifies AST equivalence on every file before writing — if the check fails, the file is skipped
+
+## Measured results
+
+Benchmarked with twin-codebase experiments (identical prompts, blind agents, real test suites as judges — full methodology in the repo):
+
+| Scenario | Session-token savings |
+|---|---|
+| Surgical one-file fix | ~0% (agent barely reads code) |
+| Maintenance on a fully compacted codebase | **-31%** |
+| Codebase exploration / audit (replicated n=3) | **-56%** |
+
+Quality: across 16 paired sessions, agents on compacted code passed **exactly the same test suites and evals** as agents on normal code. Zero measured loss.
+
+Codebase density gain: -19% to -41% tokens depending on the project (less if docstrings must be kept).
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `compact-code compact [paths]` | compact `**/*.py` in place (skips `.venv`, `.git`, etc.) |
+| `compact-code compact --check` | dry run: report savings without writing |
+| `compact-code compact --keep-docstrings` | keep docstrings (use when they are functional: CLI help text, doctests, flit) |
+| `compact-code expand <file>` | print a readable 4-space-indented view (`--write` to rewrite in place) |
+| `compact-code stats` | tokens and $ saved per full codebase read |
+
+Install `compact-code[exact]` for exact token counts (tiktoken).
+
+## The honest fine print
+
+- The gain is proportional to how much code your agent actually **reads**. Exploration-heavy and read-heavy sessions save the most; one-file quick fixes save nothing.
+- Some docstrings are functional (click help text, doctests, flit metadata). Always run your test suite after compacting; on failure, restore via git and re-run with `--keep-docstrings`.
+- Token counts use tiktoken `o200k_base` as a proxy for your model's tokenizer; ratios are indicative.
+
+## License
+
+MIT

compact_code-0.1.0/README.md

@@ -0,0 +1,58 @@
+# compact-code
+
+**Dense Python for AI agents, readable view for humans. Same logic, fewer tokens.**
+
+AI coding agents (Claude Code, Cursor, Codex...) read your code far more often than they write it. Every `Read`, every context refill re-pays the token weight of your code. But your code is written for humans: comments, docstrings, type annotations, blank lines, 4-space indentation — none of which the agent needs to understand the logic.
+
+`compact-code` removes all of it, **provably without changing the program** (the AST is preserved), and gives you back a readable view on demand. Think *minified JS + source maps, applied to the AI workflow*.
+
+```
+pip install compact-code
+cd your-project
+compact-code compact        # 3 seconds, AST-safe, reversible via git
+pytest                      # your tests are the safety net
+compact-code stats          # see what you now save on every read
+```
+
+## What it does
+
+- removes comments, docstrings and type annotations (keeps functional class-field annotations: dataclasses, NamedTuple, pydantic)
+- strips blank lines, re-indents to 1 space
+- never touches string contents, public names, parameters or logic
+- verifies AST equivalence on every file before writing — if the check fails, the file is skipped
+
+## Measured results
+
+Benchmarked with twin-codebase experiments (identical prompts, blind agents, real test suites as judges — full methodology in the repo):
+
+| Scenario | Session-token savings |
+|---|---|
+| Surgical one-file fix | ~0% (agent barely reads code) |
+| Maintenance on a fully compacted codebase | **-31%** |
+| Codebase exploration / audit (replicated n=3) | **-56%** |
+
+Quality: across 16 paired sessions, agents on compacted code passed **exactly the same test suites and evals** as agents on normal code. Zero measured loss.
+
+Codebase density gain: -19% to -41% tokens depending on the project (less if docstrings must be kept).
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `compact-code compact [paths]` | compact `**/*.py` in place (skips `.venv`, `.git`, etc.) |
+| `compact-code compact --check` | dry run: report savings without writing |
+| `compact-code compact --keep-docstrings` | keep docstrings (use when they are functional: CLI help text, doctests, flit) |
+| `compact-code expand <file>` | print a readable 4-space-indented view (`--write` to rewrite in place) |
+| `compact-code stats` | tokens and $ saved per full codebase read |
+
+Install `compact-code[exact]` for exact token counts (tiktoken).
+
+## The honest fine print
+
+- The gain is proportional to how much code your agent actually **reads**. Exploration-heavy and read-heavy sessions save the most; one-file quick fixes save nothing.
+- Some docstrings are functional (click help text, doctests, flit metadata). Always run your test suite after compacting; on failure, restore via git and re-run with `--keep-docstrings`.
+- Token counts use tiktoken `o200k_base` as a proxy for your model's tokenizer; ratios are indicative.
+
+## License
+
+MIT

compact_code-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "compact-code"
+version = "0.1.0"
+description = "Dense Python for AI agents, readable view for humans. Same logic, ~30-55% fewer tokens."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Emmanuel Freund", email = "freund.emmanuel@gmail.com" }]
+keywords = ["llm", "tokens", "claude", "agents", "ast", "minify"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Code Generators",
+]
+
+[project.optional-dependencies]
+exact = ["tiktoken>=0.5"]
+
+[project.scripts]
+compact-code = "compact_code.cli:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/compact_code"]

compact_code-0.1.0/src/compact_code/__init__.py

@@ -0,0 +1,7 @@
+"""compact-code: dense Python for AI agents, readable view for humans."""
+
+from .compactor import compact_source, verify_equivalence
+from .expand import expand_source
+
+__version__ = "0.1.0"
+__all__ = ["compact_source", "verify_equivalence", "expand_source", "__version__"]

compact_code-0.1.0/src/compact_code/cli.py

@@ -0,0 +1,153 @@
+"""compact-code CLI: compact / expand / stats."""
+
+import argparse
+import glob
+import json
+import os
+import sys
+
+from . import compactor, expand, tokens
+
+MANIFEST = ".compact-code.json"
+EXCLUDED_DIRS = {
+    ".git", ".venv", "venv", "__pycache__", "site-packages",
+    ".tox", ".mypy_cache", ".pytest_cache", "node_modules", ".eggs",
+}
+DEFAULT_PRICE_PER_MTOK = 3.0  # USD, input tokens
+
+
+def _iter_files(patterns):
+    if not patterns:
+        patterns = ["**/*.py"]
+    seen = set()
+    for pattern in patterns:
+        if os.path.isdir(pattern):
+            pattern = os.path.join(pattern, "**", "*.py")
+        for path in glob.glob(pattern, recursive=True):
+            path = os.path.normpath(path)
+            parts = set(path.replace("\\", "/").split("/"))
+            if parts & EXCLUDED_DIRS or path in seen or not path.endswith(".py"):
+                continue
+            seen.add(path)
+            yield path
+
+
+def _load_manifest():
+    if os.path.exists(MANIFEST):
+        with open(MANIFEST, encoding="utf-8") as f:
+            return json.load(f)
+    return {"files": {}}
+
+
+def _save_manifest(manifest):
+    with open(MANIFEST, "w", encoding="utf-8", newline="\n") as f:
+        json.dump(manifest, f, indent=1)
+
+
+def cmd_compact(args):
+    manifest = _load_manifest()
+    changed = skipped = before_total = after_total = 0
+    for path in _iter_files(args.paths):
+        source = open(path, encoding="utf-8").read()
+        keep_module_doc = os.path.basename(path) == "__init__.py"
+        try:
+            out = compactor.compact_source(
+                source,
+                keep_docstrings=args.keep_docstrings,
+                keep_module_docstring=keep_module_doc,
+            )
+        except SyntaxError as e:
+            print(f"SKIP {path}: {e}")
+            skipped += 1
+            continue
+        if not compactor.verify_equivalence(source, out):
+            print(f"SKIP {path}: equivalence check failed (please report this bug)")
+            skipped += 1
+            continue
+        before, after = tokens.count(source), tokens.count(out)
+        before_total += before
+        after_total += after
+        if out != source:
+            if not args.check:
+                with open(path, "w", encoding="utf-8", newline="\n") as f:
+                    f.write(out)
+                manifest["files"][path.replace("\\", "/")] = {
+                    "before": before, "after": after,
+                }
+            changed += 1
+    if not args.check and changed:
+        _save_manifest(manifest)
+    saved = before_total - after_total
+    pct = saved / before_total * 100 if before_total else 0
+    verb = "would be compacted" if args.check else "compacted"
+    approx = "" if tokens.is_exact() else " (approx., install tiktoken for exact counts)"
+    print(f"{changed} files {verb}, {skipped} skipped")
+    print(f"{before_total:,} -> {after_total:,} tokens ({pct:.1f}% smaller){approx}")
+    if not args.check and changed:
+        print("Now run your test suite. If anything fails, restore (git) and retry "
+              "with --keep-docstrings (some docstrings are functional: CLI help, doctests).")
+    return 0
+
+
+def cmd_expand(args):
+    source = open(args.file, encoding="utf-8").read()
+    out = expand.expand_source(source)
+    if args.write:
+        with open(args.file, "w", encoding="utf-8", newline="\n") as f:
+            f.write(out)
+        print(f"{args.file} expanded in place")
+    else:
+        sys.stdout.write(out)
+    return 0
+
+
+def cmd_stats(args):
+    manifest = _load_manifest()
+    files = manifest["files"]
+    if not files:
+        print("No compaction recorded here yet. Run `compact-code compact` first.")
+        return 1
+    before = sum(f["before"] for f in files.values())
+    after = sum(f["after"] for f in files.values())
+    saved = before - after
+    pct = saved / before * 100 if before else 0
+    dollars = saved / 1_000_000 * args.price
+    print(f"Codebase: {before:,} -> {after:,} tokens ({pct:.1f}% smaller, {len(files)} files)")
+    print(f"Every full read of this codebase by your agent now saves "
+          f"{saved:,} tokens (~${dollars:.2f} at ${args.price}/Mtok input).")
+    print("Sessions that read a lot of code (exploration, onboarding, refactors) "
+          "benefit the most; surgical one-file fixes benefit little.")
+    return 0
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(
+        prog="compact-code",
+        description="Dense Python for AI agents, readable view for humans. "
+        "Same logic, fewer tokens.",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p = sub.add_parser("compact", help="compact Python files in place (AST-safe)")
+    p.add_argument("paths", nargs="*", help="files, dirs or globs (default: **/*.py)")
+    p.add_argument("--keep-docstrings", action="store_true",
+                   help="keep docstrings (use when they are functional: CLI help, doctests)")
+    p.add_argument("--check", action="store_true", help="dry run, report savings only")
+    p.set_defaults(func=cmd_compact)
+
+    p = sub.add_parser("expand", help="print a readable view of a compacted file")
+    p.add_argument("file")
+    p.add_argument("--write", action="store_true", help="rewrite the file expanded")
+    p.set_defaults(func=cmd_expand)
+
+    p = sub.add_parser("stats", help="show tokens (and $) saved in this project")
+    p.add_argument("--price", type=float, default=DEFAULT_PRICE_PER_MTOK,
+                   help="USD per million input tokens (default: 3.0)")
+    p.set_defaults(func=cmd_stats)
+
+    args = parser.parse_args(argv)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

compact_code-0.1.0/src/compact_code/compactor.py

@@ -0,0 +1,118 @@
+"""Deterministic AST-based Python compactor.
+
+Removes comments, docstrings and type annotations (except functional
+class-field annotations), strips blank lines, and re-indents to 1 space.
+The program's AST is preserved: same logic, fewer tokens.
+"""
+
+import ast
+import io
+import tokenize
+
+
+class _Transformer(ast.NodeTransformer):
+    def __init__(self, keep_docstrings=False):
+        self.keep_docstrings = keep_docstrings
+
+    def _strip_doc(self, node):
+        body = node.body
+        if (
+            not self.keep_docstrings
+            and not getattr(node, "_keep_doc", False)
+            and body
+            and isinstance(body[0], ast.Expr)
+            and isinstance(body[0].value, ast.Constant)
+            and isinstance(body[0].value.value, str)
+        ):
+            body = body[1:] or [ast.Pass()]
+        node.body = [self.visit(stmt) for stmt in body]
+        return node
+
+    def visit_Module(self, node):
+        return self._strip_doc(node)
+
+    def visit_ClassDef(self, node):
+        return self._strip_doc(node)
+
+    def visit_FunctionDef(self, node):
+        return self._function(node)
+
+    def visit_AsyncFunctionDef(self, node):
+        return self._function(node)
+
+    def _function(self, node):
+        for arg in node.args.args + node.args.posonlyargs + node.args.kwonlyargs:
+            arg.annotation = None
+        if node.args.vararg:
+            node.args.vararg.annotation = None
+        if node.args.kwarg:
+            node.args.kwarg.annotation = None
+        node.returns = None
+        return self._strip_doc(node)
+
+    def visit_AnnAssign(self, node):
+        self.generic_visit(node)
+        # Bare class-field annotations are functional (dataclasses, NamedTuple,
+        # TypedDict, pydantic): keep them.
+        if node.value is None:
+            return node
+        if getattr(node, "_in_class", False):
+            return node
+        return ast.Assign(targets=[node.target], value=node.value)
+
+
+def _mark_class_fields(tree):
+    for node in ast.walk(tree):
+        if isinstance(node, ast.ClassDef):
+            for stmt in node.body:
+                if isinstance(stmt, ast.AnnAssign):
+                    stmt._in_class = True
+
+
+def _protected_lines(source):
+    """Line numbers interior to multi-line string tokens: their content must
+    not be re-indented or blank-stripped."""
+    protected = set()
+    for tok in tokenize.generate_tokens(io.StringIO(source).readline):
+        if tok.type == tokenize.STRING and tok.end[0] > tok.start[0]:
+            protected.update(range(tok.start[0] + 1, tok.end[0] + 1))
+    return protected
+
+
+def compact_source(source, keep_docstrings=False, keep_module_docstring=False):
+    """Return the compacted version of *source*.
+
+    Raises SyntaxError if *source* is not valid Python.
+    """
+    tree = ast.parse(source)
+    _mark_class_fields(tree)
+    if keep_module_docstring:
+        tree._keep_doc = True
+    tree = _Transformer(keep_docstrings).visit(tree)
+    ast.fix_missing_locations(tree)
+    out = ast.unparse(tree)
+    protected = _protected_lines(out)
+    lines = []
+    for lineno, line in enumerate(out.split("\n"), 1):
+        if lineno in protected:
+            lines.append(line)
+            continue
+        if not line.strip():
+            continue
+        indent = len(line) - len(line.lstrip())
+        lines.append(" " * (indent // 4) + line.lstrip())
+    return "\n".join(lines) + "\n"
+
+
+def _normalized_dump(source):
+    """AST dump with docstrings/annotations stripped — the equivalence
+    invariant shared by original and compacted code."""
+    tree = ast.parse(source)
+    _mark_class_fields(tree)
+    tree = _Transformer(keep_docstrings=False).visit(tree)
+    return ast.dump(tree)
+
+
+def verify_equivalence(original, compacted):
+    """True if *compacted* preserves the (normalized) AST of *original*."""
+    return _normalized_dump(original) == _normalized_dump(compacted)

compact_code-0.1.0/src/compact_code/expand.py

@@ -0,0 +1,38 @@
+"""Deterministic readable view of compacted code: 4-space indentation and
+blank lines between definitions. Instant, no LLM."""
+
+import io
+import tokenize
+
+
+def _protected_lines(source):
+    protected = set()
+    for tok in tokenize.generate_tokens(io.StringIO(source).readline):
+        if tok.type == tokenize.STRING and tok.end[0] > tok.start[0]:
+            protected.update(range(tok.start[0] + 1, tok.end[0] + 1))
+    return protected
+
+
+def expand_source(source):
+    """Return a human-readable view of *source* (does not modify logic)."""
+    protected = _protected_lines(source)
+    out = []
+    previous_indent = 0
+    for lineno, line in enumerate(source.split("\n"), 1):
+        if lineno in protected:
+            out.append(line)
+            continue
+        stripped = line.lstrip()
+        indent = len(line) - len(stripped)
+        if stripped.startswith(("def ", "async def ", "class ", "@")) and out:
+            # blank line before a definition, two before a top-level one
+            blanks = 2 if indent == 0 else 1
+            if not stripped.startswith("@") or previous_indent != indent:
+                out.extend([""] * blanks)
+        out.append("    " * indent + stripped)
+        if stripped:
+            previous_indent = indent
+    text = "\n".join(out)
+    while "\n\n\n\n" in text:
+        text = text.replace("\n\n\n\n", "\n\n\n")
+    return text if text.endswith("\n") else text + "\n"

compact_code-0.1.0/src/compact_code/tokens.py

@@ -0,0 +1,29 @@
+"""Token counting: tiktoken (o200k_base) if available, chars/4 heuristic otherwise."""
+
+_encoder = None
+_exact = None
+
+
+def _setup():
+    global _encoder, _exact
+    if _exact is not None:
+        return
+    try:
+        import tiktoken
+
+        _encoder = tiktoken.get_encoding("o200k_base")
+        _exact = True
+    except Exception:
+        _exact = False
+
+
+def count(text):
+    _setup()
+    if _exact:
+        return len(_encoder.encode(text))
+    return len(text) // 4
+
+
+def is_exact():
+    _setup()
+    return _exact

compact_code-0.1.0/tests/test_cli.py

@@ -0,0 +1,77 @@
+import json
+import os
+import subprocess
+import sys
+
+import pytest
+
+from compact_code import cli
+
+SRC = "def f(x: int) -> int:\n    \"\"\"Doubles.\"\"\"\n    # comment\n    return x * 2\n"
+
+
+@pytest.fixture
+def project(tmp_path, monkeypatch):
+    (tmp_path / "pkg").mkdir()
+    (tmp_path / "pkg" / "mod.py").write_text(SRC, encoding="utf-8")
+    (tmp_path / "pkg" / "__init__.py").write_text('"""Pkg."""\n', encoding="utf-8")
+    (tmp_path / ".venv" / "lib").mkdir(parents=True)
+    (tmp_path / ".venv" / "lib" / "junk.py").write_text(SRC, encoding="utf-8")
+    monkeypatch.chdir(tmp_path)
+    return tmp_path
+
+
+def test_compact_check_does_not_write(project):
+    assert cli.main(["compact", "--check"]) == 0
+    assert (project / "pkg" / "mod.py").read_text(encoding="utf-8") == SRC
+    assert not (project / cli.MANIFEST).exists()
+
+
+def test_compact_writes_and_records_manifest(project):
+    assert cli.main(["compact"]) == 0
+    out = (project / "pkg" / "mod.py").read_text(encoding="utf-8")
+    assert "comment" not in out and "Doubles" not in out
+    manifest = json.loads((project / cli.MANIFEST).read_text(encoding="utf-8"))
+    assert any(p.endswith("mod.py") for p in manifest["files"])
+
+
+def test_compact_skips_excluded_dirs(project):
+    cli.main(["compact"])
+    junk = (project / ".venv" / "lib" / "junk.py").read_text(encoding="utf-8")
+    assert junk == SRC
+
+
+def test_init_module_docstring_kept(project):
+    cli.main(["compact"])
+    init = (project / "pkg" / "__init__.py").read_text(encoding="utf-8")
+    assert "Pkg." in init
+
+
+def test_stats_after_compact(project, capsys):
+    cli.main(["compact"])
+    capsys.readouterr()
+    assert cli.main(["stats"]) == 0
+    out = capsys.readouterr().out
+    assert "saves" in out and "$" in out
+
+
+def test_stats_without_manifest(project, capsys):
+    assert cli.main(["stats"]) == 1
+
+
+def test_expand_stdout(project, capsys):
+    cli.main(["compact"])
+    capsys.readouterr()
+    assert cli.main(["expand", os.path.join("pkg", "mod.py")]) == 0
+    out = capsys.readouterr().out
+    assert "    return x * 2" in out
+
+
+def test_entry_point_runs():
+    result = subprocess.run(
+        [sys.executable, "-m", "compact_code.cli", "--help"],
+        capture_output=True, text=True,
+        env={**os.environ, "PYTHONPATH": os.path.join(os.path.dirname(__file__), "..", "src")},
+    )
+    assert result.returncode == 0
+    assert "compact" in result.stdout