diffctx 1.12.1 → 1.12.3

package/README.md

@@ -3,15 +3,16 @@
 [![CI](https://github.com/nikolay-e/diffctx/actions/workflows/ci.yml/badge.svg)](https://github.com/nikolay-e/diffctx/actions/workflows/ci.yml)
 [![PyPI](https://img.shields.io/pypi/v/diffctx)](https://pypi.org/project/diffctx/)
 [![crates.io](https://img.shields.io/crates/v/diffctx)](https://crates.io/crates/diffctx)
+[![npm](https://img.shields.io/npm/v/diffctx)](https://www.npmjs.com/package/diffctx)
 [![License](https://img.shields.io/pypi/l/diffctx)](https://pypi.org/project/diffctx/)
 
 **diffctx selects the minimum code an LLM needs to review a git diff.**
-Instead of pasting whole files, it walks the dependency graph from the changed
-lines outward and stops as soon as additional context stops paying for itself.
+Instead of pasting whole files, it walks the dependency graph outward from the
+changed lines and stops once more context stops paying for itself.
 
 > Coming from [`treemapper`](https://pypi.org/project/treemapper/)? That name is
-> deprecated — it was a thin wrapper around diffctx. Every command, flag, and
-> API call works unchanged: `treemapper` → `diffctx`, `treemapper-mcp` → `diffctx-mcp`.
+> deprecated. Every command, flag, and API call works unchanged:
+> `treemapper` → `diffctx`, `treemapper-mcp` → `diffctx-mcp`.
 
 ## Why not just use `tree` or repomix?
 
@@ -26,6 +27,9 @@ lines outward and stops as soon as additional context stops paying for itself.
 | Python API | ✗ | ✗ | ✗ | ✓ |
 | MCP server | ✗ | ✗ | ✗ | ✓ |
 
+Fuller positioning — measured results, and when a whole-repo packer or a
+persistent code-graph server fits better: [COMPARISON.md](COMPARISON.md).
+
 ## Install
 
 ```bash
@@ -35,21 +39,32 @@ pip install diffctx                     # or: into an active environment
 pipx install 'diffctx[mcp]'             # + MCP server for AI assistants
 ```
 
-The `[tree-sitter]` extra adds AST-level parsing for more accurate context
-selection across 30+ languages.
-
 Without Python:
 
 ```bash
 cargo install diffctx                   # native CLI from crates.io
+npx diffctx . --diff HEAD~1             # npm wrapper over the native binary
 docker run --rm -v "$PWD:/repo" ghcr.io/nikolay-e/diffctx . --diff HEAD~1
 ```
 
+On Windows, via Scoop (this repository is the bucket):
+
+```powershell
+scoop bucket add diffctx https://github.com/nikolay-e/diffctx
+scoop install diffctx/diffctx
+```
+
+The image runs as a non-root user (uid 10001) and writes to stdout — the
+native binary has no `-o` flag, so redirect to capture: `... --diff HEAD~1 >
+context.yaml`.
+
 Prebuilt binaries for linux (x86_64/aarch64), macOS (arm64) and Windows (x64)
 are attached to every [release](https://github.com/nikolay-e/diffctx/releases/latest).
 The native binary covers diff mode with YAML/JSON output; tree mode, Markdown
 output, the `graph` subcommand and the MCP server live in the Python package.
-`cargo add diffctx` embeds the selection pipeline in a Rust project
+`cargo add diffctx` embeds the pipeline in a Rust project — the library is
+imported as `_diffctx` (`use _diffctx::pipeline::build_diff_context`), since
+the crate doubles as the Python extension module
 ([docs.rs](https://docs.rs/diffctx)).
 
 ## Quick start
@@ -59,32 +74,34 @@ diffctx . --diff HEAD~1       # smart context for last commit → paste into Cla
 diffctx . -f md -c            # full codebase export → clipboard in Markdown
 ```
 
-![diffctx demo](https://raw.githubusercontent.com/nikolay-e/diffctx/main/docs/demo.gif)
+![diffctx demo](https://raw.githubusercontent.com/nikolay-e/diffctx/main/docs/demo/demo.gif)
 
-*`diffctx . --diff HEAD~1` selects only the fragments — functions, imports,
-type definitions — that an LLM actually needs to review the last commit,
-instead of dumping every changed file in full.*
+*`diffctx . --diff HEAD~1` selects only the fragments an LLM needs to review the
+last commit, instead of dumping every changed file in full.*
 
 ## Diff context mode
 
-Finds the minimal set of code fragments needed to understand a change —
-imports, callers, type definitions, config dependencies — across 50+ file
-types. It builds a code graph (imports, co-changes, type refs), propagates
-relevance from the changed lines outward, and stops when relevance drops below
-`--tau` or the `--budget` token cap is reached.
+Finds the minimal set of fragments needed to understand a change — imports,
+callers, type definitions, config dependencies — across 50+ file types. It
+builds a code graph (imports, co-changes, type refs), propagates relevance
+outward from the changed lines, and stops when relevance drops below `--tau` or
+the `--budget` token cap is hit.
 
 | Flag        | Default | Description                                                              |
 |-------------|---------|--------------------------------------------------------------------------|
 | `--scoring` | `ego`   | `ego` = bounded expansion around changed nodes (fast, predictable radius); `ppr` = Personalized PageRank (global, smoother decay, slower); `bm25` = lexical retrieval against the diff hunks (baseline for sparse graphs) |
-| `--budget`  | auto    | Hard token cap: `N` enforces a fixed cap, `-1` disables it, `0` is a strict-zero floor (empty selection; use `--full` for changed files only) |
+| `--budget`  | auto    | Hard cap in o200k_base tokens (see [Token counting](docs/product/token-budget.md)): `N` enforces a fixed cap, `-1` disables it, `0` is a strict-zero floor (empty selection; use `--full` for changed files only) |
 | `--alpha`   | 0.60    | PPR damping; higher = context clusters tighter around changes (`--scoring ppr` only) |
 | `--tau`     | 0.12    | Relevance threshold for full fragment content; lower-scoring fragments are stubbed or dropped (lower = more context) |
 | `--full`    | false   | Only the changed files, every fragment, no related-code context          |
 | `--timeout` | 300     | Wall-clock deadline in seconds; on expiry diffctx exits 124 instead of hanging |
+| `--with-raw-diff` | false | Also embed git's raw unified diff ahead of the selected fragments — additive (selection unchanged), not charged to `--budget`, lock/ignored/secret-like sections omitted. Python CLI only |
 
 Calibration of `--alpha`, `--tau`, and the edge-weight priors:
-[`docs/parameter-strategy.md`](docs/parameter-strategy.md). Theory:
-[Context-Selection for Git Diff (Zenodo, 2026)](https://doi.org/10.5281/zenodo.18824580).
+[`docs/engineering/parameter-strategy.md`](docs/engineering/parameter-strategy.md).
+Theory:
+[diffctx: Budgeted Typed-Graph Retrieval for Diff-Aware Code Context
+Selection (Zenodo, 2026)](https://doi.org/10.5281/zenodo.18824579).
 
 ### `graph` subcommand
 
@@ -102,28 +119,34 @@ diffctx graph . --level file -f graphml -o g.xml # file-level graph as GraphML
 <!-- BEGIN USAGE -->
 ```bash
 # full codebase export:
-diffctx .                                # Markdown to stdout + token count
-diffctx . -f md -c                       # Markdown → clipboard
-diffctx . -f json -o tree.json           # JSON → file
-diffctx . --no-content                   # structure only, no file contents
-diffctx . --max-depth 3                  # limit depth
-diffctx . -i custom.ignore               # custom ignore patterns
+diffctx .                                 # Markdown to stdout + token count
+diffctx . -f md -c                        # Markdown → clipboard
+diffctx . -f json -o tree.json            # JSON → file
+diffctx . --no-content                    # structure only, no file contents
+diffctx . --max-depth 3                   # limit depth
+diffctx . -i custom.ignore                # custom ignore patterns
 
 # diff context mode (requires git repo):
-diffctx . --diff                         # uncommitted changes (working tree vs HEAD)
-diffctx . --diff HEAD~1                  # context for last commit
-diffctx . --diff main..feature           # context for feature branch
-diffctx . --diff HEAD~1 --budget 30000   # limit to ~30k tokens
-diffctx . --diff HEAD~1 -c               # diff context to clipboard
+diffctx . --diff                          # uncommitted changes (working tree vs HEAD)
+diffctx . --diff HEAD~1                   # context for last commit
+diffctx . --diff main..feature            # context for feature branch
+diffctx . --diff HEAD~1 --budget 30000    # limit to ~30k tokens
+diffctx . --diff HEAD~1 -c                # diff context to clipboard
+diffctx . --diff HEAD~1 --with-raw-diff   # raw patch + selected context
 ```
 <!-- END USAGE -->
 
 Every run reports token count and size on stderr — `12,847 tokens
-(o200k_base), 52.3 KB` (tiktoken, the GPT-4o tokenizer; `~`-prefixed
-approximation above 1 MB). `-c/--copy` sends output to the clipboard via
-`pbcopy` (macOS), `clip` (Windows), or `wl-copy`/`xclip`/`xsel` (Linux).
-Unreadable files are replaced by placeholders such as `<binary file: N bytes>`,
-`<file too large: N bytes>`, or `<unreadable content: not utf-8>`.
+(o200k_base), 52.3 KB` (tiktoken, the GPT-4o tokenizer; `~`-prefixed above
+1 MB). `-c/--copy` copies output via `pbcopy` (macOS), `clip` (Windows), or
+`wl-copy`/`xclip`/`xsel` (Linux). Unreadable files become placeholders like
+`<binary file: N bytes>`, `<file too large: N bytes>`, or
+`<unreadable content: not utf-8>`.
+
+Counts come from tiktoken's `o200k_base` encoder and are exact only for the
+GPT-4o family; Claude, Gemini and others tokenize differently, so treat
+`--budget` as an upper bound in o200k tokens and leave headroom. Details:
+[`docs/product/token-budget.md`](docs/product/token-budget.md).
 
 ## Python API
 
@@ -140,6 +163,7 @@ ctx = build_diff_context(
     full=False,
     scoring_mode="ego",
     timeout=300,
+    with_raw_diff=False,      # True also embeds the raw unified diff (not charged to budget)
 )
 print(to_markdown(ctx))
 
@@ -159,32 +183,48 @@ print(to_yaml(tree))
 
 diffctx includes an [MCP](https://modelcontextprotocol.io) server that lets AI
 assistants (Claude Code, Cursor, Windsurf, etc.) call diff context analysis
-automatically during code review. Install with `pip install 'diffctx[mcp]'`
-and add to your MCP client config (e.g. `~/.claude/mcp.json` for Claude Code):
-
-```json
-{
-  "mcpServers": {
-    "diffctx": {
-      "command": "diffctx-mcp"
-    }
-  }
-}
+automatically during code review. Install with `pip install 'diffctx[mcp]'`,
+then register it — for Claude Code:
+
+```bash
+claude mcp add diffctx -- diffctx-mcp
 ```
 
-The server exposes a `get_diff_context` tool that assistants call when
-reviewing PRs, explaining changes, or investigating broken tests. Configs for
+The server exposes three tools — `get_diff_context`, `get_tree_map`, and
+`get_file_context` — that assistants call when reviewing PRs, explaining
+changes, or investigating broken tests. Tool reference and configs for
 Cursor, Continue, Windsurf, and Zed:
 [`src/diffctx/mcp/README.md`](src/diffctx/mcp/README.md).
 
 ## Ignore patterns
 
 Respects `.gitignore` and `.diffctx/ignore` automatically — hierarchically at
-every directory level, with gitignore semantics (negation `!important.log`,
-anchored `/root_only.txt`). `.diffctx/whitelist` acts as an include-only
-filter, and the output file is always auto-ignored. `--no-default-ignores`
-disables the built-in patterns; `--no-ignores` disables all ignore rules
-(tree mode only).
+every directory level, with full gitignore semantics (negation `!important.log`,
+anchored `/root_only.txt`). `.diffctx/whitelist` acts as an include-only filter,
+and the output file is always auto-ignored. `--no-default-ignores` disables the
+built-in patterns; `--no-ignores` disables all ignore rules (tree mode only).
+
+An excluded path never appears in the output in any role: in diff mode it is
+dropped both from `changed_files` and from the candidate universe, so it cannot
+come back as a related-context fragment either (including under `--full`). The
+same guarantee covers secret-like paths (`id_rsa`, `*.pem`, `*.key`, ...),
+which are filtered even without an ignore entry.
+
+## Token cache
+
+Diff mode caches per-blob tokenization under
+`~/Library/Caches/diffctx/token-cache` (macOS),
+`$XDG_CACHE_HOME/diffctx/token-cache` (Linux) or
+`%LOCALAPPDATA%\diffctx\token-cache` (Windows). It is a pure speedup: deleting
+it only costs one cold run.
+
+| Variable | Effect |
+|----------|--------|
+| `DIFFCTX_TOKEN_CACHE_DIR` | Relocate the cache |
+| `DIFFCTX_TOKEN_CACHE_MAX_BYTES` | Size cap, default `536870912` (512 MB); `0` disables eviction |
+
+Eviction is amortized: each run trims one of the cache's 256 shards back under
+its share of the cap, oldest entries first.
 
 ## Exit codes
 
@@ -194,17 +234,38 @@ disables the built-in patterns; `--no-ignores` disables all ignore rules
 | `1`  | Runtime error (bad path, permission denied, etc.) |
 | `2`  | Usage error (invalid flags/arguments) |
 | `3`  | Environment error (`--diff` outside a git repo, git not installed, no commits yet) |
-| `4`  | `--diff` produced no semantic context (clean tree, binary-only, everything filtered); output is still emitted. Deletion/rename-only diffs list `deleted_files`/`renamed_files` and exit `0` |
+| `4`  | `--diff` produced no semantic context (clean tree, binary-only, everything filtered); output is still emitted. Deletion/rename/lockfile-only diffs list `deleted_files`/`renamed_files`/`lockfile_changes` and exit `0` |
+| `124`| `--diff` exceeded the `--timeout` wall-clock deadline |
 | `130`| Interrupted (Ctrl-C) |
 | `141`| Broken pipe (e.g. piping into `head`) |
 
+## Repository layout
+
+Rust product code lives in `crates/diffctx-native/`; the Python CLI and MCP
+package live in `src/diffctx/`. Evaluation code, immutable datasets, and paper
+artifacts are intentionally separated under `eval/`, `datasets/`, and `paper/`.
+See [repository ownership boundaries](docs/architecture/repository-layout.md)
+for the lifecycle and entry point of each area.
+
 ## License
 
 Apache 2.0
 
+<!-- mcp-name: io.github.nikolay-e/diffctx -->
+<!-- Ownership marker read from the PyPI description by the MCP registry. -->
+<!-- Must survive edits verbatim: one space after the colon, case-sensitive. -->
+
 ---
 
+- [Documentation site](https://nikolay-e.github.io/diffctx/) — the pipeline
+  end to end: diff → fragments → graph → relevance → selection
+- [GitHub Action](docs/product/github-action.md) — diff context as a CI step
+  for LLM review
+- [Token counting](docs/product/token-budget.md) — which encoder, and what
+  `--budget` means for non-GPT models
+- [Comparison](COMPARISON.md) — measured results, and when a whole-repo packer
+  or a persistent code-graph server fits better
 - [Changelog](CHANGELOG.md)
 - [Security policy](SECURITY.md) — threat model and vulnerability reporting
-- [Parameter strategy](docs/parameter-strategy.md) — how `--alpha`,
+- [Parameter strategy](docs/engineering/parameter-strategy.md) — how `--alpha`,
   `--tau`, and edge weights are calibrated

package/bin/diffctx.js

@@ -1,9 +1,9 @@
 #!/usr/bin/env node
 'use strict';
 
-const fs = require('fs');
-const path = require('path');
-const { spawnSync } = require('child_process');
+const fs = require('node:fs');
+const path = require('node:path');
+const { spawnSync } = require('node:child_process');
 
 const binaryName = process.platform === 'win32' ? 'diffctx.exe' : 'diffctx';
 const binaryPath = path.join(__dirname, binaryName);

package/checksums.json

@@ -1,6 +1,6 @@
 {
-  "diffctx-1.12.1-aarch64-apple-darwin.tar.gz": "93eec66793bbf7f7d62f4288e990852da7d7a6ab79d3b6b29bea03ae2ac7e434",
-  "diffctx-1.12.1-aarch64-unknown-linux-gnu.tar.gz": "55610bfe7856f96a73e87451eddc1aa1a546b84ddc2778cf67738e2333f16f02",
-  "diffctx-1.12.1-x86_64-pc-windows-msvc.zip": "731177d8af8807e7bc1ba7f021a1723c48da3dfc75bb544415b55b268d5f03b4",
-  "diffctx-1.12.1-x86_64-unknown-linux-gnu.tar.gz": "f37da41a6381497c15df881f35b8caf324d8ac5fa331118d2736537a246a71b1"
+  "diffctx-1.12.3-aarch64-apple-darwin.tar.gz": "d62da2cd29e08130dc82936080d0b5734dc91bf392b9d7beb7696d7869bbd56c",
+  "diffctx-1.12.3-aarch64-unknown-linux-gnu.tar.gz": "a064e1edf242ddd448d8ed12445dacdcc32c5084ee82c62d54854e3b6b15abca",
+  "diffctx-1.12.3-x86_64-pc-windows-msvc.zip": "daf1064d4a4983305812443f79cb9a3d350c53468a513a127a4d5f079a19489c",
+  "diffctx-1.12.3-x86_64-unknown-linux-gnu.tar.gz": "83019a9eb2fa8a49eaa7999f5f6f9d88c72a88719c6ab456135ef0ed38f6f7fa"
 }

package/install.js

@@ -1,12 +1,12 @@
 #!/usr/bin/env node
 'use strict';
 
-const fs = require('fs');
-const os = require('os');
-const path = require('path');
-const crypto = require('crypto');
-const https = require('https');
-const { execFileSync } = require('child_process');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const crypto = require('node:crypto');
+const https = require('node:https');
+const { execFileSync } = require('node:child_process');
 
 const { version } = require('./package.json');
 const checksums = require('./checksums.json');
@@ -65,16 +65,33 @@ function verify(archivePath, expectedSha256) {
   }
 }
 
+function systemBinary(candidates, name) {
+  const found = candidates.find((candidate) => fs.existsSync(candidate));
+  if (!found) {
+    throw new Error(`cannot locate ${name}; expected one of ${candidates.join(', ')}`);
+  }
+  return found;
+}
+
 function extract(archivePath, archiveKind, destinationDir) {
   if (archiveKind === 'zip') {
+    const systemRoot = process.env.SystemRoot || String.raw`C:\Windows`;
+    const powershell = systemBinary(
+      [
+        path.join(systemRoot, 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe'),
+        path.join(systemRoot, 'SysWOW64', 'WindowsPowerShell', 'v1.0', 'powershell.exe'),
+      ],
+      'powershell.exe'
+    );
     execFileSync(
-      'powershell',
+      powershell,
       ['-NoProfile', '-Command', `Expand-Archive -LiteralPath '${archivePath}' -DestinationPath '${destinationDir}' -Force`],
       { stdio: 'inherit' }
     );
     return;
   }
-  execFileSync('tar', ['-xzf', archivePath, '-C', destinationDir], { stdio: 'inherit' });
+  const tar = systemBinary(['/usr/bin/tar', '/bin/tar'], 'tar');
+  execFileSync(tar, ['-xzf', archivePath, '-C', destinationDir], { stdio: 'inherit' });
 }
 
 async function main() {
@@ -96,10 +113,12 @@ async function main() {
   extract(archivePath, archive, workDir);
 
   const binaryName = process.platform === 'win32' ? 'diffctx.exe' : 'diffctx';
-  fs.copyFileSync(path.join(workDir, binaryName), path.join(binDir, binaryName));
-  if (process.platform !== 'win32') {
-    fs.chmodSync(path.join(binDir, binaryName), 0o755);
-  }
+  const extracted = path.join(workDir, binaryName);
+  const installed = path.join(binDir, binaryName);
+  fs.copyFileSync(extracted, installed);
+  // Carry over the archive's mode instead of a hardcoded literal, so the
+  // binary stays executable without widening permissions beyond the release.
+  fs.chmodSync(installed, fs.statSync(extracted).mode & 0o777);
   fs.rmSync(workDir, { recursive: true, force: true });
 }
 

package/package.json

@@ -1,13 +1,21 @@
 {
   "name": "diffctx",
-  "version": "1.12.1",
+  "mcpName": "io.github.nikolay-e/diffctx",
+  "version": "1.12.3",
   "description": "Selects the minimum code an LLM needs to review a git diff",
   "keywords": [
     "git",
     "diff",
     "llm",
     "code-review",
-    "context"
+    "context",
+    "mcp",
+    "code-context",
+    "context-engineering",
+    "ai-agents",
+    "diff-context",
+    "llm-context",
+    "codebase"
   ],
   "homepage": "https://github.com/nikolay-e/diffctx",
   "bugs": "https://github.com/nikolay-e/diffctx/issues",