patchling 0.2.0 → 0.3.0

package/README.md

@@ -8,13 +8,14 @@
 
 **Natural-language code transformation, browser-first.** Hand patchling an in-memory `{ path: content }` file map plus a plain-English goal; **`generateDiff`** returns a unified git diff, and **`smartapply`** melds that diff into your source — even when `git apply` would reject it (drifted lines, fuzzy hunks, renames, new files, deletions). It's a bounded primitive you embed in your own app, not an open-ended coding agent. Wired to [NanoGPT](https://nano-gpt.com) for completions (including "Sign in with NanoGPT" OAuth PKCE).
 
-Everything runs in the browser (and in Node 18+): no filesystem, no build step, zero runtime dependencies. The diff engine is a faithful port of the Python [gptdiff](https://github.com/255BITS/gptdiff); it operates on an in-memory file map instead of a directory on disk.
+Everything runs in the browser (and in Node 18+): no filesystem, no build step, zero runtime dependencies. The diff engine is a faithful port of the [Python patchling](https://github.com/255BITS/patchling-py) (formerly gptdiff); it operates on an in-memory file map instead of a directory on disk.
 
-See it in action: **[live browser demos →](https://255bits.github.io/gptdiff-js-examples/)**.
+See it in action: **[patchling.app](https://patchling.app)** (live demo on the homepage) · **[live browser demos →](https://255bits.github.io/patchling-examples/)**
 
 > **The family** —
-> [**gptdiff**](https://github.com/255BITS/gptdiff) (Python library + CLI) ·
-> **patchling** (this package — the browser/Node runtime) ·
+> [**patchling.app**](https://patchling.app) (project home + live demo) ·
+> **patchling** (this package — the browser/Node runtime, [npm](https://www.npmjs.com/package/patchling)) ·
+> [**patchling for Python**](https://github.com/255BITS/patchling-py) (library + CLI, [PyPI](https://pypi.org/project/patchling/)) ·
 > [**nanoodle.com**](https://nanoodle.com) (visual AI workflow editor built on it)
 
 ```js
@@ -40,7 +41,7 @@ npm install patchling
 import { generateDiff, smartapply } from 'patchling';
 ```
 
-**Browser, no build step** — it's zero-dependency ESM, so a CDN import works directly (once the package is published to npm):
+**Browser, no build step** — it's zero-dependency ESM, so a CDN import works directly:
 
 ```js
 import { generateDiff, smartapply } from 'https://esm.sh/patchling';
@@ -63,6 +64,7 @@ Configuration comes from environment variables (Node) or `setEnv(...)` overrides
 | `GPTDIFF_LLM_API_KEY` | NanoGPT API key (`sk-nano-…`) | — (required for live calls) |
 | `GPTDIFF_LLM_BASE_URL` | OpenAI-compatible base URL | `https://nano-gpt.com/api/v1/` |
 | `GPTDIFF_MODEL` | Model id | `xiaomi/mimo-v2.5-pro-ultraspeed` |
+| `GPTDIFF_FILTER_MODEL` | Cheap model for `filterRelevant` | `xiaomi/mimo-v2.6-flash` |
 
 ```js
 import { setEnv } from 'patchling';
@@ -100,6 +102,49 @@ Builds the prompt, calls the LLM, and returns the unified diff extracted from th
 - `opts.callLlm` — inject a custom/mock completion client (used heavily in tests).
 - `opts.onUsage` — optional callback invoked with `{ promptTokens, completionTokens, totalTokens }` after the LLM call completes, so callers can surface cost/usage without changing the `Promise<string>` return type.
 
+### `filterRelevant(files, task, opts?) → Promise<Result>`
+Opt-in pass for a large file map. Each candidate file is sent with the task to a cheap model that answers `is this file relevant to the task? yes/no`. `result.files` is the `yes` map, in input order, with the original contents. Hand that to `generateDiff`. `smartapply` still takes the full map, so files the diff never mentions are left as they were.
+
+```js
+import { filterRelevant, buildEnvironment, generateDiff, smartapply } from 'patchling';
+
+const result = await filterRelevant(files, task, {
+  // model: 'xiaomi/mimo-v2.6-flash',       // GPTDIFF_FILTER_MODEL, else this default
+  concurrency: 8,                           // max in-flight calls
+  // cache: null,                           // omit for a fresh in-memory cache; a path string persists it
+  // maxBytes: 200000,                      // 0 or null disables
+  // maxTokens: 12000,                      // input-token cap; 0 or null disables
+  // onOversize: 'truncate',                // include | exclude | truncate
+  // onBinary: 'exclude',                   // include | exclude | truncate
+  onDecision(d) { console.log(d.path, d.decision); },
+});
+const diff = await generateDiff(buildEnvironment(result.files), task, { model });
+const next = await smartapply(diff, files);
+```
+
+- `opts.model`, `opts.apiKey`, `opts.baseUrl` — same key and base URL resolution as `generateDiff` (`GPTDIFF_LLM_API_KEY`, `GPTDIFF_LLM_BASE_URL`). The model falls back to `GPTDIFF_FILTER_MODEL`, then `xiaomi/mimo-v2.6-flash`.
+- `opts.concurrency` — at most this many classifications in flight (default 8).
+- `opts.cache` — a `RelevanceCache`, a `{ get, set }` object, or a filesystem path (Node writes `{ version: 1, entries }` atomically; a corrupt file is ignored). Omit it, or pass `true`, for a fresh in-memory cache used only by this call. `null` and `false` turn caching off. `createMemoryCache()` builds another memory cache you can share. The key is the SHA-256 of the length-prefixed classification inputs. Each field is written as its UTF-8 byte length, a colon, the bytes, and a newline, in this order: prompt version (`3`), model, output-token limit, the reasoning effort actually sent (`""` when that parameter was omitted or rejected), task, file path, mode (`full`, `truncated`, or `binary`), the preview note (or `""`), and the body that was sent. A later lookup tries the configured effort first, then the empty effort. A full file and a preview truncated to the same bytes do not share a key. Each path is its own decision, including paths with identical contents. A cache read failure is a miss. A cache write failure is logged and the run continues. Fallback and error decisions are not stored.
+- `opts.maxBytes` — UTF-8 size cap (default 200000). `opts.maxTokens` — input-token cap (default 12000; `ceil(code points / 4)`, so a surrogate pair is one code point). `0` or `null` disables either limit. Token truncation keeps the first `maxTokens * 4` code points. Byte truncation stays on a UTF-8 code-point boundary.
+- `opts.onOversize` and `opts.onBinary` — `include` (keep, no model call), `exclude` (drop, no model call), or `truncate` (ask about a preview). Defaults are `truncate` and `exclude`. A `yes` on a truncated preview still forwards the original text in `result.files`. An included binary file keeps the original bytes.
+- `opts.ignore` — lockfiles and generated paths are excluded with `skippedReason: "ignored"`. Matching is case-insensitive. `**` crosses directories and `*` does not. `false` classifies every path. An array replaces the default list. The canonical list is `DEFAULT_IGNORE_PATTERNS`: `**/package-lock.json`, `**/yarn.lock`, `**/pnpm-lock.yaml`, `**/bun.lock`, `**/bun.lockb`, `**/npm-shrinkwrap.json`, `**/Cargo.lock`, `**/Gemfile.lock`, `**/poetry.lock`, `**/Pipfile.lock`, `**/composer.lock`, `**/go.sum`, `**/uv.lock`, `**/pdm.lock`, `**/flake.lock`, `**/pubspec.lock`, `**/packages.lock.json`, `**/Package.resolved`, `**/*.min.js`, `**/*.min.mjs`, `**/*.min.css`, `**/*.map`, `**/dist/**`, `**/build/**`, `**/.next/**`, `**/coverage/**`, `**/__pycache__/**`, `**/*.pyc`, `**/*.pyo`, `**/*.generated.*`. The same list is in `test/fixtures/parity/relevance.json`.
+- `opts.maxOutputTokens` — completion cap (default 1024). The call also sends `reasoning_effort: "low"` so a reasoning model does not spend the whole budget before the yes/no. A 4xx that names that parameter is retried once without it. An empty answer is an error.
+- `opts.retries` — extra attempts after a timeout, an HTTP 5xx (including 503), or a billing-reservation error (default 2), with exponential backoff.
+- `opts.onDecision` — called as each file finishes, in completion order. `result.decisions` stays in input order.
+- `opts.callLlm` — inject a mock client.
+
+Each decision is `{ path, decision, reason, raw, tokens, promptTokens, completionTokens, cached, skippedReason, error }`.
+
+- `decision` is `"yes"` or `"no"`. After `<think>` / `<thinking>` blocks are stripped, the decision is a leading standalone `yes` or `no` (quotes, markdown bold, backticks, and an `Answer:` or `Decision:` prefix are wrappers, including when a reason follows, as in `**no** unrelated`). Any other wording, including a negated "no" later in the sentence, is malformed: the file is included, `failedOpen` is true, and the judgment is not cached. An empty answer, a non-string message content, and a thrown call do the same. `error` is set when the call throws or the answer is empty. `result.failedOpen` counts those files, and `formatFilterSummary` includes that count.
+- `skippedReason` is `"binary"`, `"oversize"`, `"truncated"`, `"ignored"`, or `null`.
+- `tokens`, `promptTokens`, and `completionTokens` are `0` when no model call was made. `reason` is a string. `failedOpen` is true only when the file was included without a definitive yes/no.
+- `result.billedTokens` sums token counts for decisions that were not cache hits. Identical contents at different paths are separate calls.
+- `buildEnvironment(files, { decisions })` keeps only `"yes"` paths. Passing the decisions array itself does the same. With no second argument, every file is included. `buildEnvironment(result.files)` is the same set.
+
+Python (`filter_relevant` in [patchling-py](https://github.com/255BITS/patchling-py)) returns the same result. Its fields are snake_case: `max_bytes`, `max_tokens`, `on_oversize`, `on_binary`, `skipped_reason`, `prompt_tokens`, `completion_tokens`, `failed_open`. Prompt version 3, the prompt text, the cache-key encoding, the ignore list, and the code-point token estimate are the shared contract in `test/fixtures/parity/relevance.json`. The whole byte payload is validated as UTF-8. A leading BOM (U+FEFF) is kept. Invalid bytes anywhere take the binary path (an incomplete sequence is tolerated only on the 8192-byte probe boundary). A response that is not a chat-completion envelope, including non-string message content, fails that file open and is not cached.
+
+A pallets/flask labeling pass has been described as recall 1.0 for `xiaomi/mimo-v2.6-flash`, with the filter using about 0.8× the tokens of a full-context read. Those figures are **UNVERIFIED**.
+
 ### `smartapply(diffText, files, opts?) → Promise<{ path: content }>`
 Applies a diff to an in-memory file map with per-file, LLM-assisted conflict resolution (runs files concurrently). Handles creation, modification, and deletion; `<think>…</think>` and reasoning preambles are stripped automatically. Returns a **new** map; deleted files are omitted.
 
@@ -111,14 +156,14 @@ Applies a diff to an in-memory file map with per-file, LLM-assisted conflict res
 Deterministic, no-LLM patch application (the strict "basic" applier). `changed` is `true` iff something was created, modified, or deleted; `files` is the updated map (equal to the input on failure — no partial writes).
 
 ### Other exports
-`parseDiffPerFile`, `buildEnvironment`, `colorCodeDiff`, `swallowReasoning`, `stripBadOutput`, `extractDiffBlocks`, `callLlm`, `resolveApiKey`, `resolveBaseUrl`, `getEnv`, `setEnv`, and the `oauth` namespace.
+`parseDiffPerFile`, `buildEnvironment`, `filterRelevant`, `RelevanceCache`, `createMemoryCache`, `parseRelevanceAnswer`, `isBinaryContent`, `isIgnoredPath`, `countTokens`, `formatFilterSummary`, `dumpDecisions`, `PROMPT_VERSION`, `colorCodeDiff`, `swallowReasoning`, `stripBadOutput`, `extractDiffBlocks`, `callLlm`, `resolveApiKey`, `resolveBaseUrl`, `getEnv`, `setEnv`, and the `oauth` namespace.
 
 ## Differences from the Python package
 
 - **No filesystem.** `applyDiff`/`smartapply` take and return a `{ path: content }` map rather than reading/writing a project directory.
-- **Scope.** Only `generateDiff` + `smartapply` (and their dependencies) are ported — not the `gptdiff`/`gptpatch`/`plangptdiff` CLIs.
+- **Scope.** Only `generateDiff` + `smartapply` (and their dependencies) are ported — not the Python package's CLIs (`patchling`, `patchling-apply`).
 - **Dependency injection** replaces Python's `monkeypatch`: pass `callLlm` / `callLlmForApply` to test without a network.
-- **`fetch` + Web Crypto** replace `openai`/`requests`/`tiktoken`.
+- **`fetch` + Web Crypto** replace `openai`/`requests`/`tiktoken`. Relevance token counts use the same characters/4 estimate Python uses when tiktoken is absent. A disk-backed `RelevanceCache` is Node-only; the browser keeps the in-memory cache.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "patchling",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Natural-language code transformation as a browser-first JS primitive: generateDiff turns a plain-English goal into a git diff, and smartapply melds it into your source even when git apply would reject it. Wired to NanoGPT. Formerly gptdiff-js.",
   "type": "module",
   "main": "src/index.js",
@@ -25,16 +25,18 @@
   "sideEffects": false,
   "repository": {
     "type": "git",
-    "url": "git+https://github.com/255BITS/gptdiff-js.git"
+    "url": "git+https://github.com/255BITS/patchling.git"
   },
-  "homepage": "https://github.com/255BITS/gptdiff-js#readme",
+  "homepage": "https://github.com/255BITS/patchling#readme",
   "bugs": {
-    "url": "https://github.com/255BITS/gptdiff-js/issues"
+    "url": "https://github.com/255BITS/patchling/issues"
   },
   "scripts": {
     "prepublishOnly": "npm test",
     "test": "node --test",
     "test:live": "GPTDIFF_LIVE=1 node --test test/live.test.js",
+    "test:live:llm": "LIVE_LLM=1 node --test test/live-smartapply.test.js",
+    "test:live:filter": "LIVE_LLM=1 node --test test/live-filter.test.js",
     "benchmark": "node benchmark/run.js",
     "types": "rm -f src/*.d.ts && tsc -p tsconfig.json"
   },

package/src/applydiff.d.ts

@@ -37,3 +37,72 @@ export function applyDiff(files: Record<string, string>, diffText: string): {
     changed: boolean;
     files: Record<string, string>;
 };
+/**
+ * Parse the hunks of a single-file patch.
+ *
+ * Returns null when a body line is ambiguous (`++`, which is either a doubled
+ * add marker or a literal line that starts with `+`). A non-standard hunk
+ * header with no line numbers is kept and anchored at the start of the file.
+ * {@link applyPatchToFile} refuses that header instead; this parser exists so
+ * a later intent check can still see the `+`/`-` lines.
+ *
+ * @param {string} patch
+ * @returns {Array<{ hint: number, ops: Array<{ kind: string, text: string }> }> | null}
+ */
+export function parsePatchHunks(patch: string): Array<{
+    hint: number;
+    ops: Array<{
+        kind: string;
+        text: string;
+    }>;
+}> | null;
+/**
+ * Whether `updatedContent` does something other than apply this diff's
+ * `+`/`-` lines to `originalContent`.
+ *
+ * Armed when a context line does not occur anywhere in the original file —
+ * the case where a model copies the diff's stale context in, or drops lines
+ * the hunk never removed. A context line that differs only by a uniform
+ * indentation shift, or a hunk that matches somewhere other than its `@@`
+ * line, still counts as context: the edit is accepted at the file's indent.
+ * A result that deletes any other line, or an empty rewrite of a non-empty
+ * file, is still a violation.
+ *
+ * A `-` line that merely fails to match (quotes, whitespace) does not by
+ * itself arm the check when every context line is present as written, so a
+ * fuzzy model rewrite of that line can still stand.
+ *
+ * @param {string} originalContent
+ * @param {string} updatedContent
+ * @param {string} patch
+ * @returns {boolean}
+ */
+export function diffOutputViolatesIntent(originalContent: string, updatedContent: string, patch: string): boolean;
+/**
+ * Apply only the `+`/`-` lines of a patch, ignoring context lines that are
+ * not actually in the file.
+ *
+ * A context or `-` line may match with a uniform indentation shift, and the
+ * hunk may sit somewhere other than the hunk header. Added lines are written
+ * at that same indent. When drifted context pairs a diff name with the
+ * file's own name (`user_id` in the diff, `uid` in the file), `+` lines are
+ * rewritten onto those names. Returns null when a `-` line cannot be found
+ * even then, when a pure insertion has no matching context to anchor it, or
+ * when a body line is ambiguous. Never writes a context line that was not
+ * already in the original.
+ *
+ * @param {string} originalContent
+ * @param {string} patch
+ * @returns {string | null}
+ */
+export function applyIntentIgnoringBadContext(originalContent: string, patch: string): string | null;
+/**
+ * Deterministic +/- apply that rewrote at least one added identifier onto
+ * a name the file already uses. Null when the hunk cannot be anchored or
+ * when no identifier changed, so the caller can still ask the model.
+ *
+ * @param {string} originalContent
+ * @param {string} patch
+ * @returns {string | null}
+ */
+export function applyAdaptedIdentifiers(originalContent: string, patch: string): string | null;