@theaiteam/promptdiff 1.0.0-rc.1

package/CHANGELOG.md

@@ -0,0 +1,48 @@
+# Changelog
+
+## 1.0.0 — 2026-07-31
+
+First stable release. promptdiff is a Bun CLI for testing whether an LLM
+prompt or skill change actually changes behavior — with any model, not just
+Claude.
+
+### Commands
+
+- `run` — one bounded model invocation with an agent file and optional skills
+- `compare` — N-run baseline-vs-proposed comparison with deterministic
+  grading, pass-rate assertions, and honest statistics
+- `measure` — single-arm characterization: per-case pass rates, no
+  comparison semantics (#18)
+- `calibrate` — measure a judge rubric's accuracy against labeled fixtures
+  before it is trusted to grade anything (#21/#22)
+
+### Runners
+
+- `claude-p` (default): headless Claude Code — tools, artifact sandboxes,
+  skill-registry install testing, explicit budget-abort errors (#3)
+- `openai`: any OpenAI-compatible endpoint (OpenAI, ollama, vLLM, llama.cpp,
+  OpenRouter) — vision inputs, `requestParams` passthrough, transient-failure
+  retries with backoff (#17), and usage-based cost accounting so
+  `maxBudgetUsd` enforces (#20)
+- Per-arm model/runner/endpoint for model-vs-model comparisons, with
+  cross-runner caveats surfaced in the summary
+
+### Graders
+
+- `text` — contains / notContains / regex over the final output
+- `command` — shell exit codes in the per-run sandbox, with
+  `$PROMPTDIFF_OUTPUT_FILE` for completion-style runs
+- `json` — path assertions over the last balanced JSON value (#19)
+- `judge` — LLM judge with a rubric, labeled calibration fixtures, and a
+  refuse-to-grade gate on per-class accuracy (#21/#22)
+
+### Trustworthy receipts
+
+- Template rendering (`render.vars`, `--var`) so scenarios test the prompt
+  files that actually ship (#15)
+- Sampling-noise labels via exact Fisher tests (#13)
+- `productionModel` divergence warnings (#14)
+- Grader evidence for failing runs in the summary (#4)
+- `--report ndjson` append-only run history (#11)
+- `--receipts` content-addressed eval receipts keyed on prompt hashes (#10)
+- `--cache` opt-in baseline-arm caching keyed on content (#12)

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Josh Owens
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.