prompt-lock 0.2.0 → 0.4.0

package/CHANGELOG.md

@@ -5,6 +5,42 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2026-04-09
+
+### Added
+
+- **A/B testing mode** — compare two prompt variants side-by-side with `prompt-lock run --ab v1:v2`. Shows cost, latency, token, and pass-rate deltas; picks a winner based on pass rate → cost (5% threshold) → latency (10% threshold).
+- **`runAB()` programmatic API** — call `runAB(variantA, variantB, opts)` to get an `ABComparisonResult` with winner and deltas.
+- **A/B markdown reports** — `generateABMarkdownReport()` writes comparison tables to `.promptlock/reports/ab-<timestamp>.md`, ideal for PR comments.
+- **JSON Schema for configs** — `schemas/promptlock.schema.json` ships with the package. YAML/JSON users get IDE autocomplete and validation by adding:
+  ```yaml
+  # yaml-language-server: $schema=https://raw.githubusercontent.com/shmulikdav/Promptlock/main/schemas/promptlock.schema.json
+  ```
+
+### Changed
+
+- `files` field in package.json now includes `schemas/` directory so the schema ships with npm.
+
+## [0.3.0] - 2026-04-05
+
+### Added
+
+- **LLM-as-judge assertion** (`llm-judge`) — use a separate LLM to evaluate output quality with configurable criteria and threshold
+- **YAML/JSON config files** — auto-discovers `promptlock.yaml`/`.yml`/`.json` in project root; no more JS-only configs
+- **CSV/JSON dataset import** — `dataset: './data/test-inputs.csv'` loads external test data files
+- **Cost & token tracking** — captures token usage from OpenAI and Anthropic responses, estimates cost via built-in pricing table
+- **`max-cost` assertion** — fail if a prompt exceeds a dollar threshold (e.g. `{ type: 'max-cost', dollars: 0.05 }`)
+- **Markdown report format** — `--report markdown` generates GitHub-flavored markdown tables
+- **Watch mode** (`--watch`) — auto-reruns on config/prompt file changes with debounce
+- Cost summary in console, JSON, and HTML reports
+- `estimateCost()`, `getPricingTable()`, `loadDataset()`, `loadConfigFile()`, `discoverConfigFile()` exports
+
+### Changed
+
+- `LLMProvider` interface now supports optional `callWithMeta()` for token usage (backward compatible)
+- `dataset` field accepts file path strings (`.csv`, `.json`) in addition to inline arrays
+- Config loader scans for `.yaml`/`.yml` files in `prompts/` directory
+
 ## [0.2.0] - 2026-04-05
 
 ### Added

package/README.md

@@ -1,5 +1,9 @@
 # prompt-lock
 
+[![CI](https://github.com/shmulikdav/Promptlock/actions/workflows/ci.yml/badge.svg)](https://github.com/shmulikdav/Promptlock/actions/workflows/ci.yml)
+[![npm version](https://img.shields.io/npm/v/prompt-lock)](https://www.npmjs.com/package/prompt-lock)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
 **Version control and behavioral regression testing for LLM prompts.**
 
 prompt-lock wraps your prompts with behavioral assertions and snapshot baselines. On every change, it runs the assertion suite and flags regressions — like Jest for LLM behavior.
@@ -57,7 +61,33 @@ npx prompt-lock snapshot
 
 ## Defining Prompts
 
-Create `.js` files in your `prompts/` directory. You can export a single config or an array of configs:
+Create config files as `.js`, `.yaml`, `.yml`, or `.json`. prompt-lock auto-discovers `promptlock.yaml` in your project root, or scans the `prompts/` directory.
+
+### YAML Config (recommended)
+
+```yaml
+# promptlock.yaml
+id: article-summarizer
+version: '1.0.0'
+provider: anthropic
+model: claude-sonnet-4-20250514
+prompt: |
+  Summarize the following article in 3 bullet points.
+  Article: {{article}}
+defaultVars:
+  article: The quick brown fox jumped over the lazy dog.
+assertions:
+  - type: contains
+    value: '•'
+  - type: max-length
+    chars: 500
+  - type: max-cost
+    dollars: 0.05
+```
+
+### JavaScript Config
+
+Create `.js` files in your `prompts/` directory:
 
 ```javascript
 // prompts/summarizer.js
@@ -108,7 +138,7 @@ module.exports = {
 
 ### Testing with Datasets
 
-Test a prompt against multiple inputs:
+Test a prompt against multiple inputs — inline or from external CSV/JSON files:
 
 ```javascript
 module.exports = {
@@ -117,11 +147,18 @@ module.exports = {
   model: 'gpt-4o-mini',
   prompt: 'Classify this ticket: {{ticket}}',
   defaultVars: { ticket: 'My payment failed' },
+
+  // Inline dataset
   dataset: [
     { ticket: 'My payment failed' },
     { ticket: 'How do I reset my password?' },
     { ticket: 'Your product is great!' },
   ],
+
+  // Or load from a file:
+  // dataset: './data/test-tickets.csv',
+  // dataset: './data/test-tickets.json',
+
   assertions: [
     { type: 'json-valid' },
     { type: 'max-latency', ms: 5000 },
@@ -129,6 +166,38 @@ module.exports = {
 };
 ```
 
+CSV files use the first row as headers (= template variable names):
+
+```csv
+ticket,expected_category
+My payment failed,billing
+How do I reset my password?,account
+Your product is great!,feedback
+```
+
+### LLM-as-Judge
+
+Use a separate LLM to evaluate output quality:
+
+```javascript
+module.exports = {
+  id: 'creative-writer',
+  provider: 'anthropic',
+  model: 'claude-sonnet-4-20250514',
+  prompt: 'Write a haiku about {{topic}}',
+  defaultVars: { topic: 'coding' },
+  assertions: [
+    { type: 'max-length', chars: 200 },
+    {
+      type: 'llm-judge',
+      judge: { provider: 'openai', model: 'gpt-4o-mini' },
+      criteria: 'Is this a valid haiku with 5-7-5 syllable structure?',
+      threshold: 0.7,  // pass if score >= 0.7
+    },
+  ],
+};
+```
+
 ## Programmatic Usage
 
 ```typescript
@@ -167,12 +236,15 @@ prompt-lock run --id my-prompt     # Run a specific prompt
 prompt-lock run --ci               # Exit code 1 on failure
 prompt-lock run --report html      # Generate HTML report
 prompt-lock run --report json      # Generate JSON report
-prompt-lock run --report both      # Generate both reports
+prompt-lock run --report markdown  # Generate Markdown report
+prompt-lock run --report both      # Generate JSON + HTML reports
 prompt-lock run --dry-run          # Show what would be tested without calling LLMs
 prompt-lock run --verbose          # Show detailed output per prompt
 prompt-lock run --parallel         # Run prompts in parallel
 prompt-lock run --concurrency 10   # Max concurrent runs (default: 5)
 prompt-lock run --cache            # Cache LLM outputs (skip unchanged prompts)
+prompt-lock run --watch            # Watch for file changes and re-run
+prompt-lock run --ab v1:v2         # A/B compare two prompt IDs side-by-side
 prompt-lock run --github-pr owner/repo#123  # Post results as PR comment
 ```
 
@@ -226,6 +298,89 @@ prompt-lock run --cache
 prompt-lock cache clear
 ```
 
+### A/B Testing Mode
+
+Compare two prompt variants side-by-side and pick a winner:
+
+```bash
+prompt-lock run --ab summarizer-v1:summarizer-v2
+```
+
+Output:
+
+```
+A/B Comparison: summarizer-v1 vs summarizer-v2
+
+| Metric  | Variant A         | Variant B         | Delta     |
+| ------- | ----------------- | ----------------- | --------- |
+| Status  | ✅ 5/5 passed     | ✅ 5/5 passed     | —         |
+| Latency | 1250ms            | 980ms             | -270ms    |
+| Cost    | $0.004500         | $0.003200         | -$0.00130 |
+| Tokens  | 1234              | 987               | -247      |
+
+Winner: Variant B
+```
+
+**Winner logic:**
+1. Higher pass rate wins
+2. Otherwise: >5% cheaper wins
+3. Otherwise: >10% faster wins
+4. Otherwise: tie
+
+For signal over noise, use a `dataset` of size 5+ when A/B testing — a single LLM call isn't statistically meaningful.
+
+Programmatic API:
+
+```typescript
+import { runAB } from 'prompt-lock';
+
+const result = await runAB(variantA, variantB);
+console.log(result.winner);  // 'A' | 'B' | 'tie'
+console.log(result.deltas.costDollars);
+```
+
+### YAML Autocomplete (JSON Schema)
+
+Get IDE autocomplete, validation, and inline docs for YAML configs. Add this comment at the top of your `promptlock.yaml`:
+
+```yaml
+# yaml-language-server: $schema=https://raw.githubusercontent.com/shmulikdav/Promptlock/main/schemas/promptlock.schema.json
+id: my-prompt
+provider: anthropic
+model: claude-sonnet-4-20250514
+# VS Code now suggests all valid fields and assertion types
+```
+
+Install the [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) in VS Code to enable schema support.
+
+### Cost & Token Tracking
+
+prompt-lock automatically tracks token usage and estimates cost for OpenAI and Anthropic calls. Costs are shown in console output and included in all report formats.
+
+```bash
+prompt-lock run --verbose   # Shows per-prompt token counts and cost
+```
+
+Use the `max-cost` assertion to enforce budget limits:
+
+```yaml
+assertions:
+  - type: max-cost
+    dollars: 0.05
+```
+
+Built-in pricing for GPT-4o, GPT-4o-mini, Claude Sonnet 4, Claude Haiku, and more. Unknown models show zero cost.
+
+### Watch Mode
+
+Auto-rerun on file changes during prompt development:
+
+```bash
+prompt-lock run --watch
+```
+
+Watches your config files and `prompts/` directory. Debounces rapid changes (500ms).
+
 ### Retry Logic
 
 All LLM provider calls automatically retry on transient errors (rate limits, timeouts, network errors) with exponential backoff. Default: 3 retries. Use `--verbose` to see retry activity.
@@ -258,7 +413,9 @@ This posts a markdown table with pass/fail results and failure details. If a com
 | `no-hallucination-words` | `words?: string[]` | Output does NOT contain hallucination indicators |
 | `no-duplicates` | `separator?: string` | Output has no duplicate items (split by separator, default `\n`) |
 | `max-latency` | `ms: number` | LLM response time is under N milliseconds |
-| `custom` | `name: string, fn: (output) => boolean` | User-provided function returning boolean |
+| `max-cost` | `dollars: number` | LLM call cost is under N dollars |
+| `llm-judge` | `judge: {provider, model}, criteria: string, threshold?: number` | Another LLM scores output quality (0-1) |
+| `custom` | `name: string, fn: (output) => boolean` | User-provided function returning boolean (JS configs only) |
 
 ## Provider Setup
 

package/dist/assertions/builtin.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"builtin.d.ts","sourceRoot":"","sources":["../../src/assertions/builtin.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,eAAe,EAAE,MAAM,UAAU,CAAC;AAc3C,KAAK,gBAAgB,GAAG,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,eAAe,CAAC;AAE7F,eAAO,MAAM,iBAAiB,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAoK9D,CAAC"}
+{"version":3,"file":"builtin.d.ts","sourceRoot":"","sources":["../../src/assertions/builtin.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,eAAe,EAAE,MAAM,UAAU,CAAC;AAc3C,KAAK,gBAAgB,GAAG,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,eAAe,CAAC;AAE7F,eAAO,MAAM,iBAAiB,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAgL9D,CAAC"}

package/dist/assertions/builtin.js

@@ -157,6 +157,17 @@ exports.builtinAssertions = {
             actual: `${actual}ms`,
         };
     },
+    'max-cost': (output, config) => {
+        const dollars = config.dollars;
+        const actual = config.__cost ?? 0;
+        return {
+            type: 'max-cost',
+            name: `max-cost $${dollars}`,
+            passed: actual <= dollars,
+            expected: `<= $${dollars}`,
+            actual: `$${actual.toFixed(6)}`,
+        };
+    },
     'no-hallucination-words': (output, config) => {
         const words = config.words ?? DEFAULT_HALLUCINATION_WORDS;
         const found = words.filter(w => output.includes(w));

package/dist/assertions/builtin.js.map

@@ -1 +1 @@
-{"version":3,"file":"builtin.js","sourceRoot":"","sources":["../../src/assertions/builtin.ts"],"names":[],"mappings":";;;AAEA,MAAM,2BAA2B,GAAG;IAClC,UAAU;IACV,UAAU;IACV,UAAU;IACV,UAAU;IACV,sBAAsB;IACtB,kBAAkB;IAClB,kBAAkB;IAClB,qBAAqB;IACrB,qBAAqB;CACtB,CAAC;AAIW,QAAA,iBAAiB,GAAqC;IACjE,UAAU,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC7B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,UAAU;YAChB,IAAI,EAAE,aAAa,KAAK,GAAG;YAC3B,MAAM,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC;YAC9B,QAAQ,EAAE,sBAAsB,KAAK,GAAG;YACxC,MAAM,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,WAAW;SACvD,CAAC;IACJ,CAAC;IAED,cAAc,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QACjC,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,cAAc;YACpB,IAAI,EAAE,iBAAiB,KAAK,GAAG;YAC/B,MAAM,EAAE,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC;YAC/B,QAAQ,EAAE,0BAA0B,KAAK,GAAG;YAC5C,MAAM,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,WAAW;SACvD,CAAC;IACJ,CAAC;IAED,aAAa,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAChC,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,aAAa;YACnB,IAAI,EAAE,gBAAgB,KAAK,GAAG;YAC9B,MAAM,EAAE,MAAM,CAAC,UAAU,CAAC,KAAK,CAAC;YAChC,QAAQ,EAAE,yBAAyB,KAAK,GAAG;YAC3C,MAAM,EAAE,gBAAgB,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG;SACzD,CAAC;IACJ,CAAC;IAED,WAAW,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC9B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,WAAW;YACjB,IAAI,EAAE,cAAc,KAAK,GAAG;YAC5B,MAAM,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC;YAC9B,QAAQ,EAAE,uBAAuB,KAAK,GAAG;YACzC,MAAM,EAAE,cAAc,MAAM,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG;SACrD,CAAC;IACJ,CAAC;IAED,eAAe,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAClC,MAAM,OAAO,GAAG,MAAM,CAAC,OAAiB,CAAC;QACzC,IAAI,KAAa,CAAC;QAClB,IAAI,CAAC;YACH,KAAK,GAAG,IAAI,MAAM,CAAC,OAAO,CAAC,CAAC;QAC9B,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,OAAO;gBACL,IAAI,EAAE,eAAe;gBACrB,IAAI,EAAE,kBAAkB,OAAO,GAAG;gBAClC,MAAM,EAAE,KAAK;gBACb,QAAQ,EAAE,oBAAoB,OAAO,GAAG;gBACxC,MAAM,EAAE,kBAAmB,CAAW,CAAC,OAAO,EAAE;gBAChD,OAAO,EAAE,0BAA2B,CAAW,CAAC,OAAO,EAAE;aAC1D,CAAC;QACJ,CAAC;QACD,MAAM,OAAO,GAAG,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACnC,OAAO;YACL,IAAI,EAAE,eAAe;YACrB,IAAI,EAAE,kBAAkB,OAAO,GAAG;YAClC,MAAM,EAAE,OAAO;YACf,QAAQ,EAAE,oBAAoB,OAAO,GAAG;YACxC,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU;SACzC,CAAC;IACJ,CAAC;IAED,YAAY,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC/B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,YAAY;YAClB,IAAI,EAAE,cAAc,KAAK,EAAE;YAC3B,MAAM,EAAE,MAAM,CAAC,MAAM,IAAI,KAAK;YAC9B,QAAQ,EAAE,MAAM,KAAK,QAAQ;YAC7B,MAAM,EAAE,GAAG,MAAM,CAAC,MAAM,QAAQ;SACjC,CAAC;IACJ,CAAC;IAED,YAAY,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC/B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,YAAY;YAClB,IAAI,EAAE,cAAc,KAAK,EAAE;YAC3B,MAAM,EAAE,MAAM,CAAC,MAAM,IAAI,KAAK;YAC9B,QAAQ,EAAE,MAAM,KAAK,QAAQ;YAC7B,MAAM,EAAE,GAAG,MAAM,CAAC,MAAM,QAAQ;SACjC,CAAC;IACJ,CAAC;IAED,YAAY,EAAE,CAAC,MAAM,EAAE,EAAE;QACvB,IAAI,MAAM,GAAG,KAAK,CAAC;QACnB,IAAI,OAA2B,CAAC;QAChC,IAAI,CAAC;YACH,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;YACnB,MAAM,GAAG,IAAI,CAAC;QAChB,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,OAAO,GAAI,CAAW,CAAC,OAAO,CAAC;QACjC,CAAC;QACD,OAAO;YACL,IAAI,EAAE,YAAY;YAClB,IAAI,EAAE,YAAY;YAClB,MAAM;YACN,QAAQ,EAAE,YAAY;YACtB,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,iBAAiB,OAAO,EAAE;SAC3D,CAAC;IACJ,CAAC;IAED,cAAc,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QACjC,MAAM,MAAM,GAAG,MAAM,CAAC,MAAkB,CAAC;QACzC,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;QACxD,OAAO;YACL,IAAI,EAAE,cAAc;YACpB,IAAI,EAAE,iBAAiB,MAAM,CAAC,MAAM,SAAS;YAC7C,MAAM,EAAE,OAAO,CAAC,MAAM,KAAK,CAAC;YAC5B,QAAQ,EAAE,6BAA6B,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE;YAC1D,MAAM,EAAE,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,YAAY,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,WAAW;SAC5E,CAAC;IACJ,CAAC;IAED,eAAe,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAClC,MAAM,SAAS,GAAI,MAAM,CAAC,SAAgC,IAAI,IAAI,CAAC;QACnE,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACzE,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;QAC/B,MAAM,UAAU,GAAa,EAAE,CAAC;QAChC,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC;gBAAE,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YAC1C,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACjB,CAAC;QACD,OAAO;YACL,IAAI,EAAE,eAAe;YACrB,IAAI,EAAE,eAAe;YACrB,MAAM,EAAE,UAAU,CAAC,MAAM,KAAK,CAAC;YAC/B,QAAQ,EAAE,oBAAoB;YAC9B,MAAM,EAAE,UAAU,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,eAAe,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,eAAe;SACzF,CAAC;IACJ,CAAC;IAED,aAAa,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAChC,uEAAuE;QACvE,MAAM,EAAE,GAAG,MAAM,CAAC,EAAY,CAAC;QAC/B,MAAM,MAAM,GAAI,MAAM,CAAC,UAAqB,IAAI,CAAC,CAAC;QAClD,OAAO;YACL,IAAI,EAAE,aAAa;YACnB,IAAI,EAAE,eAAe,EAAE,IAAI;YAC3B,MAAM,EAAE,MAAM,IAAI,EAAE;YACpB,QAAQ,EAAE,MAAM,EAAE,IAAI;YACtB,MAAM,EAAE,GAAG,MAAM,IAAI;SACtB,CAAC;IACJ,CAAC;IAED,wBAAwB,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC3C,MAAM,KAAK,GAAI,MAAM,CAAC,KAA8B,IAAI,2BAA2B,CAAC;QACpF,MAAM,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;QACpD,OAAO;YACL,IAAI,EAAE,wBAAwB;YAC9B,IAAI,EAAE,wBAAwB;YAC9B,MAAM,EAAE,KAAK,CAAC,MAAM,KAAK,CAAC;YAC1B,QAAQ,EAAE,wBAAwB;YAClC,MAAM,EAAE,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,UAAU,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,YAAY;SACvE,CAAC;IACJ,CAAC;CACF,CAAC"}
+{"version":3,"file":"builtin.js","sourceRoot":"","sources":["../../src/assertions/builtin.ts"],"names":[],"mappings":";;;AAEA,MAAM,2BAA2B,GAAG;IAClC,UAAU;IACV,UAAU;IACV,UAAU;IACV,UAAU;IACV,sBAAsB;IACtB,kBAAkB;IAClB,kBAAkB;IAClB,qBAAqB;IACrB,qBAAqB;CACtB,CAAC;AAIW,QAAA,iBAAiB,GAAqC;IACjE,UAAU,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC7B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,UAAU;YAChB,IAAI,EAAE,aAAa,KAAK,GAAG;YAC3B,MAAM,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC;YAC9B,QAAQ,EAAE,sBAAsB,KAAK,GAAG;YACxC,MAAM,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,WAAW;SACvD,CAAC;IACJ,CAAC;IAED,cAAc,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QACjC,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,cAAc;YACpB,IAAI,EAAE,iBAAiB,KAAK,GAAG;YAC/B,MAAM,EAAE,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC;YAC/B,QAAQ,EAAE,0BAA0B,KAAK,GAAG;YAC5C,MAAM,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,WAAW;SACvD,CAAC;IACJ,CAAC;IAED,aAAa,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAChC,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,aAAa;YACnB,IAAI,EAAE,gBAAgB,KAAK,GAAG;YAC9B,MAAM,EAAE,MAAM,CAAC,UAAU,CAAC,KAAK,CAAC;YAChC,QAAQ,EAAE,yBAAyB,KAAK,GAAG;YAC3C,MAAM,EAAE,gBAAgB,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,MAAM,CAAC,GAAG;SACzD,CAAC;IACJ,CAAC;IAED,WAAW,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC9B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,WAAW;YACjB,IAAI,EAAE,cAAc,KAAK,GAAG;YAC5B,MAAM,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC;YAC9B,QAAQ,EAAE,uBAAuB,KAAK,GAAG;YACzC,MAAM,EAAE,cAAc,MAAM,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG;SACrD,CAAC;IACJ,CAAC;IAED,eAAe,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAClC,MAAM,OAAO,GAAG,MAAM,CAAC,OAAiB,CAAC;QACzC,IAAI,KAAa,CAAC;QAClB,IAAI,CAAC;YACH,KAAK,GAAG,IAAI,MAAM,CAAC,OAAO,CAAC,CAAC;QAC9B,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,OAAO;gBACL,IAAI,EAAE,eAAe;gBACrB,IAAI,EAAE,kBAAkB,OAAO,GAAG;gBAClC,MAAM,EAAE,KAAK;gBACb,QAAQ,EAAE,oBAAoB,OAAO,GAAG;gBACxC,MAAM,EAAE,kBAAmB,CAAW,CAAC,OAAO,EAAE;gBAChD,OAAO,EAAE,0BAA2B,CAAW,CAAC,OAAO,EAAE;aAC1D,CAAC;QACJ,CAAC;QACD,MAAM,OAAO,GAAG,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACnC,OAAO;YACL,IAAI,EAAE,eAAe;YACrB,IAAI,EAAE,kBAAkB,OAAO,GAAG;YAClC,MAAM,EAAE,OAAO;YACf,QAAQ,EAAE,oBAAoB,OAAO,GAAG;YACxC,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,UAAU;SACzC,CAAC;IACJ,CAAC;IAED,YAAY,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC/B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,YAAY;YAClB,IAAI,EAAE,cAAc,KAAK,EAAE;YAC3B,MAAM,EAAE,MAAM,CAAC,MAAM,IAAI,KAAK;YAC9B,QAAQ,EAAE,MAAM,KAAK,QAAQ;YAC7B,MAAM,EAAE,GAAG,MAAM,CAAC,MAAM,QAAQ;SACjC,CAAC;IACJ,CAAC;IAED,YAAY,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC/B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAe,CAAC;QACrC,OAAO;YACL,IAAI,EAAE,YAAY;YAClB,IAAI,EAAE,cAAc,KAAK,EAAE;YAC3B,MAAM,EAAE,MAAM,CAAC,MAAM,IAAI,KAAK;YAC9B,QAAQ,EAAE,MAAM,KAAK,QAAQ;YAC7B,MAAM,EAAE,GAAG,MAAM,CAAC,MAAM,QAAQ;SACjC,CAAC;IACJ,CAAC;IAED,YAAY,EAAE,CAAC,MAAM,EAAE,EAAE;QACvB,IAAI,MAAM,GAAG,KAAK,CAAC;QACnB,IAAI,OAA2B,CAAC;QAChC,IAAI,CAAC;YACH,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;YACnB,MAAM,GAAG,IAAI,CAAC;QAChB,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,OAAO,GAAI,CAAW,CAAC,OAAO,CAAC;QACjC,CAAC;QACD,OAAO;YACL,IAAI,EAAE,YAAY;YAClB,IAAI,EAAE,YAAY;YAClB,MAAM;YACN,QAAQ,EAAE,YAAY;YACtB,MAAM,EAAE,MAAM,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,iBAAiB,OAAO,EAAE;SAC3D,CAAC;IACJ,CAAC;IAED,cAAc,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QACjC,MAAM,MAAM,GAAG,MAAM,CAAC,MAAkB,CAAC;QACzC,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;QACxD,OAAO;YACL,IAAI,EAAE,cAAc;YACpB,IAAI,EAAE,iBAAiB,MAAM,CAAC,MAAM,SAAS;YAC7C,MAAM,EAAE,OAAO,CAAC,MAAM,KAAK,CAAC;YAC5B,QAAQ,EAAE,6BAA6B,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE;YAC1D,MAAM,EAAE,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,YAAY,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,WAAW;SAC5E,CAAC;IACJ,CAAC;IAED,eAAe,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAClC,MAAM,SAAS,GAAI,MAAM,CAAC,SAAgC,IAAI,IAAI,CAAC;QACnE,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACzE,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;QAC/B,MAAM,UAAU,GAAa,EAAE,CAAC;QAChC,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC;gBAAE,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YAC1C,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACjB,CAAC;QACD,OAAO;YACL,IAAI,EAAE,eAAe;YACrB,IAAI,EAAE,eAAe;YACrB,MAAM,EAAE,UAAU,CAAC,MAAM,KAAK,CAAC;YAC/B,QAAQ,EAAE,oBAAoB;YAC9B,MAAM,EAAE,UAAU,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,eAAe,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,eAAe;SACzF,CAAC;IACJ,CAAC;IAED,aAAa,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAChC,uEAAuE;QACvE,MAAM,EAAE,GAAG,MAAM,CAAC,EAAY,CAAC;QAC/B,MAAM,MAAM,GAAI,MAAM,CAAC,UAAqB,IAAI,CAAC,CAAC;QAClD,OAAO;YACL,IAAI,EAAE,aAAa;YACnB,IAAI,EAAE,eAAe,EAAE,IAAI;YAC3B,MAAM,EAAE,MAAM,IAAI,EAAE;YACpB,QAAQ,EAAE,MAAM,EAAE,IAAI;YACtB,MAAM,EAAE,GAAG,MAAM,IAAI;SACtB,CAAC;IACJ,CAAC;IAED,UAAU,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC7B,MAAM,OAAO,GAAG,MAAM,CAAC,OAAiB,CAAC;QACzC,MAAM,MAAM,GAAI,MAAM,CAAC,MAAiB,IAAI,CAAC,CAAC;QAC9C,OAAO;YACL,IAAI,EAAE,UAAU;YAChB,IAAI,EAAE,aAAa,OAAO,EAAE;YAC5B,MAAM,EAAE,MAAM,IAAI,OAAO;YACzB,QAAQ,EAAE,OAAO,OAAO,EAAE;YAC1B,MAAM,EAAE,IAAI,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE;SAChC,CAAC;IACJ,CAAC;IAED,wBAAwB,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;QAC3C,MAAM,KAAK,GAAI,MAAM,CAAC,KAA8B,IAAI,2BAA2B,CAAC;QACpF,MAAM,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;QACpD,OAAO;YACL,IAAI,EAAE,wBAAwB;YAC9B,IAAI,EAAE,wBAAwB;YAC9B,MAAM,EAAE,KAAK,CAAC,MAAM,KAAK,CAAC;YAC1B,QAAQ,EAAE,wBAAwB;YAClC,MAAM,EAAE,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,UAAU,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,YAAY;SACvE,CAAC;IACJ,CAAC;CACF,CAAC"}

package/dist/assertions/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,eAAe,EAAE,eAAe,EAAE,MAAM,UAAU,CAAC;AAK5D,wBAAsB,aAAa,CACjC,MAAM,EAAE,MAAM,EACd,UAAU,EAAE,eAAe,EAAE,GAC5B,OAAO,CAAC,eAAe,EAAE,CAAC,CA6B5B"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,eAAe,EAAE,eAAe,EAAE,MAAM,UAAU,CAAC;AAM5D,wBAAsB,aAAa,CACjC,MAAM,EAAE,MAAM,EACd,UAAU,EAAE,eAAe,EAAE,GAC5B,OAAO,CAAC,eAAe,EAAE,CAAC,CAuC5B"}

package/dist/assertions/index.js

@@ -4,6 +4,7 @@ exports.runAssertions = runAssertions;
 const builtin_1 = require("./builtin");
 const json_schema_1 = require("./json-schema");
 const custom_1 = require("./custom");
+const llm_judge_1 = require("./llm-judge");
 async function runAssertions(output, assertions) {
     const results = [];
     for (const assertion of assertions) {
@@ -11,6 +12,10 @@ async function runAssertions(output, assertions) {
             results.push((0, json_schema_1.assertJsonSchema)(output, assertion.schema));
             continue;
         }
+        if (assertion.type === 'llm-judge') {
+            results.push(await (0, llm_judge_1.assertLlmJudge)(output, assertion.judge, assertion.criteria, assertion.threshold ?? 0.7));
+            continue;
+        }
         if (assertion.type === 'custom') {
             results.push(await (0, custom_1.assertCustom)(output, assertion.name, assertion.fn));
             continue;

package/dist/assertions/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":";;AAKA,sCAgCC;AApCD,uCAA8C;AAC9C,+CAAiD;AACjD,qCAAwC;AAEjC,KAAK,UAAU,aAAa,CACjC,MAAc,EACd,UAA6B;IAE7B,MAAM,OAAO,GAAsB,EAAE,CAAC;IAEtC,KAAK,MAAM,SAAS,IAAI,UAAU,EAAE,CAAC;QACnC,IAAI,SAAS,CAAC,IAAI,KAAK,aAAa,EAAE,CAAC;YACrC,OAAO,CAAC,IAAI,CAAC,IAAA,8BAAgB,EAAC,MAAM,EAAE,SAAS,CAAC,MAAM,CAAC,CAAC,CAAC;YACzD,SAAS;QACX,CAAC;QAED,IAAI,SAAS,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;YAChC,OAAO,CAAC,IAAI,CAAC,MAAM,IAAA,qBAAY,EAAC,MAAM,EAAE,SAAS,CAAC,IAAI,EAAE,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC;YACvE,SAAS;QACX,CAAC;QAED,MAAM,OAAO,GAAG,2BAAiB,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAClD,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,OAAO,CAAC,IAAI,CAAC;gBACX,IAAI,EAAE,SAAS,CAAC,IAAI;gBACpB,IAAI,EAAE,SAAS,CAAC,IAAI;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,4BAA4B,SAAS,CAAC,IAAI,GAAG;aACvD,CAAC,CAAC;YACH,SAAS;QACX,CAAC;QAED,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,SAA+C,CAAC,CAAC,CAAC;IACjF,CAAC;IAED,OAAO,OAAO,CAAC;AACjB,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":";;AAMA,sCA0CC;AA/CD,uCAA8C;AAC9C,+CAAiD;AACjD,qCAAwC;AACxC,2CAA6C;AAEtC,KAAK,UAAU,aAAa,CACjC,MAAc,EACd,UAA6B;IAE7B,MAAM,OAAO,GAAsB,EAAE,CAAC;IAEtC,KAAK,MAAM,SAAS,IAAI,UAAU,EAAE,CAAC;QACnC,IAAI,SAAS,CAAC,IAAI,KAAK,aAAa,EAAE,CAAC;YACrC,OAAO,CAAC,IAAI,CAAC,IAAA,8BAAgB,EAAC,MAAM,EAAE,SAAS,CAAC,MAAM,CAAC,CAAC,CAAC;YACzD,SAAS;QACX,CAAC;QAED,IAAI,SAAS,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;YACnC,OAAO,CAAC,IAAI,CAAC,MAAM,IAAA,0BAAc,EAC/B,MAAM,EACN,SAAS,CAAC,KAAK,EACf,SAAS,CAAC,QAAQ,EAClB,SAAS,CAAC,SAAS,IAAI,GAAG,CAC3B,CAAC,CAAC;YACH,SAAS;QACX,CAAC;QAED,IAAI,SAAS,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;YAChC,OAAO,CAAC,IAAI,CAAC,MAAM,IAAA,qBAAY,EAAC,MAAM,EAAE,SAAS,CAAC,IAAI,EAAE,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC;YACvE,SAAS;QACX,CAAC;QAED,MAAM,OAAO,GAAG,2BAAiB,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAClD,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,OAAO,CAAC,IAAI,CAAC;gBACX,IAAI,EAAE,SAAS,CAAC,IAAI;gBACpB,IAAI,EAAE,SAAS,CAAC,IAAI;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,4BAA4B,SAAS,CAAC,IAAI,GAAG;aACvD,CAAC,CAAC;YACH,SAAS;QACX,CAAC;QAED,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,SAA+C,CAAC,CAAC,CAAC;IACjF,CAAC;IAED,OAAO,OAAO,CAAC;AACjB,CAAC"}

package/dist/assertions/llm-judge.d.ts

@@ -0,0 +1,6 @@
+import { AssertionResult, ProviderConfig } from '../types';
+export declare function assertLlmJudge(output: string, judgeConfig: {
+    provider: ProviderConfig;
+    model: string;
+}, criteria: string, threshold: number): Promise<AssertionResult>;
+//# sourceMappingURL=llm-judge.d.ts.map

package/dist/assertions/llm-judge.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"llm-judge.d.ts","sourceRoot":"","sources":["../../src/assertions/llm-judge.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,eAAe,EAAE,cAAc,EAAE,MAAM,UAAU,CAAC;AAW3D,wBAAsB,cAAc,CAClC,MAAM,EAAE,MAAM,EACd,WAAW,EAAE;IAAE,QAAQ,EAAE,cAAc,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,EACxD,QAAQ,EAAE,MAAM,EAChB,SAAS,EAAE,MAAM,GAChB,OAAO,CAAC,eAAe,CAAC,CAyB1B"}

package/dist/assertions/llm-judge.js

@@ -0,0 +1,75 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.assertLlmJudge = assertLlmJudge;
+const providers_1 = require("../providers");
+const JUDGE_PROMPT = `You are an evaluator. Score the following LLM output on a scale of 0.0 to 1.0 based on the given criteria.
+Respond with ONLY a JSON object: {"score": <number>, "explanation": "<brief reason>"}
+
+Criteria: {{criteria}}
+
+Output to evaluate:
+{{output}}`;
+async function assertLlmJudge(output, judgeConfig, criteria, threshold) {
+    try {
+        const judge = (0, providers_1.getProvider)(judgeConfig.provider, judgeConfig.model);
+        const prompt = JUDGE_PROMPT
+            .replace('{{criteria}}', criteria)
+            .replace('{{output}}', output);
+        const response = await judge.call(prompt);
+        const { score, explanation } = parseJudgeResponse(response);
+        return {
+            type: 'llm-judge',
+            name: `llm-judge: ${criteria.slice(0, 50)}`,
+            passed: score >= threshold,
+            expected: `>= ${threshold}`,
+            actual: `${score.toFixed(2)} — ${explanation}`,
+        };
+    }
+    catch (error) {
+        return {
+            type: 'llm-judge',
+            name: `llm-judge: ${criteria.slice(0, 50)}`,
+            passed: false,
+            message: `Judge error: ${error.message}`,
+        };
+    }
+}
+function parseJudgeResponse(response) {
+    // Try JSON parse first
+    try {
+        const json = JSON.parse(response.trim());
+        if (typeof json.score === 'number') {
+            return {
+                score: Math.max(0, Math.min(1, json.score)),
+                explanation: json.explanation || '',
+            };
+        }
+    }
+    catch {
+        // Fall through to regex
+    }
+    // Try to extract JSON from response (judge might wrap in markdown)
+    const jsonMatch = response.match(/\{[^}]*"score"\s*:\s*([\d.]+)[^}]*\}/);
+    if (jsonMatch) {
+        try {
+            const json = JSON.parse(jsonMatch[0]);
+            return {
+                score: Math.max(0, Math.min(1, json.score)),
+                explanation: json.explanation || '',
+            };
+        }
+        catch {
+            // Fall through
+        }
+    }
+    // Last resort: extract any number between 0 and 1
+    const numMatch = response.match(/\b(0(?:\.\d+)?|1(?:\.0+)?)\b/);
+    if (numMatch) {
+        return {
+            score: parseFloat(numMatch[1]),
+            explanation: 'Score extracted from unstructured response',
+        };
+    }
+    throw new Error(`Could not parse judge response: ${response.slice(0, 200)}`);
+}
+//# sourceMappingURL=llm-judge.js.map

package/dist/assertions/llm-judge.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"llm-judge.js","sourceRoot":"","sources":["../../src/assertions/llm-judge.ts"],"names":[],"mappings":";;AAWA,wCA8BC;AAxCD,4CAA2C;AAE3C,MAAM,YAAY,GAAG;;;;;;WAMV,CAAC;AAEL,KAAK,UAAU,cAAc,CAClC,MAAc,EACd,WAAwD,EACxD,QAAgB,EAChB,SAAiB;IAEjB,IAAI,CAAC;QACH,MAAM,KAAK,GAAG,IAAA,uBAAW,EAAC,WAAW,CAAC,QAAQ,EAAE,WAAW,CAAC,KAAK,CAAC,CAAC;QACnE,MAAM,MAAM,GAAG,YAAY;aACxB,OAAO,CAAC,cAAc,EAAE,QAAQ,CAAC;aACjC,OAAO,CAAC,YAAY,EAAE,MAAM,CAAC,CAAC;QAEjC,MAAM,QAAQ,GAAG,MAAM,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QAC1C,MAAM,EAAE,KAAK,EAAE,WAAW,EAAE,GAAG,kBAAkB,CAAC,QAAQ,CAAC,CAAC;QAE5D,OAAO;YACL,IAAI,EAAE,WAAW;YACjB,IAAI,EAAE,cAAc,QAAQ,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE;YAC3C,MAAM,EAAE,KAAK,IAAI,SAAS;YAC1B,QAAQ,EAAE,MAAM,SAAS,EAAE;YAC3B,MAAM,EAAE,GAAG,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,WAAW,EAAE;SAC/C,CAAC;IACJ,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO;YACL,IAAI,EAAE,WAAW;YACjB,IAAI,EAAE,cAAc,QAAQ,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE;YAC3C,MAAM,EAAE,KAAK;YACb,OAAO,EAAE,gBAAiB,KAAe,CAAC,OAAO,EAAE;SACpD,CAAC;IACJ,CAAC;AACH,CAAC;AAED,SAAS,kBAAkB,CAAC,QAAgB;IAC1C,uBAAuB;IACvB,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC,CAAC;QACzC,IAAI,OAAO,IAAI,CAAC,KAAK,KAAK,QAAQ,EAAE,CAAC;YACnC,OAAO;gBACL,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC;gBAC3C,WAAW,EAAE,IAAI,CAAC,WAAW,IAAI,EAAE;aACpC,CAAC;QACJ,CAAC;IACH,CAAC;IAAC,MAAM,CAAC;QACP,wBAAwB;IAC1B,CAAC;IAED,mEAAmE;IACnE,MAAM,SAAS,GAAG,QAAQ,CAAC,KAAK,CAAC,sCAAsC,CAAC,CAAC;IACzE,IAAI,SAAS,EAAE,CAAC;QACd,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC;YACtC,OAAO;gBACL,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC;gBAC3C,WAAW,EAAE,IAAI,CAAC,WAAW,IAAI,EAAE;aACpC,CAAC;QACJ,CAAC;QAAC,MAAM,CAAC;YACP,eAAe;QACjB,CAAC;IACH,CAAC;IAED,kDAAkD;IAClD,MAAM,QAAQ,GAAG,QAAQ,CAAC,KAAK,CAAC,8BAA8B,CAAC,CAAC;IAChE,IAAI,QAAQ,EAAE,CAAC;QACb,OAAO;YACL,KAAK,EAAE,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;YAC9B,WAAW,EAAE,4CAA4C;SAC1D,CAAC;IACJ,CAAC;IAED,MAAM,IAAI,KAAK,CAAC,mCAAmC,QAAQ,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;AAC/E,CAAC"}

package/dist/cli.js

@@ -41,18 +41,18 @@ const commander_1 = require("commander");
 const path = __importStar(require("path"));
 const fs = __importStar(require("fs"));
 const chalk_1 = __importDefault(require("chalk"));
-const config_validation_1 = require("./config-validation");
 const runner_1 = require("./runner");
 const github_1 = require("./github");
 const cache_1 = require("./cache");
 const snapshot_1 = require("./snapshot");
 const reporter_1 = require("./reporter");
+const config_loader_1 = require("./config-loader");
 const utils_1 = require("./utils");
 const program = new commander_1.Command();
 program
     .name('prompt-lock')
     .description('Version control and behavioral regression testing for LLM prompts')
-    .version('0.2.0');
+    .version('0.4.0');
 // ── init ──────────────────────────────────────────────────────────────────────
 program
     .command('init')
@@ -166,6 +166,8 @@ program
     .option('--cache', 'Cache LLM outputs (skip calls when prompt+model unchanged)')
     .option('--no-cache', 'Disable output caching')
     .option('--github-pr <pr>', 'Post results as a GitHub PR comment (e.g. owner/repo#123)')
+    .option('--watch', 'Watch for file changes and re-run automatically')
+    .option('--ab <pair>', 'Compare two prompt IDs side-by-side (e.g. "v1:v2")')
     .action(async (opts) => {
     const configs = await loadPromptConfigs(opts.config, opts.id);
     if (configs.length === 0) {
@@ -174,6 +176,54 @@ program
         process.exitCode = 1;
         return;
     }
+    // A/B testing mode
+    if (opts.ab) {
+        const [idA, idB] = String(opts.ab).split(':');
+        if (!idA || !idB) {
+            console.log(chalk_1.default.red('Invalid --ab format. Use: --ab <variant-a-id>:<variant-b-id>'));
+            process.exitCode = 1;
+            return;
+        }
+        const variantA = configs.find(c => c.id === idA);
+        const variantB = configs.find(c => c.id === idB);
+        if (!variantA) {
+            console.log(chalk_1.default.red(`Variant A not found: "${idA}"`));
+            process.exitCode = 1;
+            return;
+        }
+        if (!variantB) {
+            console.log(chalk_1.default.red(`Variant B not found: "${idB}"`));
+            process.exitCode = 1;
+            return;
+        }
+        const projectConfigAB = loadProjectConfig(opts.config);
+        const cacheDirAB = projectConfigAB?.snapshotDir
+            ? path.join(path.dirname(projectConfigAB.snapshotDir), 'cache')
+            : '.promptlock/cache';
+        const abRunOpts = {
+            verbose: opts.verbose,
+            cache: opts.cache ?? false,
+            cacheDir: cacheDirAB,
+            retry: { maxRetries: 3 },
+        };
+        const spinAB = (0, utils_1.spinner)(`Running A/B: ${idA} vs ${idB}...`);
+        const abResult = await (0, runner_1.runAB)(variantA, variantB, abRunOpts);
+        spinAB.stop(`Completed A/B comparison.`);
+        (0, reporter_1.printABReport)(abResult);
+        // Write markdown report if requested
+        const abReportDir = projectConfigAB?.reportDir ?? '.promptlock/reports';
+        const abFormats = parseReportFormats(opts.report, projectConfigAB);
+        if (abFormats.includes('markdown')) {
+            const p = await (0, reporter_1.generateABMarkdownReport)(abResult, abReportDir);
+            console.log(chalk_1.default.dim(`Report saved: ${p}`));
+        }
+        // Exit code for CI
+        const abFailed = !abResult.variantA.passed || !abResult.variantB.passed;
+        if (abFailed && (opts.ci || projectConfigAB?.ci?.failOnRegression)) {
+            process.exitCode = 1;
+        }
+        return;
+    }
     if (opts.dryRun) {
         console.log(chalk_1.default.dim(`[dry-run] Would test ${configs.length} prompt${configs.length !== 1 ? 's' : ''}:`));
         for (const c of configs) {
@@ -221,6 +271,10 @@ program
             const p = await (0, reporter_1.generateHtmlReport)(results, reportDir);
             console.log(chalk_1.default.dim(`Report saved: ${p}`));
         }
+        if (format === 'markdown') {
+            const p = await (0, reporter_1.generateMarkdownReport)(results, reportDir);
+            console.log(chalk_1.default.dim(`Report saved: ${p}`));
+        }
     }
     // Post GitHub PR comment
     if (opts.githubPr) {
@@ -256,6 +310,44 @@ program
     if (anyFailed && (opts.ci || projectConfig?.ci?.failOnRegression)) {
         process.exitCode = 1;
     }
+    // Watch mode
+    if (opts.watch) {
+        console.log('');
+        console.log(chalk_1.default.dim('Watching for changes... (press Ctrl+C to stop)'));
+        const promptsDir = path.join(process.cwd(), 'prompts');
+        const watchTargets = [promptsDir];
+        const discovered = await (0, config_loader_1.discoverConfigFile)(process.cwd());
+        if (discovered)
+            watchTargets.push(discovered);
+        let debounceTimer = null;
+        for (const target of watchTargets) {
+            try {
+                fs.watch(target, { recursive: true }, () => {
+                    if (debounceTimer)
+                        clearTimeout(debounceTimer);
+                    debounceTimer = setTimeout(async () => {
+                        console.clear();
+                        console.log(chalk_1.default.dim('Re-running...'));
+                        try {
+                            const reConfigs = await loadPromptConfigs(opts.config, opts.id);
+                            const reResults = await (0, runner_1.runAll)(reConfigs, runOpts);
+                            (0, reporter_1.printConsoleReport)(reResults);
+                        }
+                        catch (e) {
+                            console.error(chalk_1.default.red(`Error: ${e.message}`));
+                        }
+                        console.log('');
+                        console.log(chalk_1.default.dim('Watching for changes... (press Ctrl+C to stop)'));
+                    }, 500);
+                });
+            }
+            catch {
+                // Target doesn't exist, skip
+            }
+        }
+        // Keep process alive
+        await new Promise(() => { });
+    }
 });
 // ── snapshot ──────────────────────────────────────────────────────────────────
 program
@@ -379,27 +471,38 @@ function loadProjectConfig(configPath) {
     }
 }
 async function loadPromptConfigs(configPath, filterId) {
+    const configs = [];
+    // Try YAML/JSON config file first (auto-discovery)
+    const discovered = await (0, config_loader_1.discoverConfigFile)(process.cwd());
+    if (discovered) {
+        try {
+            const loaded = await (0, config_loader_1.loadConfigFile)(discovered);
+            const items = Array.isArray(loaded) ? loaded : [loaded];
+            configs.push(...items);
+        }
+        catch (e) {
+            console.error(chalk_1.default.red(`Error loading ${discovered}: ${e.message}`));
+        }
+    }
+    // Also scan prompts/ directory for JS/JSON/YAML files
     const projectConfig = loadProjectConfig(configPath);
     const promptsDir = projectConfig?.promptsDir
         ? path.resolve(process.cwd(), projectConfig.promptsDir)
         : path.join(process.cwd(), 'prompts');
-    const configs = [];
     try {
         const files = fs.readdirSync(promptsDir);
         for (const file of files) {
-            if (!file.endsWith('.js') && !file.endsWith('.json'))
+            if (!['.js', '.json', '.yaml', '.yml'].some(ext => file.endsWith(ext)))
                 continue;
             const filePath = path.join(promptsDir, file);
             try {
-                const mod = require(filePath);
-                const config = mod.default ?? mod;
-                const items = Array.isArray(config) ? config : (config && config.id) ? [config] : [];
+                const loaded = await (0, config_loader_1.loadConfigFile)(filePath);
+                const items = Array.isArray(loaded) ? loaded : [loaded];
                 for (const item of items) {
-                    const validation = (0, config_validation_1.validateConfig)(item);
-                    if (!validation.valid) {
-                        console.warn(chalk_1.default.yellow(`⚠️  ${file}: ${validation.errors.join('; ')}`));
+                    // Avoid duplicates from auto-discovery
+                    if (!configs.some(c => c.id === item.id)) {
+                        configs.push(item);
                     }
-                    configs.push(item);
                 }
             }
             catch (e) {