llm-drift-bench 0.1.0__tar.gz

llm_drift_bench-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,34 @@
+---
+name: Bug Report
+about: Report a bug, unexpected drift score, or schema validation issue in llm-drift-bench
+title: "[BUG] "
+labels: ["bug", "triage"]
+assignees: ""
+---
+
+## Bug Description
+A clear and concise description of what the bug is.
+
+## Steps to Reproduce
+1. Run command or Python snippet:
+   ```bash
+   llm-drift-bench run -b ... -c ...
+   ```
+2. Provide minimal sample input (`prompts.json` / `.jsonl` snippet):
+   ```json
+   {"prompt_id": "example_1", "output": "{...}"}
+   ```
+3. See error or unexpected output.
+
+## Expected Behavior
+A clear and concise description of what you expected to happen.
+
+## Actual Output / Traceback
+```text
+Paste terminal output or stack trace here
+```
+
+## Environment
+- **`llm-drift-bench` version**: (e.g., `0.1.0`)
+- **Python version**: (e.g., `3.11.8`)
+- **Operating System**: (e.g., `Ubuntu 22.04`, `macOS 14`, `Windows 11`)

llm_drift_bench-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,25 @@
+---
+name: Feature Request
+about: Suggest a new drift metric, schema rule, or CLI capability for llm-drift-bench
+title: "[FEAT] "
+labels: ["enhancement"]
+assignees: ""
+---
+
+## Problem Statement
+Is your feature request related to a specific LLM evaluation or quantization drift challenge? Please describe.
+
+## Proposed Solution
+Describe the CLI flag, Python API, or drift metric you would like to see added.
+
+### Example Usage
+```bash
+# Example CLI or Python usage
+llm-drift-bench run --baseline fp16.jsonl --candidate int4.jsonl
+```
+
+## Alternatives Considered
+Describe any alternative solutions, workarounds, or external libraries you've considered.
+
+## Additional Context
+Add any other context, academic references, or benchmark examples about the feature request here.

llm_drift_bench-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,83 @@
+name: CI
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+    branches: ["main"]
+
+permissions:
+  contents: read
+
+jobs:
+  lint-and-typecheck:
+    name: Lint (Ruff) & Type-Check (Mypy)
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.11
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: "pip"
+
+      - name: Install package and dev dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run Ruff linter
+        run: ruff check src tests
+
+      - name: Run Mypy strict type checker
+        run: mypy src/llm_drift_bench
+
+  test:
+    name: Test Suite (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: "pip"
+
+      - name: Install package and test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run pytest with coverage enforcement (>85%)
+        run: pytest --cov=llm_drift_bench --cov-report=term-missing --cov-fail-under=85
+
+  build:
+    name: Package Build Check
+    runs-on: ubuntu-latest
+    needs: [lint-and-typecheck, test]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tooling
+        run: |
+          python -m pip install --upgrade pip build twine
+
+      - name: Build wheel and sdist
+        run: python -m build
+
+      - name: Verify distribution metadata with twine
+        run: twine check dist/*

llm_drift_bench-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,36 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  pypi-publish:
+    name: Build & Publish to PyPI
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/llm-drift-bench
+    permissions:
+      id-token: write
+    steps:
+      - name: Check out the repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tooling
+        run: pip install --upgrade pip build
+
+      - name: Build source distribution and binary wheel
+        run: python -m build
+
+      - name: Publish distribution to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

llm_drift_bench-0.1.0/.gitignore

@@ -0,0 +1,74 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Type checkers & linters
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+.pytype/
+.ruff_cache/
+
+# IDEs & Editors
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# OS generated files
+.DS_Store
+Thumbs.db

llm_drift_bench-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,78 @@
+# Contributing to `llm-drift-bench`
+
+Thank you for your interest in contributing to **`llm-drift-bench`**! We aim to keep this package fast, modular, strictly typed, and zero-bloat.
+
+---
+
+## 1. Development Environment Setup
+
+### Prerequisites
+- Python **3.10+**
+- `git`
+
+### Clone and Install in Editable Mode
+```bash
+git clone https://github.com/anant-goel/llm-drift-bench.git
+cd llm-drift-bench
+python -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+---
+
+## 2. Running Tests, Linting & Type Checking
+
+Before opening a Pull Request, verify that all checks pass locally:
+
+### Run the Test Suite with Coverage
+```bash
+pytest --cov=llm_drift_bench --cov-report=term-missing --cov-fail-under=85
+```
+
+Run the complete test suite with pytest after installing `.[dev]`.
+
+The standard-library unittest runner does not execute the pytest-style
+function tests, and importing this package still requires its runtime
+dependencies.
+
+### Run Ruff Linter
+```bash
+ruff check src tests
+```
+
+### Run Mypy Strict Type Checking
+```bash
+mypy src/llm_drift_bench
+```
+
+---
+
+## 3. Pull Request & Commit Guidelines
+
+1. **Zero-Bloat Philosophy**: Do not add third-party runtime dependencies to `[project].dependencies` unless strictly necessary; prefer standard library implementations or optional extras.
+2. **Atomic Conventional Commits**: Every logical change (schema addition, metric function, CLI flag, test file, or documentation update) should have an atomic commit following [Conventional Commits](https://www.conventionalcommits.org/):
+   - `feat(metrics): ...`
+   - `fix(validator): ...`
+   - `test(engine): ...`
+   - `docs: ...`
+   - `ci: ...`
+3. **Test Coverage**: New features and bug fixes must include unit tests in `tests/` maintaining at least 85% total coverage.
+4. **Strict Typing**: All functions and methods in `src/llm_drift_bench/` must include full Python type annotations (`from __future__ import annotations`) and pass `mypy --strict`.
+
+---
+
+## 4. `good first issue` Roadmap
+
+Looking for a place to start contributing? Here is our curated list of `good first issue` enhancements:
+
+- [ ] **`[good first issue]` Support `uniqueItems` and `multipleOf` in `validator.py`**
+  - Extend `validate_against_schema` in `src/llm_drift_bench/validator.py` to validate JSON Schema `uniqueItems: true` on arrays and `multipleOf` on numeric fields, with corresponding unit tests in `tests/test_validator.py`.
+- [ ] **Render the canonical benchmark report directly as Markdown.**
+  - Preserve thresholds, failure reasons, optional measurements, and scores without recomputing metrics or inventing missing values.
+- [ ] **`[good first issue]` Per-Category Drift Breakdown in `DriftBenchmarkReport`**
+  - Group `SampleDriftResult` items by `category` and include per-category mean drift and conformity rates in `DriftBenchmarkReport.to_dict()`.
+- [ ] **`[good first issue]` CSV / TSV Export Option for Sample Results**
+  - Add `--csv <path>` to `llm-drift-bench run` using Python's standard library `csv` module to export per-prompt drift scores for spreadsheet analysis.
+- [ ] **`[good first issue]` JSON Schema `anyOf` / `oneOf` Combinator Support**
+  - Implement recursive support for `anyOf` and `oneOf` schema combinators in `src/llm_drift_bench/validator.py`.

llm_drift_bench-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 llm-drift-bench Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_drift_bench-0.1.0/PKG-INFO

@@ -0,0 +1,273 @@
+Metadata-Version: 2.5
+Name: llm-drift-bench
+Version: 0.1.0
+Summary: A lightweight, zero-bloat CLI and Python library to benchmark, validate JSON conformity, and detect output drift across LLM updates and quantizations.
+Project-URL: Homepage, https://github.com/anant-goel/llm-drift-bench
+Project-URL: Repository, https://github.com/anant-goel/llm-drift-bench
+Project-URL: Issues, https://github.com/anant-goel/llm-drift-bench/issues
+Author: Anant Goel
+License: MIT
+License-File: LICENSE
+Keywords: benchmark,drift-detection,evaluation,json-validation,llm,mlops,quantization
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: dev
+Requires-Dist: build>=1.2.0; extra == 'dev'
+Requires-Dist: mypy>=1.10.0; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.5.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# llm-drift-bench
+
+[![CI](https://github.com/anant-goel/llm-drift-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/anant-goel/llm-drift-bench/actions/workflows/ci.yml)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![Runtime dependencies](https://img.shields.io/badge/runtime-Pydantic%20%2B%20Rich-blue.svg)](pyproject.toml)
+[![Code style: Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
+[![Typed: Mypy Strict](https://img.shields.io/badge/typing-mypy%20strict-blue.svg)](http://mypy-lang.org/)
+
+**`llm-drift-bench`** is a Python CLI and library for comparing captured LLM outputs across model updates, prompt revisions, and quantizations. It measures lexical differences, JSON structure and field agreement, numeric changes, and recorded latency.
+
+---
+
+## Why `llm-drift-bench`?
+
+When upgrading model checkpoints or quantizing weights for production inference, subtle regressions frequently slip past perplexity benchmarks:
+- **Broken JSON Conformity**: Markdown fence artifacts or missing required keys in structured outputs.
+- **Structural Key-Path Drift**: Schema-valid payloads that drop optional nested fields or alter value types.
+- **Numeric Field Drift**: Extracted financial or confidence numbers drifting beyond acceptable tolerances.
+- **Tail Latency Regressions**: P95 latency spikes across specific prompt categories.
+
+The runtime uses Pydantic and Rich, together with their transitive dependencies.
+
+Similarity measures agreement with a baseline, not factual correctness or semantic equivalence. The validator implements a JSON Schema subset and rejects unsupported keywords before checking the response. It is not a complete JSON Schema implementation.
+
+---
+
+## Quickstart after publication
+
+```bash
+pip install llm-drift-bench
+llm-drift-bench init-sample --dir ./demo
+llm-drift-bench run -b ./demo/baseline_fp16.jsonl -c ./demo/candidate_int4.jsonl -p ./demo/prompts.json
+```
+
+---
+
+## Visual Terminal Output
+
+### 1. Drift Benchmark Comparison (`llm-drift-bench run`)
+
+```text
+==============================================================================
+ LLM DRIFT BENCHMARK REPORT  [PASSED]
+==============================================================================
+ Baseline Model  : llama-3-8b-fp16
+ Candidate Model : llama-3-8b-q4_k_m
+ Total Samples   : 2 (Drifted: 0 / 0.0%)
+------------------------------------------------------------------------------
+ Metric                               | Baseline         | Candidate / Value 
+------------------------------------------------------------------------------
+ JSON Conformity Rate                 |          100.0% |            100.0%
+ Mean Composite Drift                 |     0.0000 (ref) |             0.0426
+ Max Composite Drift                  |     0.0000 (ref) |             0.0852
+ Mean Token Jaccard Similarity        |     1.0000 (ref) |             0.9412
+ Mean Levenshtein Similarity          |     1.0000 (ref) |             0.9930
+ Mean JSON Structural Similarity      |     1.0000 (ref) |             1.0000
+ Mean JSON Numeric Relative Drift     |     0.0000 (ref) |             0.0160
+ P50 Latency (ms)                     |            130.2 |               82.8
+ P95 Latency (ms)                     |            141.3 |               88.6
+------------------------------------------------------------------------------
+ Prompt ID          | Category     | JSON OK   | Struct Sim | Drift    | State  
+------------------------------------------------------------------------------
+ extract_invoice_01 | extraction   | YES       |     1.0000 |   0.0000 | OK     
+ classify_ticket_02 | classificati | YES       |     1.0000 |   0.0852 | OK
+==============================================================================
+```
+
+### 2. Standalone JSON Conformity Validation (`llm-drift-bench validate`)
+
+```bash
+llm-drift-bench validate -i ./demo/candidate_int4.jsonl -p ./demo/prompts.json --strict
+```
+
+```text
+========================================================================
+ JSON CONFORMITY VALIDATION  [2/2 passed (100.0%)]
+========================================================================
+ [PASS] extract_invoice_01 | Conformant
+ [PASS] classify_ticket_02 | Conformant
+========================================================================
+```
+
+---
+
+## Input & Configuration Formats
+
+### Prompt Suite with JSON Schemas (`prompts.json`)
+
+```json
+[
+  {
+    "prompt_id": "extract_invoice_01",
+    "category": "extraction",
+    "prompt": "Extract invoice fields as JSON.",
+    "schema": {
+      "type": "object",
+      "required": ["invoice_id", "total_usd", "status"],
+      "properties": {
+        "invoice_id": {"type": "string", "minLength": 3, "pattern": "^INV-"},
+        "total_usd": {"type": "number", "minimum": 0},
+        "status": {"type": "string", "enum": ["paid", "pending", "overdue"]}
+      },
+      "additionalProperties": false
+    }
+  }
+]
+```
+
+### Model Outputs (`baseline_fp16.jsonl` / `candidate_int4.jsonl`)
+
+```jsonl
+{"prompt_id": "extract_invoice_01", "model_name": "llama-3-8b-fp16", "latency_ms": 142.5, "output": "{\"invoice_id\": \"INV-2026-09\", \"total_usd\": 1250.00, \"status\": \"paid\"}"}
+```
+
+---
+
+## CLI Reference & CI Gating
+
+Gate your CI/CD pipeline on model drift and JSON conformity thresholds using `--fail-on-drift`:
+
+```bash
+llm-drift-bench run \
+  --baseline ./baseline_fp16.jsonl \
+  --candidate ./candidate_int4.jsonl \
+  --prompts ./prompts.json \
+  --max-drift 0.15 \
+  --min-conformity 0.98 \
+  --max-numeric-drift 0.05 \
+  --max-latency-regression 25.0 \
+  --output ./artifacts/drift_report.json \
+  --fail-on-drift
+```
+
+| Flag | Default | Description |
+| :--- | :--- | :--- |
+| `-b`, `--baseline` | *(Required)* | Path to baseline model outputs (`.json` or `.jsonl`). |
+| `-c`, `--candidate` | *(Required)* | Path to candidate model outputs (`.json` or `.jsonl`). |
+| `-p`, `--prompts` | `None` | Optional prompt definitions with per-prompt JSON Schemas. |
+| `--max-drift` | `0.25` | Maximum allowed mean/per-sample composite drift score `[0.0, 1.0]`. |
+| `--min-conformity` | `0.95` | Minimum required candidate JSON Schema conformity rate `[0.0, 1.0]`. |
+| `--max-numeric-drift` | `0.15` | Maximum allowed relative numeric drift on shared JSON number fields. |
+| `--max-latency-regression` | `50.0` | Maximum allowed P95 latency regression percentage (`%`). |
+| `-o`, `--output` | `None` | File path to save the structured JSON benchmark report. |
+| `--format` | `table` | Output format on stdout (`table`, `rich`, `json`, or `markdown`). |
+| `--fail-on-drift` | `False` | Exit with code `1` if any threshold is violated or sample drifts. |
+
+### Current behavior of the unreleased revision
+
+Levenshtein similarity always compares characters (Unicode code points),
+including whitespace and case. Each response is limited to 16,384 characters
+for this metric. Longer responses raise an error (CLI exit code `2`), rather
+than being truncated or scored using a different unit. Unicode normalization
+is not applied.
+
+Commands are `run`, `report`, `validate`, and `init-sample`.
+`--suite` and `--prompts` are aliases for the suite path.
+
+Use existing captured-output files for both comparison targets.
+Missing files, directory paths, and model identifiers are input errors
+(CLI exit code `2`). The tool does not call model endpoints or generate
+responses. The library also accepts explicit sequences of `ModelSample`.
+`init-sample` creates illustrative fixtures, not measured model outputs.
+
+Baseline and candidate must contain exactly the same non-empty prompt IDs,
+with no duplicates. An explicitly supplied suite must cover exactly those
+IDs and cannot be empty. The `validate --prompts` command applies the same
+suite-coverage rule. Matching errors return CLI exit code `2`.
+Comparison results are ordered by prompt ID, independently of input order.
+For library callers, `evaluate_suite([], baseline, candidate)` still means
+infer the prompts from the complete matching output sets; it never selects
+only their intersection.
+
+Table and JSON output use the benchmark engine. Rich and Markdown output
+currently recompute scores through a second evaluator with different
+weights. Until these paths are unified, their scores are not interchangeable.
+
+Missing measurements appear as `n/a` in reports and `null` in JSON.
+Throughput requires an explicit `tokens_generated` count and positive measured
+`latency_ms`; text token estimates are not substituted. Zero latency remains
+a recorded value, but cannot produce throughput or a baseline-relative
+percentage regression.
+
+Suite means and latency percentiles require complete measurements for that
+side of the comparison. If either side lacks timings, the latency regression
+check is skipped; a passing result does not certify latency performance.
+Other drift and conformity thresholds still apply.
+
+Saved engine reports lose some information when converted by `report`.
+Unavailable latency means and throughput remain `n/a`; percentiles are not
+substituted for means or individual observations.
+
+---
+
+## Python Library API
+
+Use `llm-drift-bench` directly inside your evaluation pipelines or pytest suites:
+
+```python
+from llm_drift_bench import (
+    DriftBenchmarkEngine,
+    DriftThresholds,
+    ModelSample,
+    PromptCase,
+    format_report_table,
+)
+
+prompts = [
+    PromptCase(
+        prompt_id="invoice_1",
+        prompt="Extract invoice total.",
+        schema={
+            "type": "object",
+            "required": ["total"],
+            "properties": {"total": {"type": "number", "minimum": 0}},
+        },
+    )
+]
+
+baseline = [ModelSample(prompt_id="invoice_1", model_name="fp16", output='{"total": 100.0}')]
+candidate = [ModelSample(prompt_id="invoice_1", model_name="int4", output='{"total": 99.5}')]
+
+engine = DriftBenchmarkEngine(thresholds=DriftThresholds(max_composite_drift=0.10))
+report = engine.evaluate(baseline, candidate, prompts)
+
+print(format_report_table(report))
+assert report.passed
+```
+
+---
+
+## Contributing & License
+
+Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing instructions, and our `good first issue` roadmap.
+
+Licensed under the [MIT License](LICENSE).