mistrial 0.2.0__tar.gz

mistrial-0.2.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,23 @@
+---
+name: Bug report
+about: Something gave the wrong result or crashed
+labels: bug
+---
+
+**Command you ran**
+
+```bash
+mistrial run ...
+```
+
+**What happened**
+
+Paste the output or traceback.
+
+**What you expected**
+
+**Environment**
+
+- mistrial version (`mistrial --version`):
+- Python version:
+- OS:

mistrial-0.2.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,13 @@
+---
+name: Feature request
+about: A metric, adapter, loader or report change
+labels: enhancement
+---
+
+**What you want to do**
+
+**What you do today instead**
+
+**If it is a new metric**
+
+Link to the formula or paper and, if you can, a small worked example with the expected number.

mistrial-0.2.0/.github/workflows/ci.yml

@@ -0,0 +1,33 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: python -m pip install -e ".[dev]"
+      - run: python -m pytest -q
+      - name: demo smoke test
+        run: |
+          mistrial demo --runs 5 --html none --out-dir /tmp/jf --adaptive --stabilize 3 --pass-mark 0.55
+          mistrial rubric-check examples/cases.jsonl -j demo:steady --out-dir /tmp/jf -q
+          run=$(ls /tmp/jf/runs/2*.json | head -1)
+          mistrial gate-risk "$run" --pass-mark 0.5
+          mistrial migrate "$run" --pass-mark 0.6
+          mistrial escalate "$run" --out /tmp/jf/needs-human.csv
+          mistrial sentinel baseline "$run" --out /tmp/jf/base.json
+          mistrial report "$run" --html none --markdown /tmp/jf/card.md
+      - name: build
+        run: python -m pip install -q build twine && python -m build -q && python -m twine check dist/*

mistrial-0.2.0/.gitignore

@@ -0,0 +1,12 @@
+.venv/
+__pycache__/
+*.pyc
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.mistrial/
+report.html
+report.json
+.DS_Store
+docs/*.mp4

mistrial-0.2.0/CHANGELOG.md

@@ -0,0 +1,55 @@
+# Changelog
+
+All notable changes to mistrial are recorded here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+Nothing yet.
+
+## [0.2.0] - 2026-10-26
+
+First public release. Ten additions over 0.1.0, each answering a documented way LLM judges go wrong in production.
+
+### Added
+- **Rubric sensitivity** (`mistrial rubric-check`): run the judge under three rewordings of the rubric that mean the same thing, report the share of verdicts that change and Krippendorff's alpha across wordings. Built-in rewordings for `correctness` and `quality`; bring your own with `--variant`.
+- **Rubric reversal** (same command): ask with the criterion flipped and check that verdicts invert. Reports the cases where the judge gave the same answer to "is it correct?" and "is it incorrect?".
+- **Gate false-alarm rate** (`--pass-mark` on `run`/`report`, and `mistrial gate-risk`): how often your CI gate would go red with nothing changed, by re-sampling each case's observed verdicts; plus a safe mark and the runs per case that bring the rate under 5%.
+- **Judge migration** (`mistrial migrate`): old judge vs new judge on the same cases, with your pass mark translated to the new judge and a per-case cutoff mapped by quantile.
+- **Sentinel** (`mistrial sentinel baseline` / `check`): freeze a run on anchor cases, re-judge fresh on a schedule, exit 2 on JUDGE_DRIFT. Design credit: judge-drift-sentinel.
+- **Parser audit**: every LLM verdict keeps the raw reply and how it was read (strict JSON, fenced, embedded in prose, no verdict). The card shows the counts and examples.
+- **Adaptive re-ask** (`--adaptive`): two runs everywhere, extra runs only where the two disagreed. The card reports the calls skipped.
+- **Escalation** (`mistrial escalate`): a CSV of the cases a human should look at, with a column for their verdict; `--langsmith-dataset` pushes them to LangSmith.
+- **Stabilized evaluator** (`--stabilize K` and `mistrial.stabilize()`): the judge wrapped in a K-sample majority vote with the same evaluator signature, measured next to the raw one.
+- **GitHub Action** (`action.yml`): runs the card in CI, posts it as a PR comment, uploads the HTML report, fails the job on a gate. `--markdown` writes the same card anywhere.
+- `examples/tricky.jsonl`: 32 labeled cases built to be hard to judge (hedges, rounding, unit conversions, extra details, equivalent code). The bundled `examples/cases.jsonl` stays as the easy set.
+- `mistrial demo` says up front that its three judges are fake and flaky on purpose.
+- `docs/img/compare.png`, a feature comparison against judgecheck, judgebench, Assay, judge-drift-sentinel and LangSmith/promptfoo, and a "Standing on shoulders" README section with the papers and repos this builds on.
+- Judges for every OpenAI-compatible provider: `gemini:`, `groq:`, `mistral:`, `deepseek:`, `xai:`, `together:`, `openrouter:`, `ollama:` (local, no key) and `custom:` with `MISTRIAL_BASE_URL`. `<PROVIDER>_BASE_URL` overrides any base URL. `mistrial providers` lists them.
+- An 88 second animated explainer video (`docs/explainer.mp4`, not committed) rendered from `docs/explainer/explainer.html`: the problem, why it slips through, the command, the real numbers, the fix. Narration script and re-render steps in `docs/explainer/SCRIPT.md`.
+
+### Changed
+- The project is called **mistrial**. Package, command, env vars (`MISTRIAL_BASE_URL`, `MISTRIAL_API_KEY`) and the `.mistrial/` directory all follow. It was briefly called judgeflake before any release.
+- README, demo GIF and report screenshot now show real, current OpenAI judges (`gpt-5.4-nano`, `gpt-5.4-mini`, `gpt-5.5`) instead of the fake `demo` judges. The numbers in the README come from 480 live calls made on 2026-10-25.
+- README opens with the problem, why nothing catches it today, and what mistrial does, with three diagrams in `docs/img/`.
+- Saved runs are format version 2 (adds `adaptive`, `calls_saved`, and `text`/`parse` per verdict). Version 1 files still load.
+
+## [0.1.0] - 2026-10-25
+
+First tagged version, never published.
+
+### Added
+- `mistrial run`: run one or more judges over a cases file N times and print the reliability card.
+- Judges: `openai:<model>`, `anthropic:<model>`, any Python `module:function` with the LangSmith / openevals signature, and deterministic `demo` judges.
+- Cases from JSONL, JSON or CSV with common field aliases and an optional human `label`.
+- Metrics: self-agreement, flip rate, per-case disagreement, pairwise percent agreement and Cohen's kappa, Krippendorff's alpha (nominal and interval, missing values allowed), accuracy and kappa against human labels.
+- Built-in rubrics `correctness` and `quality`, or a prompt file.
+- Terminal card, single-file HTML report (light and dark), JSON report.
+- On-disk response cache with `--fresh` to bypass it.
+- `mistrial compare` for drift between two saved runs.
+- CI gates: `--min-self-agreement`, `--max-flip-rate`, `--min-agreement`.
+- `mistrial demo`: three fake judges on twenty bundled cases, no API key.
+- `mistrial report` to re-render a saved run and `mistrial rubrics` to list rubrics.
+
+[Unreleased]: https://github.com/hemanth1999k/mistrial/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/hemanth1999k/mistrial/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/hemanth1999k/mistrial/releases/tag/v0.1.0

mistrial-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,51 @@
+# Contributing
+
+Thanks for looking. mistrial is small on purpose, so most contributions are quick.
+
+## Setup
+
+```bash
+git clone https://github.com/hemanth1999k/mistrial.git
+cd mistrial
+python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
+.venv/bin/pytest -q
+.venv/bin/mistrial demo
+```
+
+Python 3.10 or newer. No API key is needed for the tests or the demo; the OpenAI and Anthropic adapters are tested against a fake HTTP server.
+
+## What is welcome
+
+- Bug reports with the command you ran and the output you got.
+- Adapters for other providers. Follow `LLMJudge._openai` in `src/mistrial/judges.py`: plain `httpx`, retries on 429 and 5xx, one JSON verdict out.
+- Rubrics. Add to `RUBRICS` in `src/mistrial/prompts.py` with a one-line description.
+- Loaders for other case formats, in `src/mistrial/cases.py`.
+- Report improvements, in `src/mistrial/report.py`. Keep it one file with no external assets.
+
+## Rules for metrics
+
+A new number on the card needs all three:
+
+1. A published formula, cited in the docstring.
+2. A test in `tests/test_stats.py` against a known example from the literature or a hand-computed case.
+3. A line in the README table saying what it means and what it does not mean.
+
+No invented scores and no letter grades. The card shows numbers and a one-line reading of them.
+
+## Pull requests
+
+- One change per PR, with a test.
+- Run `pytest -q` and `mistrial demo --runs 3 --html none` before opening it.
+- Add a line under `[Unreleased]` in `CHANGELOG.md`.
+- Keep the PR description short: what changed and why.
+
+## The demo GIF
+
+`docs/demo.gif` is rendered from `docs/demo.tape` with [VHS](https://github.com/charmbracelet/vhs): `brew install vhs && vhs docs/demo.tape`. Re-render it when the card's layout changes.
+
+## Releasing (maintainers)
+
+1. Bump `version` in `pyproject.toml`, move `[Unreleased]` entries under the new version in `CHANGELOG.md`.
+2. `git tag v0.x.y && git push --tags`.
+3. `python -m build && python -m twine upload dist/*`.
+4. Create the GitHub release from the tag with the changelog section as notes.

mistrial-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hemanth Dhanasekaran
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.