poorjev 0.1.0__tar.gz

poorjev-0.1.0/.github/workflows/tests.yml

@@ -0,0 +1,26 @@
+name: tests
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+  workflow_dispatch:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+      - run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+      - run: pytest -q

poorjev-0.1.0/.gitignore

@@ -0,0 +1,17 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.pytest_cache/
+.coverage
+htmlcov/
+.venv/
+venv/
+env/
+.DS_Store
+*.png
+!docs/*.png
+models/
+.hf_cache/

poorjev-0.1.0/CLAUDE.md

@@ -0,0 +1,61 @@
+# CLAUDE.md
+
+Guidance for Claude Code (and any agent) working in this repo.
+
+## What poorjev is
+
+An open source, local-first "System One" decision layer: ask typed questions
+about some state (Choice / Score / Noul), get typed answers with **calibrated**
+confidence, in one pass, no API key by default. The differentiator is honest,
+measured calibration. See `PRD.md` for the full spec and `README.md` for the pitch.
+
+## Architecture (read in this order)
+
+- `src/poorjev/primitives.py` — the contract. `Choice` / `Score` / `Noul` and
+  their frozen typed answers. Schema validity is **structural**: a decided
+  answer's `value` is always in the declared set, even for adversarial scores.
+- `src/poorjev/client.py` — `Client.ask(state, questions)`. Builds one batched
+  set of (state, hypothesis) pairs across all questions, calls the backend once,
+  applies the fitted `temperature`, and decides. Backends are a Protocol.
+- `src/poorjev/backends/local_nli.py` — default keyless backend. A zero-shot NLI
+  model scores each hypothesis for entailment. Lazy torch/transformers import.
+- `src/poorjev/calibration.py` — the moat. Temperature scaling (fit by k-fold
+  CV) + conformal abstention.
+- `src/poorjev/metrics.py` — accuracy, ECE, Brier, risk-coverage (stdlib only).
+- `src/poorjev/evaluate.py` — run the eval set through a backend, score vs gold.
+- `src/poorjev/plots.py` — reliability diagram + risk-coverage plot.
+- `src/poorjev/mcp_server.py` — expose poorjev as MCP tools (gate/judge/classify/
+  rate/decide). Logic is in plain `do_*` functions; MCP wrappers are thin.
+- `src/poorjev/cli.py` — `poorjev eval | calibrate | ask | serve`.
+- `evalset/source.py` -> `evalset/tasks.jsonl` — hand-labelled data.
+
+## Commands
+
+```bash
+pip install -e ".[local,plots,mcp,dev]"   # full dev install
+pytest -q                                  # fast suite, no model, no key
+poorjev eval --set evalset/tasks.jsonl     # accuracy + ECE (downloads model)
+poorjev calibrate --set evalset/tasks.jsonl --plots   # before/after ECE + diagrams
+poorjev serve                              # run the MCP server (stdio)
+python evalset/source.py                   # regenerate tasks.jsonl
+```
+
+## Conventions
+
+- **Tests never need a model or a key.** Unit tests use a fake/scripted backend.
+  Anything that loads the real model is an example or a manual run, not a test.
+- **Honesty is the product.** Every headline number must be reproducible by
+  `poorjev eval` / `poorjev calibrate` on the shipped set. No cherry-picking. If
+  a number is bad, print it. Keep the honest-limitations sections truthful.
+- **No em-dashes in README / marketing copy.** Use commas, colons, short
+  sentences. (Source code and this file are exempt.)
+- **Ask concrete questions.** The local NLI model handles "this moves money"
+  well and "this is dangerous" poorly. Prefer concrete hypotheses; decompose
+  abstract ones and apply a rule.
+- **Commits:** author as the repo owner only; do not add co-author trailers.
+
+## Good first contributions
+
+- More labelled decision tasks in `evalset/` (raises the eval's credibility).
+- Per-question temperature in `calibration.py` (likely lowers ECE further).
+- A new backend implementing `entail_probs(pairs)` (e.g. the optional LLM one).

poorjev-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Rupesh Poojary
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

poorjev-0.1.0/PKG-INFO

@@ -0,0 +1,264 @@
+Metadata-Version: 2.5
+Name: poorjev
+Version: 0.1.0
+Summary: The poor man's Jev: a local-first System One decision layer with provably calibrated confidence.
+Project-URL: Homepage, https://github.com/rupeshpoojary9/poorjev
+Author: Rupesh Poojary
+License: MIT
+License-File: LICENSE
+Keywords: calibration,classification,conformal-prediction,guardrails,llm,mcp,structured-output,system-one,zero-shot
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: llm
+Requires-Dist: anthropic>=0.40; extra == 'llm'
+Requires-Dist: openai>=1.40; extra == 'llm'
+Provides-Extra: local
+Requires-Dist: protobuf>=4.0; extra == 'local'
+Requires-Dist: sentencepiece>=0.2; extra == 'local'
+Requires-Dist: torch>=2.2; extra == 'local'
+Requires-Dist: transformers>=4.40; extra == 'local'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.2; extra == 'mcp'
+Provides-Extra: plots
+Requires-Dist: matplotlib>=3.8; extra == 'plots'
+Requires-Dist: numpy>=1.26; extra == 'plots'
+Description-Content-Type: text/markdown
+
+<h1 align="center">poorjev</h1>
+
+<p align="center"><b>The poor man's Jev.</b> An open source, local-first "System One" decision layer for LLM apps: typed decisions with <b>provably calibrated confidence</b>. No API key. No waitlist.</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/license-MIT-green" alt="MIT license">
+  <img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+">
+  <a href="https://github.com/rupeshpoojary9/poorjev/actions/workflows/tests.yml"><img src="https://github.com/rupeshpoojary9/poorjev/actions/workflows/tests.yml/badge.svg" alt="tests"></a>
+  <img src="https://img.shields.io/badge/ECE-0.170%20%E2%86%92%200.071-orange" alt="ECE 0.170 to 0.071">
+  <img src="https://img.shields.io/badge/API%20key-not%20required-blueviolet" alt="no API key required">
+</p>
+
+---
+
+**Your model's `0.9` is a vibe. poorjev's `0.9` is a measurement.**
+
+Every LLM-in-JSON-mode hands you a confidence score and hopes you don't check it. poorjev checks it. On the shipped eval set it cuts calibration error (ECE) from **0.170 to 0.071** with zero loss of accuracy, and it runs on your laptop with no API key.
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/rupeshpoojary9/poorjev/main/docs/reliability_before_after.png" alt="Reliability diagram: raw confidences are overconfident, calibrated confidences hug the diagonal" width="760">
+</p>
+
+<p align="center"><i>Left: raw confidences, overconfident. Right: calibrated, a stated 0.8 really is right about 80% of the time.</i></p>
+
+## Quickstart
+
+```bash
+pip install "poorjev[local]"
+```
+
+> PyPI publish is pending. Until then: `pip install "poorjev[local] @ git+https://github.com/rupeshpoojary9/poorjev"`
+
+```python
+from poorjev import Client, Choice, Score, Noul
+
+client = Client()  # local model, no key, offline after one download
+
+result = client.ask(
+    state="I've emailed three times and I'm STILL being double-charged. Cancel my account today.",
+    questions={
+        "topic":        Choice(["billing", "technical", "account", "shipping", "other"]),
+        "frustration":  Score(levels=["low", "medium", "high"]),
+        "is_urgent":    Noul("The customer needs a response today."),
+        "wants_cancel": Noul("The customer wants to cancel their account."),
+    },
+)
+
+result["topic"].value          # "billing"      always one of your options, by construction
+result["topic"].confidence     # 0.86           calibrated, not a vibe
+result["frustration"].value    # "high"
+result["is_urgent"].value      # True
+result["wants_cancel"].value   # True
+```
+
+One call, one model pass, four typed answers. No prompt engineering, no JSON parsing, no "the model returned prose."
+
+## Use it in Claude Code (MCP server)
+
+poorjev ships an MCP server, so a Claude Code (or Claude Desktop) agent can make
+fast, local, calibrated decisions as tools, with no API key and no token cost.
+The obvious use: gate a risky tool call before the agent runs it.
+
+```bash
+pip install "poorjev[local,mcp]"
+claude mcp add poorjev -- poorjev serve
+```
+
+Or add it to `.mcp.json` by hand:
+
+```json
+{
+  "mcpServers": {
+    "poorjev": { "command": "poorjev", "args": ["serve"] }
+  }
+}
+```
+
+The agent then has these local tools:
+
+| Tool | What it does |
+|---|---|
+| `gate(action)` | guardrail: should this action be blocked (moves money, deletes data)? |
+| `judge(text, statement)` | a yes/no question, with calibrated `P(true)` |
+| `classify(text, options)` | pick one option, with calibrated confidence |
+| `rate(text, levels)` | an ordinal score (low / medium / high) |
+| `decide(text, questions)` | several typed questions at once, one pass |
+
+Why this beats asking an LLM to judge: it is local (private), free (no tokens),
+fast, and the confidence is calibrated instead of made up.
+
+## Why poorjev exists
+
+Most production AI work is not chat. It is fast structured decisions: **route** a ticket, **classify** an intent, **score** a sentiment, **extract** a field, **gate** a tool call. TypeSafe's **Jev** named this category ("System One" models) and nailed the thesis, but Jev is closed, hosted, and behind a waitlist.
+
+poorjev gives you the same developer interface, locally and openly, and it wins on the one thing that actually matters for routing and gating: **confidence you can trust.** A model that is right 78% of the time but *honest about which 78%* is worth more in production than a smarter model that is silently overconfident.
+
+## poorjev vs Jev
+
+| | **Jev** (TypeSafe) | **poorjev** |
+|---|---|---|
+| Interface (typed questions, one pass) | yes | yes |
+| Calibrated confidence | yes (claimed) | **yes (measured, reproducible)** |
+| Schema-valid output, 0 type errors | yes | **yes, by construction** |
+| Runs locally, no API key | no | **yes** |
+| Your data stays in your environment | no | **yes** |
+| Waitlist / signup | yes | **no** |
+| Open source | no | **yes (MIT)** |
+| Speed | very fast (custom model) | slower, honest about it |
+
+poorjev is not a Jev clone and makes no speed claims. It reproduces the **interface** and the **calibrated-confidence guarantee** on commodity models, and proves the calibration with numbers.
+
+## The three primitives
+
+| Primitive | Use it for | Returns |
+|---|---|---|
+| `Choice(options)` | classification, routing | winning option, per-option probabilities, calibrated confidence |
+| `Score(levels)` | ordinal rating, severity | winning level, a continuous score on the scale, confidence |
+| `Noul(statement)` | yes/no gates, guardrails | `P(true)`, thresholded to a bool |
+
+The returned `value` is **always** drawn from the set you declared. An invalid category is structurally impossible, not "usually avoided." This is tested against adversarial inputs (NaN, infinity, negatives, all-zero score vectors).
+
+## How it works
+
+```
+state + typed questions
+        |
+        v
+   one batched pass through a local zero-shot NLI model   (no API key)
+        |
+        v
+   raw probabilities per option
+        |
+        v
+   calibration: temperature scaling + conformal abstention
+        |
+        v
+   typed, schema-valid answers + calibrated confidence
+```
+
+- **Local backend (default):** one small natural-language-inference model scores every option as an entailment hypothesis, in a single batched forward pass. Fully offline after a one-time ~400MB download. No key, no vendor, your text never leaves your machine.
+- **Calibration (the moat):** temperature scaling fits one scalar so predicted confidence matches real accuracy; conformal thresholding turns a target risk budget into an "I don't know, escalate" signal.
+- **LLM backend (optional, roadmap):** when you need more reasoning, point poorjev at an LLM and it makes that model's confidence honest too. That is the intelligence dial, not the default.
+
+## Benchmarks
+
+Reproduce everything with two commands:
+
+```bash
+poorjev eval       --set evalset/tasks.jsonl          # accuracy, ECE, Brier, risk-coverage
+poorjev calibrate  --set evalset/tasks.jsonl --plots  # before/after ECE + the diagrams
+```
+
+On the shipped eval set (55 hand-labelled items, 160 decisions), local NLI backend, keyless:
+
+| Metric | Raw | Calibrated |
+|---|---:|---:|
+| Accuracy | 0.781 | 0.781 |
+| **ECE (calibration error)** | **0.170** | **0.071** |
+| Brier | 0.184 | lower |
+| Temperature | 1.00 | 2.71 |
+
+Temperature is fit by 5-fold cross-validation, so the "after" number is measured on held-out data, never on data it was fit on. Full tables and the honest limitations are in [RESULTS.md](RESULTS.md).
+
+## Selective prediction: it knows when it doesn't know
+
+Set a risk budget and poorjev abstains on its least confident decisions instead of guessing:
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/rupeshpoojary9/poorjev/main/docs/risk_coverage.png" alt="Risk-coverage curve: error rate drops as the model abstains on low-confidence decisions" width="440">
+</p>
+
+At a 10% error budget it confidently answers 55% of decisions and escalates the rest. That is the natural bridge from System One (fast automatic answer) to System Two (a human, or a bigger model).
+
+## Real examples
+
+```bash
+python examples/ticket_router.py   # full triage on a support ticket
+python examples/tool_gate.py       # gate a risky tool call before it runs
+python examples/demo.py            # raw vs calibrated, side by side
+```
+
+The tool-gate example encodes a practical lesson: the local model is strong at **concrete** questions ("this action moves money", "this deletes data") and weak at **abstract** ones ("this is dangerous"). Ask concrete questions and let a one-line rule apply the policy.
+
+## Honest limitations
+
+No hype. Here is what this is not.
+
+- **Not as fast as Jev.** Jev uses a custom model. poorjev uses commodity ones. We report latency, we do not market it.
+- **The eval set is small** (tens of items, one labeller, English, support flavoured). Enough to show calibration direction and schema validity, not a leaderboard.
+- **After-ECE is 0.071, not below 0.05.** That is the real cross-validated number, reported as measured. Per-question temperature would likely push it lower.
+- **The local model is moderately intelligent.** It does real semantic entailment, not deep reasoning. Calibration and abstention are what make that safe.
+
+## FAQ
+
+**Is this a Jev clone?** No. It reproduces Jev's developer interface and its calibrated-confidence guarantee on open, local models. It does not copy Jev's architecture or its speed.
+
+**Do I need an API key or GPU?** No. The default backend runs on CPU, offline, after one model download.
+
+**How is this different from an LLM in JSON mode?** Two ways. Output is schema-valid by construction, not by parsing. And the confidence is calibrated and proven, not a number the model made up.
+
+**What is a "System One" model?** A model for fast, automatic, structured decisions (classify, route, score, gate), as opposed to slow, deliberative chat. The name is from Kahneman's System 1 / System 2.
+
+**What is ECE?** Expected Calibration Error: the average gap between a model's confidence and its actual accuracy. Lower is better. poorjev's whole job is to shrink it.
+
+**Can I use my own model?** Yes. Backends are pluggable; a backend only implements `entail_probs(pairs)`.
+
+## Roadmap
+
+- [x] Typed primitives, schema-valid by construction
+- [x] Local NLI backend, single pass, keyless
+- [x] Eval set + metrics (accuracy, ECE, Brier, risk-coverage)
+- [x] Calibration: temperature scaling + conformal abstention
+- [x] MCP server: use poorjev as local tools in Claude Code
+- [ ] Optional LLM backend (the intelligence dial)
+- [ ] `system-one-bench`: a standalone calibration benchmark for the category
+
+## Contributing
+
+Issues and PRs welcome, especially new labelled decision tasks for the eval set. If you find a case where the confidence is not honest, that is a bug worth filing.
+
+## License
+
+MIT. Use it, ship it, sell it.
+
+---
+
+<p align="center"><i>poorjev: poor in price, rich in honesty. If your model's confidence is a vibe, come check it.</i></p>

poorjev-0.1.0/PRD.md

@@ -0,0 +1,271 @@
+# PRD — `poorjev` (the poor man's Jev)
+
+**An open-source, local-first "System One" decision layer.** Same developer interface as
+Jev (typed questions over program state, answered in one pass with **calibrated**
+probabilities) but backed by small local models or any LLM, and where the one thing we get
+*provably* right is the exact thing Jev sells on: **honest confidence.**
+
+- **Owner:** Rupesh Poojary
+- **Date:** 2026-09-19
+- **Package / repo:** `poorjev` → `pip install poorjev` → `github.com/rupeshpoojary9/poorjev`
+  - *Name is free on PyPI + GitHub as of 2026-09-19. Backups: `poormansjev`, `poorjevai`.*
+- **Status:** 🔴 not started — this PRD is the build spec
+- **Lane:** LLM-evals authority (pairs with `rag-eval-benchmark`, `awesome-llm-evals`). Sister project to a future `system-one-bench`.
+
+---
+
+## 0. The viral wedge (read this first)
+
+**Name:** `poorjev`. It carries the whole story in seven characters: open, cheap, Jev-adjacent, underdog. "Poor man's X" is a badge in dev culture, not an insult, it signals resourceful-hacker "I got 90% of the expensive thing for free." Nobody needs the joke explained; it lands in zero seconds.
+
+**The one spicy, true claim we own:** *every* "System One" project on jevusecases.com (1350+ of them) quotes speed and cost. Almost none prove their **confidence is calibrated**. Jev sells calibrated confidence but gates it behind a waitlist. So the line that spreads is:
+
+> Everyone ships an LLM in JSON mode and calls the `0.9` a "confidence." It's a vibe.
+> `poorjev` is the poor man's Jev, and it's the only one that proves its `0.9` actually means `0.9`. Runs on your laptop. No waitlist.
+
+**The one screenshot that sells it:** a **reliability diagram**, our predicted probability vs. actual accuracy, before and after calibration (ECE ~0.17 → <0.05). That single image is the launch asset. Build toward producing it (see §7, M4).
+
+### Launch kit (draft copy, keep em-dash-free per house style)
+
+**README hero (top of the repo):**
+```
+poorjev
+the poor man's Jev.
+
+Poor in price. Rich in honesty.
+Typed decisions with confidence that is actually calibrated, not vibes.
+Runs on your laptop. No API key. No waitlist.
+
+pip install poorjev
+```
+
+**Launch tweet / X post:**
+```
+Jev is a $40M waitlist.
+
+poorjev is the poor man's version: same typed-question interface, runs
+local with no key, and it is the ONLY one that proves its confidence is
+real (ECE 0.17 -> 0.04, reliability diagram below).
+
+Your model's 0.9 is a vibe. Mine is a measurement.
+
+pip install poorjev
+[reliability-diagram.png]
+```
+
+**Hacker News title:**
+```
+Show HN: poorjev, a local "System One" decision layer that proves its confidence is calibrated
+```
+
+**One-line elevator (for GitHub About + resume):**
+```
+Open, local alternative to Jev's System One API: typed decisions with
+provably calibrated confidence. 100% schema-valid by construction.
+```
+
+**Distribution plan (from [[github-followers-strategy]]: distribution, not more repos):**
+1. Ship the repo with the reliability diagram in the README above the fold.
+2. Submit to jevusecases.com (1350+ dir, the exact audience) as the calibration-first entry.
+3. Show HN + one X thread built around the "your 0.9 is a vibe" line + the diagram.
+4. Add to `awesome-llm-evals` under a new "calibration / selective prediction" section.
+5. Reply-guy value: whenever someone benchmarks Jev on speed, add the calibration axis nobody else measures.
+
+---
+
+## 1. Why this exists
+
+TypeSafe launched **Jev** (Sept 15 2026, $40M DCVC): a "System One model" that takes program
+state + typed questions and returns **typed answers with calibrated probabilities in one
+parallel pass**, 70-500ms, output tokens free, ~0% type errors. The category thesis is
+correct: **most production AI work is fast structured decisions** (route, classify, extract,
+score, moderate, gate a tool call), not chat. But Jev is **gated behind a waitlist**, closed,
+and hosted-only.
+
+The gap: developers want that *interface*, "ask N typed questions about this state, get typed
+answers + trustworthy confidence, cheap and local", today, without a waitlist, and without
+shipping their data to a new vendor. And nobody in the ecosystem is proving the confidence
+half of the promise, which is the half that actually matters for routing and gating.
+
+**What we do NOT claim:** we are not reproducing Jev's non-autoregressive parallel
+architecture or its speed. We reproduce the **interface** and the **calibrated-confidence
+guarantee**, on commodity models, and we prove the calibration with numbers. That honesty is
+the whole pitch, see §9.
+
+## 2. Goal / non-goals
+
+**Goal.** A `pip install poorjev`-able Python library + CLI that:
+1. exposes the three System One primitives (**Choice / Score / Noul**) over arbitrary state,
+2. answers a batch of typed questions in a **single call** with typed, schema-valid outputs,
+3. attaches a **calibrated** probability/confidence to every answer,
+4. supports **abstention** (selective prediction) when confidence is low,
+5. runs **fully local with no API key** by default, and swaps to a real LLM backend with **one env var**,
+6. ships a **self-eval** that reports calibration + accuracy on a small labelled set, so the confidence claim is defensible,
+7. produces the **reliability diagram** from §0 as a committed artifact.
+
+**Non-goals (v1).**
+- Not a hosted API / not a Jev clone weights-wise.
+- No multimodal state (text / JSON / arrays only, matches Jev v1).
+- No training a bespoke model from scratch (we adapt existing encoders + optional LLM).
+- No sub-100ms latency promise. Latency is *reported*, not marketed.
+
+## 3. Users
+
+- App devs who want a cheap, local classify/route/score/gate layer with real confidence.
+- Anyone building LLM-agent guardrails ("is this tool call risky?" as a Noul gate).
+- **You**, in interviews: a working, measured artifact on the freshest model category, in your evals lane. Also a candidate decision layer for Vereno's live-demo loop.
+
+## 4. The interface (the product surface)
+
+Mirror the LangChain/TypeSafe shape closely enough to be a genuine drop-in mental model.
+
+```python
+from poorjev import Client, Choice, Score, Noul
+
+client = Client()                       # local backend, no key, by default
+
+result = client.ask(
+    state="Customer: 'I've emailed three times and STILL been double-charged. Cancel my account.'",
+    questions={
+        "topic":      Choice(["billing", "technical", "account", "other"]),
+        "frustration": Score(levels=["low", "medium", "high"]),
+        "is_urgent":  Noul("The customer needs a response today."),
+        "wants_cancel": Noul("The customer is asking to cancel."),
+    },
+)
+
+result["topic"].value          # -> "billing"
+result["topic"].probs          # -> {"billing": 0.86, "account": 0.09, ...}
+result["topic"].confidence     # -> 0.86   (calibrated)
+result["frustration"].value    # -> "high"
+result["frustration"].score    # -> 1.72   (continuous, on the ordinal scale)
+result["is_urgent"].value      # -> True
+result["is_urgent"].prob       # -> 0.94   (calibrated P(true))
+result["wants_cancel"].abstained  # -> False   (True if below the conformal threshold)
+```
+
+**Primitives (must match Jev semantics):**
+| Primitive | Input | Output |
+|---|---|---|
+| `Choice(options)` | discrete options | winning option + per-option probs + confidence |
+| `Score(levels=[...])` | ordered levels | continuous score on the scale + distribution + confidence |
+| `Noul(statement)` | a yes/no statement | `P(true)` (a bool once thresholded) |
+
+**Contract guarantees (enforced, tested):**
+- Output is always schema-valid, the returned `value` is *always* one of the declared options / within the scale / a bool. No parsing, no "the model returned prose." (This is Jev's "0 type errors" claim, and for us it's true by construction, not by hope, see §5.)
+- All questions in one `ask()` share one state encode and are answered in one pass.
+- Every answer carries a probability; `--calibrate` makes those probabilities calibrated.
+
+## 5. Architecture
+
+```
+                          ┌──────────── backends (pluggable) ────────────┐
+state + questions ─▶ ask() ┤ local  (default): NLI / zero-shot encoder    ├─▶ raw scores
+                          │ llm    (opt-in):  constrained decode + logprobs│
+                          └───────────────────────────────────────────────┘
+                                              │ raw scores
+                          calibration layer ──┤ temperature scaling (fit on dev set)
+                                              │ conformal prediction (abstention sets)
+                                              ▼
+                          typed, schema-valid answers + calibrated confidence
+```
+
+**5.1 Local backend (default, no key), the clever bit.**
+Reuse an NLI / zero-shot model as a general decision engine. One small model does all three:
+- **Noul** → natural-language inference: `P(entailment)` of the statement given the state = `P(true)`.
+- **Choice** → zero-shot classification: score each option as an entailment hypothesis, softmax over options.
+- **Score** → run the ordered levels as hypotheses, take the expected level index as the continuous score, argmax→winning level.
+
+Default model: a small NLI checkpoint (e.g. `MoritzLaurer/deberta-v3-base` zero-shot or `all-MiniLM` + a cross-encoder). ~single-digit hundred MB, downloads once, then fully offline. **Schema validity is structural**, options/levels are fixed sets we softmax over, so an out-of-set answer is impossible by construction.
+
+**5.2 LLM backend (opt-in, `POORJEV_BACKEND=llm` + key).**
+Any chat LLM (Anthropic default; see `claude-api` for model IDs). Use **constrained/structured output** so the raw answer is always valid, and pull token **logprobs** for the probability. LLMs are notoriously *miscalibrated* (over-confident verbalized probs), so this backend leans hardest on §5.3 to fix that, which is itself a demonstrable result.
+
+**5.3 Calibration layer (the differentiator, backend-agnostic).**
+This is what makes us more than "an LLM in JSON mode":
+- **Temperature scaling**: fit one scalar T per question-type on a small labelled dev split so predicted probs match empirical accuracy. Report **before/after ECE**.
+- **Conformal prediction**: turn a target risk level (e.g. 90%) into a per-question confidence threshold; below it, `abstained=True`. Gives honest "I don't know → escalate to System Two" behaviour, the natural bridge to a router.
+
+**5.4 Single-pass batching.** Encode state once; evaluate all questions against that encoding in one batched forward pass (local) or one structured request (LLM). No per-question round-trips.
+
+## 6. Repo layout (matches your `rag-eval-benchmark` conventions)
+
+```
+poorjev/
+├── README.md                 # the §0 hero + reliability diagram above the fold, quickstart, honest claims
+├── RESULTS.md                # real calibration + accuracy tables, reliability diagrams
+├── PRD.md                    # this file
+├── LICENSE                   # MIT
+├── pyproject.toml            # hatchling; extras: [llm], [plots], [dev]; script: poorjev=poorjev.cli:main
+├── src/poorjev/
+│   ├── __init__.py           # exports Client, Choice, Score, Noul
+│   ├── primitives.py         # the three question types + typed Answer objects
+│   ├── client.py             # ask(): orchestrates encode → backend → calibrate → type
+│   ├── backends/
+│   │   ├── local_nli.py      # default no-key backend
+│   │   └── llm.py            # opt-in Anthropic/structured-output backend
+│   ├── calibration.py        # temperature scaling + conformal prediction
+│   ├── metrics.py            # ECE, Brier, accuracy, coverage/risk
+│   ├── plots.py              # reliability diagrams, risk-coverage curves (THE launch asset)
+│   └── cli.py                # poorjev ask / eval / calibrate
+├── evalset/                  # small hand-labelled decision set (see §7)
+│   ├── tasks.jsonl           # state + questions + gold answers
+│   └── README.md             # how it was labelled
+├── examples/
+│   ├── ticket_router.py      # Choice+Score+Noul over support tickets
+│   └── tool_gate.py          # Noul guardrail: "is this tool call risky?" (agent middleware)
+└── tests/                    # schema-validity invariants, calibration math, primitives
+```
+
+**CLI:**
+```bash
+poorjev ask --state-file ticket.txt --questions questions.yaml     # one-off decision, JSON out
+poorjev eval  --set evalset/tasks.jsonl --plots                    # accuracy + ECE + Brier + coverage + reliability.png
+poorjev calibrate --set evalset/tasks.jsonl                        # fit + save temperature/conformal params
+poorjev eval  --backend llm --plots                                # same, LLM backend (needs key)
+```
+
+## 7. Eval set (the part that makes it credible)
+
+Hand-label a **small, honest** decision set, the same discipline that made `rag-eval-benchmark`
+defensible. ~60-100 items across a few realistic decision tasks (support-ticket triage,
+content moderation flags, intent detection, tool-risk gating). Each item = state + the typed
+questions + **gold answers**. This is the hard, valuable part most write-ups skip; it's what
+lets us report calibration honestly and it's the seed corpus for the future `system-one-bench`.
+
+## 8. Success metrics / definition of done
+
+v1 ships when, on the local backend over the eval set, `poorjev eval` reports real numbers and:
+
+- 🎯 **Schema validity = 100%** by construction (invariant test: no `ask()` ever returns an out-of-set/wrong-type value, incl. adversarial states). This is our honest version of Jev's "0 type errors."
+- 🎯 **Calibration improves measurably** post-temperature-scaling: report **ECE before → after** and **Brier**; target a clear ECE reduction (e.g. ~0.15 → <0.05), the exact number is whatever we honestly measure.
+- 🎯 **The reliability diagram exists** and is committed to the repo + README (the §0 launch asset).
+- 🎯 **Selective prediction works**: a risk-coverage curve showing accuracy rises as we abstain on low-confidence items.
+- 🎯 **Accuracy is competitive** with an "LLM-in-JSON-mode" baseline while being local/free, and where it loses, we say so.
+- 🎯 **Latency + cost reported** (not marketed) for local vs LLM backend.
+- 🎯 Runs **offline, no key**, first-run downloads one small model; `pip install -e ".[dev]" && poorjev eval` reproduces the tables.
+- 🎯 README + RESULTS.md written; tests green; MIT; pushed public.
+
+## 9. Honest-claims guardrails (non-negotiable, your no-hype rule)
+
+- Never claim to match Jev's speed or architecture. Frame explicitly as: *interface-compatible + honestly-calibrated, on commodity models.*
+- The name is playful; the numbers are not. Every headline number in README/RESULTS is one `poorjev eval` produces on the shipped eval set. No cherry-picking, no vibes.
+- "Poor man's" is positioning, never an excuse for a weak result. If a number is bad, we print it.
+- State the eval set is small and name what that does/doesn't prove.
+- No em-dashes in any rendered marketing/README/tweet copy (commas/colons). *(This PRD is internal, so dashes here are fine.)*
+
+## 10. Milestones (suggested build order)
+
+1. **M1 — Contract & primitives.** `primitives.py`, typed `Answer` objects, schema-validity invariant tests. *(Prove the "0 type errors" claim first.)*
+2. **M2 — Local backend.** NLI/zero-shot engine for all three primitives; `ask()` single-pass. First end-to-end decision. *(Study `openjev-sglang` first, differentiate on calibration not speed.)*
+3. **M3 — Eval set + metrics.** Hand-label `tasks.jsonl`; implement ECE/Brier/accuracy/coverage; `poorjev eval` raw (uncalibrated) numbers.
+4. **M4 — Calibration + the money screenshot.** Temperature scaling + conformal abstention; before/after ECE; **reliability diagram + risk-coverage plot**. *(This is the differentiator AND the launch asset, don't skip to polish before it works.)*
+5. **M5 — LLM backend.** Opt-in Anthropic/structured-output + logprobs; show it's miscalibrated raw and fixed by M4.
+6. **M6 — Launch.** `ticket_router.py`, `tool_gate.py`, honest write-up, tests green, push public, then run the §0 distribution plan (jevusecases.com + Show HN + X thread + awesome-llm-evals).
+
+## 11. Stretch / follow-ons
+
+- **`system-one-bench`**: promote `evalset/` into the standalone benchmark. Score `poorjev`, an LLM baseline, and (when access lands) Jev, on accuracy + calibration + latency/cost. You'd own the yardstick for the category.
+- **System-One/System-Two router** built on conformal abstention.
+- **Vereno**: wire `poorjev` as the sub-500ms decision loop in the live-demo console.
+- LangChain-shim so `poorjev` drops into the `TypeSafeClassifier` call site.