sooth 0.2.1__tar.gz

sooth-0.2.1/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,23 @@
+---
+name: Bug report
+about: Something is broken or a verdict is wrong
+labels: bug
+---
+
+**What happened**
+
+**What you expected**
+
+**Repro**
+
+```bash
+sooth --source ... --text ...
+```
+
+(Attach the draft/source snippets if shareable — remove secrets first.)
+
+**Environment**
+
+- sooth version:
+- Python version:
+- OS:

sooth-0.2.1/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,11 @@
+---
+name: Feature request
+about: Propose an improvement
+labels: enhancement
+---
+
+**Problem it solves**
+
+**Proposed behavior**
+
+**Alternatives you considered**

sooth-0.2.1/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,9 @@
+## What
+
+## Why
+
+## Checklist
+
+- [ ] `python3 tests/test_core.py` passes
+- [ ] New logic covered by an assert in `tests/test_core.py`
+- [ ] Docs updated if behavior changed (`docs/PRD.md`, `docs/DESIGN.md`, README)

sooth-0.2.1/.github/workflows/ci.yml

@@ -0,0 +1,21 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: ruff check src tests
+      - run: pytest

sooth-0.2.1/.gitignore

@@ -0,0 +1,10 @@
+__pycache__/
+.venv/
+build/
+dist/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.env
+decisions.jsonl
+.DS_Store

sooth-0.2.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Naufal Hilmiaji
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sooth-0.2.1/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.5
+Name: sooth
+Version: 0.2.1
+Summary: Verify AI-generated text against source material. Claim-by-claim trust reports powered by Jev (TypeSafe System One).
+Project-URL: Homepage, https://github.com/naufalhilmiaji/sooth
+Project-URL: Documentation, https://github.com/naufalhilmiaji/sooth/tree/main/docs
+Author-email: Naufal Hilmiaji <nhilmiaji@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: ai,jev,llm,trust,typesafe,verification
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.11
+Requires-Dist: typesafe-sdk
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Sooth
+
+Verify AI-generated text against source material. Claim-by-claim **PASS / FAIL / REVIEW**, with calibrated probabilities you can act on in CI.
+
+Built on [Jev](https://docs.typesafe.ai) (TypeSafe System One) — typed judgments and probabilities instead of generated prose. Every verdict shows its probability distribution, and the combination rules are plain code you can read. No black box.
+
+```
+sooth --source policy.md --text draft-reply.md
+```
+
+Real output (planted errors in the draft vs a news source):
+
+```
+# Sooth
+
+**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70
+
+| # | Claim | Verdict | P | Why (P distribution) | Evidence |
+|---|-------|---------|---|----------------------|----------|
+| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
+| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
+| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham. | ⚠️ REVIEW | ████████ 1.00 | contradicts 0.00 / not_found 1.00 / supports 0.00 · details 0.02 · missing #s: 5.000 |  |
+| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.24 | P(checkable)=0.24 |  |
+```
+
+## Why
+
+AI writes fast, nobody checks. Claims ship wrong. Sooth checks each claim against the evidence you give it — and when it is unsure, it says `REVIEW` instead of guessing.
+
+## Install
+
+```bash
+pip install sooth          # or: pip install -e ".[dev]" from source
+export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
+```
+
+## Usage
+
+```bash
+sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
+
+# CI-friendly exit codes
+sooth --source policy.md --text draft.md --format plain
+#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
+
+# Options
+#   --confidence T   REVIEW below this confidence (default 0.7)
+#   --format md|plain
+#   -o FILE          write report to file
+#   --log FILE       append full judgment trace (one JSONL line per run)
+```
+
+## How it works
+
+1. Draft is split into claims (one sentence each).
+2. Each claim gets three questions to Jev, all fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), and *do all details match exactly?*
+3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
+4. The report shows the full probability distribution per claim — not just a label.
+
+The decision logic lives in [`src/sooth/verify.py`](src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
+
+## Known limits (alpha)
+
+- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
+- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
+- Source + questions must fit ~64k tokens — split long documents yourself.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python3 tests/test_core.py     # pure checks, no network
+pytest                         # same suite
+bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
+PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
+```
+
+Docs: [PRD](docs/PRD.md) · [Design](docs/DESIGN.md) · [Testing](docs/TESTING.md)
+
+## Roadmap
+
+- v0.2 shipped: source-span evidence (the exact source line behind each verdict, shown in the report)
+- later: hosted web app — paste UI, history, team review queues
+
+## License
+
+MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.

sooth-0.2.1/README.md

@@ -0,0 +1,87 @@
+# Sooth
+
+Verify AI-generated text against source material. Claim-by-claim **PASS / FAIL / REVIEW**, with calibrated probabilities you can act on in CI.
+
+Built on [Jev](https://docs.typesafe.ai) (TypeSafe System One) — typed judgments and probabilities instead of generated prose. Every verdict shows its probability distribution, and the combination rules are plain code you can read. No black box.
+
+```
+sooth --source policy.md --text draft-reply.md
+```
+
+Real output (planted errors in the draft vs a news source):
+
+```
+# Sooth
+
+**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70
+
+| # | Claim | Verdict | P | Why (P distribution) | Evidence |
+|---|-------|---------|---|----------------------|----------|
+| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
+| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
+| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham. | ⚠️ REVIEW | ████████ 1.00 | contradicts 0.00 / not_found 1.00 / supports 0.00 · details 0.02 · missing #s: 5.000 |  |
+| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.24 | P(checkable)=0.24 |  |
+```
+
+## Why
+
+AI writes fast, nobody checks. Claims ship wrong. Sooth checks each claim against the evidence you give it — and when it is unsure, it says `REVIEW` instead of guessing.
+
+## Install
+
+```bash
+pip install sooth          # or: pip install -e ".[dev]" from source
+export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
+```
+
+## Usage
+
+```bash
+sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
+
+# CI-friendly exit codes
+sooth --source policy.md --text draft.md --format plain
+#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
+
+# Options
+#   --confidence T   REVIEW below this confidence (default 0.7)
+#   --format md|plain
+#   -o FILE          write report to file
+#   --log FILE       append full judgment trace (one JSONL line per run)
+```
+
+## How it works
+
+1. Draft is split into claims (one sentence each).
+2. Each claim gets three questions to Jev, all fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), and *do all details match exactly?*
+3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
+4. The report shows the full probability distribution per claim — not just a label.
+
+The decision logic lives in [`src/sooth/verify.py`](src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
+
+## Known limits (alpha)
+
+- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
+- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
+- Source + questions must fit ~64k tokens — split long documents yourself.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python3 tests/test_core.py     # pure checks, no network
+pytest                         # same suite
+bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
+PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
+```
+
+Docs: [PRD](docs/PRD.md) · [Design](docs/DESIGN.md) · [Testing](docs/TESTING.md)
+
+## Roadmap
+
+- v0.2 shipped: source-span evidence (the exact source line behind each verdict, shown in the report)
+- later: hosted web app — paste UI, history, team review queues
+
+## License
+
+MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.

sooth-0.2.1/docs/DESIGN.md

@@ -0,0 +1,159 @@
+# Technical Design — Sooth v0.1
+
+Companion to `PRD.md`. Stack and data flow only. No speculative layers.
+
+## Stack
+
+- Python 3.11+, official `typesafe-sdk` package (`TypeSafeClient.system_one`), stdlib otherwise (argparse, re, json, pathlib).
+- No web framework, no DB, no plugin system. Four modules + CLI entry is enough.
+
+## Layout (src layout, standard packaging)
+
+```
+src/sooth/
+  __init__.py   # package API
+  claims.py     # split text → claims (pure)
+  verify.py     # build Jev questions, call API, map answers → verdicts (pure map + one I/O call)
+  report.py     # verdicts → markdown / plain text + JSONL log record (pure)
+  cli.py        # argparse, file I/O, exit codes, --log writer
+tests/
+  conftest.py
+  test_core.py  # asserts for pure functions + verdict mapping
+  smoke.sh      # optional live smoke (real key)
+examples/       # fixture: source.md, draft.md
+docs/           # PRD, DESIGN, TESTING
+```
+
+## Flow
+
+```
+sources + draft
+   → claims.split(draft)            → list[Claim]
+   → verify.batch(claims, sources)  → list[Verdict]     # 1..n Jev calls
+   → report.render(verdicts)        → markdown string
+   → stdout / -o file
+   → optional JSONL log append
+```
+
+Pure functions everywhere except `verify.call_jev()` and file I/O. Testable without network.
+
+## Claim splitting (`claims.py`)
+
+v0.1: sentence splitter on `.`/`!`/`?` + newline, keeping numbers intact (`3.5 days`). Drop empties and pure questions? No — questions are claims too ("Does it support X?" is checkable as written intent… actually drop interrogatives and headings, they are not claims). Rules:
+
+- Keep: declarative sentences, ≥ 4 words.
+- Skip: headings (`# …`), list bullets' leading markers kept as text, sentences < 4 words, lines that end with `?`.
+- Each claim keeps `text` + `source_span` = `{file, line}` of origin (for "claim #3 came from draft line 12").
+
+```python
+# ponytail: regex splitter, mis-splits quotes/abbreviations; upgrade to clause-level when real drafts demand
+```
+
+## Jev request (`verify.py`)
+
+One request per batch of ≤ 30 claims. State is shared; questions reference claims by path.
+
+```jsonc
+// state
+{
+  "sources": [
+    { "name": "policy.md", "text": "…" },          // concatenated if many; name kept for evidence line
+    { "name": "tickets/t123.md", "text": "…" }
+  ],
+  "claims": [
+    { "id": "c1", "text": "Refunds are processed in 3 days." },
+    { "id": "c2", "text": "We support Bitcoin." }
+  ]
+}
+```
+
+Per claim, **four parallel questions** (fan-out pattern; statement text is embedded in the instructions):
+
+```jsonc
+"c1_checkable": {
+  "type": "noul",
+  "instructions": "Statement: <claim text>\n\nIs the statement a concrete factual claim that the evidence in `sources` could support or contradict? ...",
+  // noul answer = P(checkable)
+},
+"c1_verdict": {
+  "type": "choice",
+  "instructions": "Statement: <claim text>\n\nDoes the evidence in `sources` support the statement?",
+  "criteria": {
+    "supports":    "...",
+    "contradicts": "...",
+    "not_found":   "..."
+  }
+  // choice + probabilities + confidence
+},
+"c1_details": {
+  "type": "noul",
+  "instructions": "Statement: <claim text>\n\nDoes EVERY specific detail — names, numbers, dates, comparisons such as 'more than' or 'about' — exactly match the evidence in `sources`? ...",
+  // noul answer = P(all details match)
+},
+"c1_evidence": {
+  "type": "choice",
+  "instructions": "Statement: <claim text>\n\nWhich candidate segment best supports or contradicts the statement? Full text of each id is in `segments`. Choose 'none' if no segment is relevant.",
+  "criteria": { "s12": "<first 80 chars of candidate>", "...": "…", "none": "No segment is relevant to the statement." }
+  // code pre-filters ~6 candidate ids per claim (word overlap + number hits)
+}
+```
+
+- 30 claims → 120 questions, one call. Over batch cap or 422 → split and retry half (SDK retries 429/529 already).
+- Pin model `jev-1.13.0` (thresholds tuned against it). Constant in `verify.py`.
+- API key: env `TYPESAFE_API_KEY`. Missing → exit 3 with one-line hint.
+
+## Verdict mapping (pure)
+
+```
+if p_checkable < 0.5:              → UNCHECKABLE
+elif confidence < threshold:       → REVIEW
+elif choice == "supports":         → PASS
+elif choice == "contradicts":      → FAIL
+else:                              → REVIEW          # not_found
+```
+
+Then safeguards (`apply_safeguards`, pure) — demote `PASS` → `REVIEW` when:
+
+- `details_p < 0.5` (detail-gate Noul says some detail drifted), or
+- `missing_numbers(claim, sources)` non-empty — claim numbers absent from every source (regex extraction, separator-normalized; no model involved).
+
+FAIL/REVIEW/UNCHECKABLE pass through untouched.
+
+`Verdict = {claim_id, claim_text, draft_line, kind, p_checkable?, choice?, probabilities, confidence, details_p?, missing_numbers}`
+
+Evidence (v0.2, pre-parsed selection pattern): sources are split into sentence `segments`; code ranks ~6 candidates per claim (word overlap, numbers weighted ×3); a per-claim Choice selects the best span (`none` allowed). Report shows `source:line` + snippet. Best-effort — `none` is valid when no span matches.
+
+(If PRD table showed source snippets — that is v0.2. v0.1 report substitutes distribution; PRD example is target UX.)
+
+## Report (`report.py`)
+
+Markdown: summary line (`PASS n · FAIL n · REVIEW n · UNCHECKABLE n`), then one table as in PRD, ASCII probability bar in P column (`██████░░ 0.91` is enough; no unicode-only requirement). FAIL rows first? No — keep draft order, print summary counts first. `REVIEW` and `UNCHECKABLE` share a section below the table if any.
+
+## Decision log (`--log`)
+
+Append-only JSONL, one object per run (not per claim — one line, pretty fields):
+
+```json
+{"ts": "…", "model": "jev-1.13.0", "threshold": 0.7, "sources": ["policy.md"], "draft": "draft-reply.md", "usage": {"input_tokens": 0, "output_tokens": 0}, "results": [ …verdicts with full probabilities… ]}
+```
+
+This is the decision-ledger seed (audit trail). No viewer in v0.1.
+
+## Exit codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | all PASS or UNCHECKABLE only |
+| 1 | any FAIL |
+| 2 | any REVIEW (no FAIL) |
+| 3 | usage / missing key / unreadable file |
+
+## Errors
+
+- SDK raises typed errors; catch at CLI boundary, print one line to stderr, exit 3 (or 2 on API failure mid-run? → exit 3, "could not verify", never print a partial report marked complete).
+- Empty draft → usage error. Empty sources → usage error (nothing to check against).
+
+## Security
+
+- Key via env only, never logged, never in `--log`.
+- Files read as UTF-8 text. No shell-out, no eval. `--log` path user-controlled write (documented).

sooth-0.2.1/docs/DOGFOOD.md

@@ -0,0 +1,18 @@
+# Dogfood log — week of 2026-09-25
+
+Run Sooth on your own real AI output before building anything new. One row per session. Findings here drive Phase 4/5 decisions.
+
+## Loop (5 min per draft)
+
+1. Take an AI draft you were about to trust (reply, summary, PR description, doc).
+2. `sooth --source <the ground truth> --text <the draft> --log dogfood.jsonl`
+3. Read the report. Mark each verdict right/wrong by your own eyes.
+4. Log one row below.
+
+| Date | Draft type | Verdicts right/wrong | Surprise (if any) | Want next |
+|------|------------|----------------------|-------------------|-----------|
+|      |            |                      |                   |           |
+
+## Question this answers
+
+Does anyone actually paste ground truth + draft and act on the report? If the tool dies on real drafts (formatting, length, language), Phase 4 chunking/URL input jumps the queue. If the report is ignored even when right, the web app idea is wrong.

sooth-0.2.1/docs/PRD.md

@@ -0,0 +1,90 @@
+# PRD — Sooth (v0.1)
+
+## Problem
+
+AI writes text fast. Nobody checks it. Wrong claims ship. In 2026 most drafts (support replies, docs, reports, PR descriptions) are AI-written and reviewed by skimming humans.
+
+Sooth is a cheap, automatic checker: does each claim in the AI draft actually follow from the source material?
+
+It targets the AI-era verification gap: AI writes fast, nobody checks before shipping. Opaque AI decisions are addressed by design: every verdict shows its probability distribution, and `--log` records the full judgment trace.
+
+## What it is
+
+CLI tool. Input: source text(s) + AI-generated draft. Output: a **trust report** — every claim marked `PASS` / `FAIL` / `REVIEW`, with probability, reason, and source snippet.
+
+Engine by [Jev](https://docs.typesafe.ai) (TypeSafe System One): typed judgments + calibrated probabilities, not generated prose.
+
+## Who it is for (v0.1)
+
+Developers and technical teams who already work in a terminal and pipe AI output into things: CI jobs, agent pipelines, release notes, support macros.
+
+## User story
+
+```
+sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
+```
+
+1. User gives sources (ground truth) and draft (AI text to verify).
+2. Tool splits draft into claims (sentence per claim in v0.1).
+3. Each claim judged against sources: supported / contradicted / not found — plus "is this even checkable?".
+4. Report printed as markdown. Low-confidence claims land in `REVIEW` instead of fake-precise verdicts.
+5. Optional `--log decisions.jsonl` appends full judgment trace (audit trail seed).
+
+## Report shape (contract)
+
+| # | Claim | Verdict | P | Why (v0.1) |
+|---|-------|---------|---|------------|
+| 1 | Refunds are processed in 3 days. | ✅ PASS | 0.91 | supports .91 / contradicts .02 / not_found .07 |
+| 2 | We support Bitcoin. | ❌ FAIL | 0.87 | supports .03 / contradicts .87 / not_found .10 |
+| 3 | Customers love the change. | ⚠️ REVIEW | 0.54 | not found in sources |
+
+"Why" = probability distribution. Evidence = the source span the model selected (`source:line` + snippet).
+
+Verdict rules (fixed, readable in code):
+
+- claim not checkable (opinion, vague) → `UNCHECKABLE` (listed under REVIEW section)
+- `supports` + confidence ≥ threshold → `PASS`
+- `contradicts` + confidence ≥ threshold → `FAIL`
+- `not in source` **or** confidence < threshold → `REVIEW`
+
+Default threshold 0.7, flag `--confidence`.
+
+## MVP scope
+
+In:
+
+- CLI: `sooth --source … [--source …] --text … [--confidence T] [--log FILE] [--format md|plain]`
+- Plain text / markdown input (UTF-8). Multiple sources.
+- Sentence-per-claim splitting.
+- Jev fan-out verification (batched).
+- Markdown report to stdout (or `-o FILE`).
+- JSONL decision log (`--log`).
+- Exit codes for CI: `0` all PASS/CLEAN, `1` any FAIL, `2` any REVIEW remaining, `3` usage/config error.
+
+Out (v0.1 — later or never):
+
+- Web UI (paid phase), PDF/HTML parsing, images
+- Smarter claim splitting (clauses, bullet merging)
+- Team features, history, webhooks
+- Non-English tuning (Jev is English-strongest)
+- Custom question editing UI
+
+## Success criteria (v0.1 done when)
+
+1. `sooth` runs end-to-end against a sample fixture and prints the report above.
+2. On a hand-labeled set of 30 claims (10 supported / 10 contradicted / 10 absent-or-fluff): ≥ 24 correct verdicts after threshold tuning, no confident (≥0.7) wrong `PASS` on a contradicted claim in the set.
+3. CI-usable: exit codes behave; works without network when `--log`-replaying a fixture later (replay is stretch).
+4. One new user understands the output with no explanation beyond `--help`.
+
+## Monetization (later — not built now)
+
+Free OSS: CLI + library. Paid web app later: paste UI, history, team folders, "verify before publish" API. Same engine. No paywall logic in v0.1.
+
+## Open decisions (defaults chosen, reversible)
+
+| Decision | Default | Note |
+|----------|---------|------|
+| Language | Python 3.11+ | official Jev Python SDK |
+| Claim split | sentence regex | upgrade when it mis-splits real drafts |
+| Batch size | 30 claims/request | stay well under 64k token cap |
+| REVIEW exit code | 2 | CI can choose to ignore |

sooth-0.2.1/docs/TESTING.md

@@ -0,0 +1,51 @@
+# Tests & Verification Plan — Sooth v0.1
+
+Three layers: fast pure checks (always), live API smoke (optional), accuracy calibration (human, before release). Matches `DESIGN.md` module split.
+
+## 1. Unit / pure checks — `tests/test_core.py`
+
+One file, plain asserts, run with `python -m tests.test_core` or pytest if already installed. No network. Covers the logic that would silently corrupt verdicts:
+
+| Area | Cases (assert each) |
+|------|---------------------|
+| `claims.split` | sentence boundaries; `3.5 days` stays one claim; headings skipped; `?` lines skipped; `< 4` words skipped; `draft_line` correct |
+| verdict mapping | checkable<0.5 → UNCHECKABLE; low confidence → REVIEW; supports/contradicts/not_found → PASS/FAIL/REVIEW at high confidence; threshold boundary inclusive |
+| `report.render` | counts line correct; every claim appears once; bar length ∝ probability; empty verdicts list → still prints summary |
+| log writer | one JSON line per run; round-trips `json.loads`; no `api_key` field anywhere |
+
+Entry bar: `python -m tests.test_core` exits 0 on a clean tree. This is the one runnable check left behind with the code.
+
+## 2. Live smoke (optional, real key)
+
+- Fixture `examples/`: `source.md` (short policy: refunds in 3 days, card-and-bank only), `draft.md` (3 claims: one supported, one contradicted, one fluff).
+- `sooth --source examples/source.md --text examples/draft.md`
+- Pass when: process exits with code matching expected verdicts (1 FAIL present → exit 1), report table has 3 rows, `--log /tmp/t.jsonl` has exactly 1 line with `model` = pinned version.
+- Script it as `tests/smoke.sh` (5 lines). Not run in unit CI. Skip entirely when `TYPESAFE_API_KEY` unset.
+
+## 3. Accuracy calibration (live, labeled set)
+
+Goal from PRD: ≥ 24/30 correct on a labeled set, zero confident-wrong `PASS` on contradicted claims.
+
+- Labeled set: `examples/calibration.json` — 30 claims across `news-1.md`, `policy.md`, `pricing.md` (10 supported / 10 contradicted / 10 absent-or-opinion, each with an `expected` verdict and a note).
+- Runner: `PYTHONPATH=src python3 tests/calibrate.py [--confidence T]` — batches live Jev calls, prints the confusion matrix, exits 0 only when the bar is met.
+- Frozen matrix + notes: `examples/calibration.md` (threshold 0.7, `jev-1.13.0`, 30/30).
+- Never move the threshold to hide a confident-wrong `PASS` — fix question wording instead (`instructions`/`criteria` in `verify.py`). Bump model version → re-run this layer.
+
+## 4. Manual UX check (10 min)
+
+- [ ] `sooth --help` alone explains enough to run.
+- [ ] Report readable by someone who never saw the tool (show one person).
+- [ ] Missing key → one-line hint mentioning `TYPESAFE_API_KEY`, exit 3, no stack trace.
+- [ ] Exit codes: all-PASS fixture → 0; with-REVIEW fixture → 2; with-FAIL → 1.
+
+## 5. Definition of done (v0.1)
+
+1. `tests/test_core.py` green.
+2. Smoke fixture behaves (§2) with a real key at least once.
+3. Calibration table recorded (§3) and default threshold justified by it.
+4. Manual checklist (§4) done.
+5. Docs match reality: report column that shows probability distribution (not source snippets) — snippets deferred to v0.2 per `DESIGN.md`.
+
+## Out of scope for testing now
+
+Load/perf (CLI is one-shot), fuzzing, multi-language accuracy, regression CI on live API (costs money; add when thresholds ship to users).