sooth 0.2.1__tar.gz → 0.2.2__tar.gz

{sooth-0.2.1 → sooth-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: sooth
-Version: 0.2.1
+Version: 0.2.2
 Summary: Verify AI-generated text against source material. Claim-by-claim trust reports powered by Jev (TypeSafe System One).
 Project-URL: Homepage, https://github.com/naufalhilmiaji/sooth
 Project-URL: Documentation, https://github.com/naufalhilmiaji/sooth/tree/main/docs
@@ -23,15 +23,23 @@ Description-Content-Type: text/markdown
 
 # Sooth
 
-Verify AI-generated text against source material. Claim-by-claim **PASS / FAIL / REVIEW**, with calibrated probabilities you can act on in CI.
+**LLMs generate. Sooth verifies.**
 
-Built on [Jev](https://docs.typesafe.ai) (TypeSafe System One) — typed judgments and probabilities instead of generated prose. Every verdict shows its probability distribution, and the combination rules are plain code you can read. No black box.
+Claim-by-claim fact-checking for AI output, designed for CI. Sooth checks every sentence in a draft against your source material and returns `PASS` / `FAIL` / `REVIEW` with calibrated probabilities and the exact source line behind each verdict.
 
-```
-sooth --source policy.md --text draft-reply.md
+[![PyPI](https://img.shields.io/pypi/v/sooth.svg)](https://pypi.org/project/sooth/)
+[![Python](https://img.shields.io/pypi/pyversions/sooth.svg)](https://pypi.org/project/sooth/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/naufalhilmiaji/sooth/blob/main/LICENSE)
+[![CI](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml/badge.svg)](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml)
+
+## See it catch a hallucination
+
+```bash
+pip install sooth
+sooth demo        # no API key needed
 ```
 
-Real output (planted errors in the draft vs a news source):
+Real output — seven claims in an AI-written news summary, three numbers quietly wrong:
 
 ```
 # Sooth
@@ -46,14 +54,18 @@ Real output (planted errors in the draft vs a news source):
 | 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.24 | P(checkable)=0.24 |  |
 ```
 
+Every verdict carries its probability distribution and the source span it was judged against. When Sooth is unsure, it says `REVIEW` instead of guessing.
+
 ## Why
 
-AI writes fast, nobody checks. Claims ship wrong. Sooth checks each claim against the evidence you give it — and when it is unsure, it says `REVIEW` instead of guessing.
+AI writes fast, nobody checks. Claims ship wrong — usually a number, a hedge, or a name that drifted. Asking another LLM "is this right?" just produces more prose.
+
+Sooth takes a different bet: **the model makes typed judgments, code makes the decision.** Every rule that turns a probability into a verdict is a readable line of Python, not a prompt.
 
 ## Install
 
 ```bash
-pip install sooth          # or: pip install -e ".[dev]" from source
+pip install sooth
 export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
 ```
 
@@ -73,14 +85,49 @@ sooth --source policy.md --text draft.md --format plain
 #   --log FILE       append full judgment trace (one JSONL line per run)
 ```
 
+Wire it into CI as a quality gate for generated content:
+
+```yaml
+- name: Verify AI output
+  run: sooth --source docs/policy.md --text generated-reply.md
+  env:
+    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+```
+
 ## How it works
 
 1. Draft is split into claims (one sentence each).
-2. Each claim gets three questions to Jev, all fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), and *do all details match exactly?*
+2. Each claim gets four questions to Jev, fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), *do all details match exactly?*, and *which source span is the evidence?*
 3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
 4. The report shows the full probability distribution per claim — not just a label.
 
-The decision logic lives in [`src/sooth/verify.py`](src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
+The decision logic lives in [`src/sooth/verify.py`](https://github.com/naufalhilmiaji/sooth/blob/main/src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
+
+Judgments come from [Jev](https://docs.typesafe.ai) (TypeSafe System One), which returns typed distributions instead of generated prose.
+
+## Why not just use an LLM judge?
+
+Not claiming Sooth is universally more accurate. Claiming it gives you **architectural guarantees** a prompt cannot:
+
+| | Sooth | Generic LLM judge |
+|---|-------|-------------------|
+| Claim-level verification | ✅ | Sometimes |
+| Source span behind each verdict | ✅ | Sometimes |
+| `PASS` / `FAIL` / `REVIEW` | ✅ fixed vocabulary | Generated labels |
+| Probability distribution | ✅ | Usually none |
+| Verdict rules in readable code | ✅ | Prompt-dependent |
+| CI exit codes | ✅ | ❌ |
+| Audit log (`--log` JSONL) | ✅ | DIY |
+
+## Who it's for
+
+Developers building systems that generate text from trusted material:
+
+- **RAG pipelines** — retrieved the right document, still wrote the wrong number
+- **AI agents** — before an agent's reply reaches a human
+- **Support / ops copilots** — policy-backed answers, checked against the policy
+- **Report & summary generation** — figures that must match the source
+- **Compliance workflows** — evidence trail for every claim, not vibes
 
 ## Known limits (alpha)
 
@@ -98,11 +145,13 @@ bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
 PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
 ```
 
-Docs: [PRD](docs/PRD.md) · [Design](docs/DESIGN.md) · [Testing](docs/TESTING.md)
+Docs: [PRD](https://github.com/naufalhilmiaji/sooth/blob/main/docs/PRD.md) · [Design](https://github.com/naufalhilmiaji/sooth/blob/main/docs/DESIGN.md) · [Testing](https://github.com/naufalhilmiaji/sooth/blob/main/docs/TESTING.md)
 
 ## Roadmap
 
-- v0.2 shipped: source-span evidence (the exact source line behind each verdict, shown in the report)
+- v0.2.2 shipped: `sooth demo` (offline, no API key)
+- v0.2.1 shipped: source-span evidence, published on PyPI
+- GitHub Action
 - later: hosted web app — paste UI, history, team review queues
 
 ## License

sooth-0.2.2/README.md

@@ -0,0 +1,136 @@
+# Sooth
+
+**LLMs generate. Sooth verifies.**
+
+Claim-by-claim fact-checking for AI output, designed for CI. Sooth checks every sentence in a draft against your source material and returns `PASS` / `FAIL` / `REVIEW` with calibrated probabilities and the exact source line behind each verdict.
+
+[![PyPI](https://img.shields.io/pypi/v/sooth.svg)](https://pypi.org/project/sooth/)
+[![Python](https://img.shields.io/pypi/pyversions/sooth.svg)](https://pypi.org/project/sooth/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/naufalhilmiaji/sooth/blob/main/LICENSE)
+[![CI](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml/badge.svg)](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml)
+
+## See it catch a hallucination
+
+```bash
+pip install sooth
+sooth demo        # no API key needed
+```
+
+Real output — seven claims in an AI-written news summary, three numbers quietly wrong:
+
+```
+# Sooth
+
+**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70
+
+| # | Claim | Verdict | P | Why (P distribution) | Evidence |
+|---|-------|---------|---|----------------------|----------|
+| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
+| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
+| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham. | ⚠️ REVIEW | ████████ 1.00 | contradicts 0.00 / not_found 1.00 / supports 0.00 · details 0.02 · missing #s: 5.000 |  |
+| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.24 | P(checkable)=0.24 |  |
+```
+
+Every verdict carries its probability distribution and the source span it was judged against. When Sooth is unsure, it says `REVIEW` instead of guessing.
+
+## Why
+
+AI writes fast, nobody checks. Claims ship wrong — usually a number, a hedge, or a name that drifted. Asking another LLM "is this right?" just produces more prose.
+
+Sooth takes a different bet: **the model makes typed judgments, code makes the decision.** Every rule that turns a probability into a verdict is a readable line of Python, not a prompt.
+
+## Install
+
+```bash
+pip install sooth
+export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
+```
+
+## Usage
+
+```bash
+sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
+
+# CI-friendly exit codes
+sooth --source policy.md --text draft.md --format plain
+#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
+
+# Options
+#   --confidence T   REVIEW below this confidence (default 0.7)
+#   --format md|plain
+#   -o FILE          write report to file
+#   --log FILE       append full judgment trace (one JSONL line per run)
+```
+
+Wire it into CI as a quality gate for generated content:
+
+```yaml
+- name: Verify AI output
+  run: sooth --source docs/policy.md --text generated-reply.md
+  env:
+    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+```
+
+## How it works
+
+1. Draft is split into claims (one sentence each).
+2. Each claim gets four questions to Jev, fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), *do all details match exactly?*, and *which source span is the evidence?*
+3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
+4. The report shows the full probability distribution per claim — not just a label.
+
+The decision logic lives in [`src/sooth/verify.py`](https://github.com/naufalhilmiaji/sooth/blob/main/src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
+
+Judgments come from [Jev](https://docs.typesafe.ai) (TypeSafe System One), which returns typed distributions instead of generated prose.
+
+## Why not just use an LLM judge?
+
+Not claiming Sooth is universally more accurate. Claiming it gives you **architectural guarantees** a prompt cannot:
+
+| | Sooth | Generic LLM judge |
+|---|-------|-------------------|
+| Claim-level verification | ✅ | Sometimes |
+| Source span behind each verdict | ✅ | Sometimes |
+| `PASS` / `FAIL` / `REVIEW` | ✅ fixed vocabulary | Generated labels |
+| Probability distribution | ✅ | Usually none |
+| Verdict rules in readable code | ✅ | Prompt-dependent |
+| CI exit codes | ✅ | ❌ |
+| Audit log (`--log` JSONL) | ✅ | DIY |
+
+## Who it's for
+
+Developers building systems that generate text from trusted material:
+
+- **RAG pipelines** — retrieved the right document, still wrote the wrong number
+- **AI agents** — before an agent's reply reaches a human
+- **Support / ops copilots** — policy-backed answers, checked against the policy
+- **Report & summary generation** — figures that must match the source
+- **Compliance workflows** — evidence trail for every claim, not vibes
+
+## Known limits (alpha)
+
+- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
+- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
+- Source + questions must fit ~64k tokens — split long documents yourself.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python3 tests/test_core.py     # pure checks, no network
+pytest                         # same suite
+bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
+PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
+```
+
+Docs: [PRD](https://github.com/naufalhilmiaji/sooth/blob/main/docs/PRD.md) · [Design](https://github.com/naufalhilmiaji/sooth/blob/main/docs/DESIGN.md) · [Testing](https://github.com/naufalhilmiaji/sooth/blob/main/docs/TESTING.md)
+
+## Roadmap
+
+- v0.2.2 shipped: `sooth demo` (offline, no API key)
+- v0.2.1 shipped: source-span evidence, published on PyPI
+- GitHub Action
+- later: hosted web app — paste UI, history, team review queues
+
+## License
+
+MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.

{sooth-0.2.1 → sooth-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "sooth"
-version = "0.2.1"
+version = "0.2.2"
 description = "Verify AI-generated text against source material. Claim-by-claim trust reports powered by Jev (TypeSafe System One)."
 readme = "README.md"
 license = { text = "MIT" }

{sooth-0.2.1 → sooth-0.2.2}/src/sooth/cli.py

@@ -9,9 +9,11 @@ from datetime import UTC, datetime
 from pathlib import Path
 
 from sooth.claims import split_claims
-from sooth.report import exit_code, log_record, render_markdown, render_plain
+from sooth.report import exit_code, log_record, render_markdown, render_plain, verdicts_from_record
 from sooth.verify import DEFAULT_THRESHOLD, VerifyError, verify_claims
 
+DEMO_RECORD = Path(__file__).with_name("demo.json")
+
 
 def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
     p = argparse.ArgumentParser(
@@ -19,9 +21,11 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
         description="Verify AI-generated text against source material. "
                     "Each claim gets PASS / FAIL / REVIEW with calibrated probabilities.",
     )
-    p.add_argument("--source", action="append", required=True, metavar="FILE",
+    p.add_argument("command", nargs="?", choices=["demo"],
+                   help="run the offline demo against a bundled recorded run (no API key)")
+    p.add_argument("--source", action="append", metavar="FILE",
                    help="ground-truth source file (repeatable)")
-    p.add_argument("--text", required=True, metavar="FILE", help="AI-generated draft to check")
+    p.add_argument("--text", metavar="FILE", help="AI-generated draft to check")
     p.add_argument("--confidence", type=float, default=DEFAULT_THRESHOLD, metavar="T",
                    help=f"REVIEW below this confidence (default {DEFAULT_THRESHOLD})")
     p.add_argument("--format", choices=["md", "plain"], default="md", help="report format")
@@ -30,8 +34,29 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
     return p.parse_args(argv)
 
 
+def run_demo(fmt: str) -> int:
+    """Replay a bundled judgment trace through the real report code. No network."""
+    record = json.loads(DEMO_RECORD.read_text(encoding="utf-8"))
+    print(
+        "demo: replaying a recorded run "
+        f"({record['draft']} vs {', '.join(record['sources'])}) — no API key needed",
+        file=sys.stderr,
+    )
+    verdicts = verdicts_from_record(record)
+    report = (render_markdown if fmt == "md" else render_plain)(
+        verdicts, threshold=record["threshold"]
+    )
+    print(report, end="")
+    return exit_code(verdicts)
+
+
 def main(argv: list[str] | None = None) -> int:
     args = parse_args(argv)
+    if args.command == "demo":
+        return run_demo(args.format)
+    if not args.source or not args.text:
+        print("error: --source and --text are required (or try: sooth demo)", file=sys.stderr)
+        return 3
     try:
         sources = [(path, Path(path).read_text(encoding="utf-8")) for path in args.source]
         draft = Path(args.text).read_text(encoding="utf-8")

sooth-0.2.2/src/sooth/demo.json

@@ -0,0 +1 @@
+{"model": "jev-1.13.0", "threshold": 0.7, "sources": ["examples/news-1.md"], "draft": "examples/news-1-evil.md", "usage": {"input_tokens": 11111, "output_tokens": 1107}, "results": [{"id": "c1", "text": "Aturan baru BEI menurunkan batas bawah harga saham menjadi Rp 1 per saham mulai 28 September 2026.", "line": 3, "kind": "PASS", "p_checkable": 0.98, "choice": "supports", "probabilities": {"supports": 1.0, "not_found": 0.0, "contradicts": 0.0}, "confidence": 1.0, "details_p": 0.98, "missing_numbers": [], "evidence": {"id": "s1", "text": "Saham-saham Grup Bakrie kembali menjadi sorotan menjelang pemberlakuan aturan baru Bursa Efek Indonesia (BEI) yang menurunkan batas bawah harga saham menjadi Rp 1 per saham mulai 28 September 2026.", "line": 3, "source": "examples/news-1.md"}}, {"id": "c2", "text": "Menurut analis Bumiputera Sekuritas, aturan ini berlaku untuk seluruh emiten di BEI.", "line": 5, "kind": "PASS", "p_checkable": 0.96, "choice": "supports", "probabilities": {"supports": 1.0, "not_found": 0.0, "contradicts": 0.0}, "confidence": 1.0, "details_p": 0.82, "missing_numbers": [], "evidence": {"id": "s9", "text": "Ia menyebut aturan ini berlaku untuk seluruh saham di BEI, bukan khusus Grup Bakrie.", "line": 7, "source": "examples/news-1.md"}}, {"id": "c3", "text": "BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33.", "line": 7, "kind": "FAIL", "p_checkable": 0.93, "choice": "contradicts", "probabilities": {"supports": 0.0, "not_found": 0.0, "contradicts": 1.0}, "confidence": 1.0, "details_p": 0.01, "missing_numbers": ["33"], "evidence": {"id": "s47", "text": "Saham ini juga baru menyelesaikan rights issue dalam jumlah besar di harga Rp 53.", "line": 25, "source": "examples/news-1.md"}}, {"id": "c4", "text": "Saham VIVA disuspensi di harga Rp 50.", "line": 9, "kind": "FAIL", "p_checkable": 0.92, "choice": "contradicts", "probabilities": {"supports": 0.0, "not_found": 0.0, "contradicts": 1.0}, "confidence": 1.0, "details_p": 0.02, "missing_numbers": [], "evidence": null}, {"id": "c5", "text": "BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50.", "line": 11, "kind": "FAIL", "p_checkable": 0.92, "choice": "contradicts", "probabilities": {"supports": 0.0, "not_found": 0.0, "contradicts": 1.0}, "confidence": 1.0, "details_p": 0.01, "missing_numbers": ["10"], "evidence": {"id": "s40", "text": "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 harus turun sekitar 74%, PT Darma Henwa Tbk (DEWA) sekitar 86%, PT Bumi Resources Minerals Tbk (BRMS) sekitar 93%, dan PT Energi Mega Persada Tbk (ENRG) lebih dari 96% untuk turun ke level Rp50.", "line": 23, "source": "examples/news-1.md"}}, {"id": "c6", "text": "BEI juga menetapkan batas atas harga saham Rp 5.000 per saham di pasar reguler.", "line": 13, "kind": "REVIEW", "p_checkable": 0.57, "choice": "not_found", "probabilities": {"supports": 0.0, "not_found": 1.0, "contradicts": 0.0}, "confidence": 1.0, "details_p": 0.02, "missing_numbers": ["5.000"], "evidence": null}, {"id": "c7", "text": "Para investor ritel sangat senang dengan aturan baru ini.", "line": 15, "kind": "UNCHECKABLE", "p_checkable": 0.25, "choice": "not_found", "probabilities": {"supports": 0.0, "not_found": 1.0, "contradicts": 0.0}, "confidence": 1.0, "details_p": 0.02, "missing_numbers": [], "evidence": null}], "ts": "2026-09-25T09:59:57+00:00"}

{sooth-0.2.1 → sooth-0.2.2}/src/sooth/report.py

@@ -123,3 +123,27 @@ def log_record(result: VerifyResult, threshold: float, sources: list[str], draft
             for v in result.verdicts
         ],
     }
+
+
+def verdicts_from_record(record: dict) -> list[Verdict]:
+    """Inverse of log_record's `results` — replay a recorded run without the API."""
+    out = []
+    for r in record["results"]:
+        ev = r.get("evidence")
+        out.append(Verdict(
+            claim_id=r["id"],
+            claim_text=r["text"],
+            line=r["line"],
+            kind=r["kind"],
+            p_checkable=r.get("p_checkable"),
+            choice=r.get("choice"),
+            probabilities=r.get("probabilities") or {},
+            confidence=r.get("confidence"),
+            details_p=r.get("details_p"),
+            missing_numbers=tuple(r.get("missing_numbers") or ()),
+            evidence_id=ev.get("id") if ev else None,
+            evidence_text=ev.get("text") if ev else None,
+            evidence_line=ev.get("line") if ev else None,
+            evidence_source=ev.get("source") if ev else None,
+        ))
+    return out

{sooth-0.2.1 → sooth-0.2.2}/tests/test_core.py

@@ -12,7 +12,7 @@ from pathlib import Path
 sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))  # direct-run without install
 
 from sooth.claims import split_claims, split_segments
-from sooth.report import bar, exit_code, log_record, render_markdown
+from sooth.report import bar, exit_code, log_record, render_markdown, verdicts_from_record
 from sooth.verify import (
     FAIL,
     PASS,
@@ -229,6 +229,25 @@ def test_log_record_roundtrip_no_secret():
     assert "api_key" not in json.dumps(parsed).lower()
 
 
+def test_verdicts_from_record_roundtrip():
+    """log_record → JSON → verdicts_from_record is lossless (demo replay path)."""
+    result = VerifyResult(
+        verdicts=[
+            v(PASS, confidence=0.9, probabilities={"supports": 0.9}, details_p=0.9,
+              evidence_id="s1", evidence_text="Source line.", evidence_line=3,
+              evidence_source="a.md"),
+            v(FAIL, claim_id="c2", confidence=1.0, probabilities={"contradicts": 1.0},
+              missing_numbers=("33",)),
+        ],
+        model="jev-1.13.0",
+        usage={"input_tokens": 10, "output_tokens": 1},
+    )
+    back = verdicts_from_record(json.loads(json.dumps(log_record(result, 0.7, ["a.md"], "d.md"))))
+    assert [(x.claim_id, x.kind, x.confidence) for x in back] == [("c1", PASS, 0.9), ("c2", FAIL, 1.0)]
+    assert back[0].evidence_text == "Source line." and back[0].evidence_line == 3
+    assert back[1].missing_numbers == ("33",) and back[1].evidence_text is None
+
+
 if __name__ == "__main__":
     fns = [f for name, f in sorted(globals().items()) if name.startswith("test_") and callable(f)]
     for fn in fns:

sooth-0.2.1/README.md

@@ -1,87 +0,0 @@
-# Sooth
-
-Verify AI-generated text against source material. Claim-by-claim **PASS / FAIL / REVIEW**, with calibrated probabilities you can act on in CI.
-
-Built on [Jev](https://docs.typesafe.ai) (TypeSafe System One) — typed judgments and probabilities instead of generated prose. Every verdict shows its probability distribution, and the combination rules are plain code you can read. No black box.
-
-```
-sooth --source policy.md --text draft-reply.md
-```
-
-Real output (planted errors in the draft vs a news source):
-
-```
-# Sooth
-
-**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70
-
-| # | Claim | Verdict | P | Why (P distribution) | Evidence |
-|---|-------|---------|---|----------------------|----------|
-| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
-| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
-| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham. | ⚠️ REVIEW | ████████ 1.00 | contradicts 0.00 / not_found 1.00 / supports 0.00 · details 0.02 · missing #s: 5.000 |  |
-| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.24 | P(checkable)=0.24 |  |
-```
-
-## Why
-
-AI writes fast, nobody checks. Claims ship wrong. Sooth checks each claim against the evidence you give it — and when it is unsure, it says `REVIEW` instead of guessing.
-
-## Install
-
-```bash
-pip install sooth          # or: pip install -e ".[dev]" from source
-export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
-```
-
-## Usage
-
-```bash
-sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
-
-# CI-friendly exit codes
-sooth --source policy.md --text draft.md --format plain
-#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
-
-# Options
-#   --confidence T   REVIEW below this confidence (default 0.7)
-#   --format md|plain
-#   -o FILE          write report to file
-#   --log FILE       append full judgment trace (one JSONL line per run)
-```
-
-## How it works
-
-1. Draft is split into claims (one sentence each).
-2. Each claim gets three questions to Jev, all fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), and *do all details match exactly?*
-3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
-4. The report shows the full probability distribution per claim — not just a label.
-
-The decision logic lives in [`src/sooth/verify.py`](src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
-
-## Known limits (alpha)
-
-- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
-- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
-- Source + questions must fit ~64k tokens — split long documents yourself.
-
-## Development
-
-```bash
-pip install -e ".[dev]"
-python3 tests/test_core.py     # pure checks, no network
-pytest                         # same suite
-bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
-PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
-```
-
-Docs: [PRD](docs/PRD.md) · [Design](docs/DESIGN.md) · [Testing](docs/TESTING.md)
-
-## Roadmap
-
-- v0.2 shipped: source-span evidence (the exact source line behind each verdict, shown in the report)
-- later: hosted web app — paste UI, history, team review queues
-
-## License
-
-MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.