sooth 0.2.1__tar.gz → 0.2.3__tar.gz

{sooth-0.2.1 → sooth-0.2.3}/.github/workflows/ci.yml

@@ -19,3 +19,4 @@ jobs:
       - run: pip install -e ".[dev]"
       - run: ruff check src tests
       - run: pytest
+      - run: bash tests/test_action.sh

sooth-0.2.3/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.5
+Name: sooth
+Version: 0.2.3
+Summary: Verify AI-generated text against source material. Claim-by-claim trust reports powered by Jev (TypeSafe System One).
+Project-URL: Homepage, https://github.com/naufalhilmiaji/sooth
+Project-URL: Documentation, https://github.com/naufalhilmiaji/sooth/tree/main/docs
+Author-email: Naufal Hilmiaji <nhilmiaji@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: ai,jev,llm,trust,typesafe,verification
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.11
+Requires-Dist: typesafe-sdk
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Sooth
+
+**LLMs generate. Sooth verifies.**
+
+Claim-by-claim fact-checking for AI output, designed for CI. Sooth checks every sentence in a draft against your source material and returns `PASS` / `FAIL` / `REVIEW` with calibrated probabilities and the exact source line behind each verdict.
+
+[![PyPI](https://img.shields.io/pypi/v/sooth.svg)](https://pypi.org/project/sooth/)
+[![Python](https://img.shields.io/pypi/pyversions/sooth.svg)](https://pypi.org/project/sooth/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/naufalhilmiaji/sooth/blob/main/LICENSE)
+[![CI](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml/badge.svg)](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml)
+
+## See it catch a hallucination
+
+```bash
+pip install sooth
+sooth demo        # no API key needed
+```
+
+Real output — seven claims in an AI-written news summary, three numbers quietly wrong:
+
+```
+# Sooth
+
+**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70
+
+| # | Claim | Verdict | P | Why (P distribution) | Evidence |
+|---|-------|---------|---|----------------------|----------|
+| 1 | Aturan baru BEI menurunkan batas bawah harga saham menjadi Rp 1 per saham mulai 28 September 2026. | ✅ PASS | ████████ 1.00 | supports 1.00 / not_found 0.00 / contradicts 0.00 · details 0.98 | `examples/news-1.md:3` "Saham-saham Grup Bakrie kembali menjadi sorotan menjelang…" |
+| 2 | Menurut analis Bumiputera Sekuritas, aturan ini berlaku untuk seluruh emiten di BEI. | ✅ PASS | ████████ 1.00 | supports 1.00 / not_found 0.00 / contradicts 0.00 · details 0.82 | `examples/news-1.md:7` "Ia menyebut aturan ini berlaku untuk seluruh saham di BEI…" |
+| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
+| 4 | Saham VIVA disuspensi di harga Rp 50. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.02 |  |
+| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
+| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham di pasar reguler. | ⚠️ REVIEW | ████████ 1.00 | supports 0.00 / not_found 1.00 / contradicts 0.00 · details 0.02 · missing #s: 5.000 |  |
+| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.25 | P(checkable)=0.25 |  |
+
+## Needs review
+
+- line 13: BEI juga menetapkan batas atas harga saham Rp 5.000 per saham di pasar reguler.
+- line 15: Para investor ritel sangat senang dengan aturan baru ini.
+```
+
+Every verdict carries its probability distribution and the source span it was judged against. When Sooth is unsure, it says `REVIEW` instead of guessing.
+
+## Why
+
+AI writes fast, nobody checks. Claims ship wrong — usually a number, a hedge, or a name that drifted. Asking another LLM "is this right?" just produces more prose.
+
+Sooth takes a different bet: **the model makes typed judgments, code makes the decision.** Every rule that turns a probability into a verdict is a readable line of Python, not a prompt.
+
+## Install
+
+```bash
+pip install sooth
+export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
+```
+
+## Usage
+
+```bash
+sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
+
+# CI-friendly exit codes
+sooth --source policy.md --text draft.md --format plain
+#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
+
+# Options
+#   --confidence T   REVIEW below this confidence (default 0.7)
+#   --format md|plain
+#   -o FILE          write report to file
+#   --log FILE       append full judgment trace (one JSONL line per run)
+```
+
+Use it in CI as a quality gate for generated content:
+
+```yaml
+- name: Verify AI output
+  uses: naufalhilmiaji/sooth@v0.2.3
+  with:
+    source: docs/policy.md
+    text: generated-reply.md
+  env:
+    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+```
+
+`source` takes newline-separated paths when you have several documents. The step fails the build on any `FAIL` — and on any `REVIEW`, so add `continue-on-error: true` if you only want to block on hard contradictions. The full report is appended to the job summary.
+
+Prefer plain shell? `sooth --source policy.md --text draft.md` gives the same exit codes.
+
+## How it works
+
+1. Draft is split into claims (one sentence each).
+2. Each claim gets four questions to Jev, fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), *do all details match exactly?*, and *which source span is the evidence?*
+3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
+4. The report shows the full probability distribution per claim — not just a label.
+
+The decision logic lives in [`src/sooth/verify.py`](https://github.com/naufalhilmiaji/sooth/blob/main/src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
+
+Judgments come from [Jev](https://docs.typesafe.ai) (TypeSafe System One), which returns typed distributions instead of generated prose.
+
+## Why not just use an LLM judge?
+
+Not claiming Sooth is universally more accurate. Claiming it gives you **architectural guarantees** a prompt cannot:
+
+| | Sooth | Generic LLM judge |
+|---|-------|-------------------|
+| Claim-level verification | ✅ | Sometimes |
+| Source span behind each verdict | ✅ | Sometimes |
+| `PASS` / `FAIL` / `REVIEW` | ✅ fixed vocabulary | Generated labels |
+| Probability distribution | ✅ | Usually none |
+| Verdict rules in readable code | ✅ | Prompt-dependent |
+| CI exit codes | ✅ | ❌ |
+| Audit log (`--log` JSONL) | ✅ | DIY |
+
+## Who it's for
+
+Developers building systems that generate text from trusted material:
+
+- **RAG pipelines** — retrieved the right document, still wrote the wrong number
+- **AI agents** — before an agent's reply reaches a human
+- **Support / ops copilots** — policy-backed answers, checked against the policy
+- **Report & summary generation** — figures that must match the source
+- **Compliance workflows** — evidence trail for every claim, not vibes
+
+## Known limits (alpha)
+
+- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
+- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
+- Source + questions must fit ~64k tokens — split long documents yourself.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python3 tests/test_core.py     # pure checks, no network
+pytest                         # same suite
+bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
+PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
+```
+
+Docs: [PRD](https://github.com/naufalhilmiaji/sooth/blob/main/docs/PRD.md) · [Design](https://github.com/naufalhilmiaji/sooth/blob/main/docs/DESIGN.md) · [Testing](https://github.com/naufalhilmiaji/sooth/blob/main/docs/TESTING.md)
+
+## Roadmap
+
+- v0.2.3 shipped: GitHub Action — CI quality gate in one `uses:` line
+- v0.2.2 shipped: `sooth demo` (offline, no API key)
+- v0.2.1 shipped: source-span evidence, published on PyPI
+- later: hosted web app — paste UI, history, team review queues
+
+## License
+
+MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.

sooth-0.2.3/README.md

@@ -0,0 +1,151 @@
+# Sooth
+
+**LLMs generate. Sooth verifies.**
+
+Claim-by-claim fact-checking for AI output, designed for CI. Sooth checks every sentence in a draft against your source material and returns `PASS` / `FAIL` / `REVIEW` with calibrated probabilities and the exact source line behind each verdict.
+
+[![PyPI](https://img.shields.io/pypi/v/sooth.svg)](https://pypi.org/project/sooth/)
+[![Python](https://img.shields.io/pypi/pyversions/sooth.svg)](https://pypi.org/project/sooth/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/naufalhilmiaji/sooth/blob/main/LICENSE)
+[![CI](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml/badge.svg)](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml)
+
+## See it catch a hallucination
+
+```bash
+pip install sooth
+sooth demo        # no API key needed
+```
+
+Real output — seven claims in an AI-written news summary, three numbers quietly wrong:
+
+```
+# Sooth
+
+**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70
+
+| # | Claim | Verdict | P | Why (P distribution) | Evidence |
+|---|-------|---------|---|----------------------|----------|
+| 1 | Aturan baru BEI menurunkan batas bawah harga saham menjadi Rp 1 per saham mulai 28 September 2026. | ✅ PASS | ████████ 1.00 | supports 1.00 / not_found 0.00 / contradicts 0.00 · details 0.98 | `examples/news-1.md:3` "Saham-saham Grup Bakrie kembali menjadi sorotan menjelang…" |
+| 2 | Menurut analis Bumiputera Sekuritas, aturan ini berlaku untuk seluruh emiten di BEI. | ✅ PASS | ████████ 1.00 | supports 1.00 / not_found 0.00 / contradicts 0.00 · details 0.82 | `examples/news-1.md:7` "Ia menyebut aturan ini berlaku untuk seluruh saham di BEI…" |
+| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
+| 4 | Saham VIVA disuspensi di harga Rp 50. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.02 |  |
+| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
+| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham di pasar reguler. | ⚠️ REVIEW | ████████ 1.00 | supports 0.00 / not_found 1.00 / contradicts 0.00 · details 0.02 · missing #s: 5.000 |  |
+| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.25 | P(checkable)=0.25 |  |
+
+## Needs review
+
+- line 13: BEI juga menetapkan batas atas harga saham Rp 5.000 per saham di pasar reguler.
+- line 15: Para investor ritel sangat senang dengan aturan baru ini.
+```
+
+Every verdict carries its probability distribution and the source span it was judged against. When Sooth is unsure, it says `REVIEW` instead of guessing.
+
+## Why
+
+AI writes fast, nobody checks. Claims ship wrong — usually a number, a hedge, or a name that drifted. Asking another LLM "is this right?" just produces more prose.
+
+Sooth takes a different bet: **the model makes typed judgments, code makes the decision.** Every rule that turns a probability into a verdict is a readable line of Python, not a prompt.
+
+## Install
+
+```bash
+pip install sooth
+export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
+```
+
+## Usage
+
+```bash
+sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
+
+# CI-friendly exit codes
+sooth --source policy.md --text draft.md --format plain
+#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
+
+# Options
+#   --confidence T   REVIEW below this confidence (default 0.7)
+#   --format md|plain
+#   -o FILE          write report to file
+#   --log FILE       append full judgment trace (one JSONL line per run)
+```
+
+Use it in CI as a quality gate for generated content:
+
+```yaml
+- name: Verify AI output
+  uses: naufalhilmiaji/sooth@v0.2.3
+  with:
+    source: docs/policy.md
+    text: generated-reply.md
+  env:
+    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+```
+
+`source` takes newline-separated paths when you have several documents. The step fails the build on any `FAIL` — and on any `REVIEW`, so add `continue-on-error: true` if you only want to block on hard contradictions. The full report is appended to the job summary.
+
+Prefer plain shell? `sooth --source policy.md --text draft.md` gives the same exit codes.
+
+## How it works
+
+1. Draft is split into claims (one sentence each).
+2. Each claim gets four questions to Jev, fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), *do all details match exactly?*, and *which source span is the evidence?*
+3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
+4. The report shows the full probability distribution per claim — not just a label.
+
+The decision logic lives in [`src/sooth/verify.py`](https://github.com/naufalhilmiaji/sooth/blob/main/src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
+
+Judgments come from [Jev](https://docs.typesafe.ai) (TypeSafe System One), which returns typed distributions instead of generated prose.
+
+## Why not just use an LLM judge?
+
+Not claiming Sooth is universally more accurate. Claiming it gives you **architectural guarantees** a prompt cannot:
+
+| | Sooth | Generic LLM judge |
+|---|-------|-------------------|
+| Claim-level verification | ✅ | Sometimes |
+| Source span behind each verdict | ✅ | Sometimes |
+| `PASS` / `FAIL` / `REVIEW` | ✅ fixed vocabulary | Generated labels |
+| Probability distribution | ✅ | Usually none |
+| Verdict rules in readable code | ✅ | Prompt-dependent |
+| CI exit codes | ✅ | ❌ |
+| Audit log (`--log` JSONL) | ✅ | DIY |
+
+## Who it's for
+
+Developers building systems that generate text from trusted material:
+
+- **RAG pipelines** — retrieved the right document, still wrote the wrong number
+- **AI agents** — before an agent's reply reaches a human
+- **Support / ops copilots** — policy-backed answers, checked against the policy
+- **Report & summary generation** — figures that must match the source
+- **Compliance workflows** — evidence trail for every claim, not vibes
+
+## Known limits (alpha)
+
+- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
+- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
+- Source + questions must fit ~64k tokens — split long documents yourself.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python3 tests/test_core.py     # pure checks, no network
+pytest                         # same suite
+bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
+PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
+```
+
+Docs: [PRD](https://github.com/naufalhilmiaji/sooth/blob/main/docs/PRD.md) · [Design](https://github.com/naufalhilmiaji/sooth/blob/main/docs/DESIGN.md) · [Testing](https://github.com/naufalhilmiaji/sooth/blob/main/docs/TESTING.md)
+
+## Roadmap
+
+- v0.2.3 shipped: GitHub Action — CI quality gate in one `uses:` line
+- v0.2.2 shipped: `sooth demo` (offline, no API key)
+- v0.2.1 shipped: source-span evidence, published on PyPI
+- later: hosted web app — paste UI, history, team review queues
+
+## License
+
+MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.

sooth-0.2.3/action.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Wire SOOTH_* inputs into the sooth CLI, surface the report, propagate the exit code.
+# Inputs arrive via env (never interpolated into this script) so untrusted input
+# cannot be executed as shell.
+set -uo pipefail
+
+args=(--format md)
+while IFS= read -r path; do
+  [ -z "$path" ] && continue
+  args+=(--source "$path")
+done <<< "$SOOTH_SOURCE"
+
+args+=(--text "$SOOTH_TEXT")
+if [ -n "${SOOTH_CONFIDENCE:-}" ]; then
+  args+=(--confidence "$SOOTH_CONFIDENCE")
+fi
+
+report="${RUNNER_TEMP:-/tmp}/sooth-report.md"
+sooth "${args[@]}" -o "$report"
+status=$?
+
+if [ -f "$report" ]; then
+  cat "$report"
+  if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+    cat "$report" >> "$GITHUB_STEP_SUMMARY"
+  fi
+fi
+
+exit "$status"

sooth-0.2.3/action.yml

@@ -0,0 +1,35 @@
+name: "Sooth"
+description: "Claim-by-claim verification of AI-generated text against source material. Fails CI on hallucinations."
+author: "Naufal Hilmiaji"
+branding:
+  icon: "check-circle"
+  color: "green"
+
+inputs:
+  source:
+    description: "Ground-truth source file(s). Repeat by separating paths with newlines."
+    required: true
+  text:
+    description: "AI-generated draft to check."
+    required: true
+  confidence:
+    description: "REVIEW below this confidence."
+    required: false
+    default: "0.7"
+
+runs:
+  using: "composite"
+  steps:
+    - uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+    - name: Install sooth
+      shell: bash
+      run: python -m pip install --quiet sooth
+    - name: Verify draft
+      shell: bash
+      env:
+        SOOTH_SOURCE: ${{ inputs.source }}
+        SOOTH_TEXT: ${{ inputs.text }}
+        SOOTH_CONFIDENCE: ${{ inputs.confidence }}
+      run: bash "$GITHUB_ACTION_PATH/action.sh"

{sooth-0.2.1 → sooth-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "sooth"
-version = "0.2.1"
+version = "0.2.3"
 description = "Verify AI-generated text against source material. Claim-by-claim trust reports powered by Jev (TypeSafe System One)."
 readme = "README.md"
 license = { text = "MIT" }

{sooth-0.2.1 → sooth-0.2.3}/src/sooth/cli.py

@@ -9,9 +9,11 @@ from datetime import UTC, datetime
 from pathlib import Path
 
 from sooth.claims import split_claims
-from sooth.report import exit_code, log_record, render_markdown, render_plain
+from sooth.report import exit_code, log_record, render_markdown, render_plain, verdicts_from_record
 from sooth.verify import DEFAULT_THRESHOLD, VerifyError, verify_claims
 
+DEMO_RECORD = Path(__file__).with_name("demo.json")
+
 
 def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
     p = argparse.ArgumentParser(
@@ -19,9 +21,11 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
         description="Verify AI-generated text against source material. "
                     "Each claim gets PASS / FAIL / REVIEW with calibrated probabilities.",
     )
-    p.add_argument("--source", action="append", required=True, metavar="FILE",
+    p.add_argument("command", nargs="?", choices=["demo"],
+                   help="run the offline demo against a bundled recorded run (no API key)")
+    p.add_argument("--source", action="append", metavar="FILE",
                    help="ground-truth source file (repeatable)")
-    p.add_argument("--text", required=True, metavar="FILE", help="AI-generated draft to check")
+    p.add_argument("--text", metavar="FILE", help="AI-generated draft to check")
     p.add_argument("--confidence", type=float, default=DEFAULT_THRESHOLD, metavar="T",
                    help=f"REVIEW below this confidence (default {DEFAULT_THRESHOLD})")
     p.add_argument("--format", choices=["md", "plain"], default="md", help="report format")
@@ -30,8 +34,29 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
     return p.parse_args(argv)
 
 
+def run_demo(fmt: str) -> int:
+    """Replay a bundled judgment trace through the real report code. No network."""
+    record = json.loads(DEMO_RECORD.read_text(encoding="utf-8"))
+    print(
+        "demo: replaying a recorded run "
+        f"({record['draft']} vs {', '.join(record['sources'])}) — no API key needed",
+        file=sys.stderr,
+    )
+    verdicts = verdicts_from_record(record)
+    report = (render_markdown if fmt == "md" else render_plain)(
+        verdicts, threshold=record["threshold"]
+    )
+    print(report, end="")
+    return exit_code(verdicts)
+
+
 def main(argv: list[str] | None = None) -> int:
     args = parse_args(argv)
+    if args.command == "demo":
+        return run_demo(args.format)
+    if not args.source or not args.text:
+        print("error: --source and --text are required (or try: sooth demo)", file=sys.stderr)
+        return 3
     try:
         sources = [(path, Path(path).read_text(encoding="utf-8")) for path in args.source]
         draft = Path(args.text).read_text(encoding="utf-8")

sooth-0.2.3/src/sooth/demo.json

@@ -0,0 +1 @@
+{"model": "jev-1.13.0", "threshold": 0.7, "sources": ["examples/news-1.md"], "draft": "examples/news-1-evil.md", "usage": {"input_tokens": 11111, "output_tokens": 1107}, "results": [{"id": "c1", "text": "Aturan baru BEI menurunkan batas bawah harga saham menjadi Rp 1 per saham mulai 28 September 2026.", "line": 3, "kind": "PASS", "p_checkable": 0.98, "choice": "supports", "probabilities": {"supports": 1.0, "not_found": 0.0, "contradicts": 0.0}, "confidence": 1.0, "details_p": 0.98, "missing_numbers": [], "evidence": {"id": "s1", "text": "Saham-saham Grup Bakrie kembali menjadi sorotan menjelang pemberlakuan aturan baru Bursa Efek Indonesia (BEI) yang menurunkan batas bawah harga saham menjadi Rp 1 per saham mulai 28 September 2026.", "line": 3, "source": "examples/news-1.md"}}, {"id": "c2", "text": "Menurut analis Bumiputera Sekuritas, aturan ini berlaku untuk seluruh emiten di BEI.", "line": 5, "kind": "PASS", "p_checkable": 0.96, "choice": "supports", "probabilities": {"supports": 1.0, "not_found": 0.0, "contradicts": 0.0}, "confidence": 1.0, "details_p": 0.82, "missing_numbers": [], "evidence": {"id": "s9", "text": "Ia menyebut aturan ini berlaku untuk seluruh saham di BEI, bukan khusus Grup Bakrie.", "line": 7, "source": "examples/news-1.md"}}, {"id": "c3", "text": "BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33.", "line": 7, "kind": "FAIL", "p_checkable": 0.93, "choice": "contradicts", "probabilities": {"supports": 0.0, "not_found": 0.0, "contradicts": 1.0}, "confidence": 1.0, "details_p": 0.01, "missing_numbers": ["33"], "evidence": {"id": "s47", "text": "Saham ini juga baru menyelesaikan rights issue dalam jumlah besar di harga Rp 53.", "line": 25, "source": "examples/news-1.md"}}, {"id": "c4", "text": "Saham VIVA disuspensi di harga Rp 50.", "line": 9, "kind": "FAIL", "p_checkable": 0.92, "choice": "contradicts", "probabilities": {"supports": 0.0, "not_found": 0.0, "contradicts": 1.0}, "confidence": 1.0, "details_p": 0.02, "missing_numbers": [], "evidence": null}, {"id": "c5", "text": "BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50.", "line": 11, "kind": "FAIL", "p_checkable": 0.92, "choice": "contradicts", "probabilities": {"supports": 0.0, "not_found": 0.0, "contradicts": 1.0}, "confidence": 1.0, "details_p": 0.01, "missing_numbers": ["10"], "evidence": {"id": "s40", "text": "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 harus turun sekitar 74%, PT Darma Henwa Tbk (DEWA) sekitar 86%, PT Bumi Resources Minerals Tbk (BRMS) sekitar 93%, dan PT Energi Mega Persada Tbk (ENRG) lebih dari 96% untuk turun ke level Rp50.", "line": 23, "source": "examples/news-1.md"}}, {"id": "c6", "text": "BEI juga menetapkan batas atas harga saham Rp 5.000 per saham di pasar reguler.", "line": 13, "kind": "REVIEW", "p_checkable": 0.57, "choice": "not_found", "probabilities": {"supports": 0.0, "not_found": 1.0, "contradicts": 0.0}, "confidence": 1.0, "details_p": 0.02, "missing_numbers": ["5.000"], "evidence": null}, {"id": "c7", "text": "Para investor ritel sangat senang dengan aturan baru ini.", "line": 15, "kind": "UNCHECKABLE", "p_checkable": 0.25, "choice": "not_found", "probabilities": {"supports": 0.0, "not_found": 1.0, "contradicts": 0.0}, "confidence": 1.0, "details_p": 0.02, "missing_numbers": [], "evidence": null}], "ts": "2026-09-25T09:59:57+00:00"}

{sooth-0.2.1 → sooth-0.2.3}/src/sooth/report.py

@@ -123,3 +123,27 @@ def log_record(result: VerifyResult, threshold: float, sources: list[str], draft
             for v in result.verdicts
         ],
     }
+
+
+def verdicts_from_record(record: dict) -> list[Verdict]:
+    """Inverse of log_record's `results` — replay a recorded run without the API."""
+    out = []
+    for r in record["results"]:
+        ev = r.get("evidence")
+        out.append(Verdict(
+            claim_id=r["id"],
+            claim_text=r["text"],
+            line=r["line"],
+            kind=r["kind"],
+            p_checkable=r.get("p_checkable"),
+            choice=r.get("choice"),
+            probabilities=r.get("probabilities") or {},
+            confidence=r.get("confidence"),
+            details_p=r.get("details_p"),
+            missing_numbers=tuple(r.get("missing_numbers") or ()),
+            evidence_id=ev.get("id") if ev else None,
+            evidence_text=ev.get("text") if ev else None,
+            evidence_line=ev.get("line") if ev else None,
+            evidence_source=ev.get("source") if ev else None,
+        ))
+    return out

sooth-0.2.3/tests/test_action.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Arg-wiring checks for action.sh. No network — a fake `sooth` records its argv.
+# Run: bash tests/test_action.sh
+set -uo pipefail
+
+here="$(cd "$(dirname "$0")" && pwd)"
+tmp="$(mktemp -d)"
+trap 'rm -rf "$tmp"' EXIT
+
+write_fake() {
+  cat > "$tmp/sooth" <<'EOF'
+#!/usr/bin/env bash
+printf '%s\n' "$@" > "$ARGV_FILE"
+prev=""
+for a in "$@"; do
+  if [ "$prev" = "-o" ]; then echo "# report" > "$a"; fi
+  prev="$a"
+done
+exit "${FAKE_STATUS:-0}"
+EOF
+  chmod +x "$tmp/sooth"
+}
+
+# argv as one line, plus the same with the dynamic "-o <report>" tail dropped
+argv_full() { tr '\n' ' ' < "$ARGV_FILE"; }
+args_only() { argv_full | sed 's/ -o [^ ]*//'; }
+
+export PATH="$tmp:$PATH"
+export ARGV_FILE="$tmp/argv"
+export RUNNER_TEMP="$tmp"
+export SOOTH_TEXT="draft.md"
+export SOOTH_CONFIDENCE="0.75"
+unset GITHUB_STEP_SUMMARY 2>/dev/null || true
+
+fail() { echo "FAIL: $*" >&2; exit 1; }
+
+write_fake
+
+# single source
+export SOOTH_SOURCE="policy.md"
+bash "$here/../action.sh" >/dev/null || fail "single source: exit $?"
+got="$(args_only)"
+want="--format md --source policy.md --text draft.md --confidence 0.75 "
+[ "$got" = "$want" ] || fail "single source argv: got [$got] want [$want]"
+
+# multiple sources, newline-separated, blanks ignored
+export SOOTH_SOURCE="a.md
+b.md
+
+c.md"
+bash "$here/../action.sh" >/dev/null || fail "multi source: exit $?"
+got="$(args_only)"
+want="--format md --source a.md --source b.md --source c.md --text draft.md --confidence 0.75 "
+[ "$got" = "$want" ] || fail "multi source argv: got [$got] want [$want]"
+
+# confidence is optional
+unset SOOTH_CONFIDENCE
+export SOOTH_SOURCE="policy.md"
+bash "$here/../action.sh" >/dev/null || fail "no confidence: exit $?"
+got="$(args_only)"
+want="--format md --source policy.md --text draft.md "
+[ "$got" = "$want" ] || fail "no confidence argv: got [$got] want [$want]"
+
+# a FAIL from sooth must fail the step (CI gate), and the report still prints
+export SOOTH_CONFIDENCE="0.7"
+export SOOTH_SOURCE="policy.md"
+export FAKE_STATUS=1
+out="$(bash "$here/../action.sh")"
+status=$?
+[ "$status" -eq 1 ] || fail "exit code not propagated: got $status"
+[ -n "$out" ] || fail "report not printed on failure"
+
+# hostile input must arrive as a literal argument and never execute
+export FAKE_STATUS=0
+export SOOTH_TEXT='draft.md; echo PWNED'
+out="$(bash "$here/../action.sh")" || fail "injection: exit $?"
+case "$out" in
+  *PWNED*) fail "input was executed as shell" ;;
+esac
+full="$(argv_full)"
+case "$full" in
+  *"--text draft.md; echo PWNED "*) : ;;
+  *) fail "injection argv: got [$full]" ;;
+esac
+
+echo "ok — action.sh arg wiring (5 checks)"

{sooth-0.2.1 → sooth-0.2.3}/tests/test_core.py

@@ -12,7 +12,7 @@ from pathlib import Path
 sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))  # direct-run without install
 
 from sooth.claims import split_claims, split_segments
-from sooth.report import bar, exit_code, log_record, render_markdown
+from sooth.report import bar, exit_code, log_record, render_markdown, verdicts_from_record
 from sooth.verify import (
     FAIL,
     PASS,
@@ -229,6 +229,25 @@ def test_log_record_roundtrip_no_secret():
     assert "api_key" not in json.dumps(parsed).lower()
 
 
+def test_verdicts_from_record_roundtrip():
+    """log_record → JSON → verdicts_from_record is lossless (demo replay path)."""
+    result = VerifyResult(
+        verdicts=[
+            v(PASS, confidence=0.9, probabilities={"supports": 0.9}, details_p=0.9,
+              evidence_id="s1", evidence_text="Source line.", evidence_line=3,
+              evidence_source="a.md"),
+            v(FAIL, claim_id="c2", confidence=1.0, probabilities={"contradicts": 1.0},
+              missing_numbers=("33",)),
+        ],
+        model="jev-1.13.0",
+        usage={"input_tokens": 10, "output_tokens": 1},
+    )
+    back = verdicts_from_record(json.loads(json.dumps(log_record(result, 0.7, ["a.md"], "d.md"))))
+    assert [(x.claim_id, x.kind, x.confidence) for x in back] == [("c1", PASS, 0.9), ("c2", FAIL, 1.0)]
+    assert back[0].evidence_text == "Source line." and back[0].evidence_line == 3
+    assert back[1].missing_numbers == ("33",) and back[1].evidence_text is None
+
+
 if __name__ == "__main__":
     fns = [f for name, f in sorted(globals().items()) if name.startswith("test_") and callable(f)]
     for fn in fns:

sooth-0.2.1/PKG-INFO

@@ -1,110 +0,0 @@
-Metadata-Version: 2.5
-Name: sooth
-Version: 0.2.1
-Summary: Verify AI-generated text against source material. Claim-by-claim trust reports powered by Jev (TypeSafe System One).
-Project-URL: Homepage, https://github.com/naufalhilmiaji/sooth
-Project-URL: Documentation, https://github.com/naufalhilmiaji/sooth/tree/main/docs
-Author-email: Naufal Hilmiaji <nhilmiaji@gmail.com>
-License: MIT
-License-File: LICENSE
-Keywords: ai,jev,llm,trust,typesafe,verification
-Classifier: Development Status :: 3 - Alpha
-Classifier: Environment :: Console
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Topic :: Software Development :: Quality Assurance
-Requires-Python: >=3.11
-Requires-Dist: typesafe-sdk
-Provides-Extra: dev
-Requires-Dist: pytest; extra == 'dev'
-Requires-Dist: ruff; extra == 'dev'
-Description-Content-Type: text/markdown
-
-# Sooth
-
-Verify AI-generated text against source material. Claim-by-claim **PASS / FAIL / REVIEW**, with calibrated probabilities you can act on in CI.
-
-Built on [Jev](https://docs.typesafe.ai) (TypeSafe System One) — typed judgments and probabilities instead of generated prose. Every verdict shows its probability distribution, and the combination rules are plain code you can read. No black box.
-
-```
-sooth --source policy.md --text draft-reply.md
-```
-
-Real output (planted errors in the draft vs a news source):
-
-```
-# Sooth
-
-**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70
-
-| # | Claim | Verdict | P | Why (P distribution) | Evidence |
-|---|-------|---------|---|----------------------|----------|
-| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
-| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
-| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham. | ⚠️ REVIEW | ████████ 1.00 | contradicts 0.00 / not_found 1.00 / supports 0.00 · details 0.02 · missing #s: 5.000 |  |
-| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.24 | P(checkable)=0.24 |  |
-```
-
-## Why
-
-AI writes fast, nobody checks. Claims ship wrong. Sooth checks each claim against the evidence you give it — and when it is unsure, it says `REVIEW` instead of guessing.
-
-## Install
-
-```bash
-pip install sooth          # or: pip install -e ".[dev]" from source
-export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
-```
-
-## Usage
-
-```bash
-sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
-
-# CI-friendly exit codes
-sooth --source policy.md --text draft.md --format plain
-#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
-
-# Options
-#   --confidence T   REVIEW below this confidence (default 0.7)
-#   --format md|plain
-#   -o FILE          write report to file
-#   --log FILE       append full judgment trace (one JSONL line per run)
-```
-
-## How it works
-
-1. Draft is split into claims (one sentence each).
-2. Each claim gets three questions to Jev, all fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), and *do all details match exactly?*
-3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
-4. The report shows the full probability distribution per claim — not just a label.
-
-The decision logic lives in [`src/sooth/verify.py`](src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
-
-## Known limits (alpha)
-
-- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
-- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
-- Source + questions must fit ~64k tokens — split long documents yourself.
-
-## Development
-
-```bash
-pip install -e ".[dev]"
-python3 tests/test_core.py     # pure checks, no network
-pytest                         # same suite
-bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
-PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
-```
-
-Docs: [PRD](docs/PRD.md) · [Design](docs/DESIGN.md) · [Testing](docs/TESTING.md)
-
-## Roadmap
-
-- v0.2 shipped: source-span evidence (the exact source line behind each verdict, shown in the report)
-- later: hosted web app — paste UI, history, team review queues
-
-## License
-
-MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.

sooth-0.2.1/README.md

@@ -1,87 +0,0 @@
-# Sooth
-
-Verify AI-generated text against source material. Claim-by-claim **PASS / FAIL / REVIEW**, with calibrated probabilities you can act on in CI.
-
-Built on [Jev](https://docs.typesafe.ai) (TypeSafe System One) — typed judgments and probabilities instead of generated prose. Every verdict shows its probability distribution, and the combination rules are plain code you can read. No black box.
-
-```
-sooth --source policy.md --text draft-reply.md
-```
-
-Real output (planted errors in the draft vs a news source):
-
-```
-# Sooth
-
-**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70
-
-| # | Claim | Verdict | P | Why (P distribution) | Evidence |
-|---|-------|---------|---|----------------------|----------|
-| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
-| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | contradicts 1.00 / not_found 0.00 / supports 0.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
-| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham. | ⚠️ REVIEW | ████████ 1.00 | contradicts 0.00 / not_found 1.00 / supports 0.00 · details 0.02 · missing #s: 5.000 |  |
-| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.24 | P(checkable)=0.24 |  |
-```
-
-## Why
-
-AI writes fast, nobody checks. Claims ship wrong. Sooth checks each claim against the evidence you give it — and when it is unsure, it says `REVIEW` instead of guessing.
-
-## Install
-
-```bash
-pip install sooth          # or: pip install -e ".[dev]" from source
-export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
-```
-
-## Usage
-
-```bash
-sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
-
-# CI-friendly exit codes
-sooth --source policy.md --text draft.md --format plain
-#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
-
-# Options
-#   --confidence T   REVIEW below this confidence (default 0.7)
-#   --format md|plain
-#   -o FILE          write report to file
-#   --log FILE       append full judgment trace (one JSONL line per run)
-```
-
-## How it works
-
-1. Draft is split into claims (one sentence each).
-2. Each claim gets three questions to Jev, all fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), and *do all details match exactly?*
-3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
-4. The report shows the full probability distribution per claim — not just a label.
-
-The decision logic lives in [`src/sooth/verify.py`](src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
-
-## Known limits (alpha)
-
-- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
-- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
-- Source + questions must fit ~64k tokens — split long documents yourself.
-
-## Development
-
-```bash
-pip install -e ".[dev]"
-python3 tests/test_core.py     # pure checks, no network
-pytest                         # same suite
-bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
-PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
-```
-
-Docs: [PRD](docs/PRD.md) · [Design](docs/DESIGN.md) · [Testing](docs/TESTING.md)
-
-## Roadmap
-
-- v0.2 shipped: source-span evidence (the exact source line behind each verdict, shown in the report)
-- later: hosted web app — paste UI, history, team review queues
-
-## License
-
-MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.