sooth 0.2.2__tar.gz → 0.3.0__tar.gz

sooth-0.3.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Read the PRD, design, and testing docs first
+    url: https://github.com/naufalhilmiaji/sooth/tree/main/docs
+    about: Scope decisions, the verdict contract, and what the test layers cover.
+  - name: TypeSafe / Jev API questions
+    url: https://docs.typesafe.ai
+    about: Sooth is a CLI on top of Jev. Questions about the engine, keys, or quotas belong with TypeSafe.

sooth-0.3.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,17 @@
+## What
+
+## Why
+
+## Checklist
+
+- [ ] `python3 tests/test_core.py` passes
+- [ ] `bash tests/test_action.sh` passes (if `action.yml` / `action.sh` changed)
+- [ ] `ruff check src tests` is clean
+- [ ] New logic covered by an assert in `tests/test_core.py`
+- [ ] Docs updated if behavior changed (`docs/PRD.md`, `docs/DESIGN.md`, `docs/TESTING.md`, README)
+- [ ] `CHANGELOG.md` updated under `## [Unreleased]`
+
+## For verdict or accuracy changes
+
+- [ ] No calibrated threshold was lowered to make a case pass
+- [ ] Relevant demo records re-recorded from a live run rather than hand-edited

sooth-0.3.0/.github/workflows/ci.yml

@@ -0,0 +1,60 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: ruff check src tests
+      - run: pytest
+      - run: bash tests/test_action.sh
+
+      # End-to-end smoke of the packaged CLI with no API key: both demo cases are
+      # recorded runs, so this exercises claim parsing, verdict mapping, and every
+      # renderer on a real report. It also pins the exit-code contract, which is
+      # the thing CI users actually depend on.
+      - name: Offline demo smoke
+        run: |
+          sooth --version
+          set +e
+          sooth demo > /dev/null;                    rc_md=$?
+          sooth demo --case id > /dev/null;           rc_id=$?
+          sooth demo --format plain > /dev/null;      rc_plain=$?
+          sooth demo --format json > /tmp/demo.json;  rc_json=$?
+          set -e
+          test "$rc_md" -eq 1 && test "$rc_id" -eq 1 && test "$rc_plain" -eq 1 && test "$rc_json" -eq 1
+          python -c "import json; d = json.load(open('/tmp/demo.json')); assert d['exit_code'] == 1, d['summary']; assert len(d['verdicts']) == 8, len(d['verdicts']); assert sum(d['summary'][k] for k in ('pass','fail','review','uncheckable')) == 8; assert d['verdicts'][1]['missing_numbers'] == ['2,000'], d['verdicts'][1]; print('json contract ok')"
+
+  package:
+    # The demo records are non-Python files inside the package directory. If they
+    # stop shipping in the wheel, `sooth demo` breaks only for users who installed
+    # from PyPI — so assert it here rather than by hand.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip wheel --no-deps -w dist .
+      - name: Install the built wheel and replay every demo case
+        run: |
+          pip install --force-reinstall dist/*.whl
+          cd /tmp   # leave the source tree, so the installed copy is what runs
+          set +e
+          sooth demo > /dev/null;              rc_en=$?
+          sooth demo --case id > /dev/null;    rc_id=$?
+          set -e
+          test "$rc_en" -eq 1 && test "$rc_id" -eq 1
+          echo "wheel ships both demo cases"

sooth-0.3.0/CHANGELOG.md

@@ -0,0 +1,70 @@
+# Changelog
+
+All notable changes to Sooth. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [SemVer](https://semver.org/).
+
+## [Unreleased]
+
+## [0.3.0] — 2026-09-26
+
+Presentation and machine-consumption release. No change to verification behaviour or to any calibrated threshold.
+
+### Added
+
+- `--format json` — stable machine-readable report (`summary`, `exit_code`, `verdicts[]` with probabilities, evidence, and missing numbers). One JSON contract for pipelines and agents instead of scraping Markdown.
+- `sooth demo --case en|id` — two bundled recorded runs now ship in the wheel. `en` is the default (release notes vs an AI summary, three numbers silently wrong); `id` is the Indonesian market-news case.
+- `--version`, reading the version from installed distribution metadata.
+- GitHub Action: `fail-on` input (`review` default, `fail`, `never`), and `outputs` — `verdicts`, `pass-count`, `fail-count`, `review-count`, `uncheckable-count`, `report`, `exit-code`. All from a single API call.
+- English calibration fixture (`examples/release-notes.md` + `examples/ai-summary.md`) as the default demo.
+- CI: an offline demo smoke step that pins the documented exit codes and the JSON contract, plus a `package` job that installs the built wheel and replays both demo cases (guards non-Python package data).
+- `CHANGELOG.md`, `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`.
+
+### Changed
+
+- README rewritten around the measured accuracy numbers and an English hero demo.
+- Action installs the exact version declared in its own `pyproject.toml`, so the pin can no longer drift from the tag.
+- PyPI metadata: keywords and classifiers now cover `hallucination`, `fact-checking`, `llm-evaluation`, `rag`, `guardrails`, `ci`, `github-action`.
+- `render_json` and the `--log` record now share one verdict-serialisation function, so the two JSON surfaces cannot drift apart.
+
+### Fixed
+
+- `sooth.__version__` reported `0.1.0` while the distribution was `0.2.3`. Now sourced from packaging metadata; a test fails if it ever becomes a hardcoded literal again.
+- `tests/smoke.sh` invoked the bare `python3`, which fails when `typesafe-sdk` only exists in the project venv. It now prefers an active venv, then `./.venv`, then `python3`, accepts `PYTHON=`, and explains the fix instead of exiting 3 silently.
+
+## [0.2.3] — 2026-09-25
+
+### Added
+
+- GitHub Action — a CI quality gate in one `uses:` line, with the report appended to the job summary.
+
+## [0.2.2] — 2026-09-25
+
+### Added
+
+- `sooth demo` — replays a bundled recorded run offline, no API key needed.
+- README overhaul with a recorded demo.
+
+## [0.2.1] — 2026-09-25
+
+### Added
+
+- Source-span evidence behind every verdict: sources are split into sentence segments, code pre-ranks ~6 candidates per claim, and a per-claim Choice selects the best span.
+- First PyPI release.
+
+## [0.2.0] — 2026-09-25
+
+### Added
+
+- Source-span evidence behind every verdict (`v0.2` feature branch).
+- 30-claim calibration set with a live runner and a frozen confusion matrix.
+
+### Fixed
+
+- Ruff lint violations blocking CI.
+
+## [0.1.0-alpha] — 2026-09-24
+
+### Added
+
+- Initial CLI: `sooth --source … --text …`, sentence-per-claim splitting, Jev fan-out verification (four questions per claim in one batched call), Markdown report, `--log` JSONL decision ledger, CI exit codes, `--confidence` threshold.
+- Safeguards: detail-match gate and deterministic number-smuggling check demote `PASS` to `REVIEW`.
+- PRD, technical design, and testing docs.

sooth-0.3.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,90 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, colour, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behaviour that contributes to a positive environment:
+
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologising to those affected by our mistakes,
+  and learning from the experience
+- Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behaviour:
+
+- The use of sexualised language or imagery, and sexual attention or advances
+  of any kind
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+The maintainer is responsible for clarifying and enforcing our standards of
+acceptable behaviour, and will take appropriate and fair corrective action in
+response to any behaviour that they deem inappropriate, threatening, offensive,
+or harmful.
+
+The maintainer has the right and responsibility to remove, edit, or reject
+comments, commits, code, issues, and other contributions that are not aligned
+with this Code of Conduct, and will communicate reasons for moderation decisions
+when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces — issues, pull
+requests, discussions — and also applies when an individual is officially
+representing the project in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behaviour may be
+reported to the maintainer at **nhilmiaji@gmail.com**. All complaints will be
+reviewed and investigated promptly and fairly. The maintainer is obligated to
+respect the privacy and security of the reporter of any incident.
+
+## Enforcement Guidelines
+
+The maintainer will follow these Community Impact Guidelines in determining the
+consequences for any action deemed in violation of this Code of Conduct:
+
+**1. Correction** — A private, written warning, providing clarity around the
+nature of the violation and an explanation of why the behaviour was
+inappropriate. A public apology may be requested.
+
+**2. Warning** — A warning with consequences for continued behaviour. No
+interaction with the people involved for a specified period of time. Violating
+these terms may lead to a temporary or permanent ban.
+
+**3. Temporary Ban** — A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved is allowed during this period.
+
+**4. Permanent Ban** — A permanent ban from any sort of public interaction
+within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org),
+version 2.1, available at
+https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/inclusion).

sooth-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,72 @@
+# Contributing to Sooth
+
+Thanks for looking. This is a small tool with a narrow contract, and the fastest way to get a change merged is to keep it narrow.
+
+## Set up
+
+```bash
+git clone https://github.com/naufalhilmiaji/sooth
+cd sooth
+python3 -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+## Run the checks
+
+```bash
+python3 tests/test_core.py     # pure checks — no network, no API key, under a second
+pytest                         # same suite through pytest
+bash tests/test_action.sh      # GitHub Action arg wiring, outputs, and fail-on policy
+ruff check src tests           # lint — CI runs this, line length 100
+```
+
+Two optional layers need a live key (`TYPESAFE_API_KEY` from [console.typesafe.ai](https://console.typesafe.ai)):
+
+```bash
+bash tests/smoke.sh                          # end-to-end against the live API
+PYTHONPATH=src python3 tests/calibrate.py    # 30-claim calibration; exits non-zero if the bar is missed
+```
+
+CI never calls the live API — it would cost money and add flakiness on forks. Run the live layers yourself before a release.
+
+## Ground rules
+
+**Never move the threshold to hide a failure.** If calibration regresses, the fix is question wording in `build_questions()` in `src/sooth/verify.py`, not a lower `--confidence` default and not a relaxed labelled case. The calibration matrix is frozen in `examples/calibration.md`; if you change it, say why in the diff.
+
+**No new runtime dependencies** without a discussion issue first. The runtime dependency list is exactly one package (`typesafe-sdk`); a verifier that drags in a dependency tree is a verifier nobody installs into a CI job.
+
+**Verdict mapping stays pure and readable.** `map_verdict` and `apply_safeguards` take values and return values. Network I/O lives in `verify_claims` only. If your change makes the decision path untestable without a key, it will not be merged.
+
+**Deterministic checks belong in code, not in prompts.** If a rule can be expressed as a regex or an arithmetic comparison — like the number-smuggling check in `missing_numbers()` — it belongs in Python, where it can be tested and audited.
+
+**Fixed vocabulary.** Verdicts are `PASS`, `FAIL`, `REVIEW`, `UNCHECKABLE` and nothing else. Resist adding states.
+
+## Adding a demo case
+
+`sooth demo` must keep working with no API key and no network. To add one:
+
+1. Add the source and draft documents under `examples/`.
+2. Run a real verification with `--log` and keep the output:
+   ```bash
+   sooth --source examples/your-source.md --text examples/your-draft.md --log /tmp/run.jsonl
+   ```
+3. Save that single JSONL line as `src/sooth/demo-<case>.json`.
+4. Register it in `DEMO_CASES` in `src/sooth/cli.py`.
+5. `pytest` covers the rest — `test_demo_fixtures_are_valid_records` and `test_cli_demo_replays_offline` will fail if the record or the wiring is wrong.
+
+Demo records are real recorded runs. Do not hand-edit verdicts to make a demo look better; re-run and commit what the model actually returned.
+
+## Adding calibration claims
+
+Append to `examples/calibration.json` with an `expected` verdict and a one-line note explaining why that label is correct. Then re-run `tests/calibrate.py` live and update `examples/calibration.md` with the new matrix. Cases that reveal a wrong answer are more valuable than cases that confirm the tool works — record those, and explain the failure mode.
+
+## Reporting a wrong verdict
+
+The most useful bug report includes: the source document, the draft, the verdict Sooth returned, the verdict you expected, and the `--format json` payload for that claim. Include the `Why` column — the probability distribution usually makes the cause obvious.
+
+## Pull requests
+
+- One thing per PR. A feature and a refactor in the same diff will be asked to split.
+- Add or update a test in `tests/test_core.py` for any change to logic in `claims.py`, `verify.py`, or `report.py`.
+- Update `CHANGELOG.md` under `## [Unreleased]`.
+- Commit messages: `feat:`, `fix:`, `docs:`, `chore:`, `test:` — the repo log is readable, keep it that way.

sooth-0.3.0/PKG-INFO

@@ -0,0 +1,258 @@
+Metadata-Version: 2.5
+Name: sooth
+Version: 0.3.0
+Summary: Claim-by-claim hallucination detection and fact-checking for AI-generated text, built for CI. PASS/FAIL/REVIEW verdicts with calibrated probabilities and the source line behind each one, powered by Jev (TypeSafe System One).
+Project-URL: Homepage, https://github.com/naufalhilmiaji/sooth
+Project-URL: Documentation, https://github.com/naufalhilmiaji/sooth/tree/main/docs
+Project-URL: Repository, https://github.com/naufalhilmiaji/sooth
+Project-URL: Issues, https://github.com/naufalhilmiaji/sooth/issues
+Project-URL: Changelog, https://github.com/naufalhilmiaji/sooth/releases
+Author-email: Naufal Hilmiaji <nhilmiaji@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: ai-safety,ci,fact-checking,github-action,guardrails,hallucination,hallucination-detection,jev,llm-evaluation,rag,typesafe,verification
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.11
+Requires-Dist: typesafe-sdk
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Sooth
+
+**LLMs generate. Sooth verifies.**
+
+Claim-by-claim hallucination detection for AI output. Sooth checks every sentence in a draft against your source material and returns `PASS` / `FAIL` / `REVIEW` — with calibrated probabilities, the exact source line behind each verdict, and exit codes that fail your build.
+
+[![PyPI](https://img.shields.io/pypi/v/sooth.svg)](https://pypi.org/project/sooth/)
+[![Python](https://img.shields.io/pypi/pyversions/sooth.svg)](https://pypi.org/project/sooth/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/naufalhilmiaji/sooth/blob/main/LICENSE)
+[![CI](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml/badge.svg)](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml)
+
+## Try it in 10 seconds — no API key
+
+```bash
+pip install sooth
+sooth demo
+```
+
+`demo` replays a recorded run through the real reporting code. No network, no key, no signup.
+
+## See it catch a hallucination
+
+A model was asked to summarise a product release note. Three numbers came back wrong.
+
+```
+# Sooth
+
+**PASS 4 · FAIL 3 · REVIEW 0 · UNCHECKABLE 1** — threshold 0.70
+
+| # | Claim | Verdict | P | Why (P distribution) | Evidence |
+|---|-------|---------|---|----------------------|----------|
+| 1 | Kestrel Cloud launched API v3 on 14 August 2026 across all of its regions. | ✅ PASS | ████████ 1.00 | supports 1.00 / contradicts 0.00 / not_found 0.00 · details 0.86 | `examples/release-notes.md:7` "API v3 is generally available in all 12 regions, includin…" |
+| 2 | The rate limit jumps to 2,000 requests per second per project, a twenty-fold increase over v2. | ❌ FAIL | ████████ 1.00 | supports 0.00 / contradicts 1.00 / not_found 0.00 · details 0.01 · missing #s: 2,000 | `examples/release-notes.md:11` "The rate limit rises from 100 requests per second to 500 …" |
+| 3 | Token pricing falls to $0.002 per 1,000 input tokens, making it the cheapest mainstream inference API. | ❌ FAIL | ████████ 1.00 | supports 0.00 / contradicts 1.00 / not_found 0.00 · details 0.01 · missing #s: 0.002 | `examples/release-notes.md:15` "Token pricing drops to $0.004 per 1,000 input tokens and …" |
+| 4 | Kestrel is now certified SOC 2 Type II, which unblocks enterprise procurement. | ❌ FAIL | ████████ 1.00 | supports 0.00 / contradicts 1.00 / not_found 0.00 · details 0.01 | `examples/release-notes.md:27` "SOC 2 Type II certification is still in progress and is n…" |
+| 5 | The free tier now includes 10,000 requests per month. | ✅ PASS | ████████ 1.00 | supports 1.00 / contradicts 0.00 / not_found 0.00 · details 0.97 | `examples/release-notes.md:15` "The free tier now includes 10,000 requests per month, up …" |
+| 6 | API v3 runs across 12 regions and held p99 latency of 180 ms through the beta. | ✅ PASS | ████████ 1.00 | supports 1.00 / contradicts 0.00 / not_found 0.00 · details 0.88 | `examples/release-notes.md:19` "Latency at p99 was 180 ms during the six-week beta." |
+| 7 | Developers say the migration is the smoothest they have seen. | ➖ UNCHECKABLE | █░░░░░░░ 0.16 | P(checkable)=0.16 |  |
+| 8 | Existing v2 keys keep working until 31 January 2027. | ✅ PASS | ████████ 1.00 | supports 1.00 / contradicts 0.00 / not_found 0.00 · details 0.99 | `examples/release-notes.md:7` "Existing v2 keys keep working until 31 January 2027." |
+
+## Needs review
+
+- line 3: Developers say the migration is the smoothest they have seen.
+```
+
+Exit code `1`. The build stops. No human had to notice that `2,000` should have been `500`, or that SOC 2 was still *in progress* in the source.
+
+Every verdict carries its probability distribution and the source span it was judged against. When Sooth is unsure, it says `REVIEW` instead of guessing.
+
+Reproduce that exact report offline with `sooth demo`. The inputs ship in [`examples/`](examples/): [`release-notes.md`](examples/release-notes.md) and [`ai-summary.md`](examples/ai-summary.md).
+
+## Accuracy
+
+Measured, not asserted. [`examples/calibration.json`](examples/calibration.json) is a hand-labelled set of 30 claims — 10 supported, 10 contradicted (number flips, feature mis-attribution, negation flips), 5 invented, 5 opinions — drawn from three documents (English policy, SaaS pricing, and Indonesian market news).
+
+- **30/30 correct** verdicts, live, re-run 2026-09-26 on the v0.3.0 code. Bar is ≥ 24/30; the runner exits non-zero if it is missed.
+- **Zero confident-wrong `PASS` on a contradicted claim, in every run.** This is the number that matters: a wrong `PASS` is the failure mode that ships garbage, so it is the one the runner gates on hardest.
+- One case is inherently jittery: an invented claim whose checkable probability sits on the 0.5 floor, flipping between `REVIEW` and `UNCHECKABLE` across runs. Both outcomes mean "do not act on this". It is recorded rather than tuned away.
+
+Full confusion matrix and notes: [`examples/calibration.md`](examples/calibration.md). Re-run it live with `PYTHONPATH=src python3 tests/calibrate.py`. The threshold has never been moved to hide a failure — when a case regressed, question wording in `verify.py` was fixed instead.
+
+Honest caveat: 30 claims is a small set, labelled by the author, and it is not a substitute for evaluating Sooth on your own documents. It is a regression gate, not a benchmark claim.
+
+## Install
+
+```bash
+pip install sooth
+export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
+```
+
+Requires Python 3.11+. Judgments come from [Jev](https://docs.typesafe.ai) (TypeSafe System One), which returns typed distributions instead of generated prose.
+
+## Usage
+
+```bash
+sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md
+
+# CI-friendly exit codes
+sooth --source policy.md --text draft.md --format plain
+#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error
+
+# Options
+#   --confidence T   REVIEW below this confidence (default 0.7)
+#   --format md|plain|json
+#   -o FILE          write report to file
+#   --log FILE       append full judgment trace (one JSONL line per run)
+#   --version
+```
+
+## Machine-readable output
+
+`--format json` gives pipelines and agents a stable contract — no prose parsing, and the exit code is stated in the payload:
+
+```json
+{
+  "summary": { "pass": 4, "fail": 3, "review": 0, "uncheckable": 1, "threshold": 0.7 },
+  "exit_code": 1,
+  "verdicts": [
+    {
+      "id": "c2",
+      "text": "The rate limit jumps to 2,000 requests per second per project…",
+      "line": 3,
+      "kind": "FAIL",
+      "choice": "contradicts",
+      "probabilities": { "supports": 0.0, "contradicts": 1.0, "not_found": 0.0 },
+      "confidence": 1.0,
+      "details_p": 0.01,
+      "missing_numbers": ["2,000"],
+      "evidence": {
+        "text": "The rate limit rises from 100 requests per second to 500 requests per second per project.",
+        "line": 11,
+        "source": "examples/release-notes.md"
+      }
+    }
+  ]
+}
+```
+
+## Use it as a CI quality gate
+
+```yaml
+- name: Verify AI output
+  uses: naufalhilmiaji/sooth@v0.3.0
+  with:
+    source: docs/policy.md
+    text: generated-reply.md
+    fail-on: review        # review (default) | fail | never
+  env:
+    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+```
+
+The step fails the build on `FAIL` — and on `REVIEW`, so nothing uncertain ships silently. Set `fail-on: fail` to block only on hard contradictions. The full report is appended to the job summary.
+
+Consume the counts in later steps:
+
+```yaml
+- id: sooth
+  uses: naufalhilmiaji/sooth@v0.3.0
+  with: { source: policy.md, text: draft.md, fail-on: never }
+  env: { TYPESAFE_API_KEY: "${{ secrets.TYPESAFE_API_KEY }}" }
+
+- run: echo "${{ steps.sooth.outputs.verdicts }}"   # PASS 1, FAIL 1, REVIEW 0, UNCHECKABLE 0
+```
+
+Outputs: `verdicts`, `pass-count`, `fail-count`, `review-count`, `uncheckable-count`, `report`, `exit-code`.
+
+Prefer plain shell? `sooth --source policy.md --text draft.md` gives the same exit codes.
+
+## How it works
+
+1. Draft is split into claims (one sentence each).
+2. Each claim gets four questions to Jev, fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), *do all details match exactly?*, and *which source span is the evidence?*
+3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
+4. The report shows the full probability distribution per claim — not just a label.
+
+The decision logic lives in [`src/sooth/verify.py`](https://github.com/naufalhilmiaji/sooth/blob/main/src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.
+
+## Why not just use an LLM judge?
+
+Not claiming Sooth is universally more accurate. Claiming it gives you **architectural guarantees** a prompt cannot:
+
+| | Sooth | Generic LLM judge |
+|---|-------|-------------------|
+| Claim-level verification | ✅ | Sometimes |
+| Source span behind each verdict | ✅ | Sometimes |
+| `PASS` / `FAIL` / `REVIEW` | ✅ fixed vocabulary | Generated labels |
+| Probability distribution | ✅ | Usually none |
+| Verdict rules in readable code | ✅ | Prompt-dependent |
+| CI exit codes | ✅ | ❌ |
+| Audit log (`--log` JSONL) | ✅ | DIY |
+| Runs the same verdict twice | ✅ deterministic mapping | Sampling variance |
+
+An LLM judge gives you a sentence about your text. Sooth gives you a decision your code can branch on, and the evidence it branched on.
+
+## Who it's for
+
+Developers building systems that generate text from trusted material:
+
+- **RAG pipelines** — retrieved the right document, still wrote the wrong number
+- **AI agents** — before an agent's reply reaches a human
+- **Support / ops copilots** — policy-backed answers, checked against the policy
+- **Report & summary generation** — figures that must match the source
+- **Compliance workflows** — evidence trail for every claim, not vibes
+
+## Beyond English
+
+The bundled Indonesian case proves the pipeline is not English-only: `sooth demo --case id` replays a recorded run against Indonesian market news, where three fabricated figures are caught. Non-English claim segmentation needs a whitespace-language sentence splitter (Indonesian, Malay, and most European languages are fine; Chinese and Japanese would need a different splitter — see [`src/sooth/claims.py`](https://github.com/naufalhilmiaji/sooth/blob/main/src/sooth/claims.py)).
+
+## Known limits (alpha)
+
+- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
+- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
+- Source + questions must fit ~64k tokens — split long documents yourself.
+- Sentence splitting is regex-based: abbreviations and quoted sentences can mis-split.
+- Requires a TypeSafe API key for anything beyond `sooth demo`.
+
+## Development
+
+```bash
+git clone https://github.com/naufalhilmiaji/sooth && cd sooth
+python3 -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev]"
+
+python3 tests/test_core.py     # pure checks, no network, under a second
+pytest                         # same suite
+bash tests/test_action.sh      # GitHub Action arg wiring + fail-on policy
+ruff check src tests           # lint (CI gate)
+
+export TYPESAFE_API_KEY=...    # only for the two live layers below
+bash tests/smoke.sh                          # live end-to-end, both fixtures
+PYTHONPATH=src python3 tests/calibrate.py    # live calibration, 30 labeled claims
+```
+
+Docs: [PRD](https://github.com/naufalhilmiaji/sooth/blob/main/docs/PRD.md) · [Design](https://github.com/naufalhilmiaji/sooth/blob/main/docs/DESIGN.md) · [Testing](https://github.com/naufalhilmiaji/sooth/blob/main/docs/TESTING.md) · [Changelog](https://github.com/naufalhilmiaji/sooth/blob/main/CHANGELOG.md)
+
+## Roadmap
+
+- **v0.3.0** — `--format json`, `sooth demo --case`, Action `fail-on` + outputs
+- **v0.2.3** shipped — GitHub Action: CI quality gate in one `uses:` line
+- **v0.2.2** shipped — `sooth demo` (offline, no API key)
+- **v0.2.1** shipped — source-span evidence, published on PyPI
+- later — pluggable backends (local models), hosted web app, review queues
+
+## License
+
+MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.