underwrite 0.0.1__tar.gz → 0.1.0__tar.gz

underwrite-0.1.0/PKG-INFO

@@ -0,0 +1,396 @@
+Metadata-Version: 2.4
+Name: underwrite
+Version: 0.1.0
+Summary: Local evidence normalization, measurement, and acceptance checks
+License-Expression: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: underwrite-core>=0.1.0,<0.2
+Requires-Dist: jsonschema>=4.26.0
+Requires-Dist: pyyaml>=6.0.3
+Requires-Python: >=3.12
+Project-URL: Repository, https://github.com/ntts9990/underwrite
+Project-URL: Issues, https://github.com/ntts9990/underwrite/issues
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+<h1><img src="assets/brand/underwrite-logo.png" alt="underwrite" width="640"></h1>
+
+### Know what your eval files actually support.
+
+Find mismatches. Keep missing evidence visible. Know what to check next.
+
+**Local CLI & Python library · Typed JSON · No server or model calls**
+
+[![CI](https://github.com/ntts9990/underwrite/actions/workflows/ci.yml/badge.svg)](https://github.com/ntts9990/underwrite/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/underwrite)](https://pypi.org/project/underwrite/)
+[![Python 3.12+](https://img.shields.io/badge/Python-3.12%2B-blue)](https://pypi.org/project/underwrite/)
+[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
+
+**[Try it](#quickstart)** · [Use with an agent](#agent-use) · [Supported formats](#supported-formats) · [Docs](#documentation)
+
+**English** · [한국어](README.ko.md)
+
+</div>
+
+Already have evaluation exports from **goppi, Langfuse, DeepEval, or Promptfoo**?
+A score or summary alone does not show whether supplied rows match declared
+totals or whether the planned case inventory is complete. For supported local
+profiles, underwrite checks relationships it can verify and makes unknown
+coverage explicit. It returns typed CLI/Python results with findings, input
+identity, and interpretation limits. You or your agent can use them to request
+the next piece of evidence.
+
+Underwrite complements the tools that run evaluations: goppi owns execution,
+trace/verifier binding, and reruns; other producers own their exports and scores.
+Underwrite examines supplied artifacts and can retain selected bytes. It does
+not rerun an agent, independently authenticate a producer, or assign a universal
+quality score. No Docker, dashboard, or provider credentials are needed to
+inspect the supported local files.
+
+> [!NOTE]
+> **This README describes the `0.1.0` source tree and matching wheels.** The older
+> PyPI `0.0.1` lacks these workflows; use the source quickstart or matching
+> `0.1.0` app/core wheels. The older build's `pins refresh` needs a separate
+> caution; see [Published installation](#install).
+
+## Quickstart
+
+Requires **Python 3.12+** and [uv](https://docs.astral.sh/uv/).
+This checkout is tested with CPython 3.12 on Linux and macOS. File-based CLI
+workflows require Unix/POSIX file APIs, including ordinary artifact reads.
+
+**1. Check the file.** Run a bundled synthetic example—no API key or adjacent
+repository required:
+
+```sh
+git clone https://github.com/ntts9990/underwrite.git
+cd underwrite
+uv sync --frozen --all-groups
+
+uv run --frozen --no-sync underwrite audit-promptfoo \
+  fixtures/examples/evidence/synthetic-promptfoo-count-mismatch.json --json
+```
+
+**2. Read the finding.** The producer summary declares **3 results**, including
+**1 failure**; the file contains **2 rows**, with **0 failure rows**. The command
+reports the difference without guessing which side is wrong. Selected fields
+from its actual output:
+
+```json
+{
+  "schema": "promptfoo_accounting.v1",
+  "declared_relation": "inconsistent",
+  "producer_stats": {"errors": 1, "failures": 1, "successes": 1, "total": 3},
+  "row_tallies": {"errors": 1, "failures": 0, "successes": 1, "total": 2},
+  "deltas": {"errors": 0, "failures": -1, "successes": 0, "total": -1},
+  "findings": [
+    {"code": "CATEGORY_COUNT_MISMATCH", "location": "/results/stats/failures", "row_index": null}
+  ],
+  "limitations": [
+    "PLANNED_DENOMINATOR_UNKNOWN",
+    "CAUSE_UNKNOWN",
+    "SOURCE_CLAIMS_UNVERIFIED",
+    "NO_APPROVAL_AUTHORITY"
+  ]
+}
+```
+
+**3. Next check.** Compare the export with the producer's run record and planned
+case inventory before claiming that a run was skipped or a failure row is
+missing. `CAUSE_UNKNOWN` and `PLANNED_DENOMINATOR_UNKNOWN` make that limit
+explicit. Exit `0` means the report was produced, not that the counts agree or
+the source's verdict was verified. To keep the input and analysis together,
+[save them in a local record](#keep-results-you-can-return-to).
+
+<details>
+<summary><strong>Try the native measurement → acceptance workflow</strong></summary>
+
+From the same checkout, run the included synthetic binary-event example:
+
+```sh
+mkdir -p output
+uv run --frozen --no-sync underwrite ingest \
+  --format underwrite.evidence-bundle --version v1 \
+  --file fixtures/examples/event-measurement/bundle.json --json \
+  > output/observation.json
+
+uv run --frozen --no-sync underwrite measure \
+  --observation output/observation.json \
+  --policy fixtures/examples/event-acceptance/policy.json --json \
+  > output/read-event.json
+
+uv run --frozen --no-sync underwrite accept \
+  --candidate fixtures/examples/event-acceptance/candidate-single.json \
+  --claims fixtures/examples/event-acceptance/claims.json \
+  --read output/read-event.json --json
+```
+
+This measures a synthetic binary-event proportion, not general quality or latency.
+The `read.v2` uses an explicit `measurement_policy.v3`; calibration diagnostics do
+not vote on its verdict. The final classification is `screened`, requires human
+review, and keeps `merge_authorized` false. Repeat `--read` for each required claim.
+See [measurement meaning](AGENT_USAGE.md#measurement-meaning-and-migration).
+
+</details>
+
+## Why underwrite?
+
+| Question after an evaluation | What underwrite gives you |
+| --- | --- |
+| Does the summary match the supplied records? | Profile-specific accounting: Promptfoo count deltas, DeepEval evaluator errors, or Langfuse score associations. |
+| What is still unknown? | Explicit missingness and limits such as an unknown planned denominator or unverified source claim, so a zero is not silently substituted. |
+| Can this evidence support a measurement? | A separate native binary-event path with declared semantics and policy; external scores do not become a universal quality verdict. |
+| Can I revisit the conclusion? | Typed CLI/Python JSON and optional private records tying selected input bytes to the analysis, with verification and retrieval. |
+
+## How it works
+
+```mermaid
+flowchart LR
+    A[Supported local eval file] --> B[Select exact profile]
+    B --> C[Check supplied records and missingness]
+    C --> D[Typed JSON: finding, identity, limits]
+    D --> E[Agent or person gathers next evidence]
+    C --> F[Optional private record]
+    F --> G[Verify and retrieve]
+    H[Native event evidence + policy] --> I[Measure]
+    I --> D
+```
+
+The command checks only what its selected profile supports. The calling agent
+interprets the JSON and can pursue the next evidence task; underwrite does not
+apply a proposed change or grant merge authority. Local records keep selected
+inputs and results for later review.
+
+## Agent use
+
+Give your coding agent the prepared checkout and an exported artifact. It can
+discover the available routes, execute them, and use the JSON in its response:
+
+```sh
+uv run --frozen --no-sync underwrite --help
+uv run --frozen --no-sync underwrite capabilities --json
+```
+
+Python callers can use
+`underwrite.instrument.ingest.accounting.audit_promptfoo_file(Path(...))` for
+the same result as a dictionary. Select an exact supported profile before
+interpreting any producer-specific field.
+
+Example task for an agent with terminal access:
+
+> Read `skills/underwrite-review/SKILL.md` in this checkout. Review
+> `fixtures/examples/evidence/synthetic-promptfoo-count-mismatch.json` as a
+> Promptfoo 0.123.0 export. Use `capabilities --json` and `audit-promptfoo`.
+> Report `declared_relation`, `producer_stats`, `row_tallies`, `raw_hash`,
+> `limitations`, and the next useful action. Do not rerun the producer or treat
+> exit `0` as agreement or approval.
+
+The optional [underwrite-review skill](skills/underwrite-review/SKILL.md) provides
+reusable guidance for Codex and Claude Code. [Setup and usage](AGENT_USAGE.md#optional-artifact-review-skill)
+are explicit: the skill does not install the CLI or add a background service.
+The same CLI and importable Python APIs are available to your own automation.
+
+## Supported formats
+
+Support is for the **specific local payload profiles below**, not every API,
+export, or version offered by each producer. Select the version explicitly.
+
+| Your input | Supported profile | Start here |
+| --- | --- | --- |
+| **goppi** | Completed baseline exchange manifest v2; separate paired-handoff route | [`audit-baseline`](AGENT_USAGE.md#audit-a-completed-goppi-baseline-as-common-evidence) for descriptive common JSON · [`project-baseline`](AGENT_USAGE.md#project-a-completed-goppi-baseline-into-event-observations) for native measurement · [`review-handoff`](AGENT_USAGE.md#review-a-completed-paired-handoff-in-one-call) |
+| **Langfuse** | `4.35.0` server blob exports: observations-v2 and scores JSON arrays | [`audit-langfuse`](AGENT_USAGE.md#account-for-langfuse-score-kinds-and-associations), with an optional content-minimized row index |
+| **DeepEval** | Ordinary TestRun JSON, `4.1.1` or `4.2.8` | [`audit-deepeval`](AGENT_USAGE.md#account-for-deepeval-scores-and-evaluator-errors) |
+| **Promptfoo** | Eval output JSON, `0.123.0` or `0.124.1` | [`audit-promptfoo`](AGENT_USAGE.md#source-checkout-pilot-audit-promptfoo-counts) |
+| **External common result** | `evidence_result.v1` JSON from a producer | [`inspect-result`](AGENT_USAGE.md#inspect-an-external-common-result) for contract screening and availability counts, with optional evidence-item `next_action` and declared metric-binding counts |
+| **Native evidence** | `underwrite.evidence-bundle` / `v1` | `ingest` → `measure` → `accept` |
+| **Inspect AI** | `inspect.eval-log` / `0.3.263` | `ingest` |
+| **OpenInference** | `openinference.traces` / `0.1.38` | `ingest` |
+| **OTLP JSON** | `otlp-json.traces` / `0.160.0` | `ingest` |
+
+DeepEval accounting defaults to `4.1.1`; select `--version 4.2.8` for that
+profile. For a reusable owner-asserted metric-name pack and common descriptive
+JSON, see the [DeepEval binding workflow](AGENT_USAGE.md#account-for-deepeval-scores-and-evaluator-errors).
+Promptfoo accounting defaults to `0.123.0`; select `--version 0.124.1`
+for that profile. Langfuse's version identifies the server export, not its SDK.
+See [artifact requirements](AGENT_USAGE.md#get-the-right-producer-artifact) and
+[producer checks](EVIDENCE_REVIEW.md) for the tested scope and fixture origins.
+
+Generic `ingest` / `project` normalizes supported exports to an observation;
+external observations do not enter the current `measure` path. goppi uses its
+explicit exchange adapters: `audit-baseline` produces descriptive
+`evidence_result.v1` without a policy or directional verdict, while
+`project-baseline` can produce native event observations. `prepare-event-policy`
+binds a consumer configuration while leaving statistical assumptions unknown.
+For an already produced `evidence_result.v1` file, `inspect-result` checks the
+installed contract and item counts without verifying the producer's source or
+bundle, or admitting that result to measurement or acceptance.
+Its optional action view counts recorded next actions for evidence remediation,
+not executed tool calls or an agent trajectory.
+
+## Keep results you can return to
+
+Use `records` when you want the analysis and its selected source bytes retained
+together. For example, after the source quickstart:
+
+```sh
+mkdir -p output
+uv run --frozen --no-sync underwrite records audit-promptfoo \
+  fixtures/examples/evidence/synthetic-promptfoo-count-mismatch.json \
+  --root output/review-store --id demo-001 --json
+
+uv run --frozen --no-sync underwrite records read-json \
+  --root output/review-store --id demo-001 \
+  --name analysis/accounting.json --json
+```
+
+Use a fresh ID for each capture; existing records are never overwritten.
+`records verify` checks stored integrity. `records measure` captures a native
+measurement, and `records replay-measure` recalculates it from verified saved
+inputs using the current engine, without executing the original producer.
+For an exact completed goppi baseline v2 directory, `records audit-baseline`
+captures its five files and the descriptive result together in one call; see
+[the goppi record workflow](STORAGE.md#retain-a-completed-goppi-baseline).
+
+Records retain raw inputs, which may contain private content. Choose a private
+local POSIX filesystem and a backup strategy; this checkout's ignored `output/`
+is a demo location, not a backup. [Storage contracts and recovery](STORAGE.md).
+
+## What the result means
+
+underwrite keeps different kinds of evidence distinct:
+
+| Result | What it establishes |
+| --- | --- |
+| Source accounting | Relationships among the supplied rows, scores, and declared summaries |
+| Byte hashes and record verification | Identity and integrity of selected bytes, not independent provenance |
+| Native measurement | A result under the supplied event definition, population, and statistical assumptions |
+| Acceptance classification | A policy result from declared claims and reads; no merge or deployment authority |
+
+`not_measured`, `indeterminate`, and `abstained` remain different states. Missing
+information is explicit. A source score is not automatically a calibrated
+probability, and a successful command is not approval to ship.
+
+## Install
+
+For the workflows on this page, use the [source quickstart](#quickstart) or
+matching `0.1.0` app/core wheels. `uv sync` selects the workspace
+`underwrite-core`; locally built wheels must pair app and core from the same
+checkout.
+
+<details>
+<summary>Install the older published PyPI 0.0.1</summary>
+
+```sh
+uv tool install 'underwrite==0.0.1'
+underwrite --help
+```
+
+The older published [app](https://pypi.org/project/underwrite/0.0.1/) and
+[core](https://pypi.org/project/underwrite-core/0.0.1/) do not include the
+`0.1.0` measurement, accounting, records, handoff, or `capabilities` workflows.
+Discover installed commands with `--help`. Do not pair a `0.1.0` app wheel with
+the `0.0.1` core.
+
+The published `0.0.1` `pins refresh` should not process an untrusted pin
+manifest. A controlled local audit reproduced overwrite of a writable file
+outside the pin registry using a crafted manifest. The `0.1.0` code
+refuses that path before mutation; the older `0.0.1` artifacts do not include
+the correction. The reproduced case required the crafted manifest and a
+writable target file.
+
+</details>
+
+## CLI
+
+Use `underwrite <command> --help` for options. Alongside the workflows above,
+`inspect` checks artifact manifests, `audit-counts` checks Promptfoo counts through
+a manifest with an expected hash, `check` examines release evidence linkage,
+`absence` checks population presence, `boundary scan` checks configured verdict
+vocabulary, and `drift check` / `pins refresh` work with supplied pin registries.
+[JSON contracts](contracts/) define each command family's results and errors.
+
+Legacy `absence` and `drift check` exemption files and `boundary scan` allowlists
+are bounded to 1 MiB, 64 JSON container levels, and 1024 policy entries each.
+`boundary scan` reads at most 4 MiB per selected JSON or JSONL file and 32 MiB
+of captured bytes across the command, including its allowlist. It admits up to
+4096 selected files, 100000 visited directory entries, 100000 physical JSONL
+lines and 1 MiB per raw line (including its delimiter), and 4096 leaks including
+allowed rows. Its selected JSON or human output is capped at 1 MiB including
+the final newline. Duplicate keys and nonfinite numbers are invalid; supplied
+`--now` and policy expiry dates must be real ASCII `YYYY-MM-DD` dates. Admission
+failures return a typed `UNCONFIGURED` result with exit 2. An absent `--now`
+still yields `NOW_REQUIRED` only when the semantic check needs a clock.
+
+`absence` retains `Path.glob` matching and refuses more than 4096 matching
+regular files; it does not bound directory traversal within `Path.glob` or
+suppressed internal filesystem errors. Boundary scans follow an explicitly
+selected directory link and regular file links, but do not descend into a child
+directory link. These selections do not establish path containment against a
+hostile filesystem.
+
+`drift check` admits up to 256 manifests and 4096 declared pin files while
+counting at most 100000 visited registry entries. Each manifest is limited to
+1 MiB and 64 JSON container levels; each selected source or pin copy is limited
+to 4 MiB. The 32 MiB command capture budget includes exemption bytes, manifests,
+and both copies. Read-only inspection follows selected directory and file links;
+it does not create an atomic filesystem snapshot or a hostile-path containment
+guarantee. Missing or nonregular selected copies are reported as DRIFT; malformed,
+unreadable, or excessive registry input returns a typed `UNCONFIGURED` result
+without fabricated population counts.
+
+`pins refresh` selects one registry repository and copies committed Git blobs.
+Refresh paths must be normalized relative `/`-separated components; empty, dot,
+parent, backslash, Unicode control/format, oversized, and colliding destinations
+are refused.
+The selected registry and its existing destinations are checked before mutation.
+Each pin is replaced with a new inode under its opened registry parent, and
+`PINS.json` is replaced last. Admission or Git-source failures leave the registry
+unchanged. A later filesystem failure reports `publication: may_be_partial`;
+refresh is not an all-files transaction or a durability guarantee.
+Refresh captures each committed Git blob within 4 MiB and the 32 MiB aggregate
+budget, and checks the fully serialized 1 MiB manifest before its first write.
+
+<a id="development-checkout-evidence-pilots"></a>
+
+## Documentation
+
+| Explore | Guide |
+| --- | --- |
+| Run from an agent, choose inputs, interpret JSON | [Agent usage](AGENT_USAGE.md) |
+| Compare declarations, paired evidence, and goppi handoffs | [Evidence workflows](AGENT_USAGE.md#source-checkout-pilot-compare-declarations) |
+| Save, verify, retrieve, and recalculate | [Local records](STORAGE.md) |
+| Understand producer fixtures and tested boundaries | [Producer evidence](EVIDENCE_REVIEW.md) |
+| Extend the CLI and pure calculation core | [Contributing](CONTRIBUTING.md) · [Architecture rules](AGENTS.md) |
+
+## Contributing
+
+Bring a reproducible artifact, a concrete workflow, or a fix.
+[Open an issue](https://github.com/ntts9990/underwrite/issues) with the producer
+version and expected result; share only sanitized examples. See
+[CONTRIBUTING.md](CONTRIBUTING.md) for local preflight and PR checks, and
+[SECURITY.md](SECURITY.md) for private vulnerability reports.
+
+<details>
+<summary>Maintainers: future PyPI releases</summary>
+
+Before using OIDC publishing, create the GitHub environment `pypi`, restrict it to
+`main`, and configure a PyPI GitHub Trusted Publisher on **each** package with owner
+`ntts9990`, repository `underwrite`, workflow `publish.yml`, and environment `pypi`.
+Once configured, run **Publish PyPI** manually from `main` in GitHub Actions.
+
+The workflow validates both packages, then publishes core before the application;
+only the separate publish job receives OIDC permission. No stored PyPI API token
+is required. Retry a partial upload with the same artifacts: existing registry
+files must match exactly, and changed artifacts require a new version.
+
+See the official [uv publishing guide](https://docs.astral.sh/uv/guides/package/)
+and [PyPI Trusted Publisher guide](https://docs.pypi.org/trusted-publishers/using-a-publisher/).
+
+</details>
+
+## License
+
+[Apache-2.0](LICENSE). [LICENSING.md](LICENSING.md) records the adapted policy
+templates' CC licenses.