verieffect 0.2.0a1__tar.gz

verieffect-0.2.0a1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bing Feng
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

verieffect-0.2.0a1/PKG-INFO

@@ -0,0 +1,405 @@
+Metadata-Version: 2.4
+Name: verieffect
+Version: 0.2.0a1
+Summary: Independent deterministic verification for AI agents — signed, offline-verifiable receipts
+Author: Bing Feng
+License: MIT
+Project-URL: Homepage, https://github.com/bfeng1122/verieffect
+Project-URL: Repository, https://github.com/bfeng1122/verieffect
+Keywords: ai,agents,verification,receipts,ed25519,audit
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Security
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: cryptography>=41.0
+Requires-Dist: requests>=2.34.2
+Dynamic: license-file
+
+# Verieffect
+
+**Verify the effect. Not the claim.**
+
+*Independent, deterministic verification for AI agents — with signed, offline-verifiable
+receipts as the evidence format.*
+
+When an agent says "I uploaded the file" / "the trade filled" / "the backup ran" —
+Verieffect probes ground truth *itself* and issues a receipt anyone can verify offline.
+
+```console
+$ # the backup agent's log says: "nightly backup ran ✓"
+$ verieffect claim file_exists --agent backup-cron \
+    --target ./backups/nightly.dump --expect '{"min_bytes": 1048576}' --out receipt.json
+{
+  "v": 1,
+  "claim_id": "830c4620-deb4-4d5e-a4d2-ed83c77f79da",
+  "agent_id": "backup-cron",
+  "kind": "file_exists",
+  "target": "./backups/nightly.dump",
+  "expect": { "min_bytes": 1048576 },
+  "observed": { "path": "backups\\nightly.dump", "exists": true, "size": 2, "mtime": 1793862129 },
+  "verdict": "FAIL",
+  ...
+  "sig": "ed25519:8bfac27e…"
+}
+$ echo $?
+1
+$ python verify_receipt.py --pubkey ~/.verieffect/keys/verieffect_ed25519.pub receipt.json
+VALID: ok (claim 830c4620-deb4-4d5e-a4d2-ed83c77f79da, verdict FAIL)
+```
+(output from a Windows run — observed.path uses backslashes)
+
+The file exists — it's just 2 bytes. "Backup ran" was true; "backup landed" wasn't.
+The signed `FAIL` receipt is the evidence, and exit code `1` lets cron alert on it.
+
+> **Renamed from AgentProof** (2026-10-30) after finding 5+ existing products on that
+> name. Same project and repo history; the signing key is unchanged, so pre-rename
+> receipts still verify.
+
+> **What the signature does and doesn't prove.** The signing key lives next to your
+> agents (same host, MVP). Receipts are therefore **tamper-evident against drift and
+> sloppy self-reports — not tamper-proof against an adversarial agent with local code
+> execution** (such an agent could read the key and forge receipts). Verieffect is a
+> *drift detector* for the well-documented "agent says done, side effect didn't
+> happen" failure mode. Hosted/separate-user verification is on the roadmap. Details:
+> [threat model](docs/receipt-schema.md#threat-model-what-receipts-do-and-dont-prove).
+
+- **No LLM in the verdict path.** Verdicts come from deterministic probes: filesystem
+  state, HTTP responses, database rows, command exit codes. A model never *opines*
+  that work succeeded; evidence shows it.
+- **Signed receipts.** Ed25519 signature + per-agent hash chain, so receipts can't be
+  altered without the key, and edits after the fact are detected. (Honest scope: the
+  chain detects edits and mid-chain deletions; tail truncation needs an externally
+  anchored head hash — see [Checkpointing](#checkpointing-anti-truncation) below.)
+- **Three verdicts, never two.** `PASS` / `FAIL` / `UNVERIFIABLE`, each with its own
+  exit code (`0` / `1` / `3`). A network error, timeout or missing binary is never
+  laundered into a pass (or a fail). A missing file is never excused as "couldn't
+  check." A typo'd or unknown `expect` key is never silently ignored: it is
+  `UNVERIFIABLE` and the receipt names the key.
+
+## Why
+
+Two things are true at once: agent tasks get done wrong surprisingly often, and agents
+tend to report them as done anyway.
+
+**Tasks silently done wrong.** An analysis by [Veridatum](https://veridatum.io/) — a
+similar project we compare against below — of the public τ-bench retail benchmark found
+that **118 of 456 simulated GPT-4.1 runs (26%) failed the final-state check**: the
+agent's transcript and the database disagreed. Veridatum is explicit that this is a
+public-benchmark demonstration, not a real-world error rate, and it measures wrong
+outcomes rather than false reports. Our own inference (not Veridatum's claim): nothing
+in the agent's transcript tells you *which* runs were wrong — you find out only by
+checking the system of record.
+
+**Agents report failures as success.** The overclaiming evidence comes from the
+research literature:
+
+- **LIMBO** ("Where Does Exactly-Once Live?", arXiv:2609.29095): in **90% of episodes
+  that produced a duplicate side effect, the agent reported the task as completed.**
+  This is the closest published evidence to exactly what Verieffect catches — the
+  self-report says "done," the system of record says otherwise.
+- **OverclaimBench** (arXiv:2609.20812): **80.4% of incomplete runs** were reported in
+  a misleading way.
+- **"The Completion Illusion"** (IOV Labs; a lab study, not peer-reviewed): agents
+  self-reported perfect completion while verified accuracy was 86–96%.
+
+Autonomous agent stacks — cron-driven pipelines, CI bots, personal agents — fail the
+same way: the log says "done," the side effect never happened (or happened twice).
+Verieffect makes "done" a verifiable claim.
+
+## Related work (read this before assuming we invented it)
+
+Verifying an action's *effect* instead of trusting the actor's self-report is a
+published principle, not our invention. We built a working implementation; others
+described the idea or solved adjacent problems:
+
+- **[Re-Perceived Effects: A Well-Formedness Contract for Accountable Actuation](https://doi.org/10.5281/zenodo.22768261)**
+  (Zain Dana Harper, Sept 2026). The closest prior art: replace the actuator's
+  self-report with a *re-perceived* effect read back from the world — re-hash the
+  file, check the process return code — with a total checker that yields
+  "Unverifiable" on malformed input rather than raising. Same core principle and
+  several of the same rules as our `file_exists`/`exit_code` probes. It's a short
+  conceptual note; Verieffect is a shipped, tested implementation (200+ test checks, CI,
+  dogfooded) and adds the packaging that note names as its own gap: sealed
+  receipts re-verifiable offline.
+- **[Beyond Approved Actions: Runtime Validation of Persistent Outcomes in Agent Workflows](https://arxiv.org/abs/2609.31301)**
+  (EffectMatch, Sept 2026). Broader than us: binds an application-approved outcome
+  to the execution and governs commit/continuation from the comparison. We verify
+  individual claims after the fact; they enforce a contract across a workflow.
+- **[Treeship](https://github.com/zerkerlabs/treeship)** (Apache-2.0),
+  **[Attestroot](https://pypi.org/project/attestroot/)**,
+  **[Logsiegel](https://logsiegel.com)**, **[Tanilo](https://tanilo.io)** — signed,
+  hash-chained/Merkle agent receipts with offline verification. Overlapping on the
+  *receipt* layer. The distinction: they **record what the agent did** (wrap commands,
+  seal transcripts); Attestroot states plainly that a lying agent's capture "would
+  seal and verify perfectly." Verieffect **queries the system of record itself** when
+  the claim is filed. The probe runs inside Verieffect, not inside the agent. (Offline
+  verification later checks the receipt's authenticity; it does not re-probe.)
+- **[Veridatum](https://veridatum.io/)** — the same principle (check the claim against
+  the system of record) as closed SaaS for business outcomes (Stripe refunds, orders).
+  We're the open-source, developer-facing, technical-claim version.
+- Standards: [in-toto](https://in-toto.io)/[SLSA](https://slsa.dev) attest *artifacts
+  and builds*; [C2PA](https://c2pa.org) attests *content provenance*. None answers
+  "did this runtime side effect land in my DB at 3am." Adopting their envelope formats
+  (RFC 8785 canonical JSON, DSSE) is on our roadmap — we'd rather compose than compete.
+
+**What's left that's ours:** a Python-native probe library you can extend, strict
+three-verdict semantics, signed hash-chained receipts with agent-binding and
+truncation detection, a dependency-light offline verifier, and exit codes built for
+cron/CI. Implementation and developer experience — not the underlying idea.
+
+## Quick start
+
+```bash
+pip install verieffect        # from PyPI; gives you the `verieffect` console script
+```
+
+Or from a clone (for contributors):
+
+```bash
+python -m venv .venv
+.venv/bin/python -m pip install -r requirements.txt   # Windows: .venv/Scripts/python
+# Optional: `pip install .` also gives you a `verieffect` console script.
+```
+
+```bash
+# 1. File a claim — targets a file that ships with the repo, so this PASSES on a
+#    fresh clone (exit code 0). Keys/receipts land under ~/.verieffect (see below).
+.venv/bin/python -m verieffect.cli claim file_exists --agent my-agent \
+  --target ./README.md --expect '{"min_bytes": 1024}' --out receipt.json
+
+# 2. Verify the receipt offline (completes the claim→verify loop)
+.venv/bin/python verify_receipt.py \
+  --pubkey ~/.verieffect/keys/verieffect_ed25519.pub receipt.json
+
+# 3. Audit the whole chain for that agent
+.venv/bin/python -m verieffect.cli audit --agent my-agent
+```
+
+On Windows, substitute `.venv/Scripts/python` for `.venv/bin/python` throughout.
+
+> **Windows cmd/PowerShell:** single-quoted JSON doesn't work — use double quotes
+> with escaped inner quotes: `--expect "{\"min_bytes\": 1024}"`, and replace `\`
+> line continuations with `^` (cmd) or a backtick (PowerShell).
+
+Output (truncated):
+
+```json
+{
+  "v": 1,
+  "claim_id": "…",
+  "agent_id": "my-agent",
+  "kind": "file_exists",
+  "target": "./README.md",
+  "expect": {"min_bytes": 1024},
+  "observed": {"path": "README.md", "exists": true, "is_dir": false, "is_regular_file": true,
+               "size": 4096, "mtime": 1792090847},
+  "verdict": "PASS",
+  "checked_at": "2026-10-20T23:12:06Z",
+  "prev_receipt_hash": "…",
+  "sig": "ed25519:…"
+}
+```
+
+Exit codes — `claim` (`python -m verieffect.cli claim …`), so shell scripts and
+cron jobs can branch on verification results directly:
+
+| Code | Meaning |
+|---|---|
+| `0` | PASS |
+| `1` | FAIL — the observed state contradicts the expectation |
+| `3` | UNVERIFIABLE — ground truth couldn't be reached or judged (network error, timeout, binary not found, malformed/unknown `expect` key, non-object `expect` such as `null`, malformed SHA-256, no assertion, a refused probe — `exit_code` without `allow_shell`, non-GET/HEAD `http_ok` — or a response/output over its size cap). A receipt is still written. |
+| `2` | usage error, or claim aborted before any receipt was written (bad `--expect` JSON incl. duplicate keys, corrupt chain tail, lock timeout) |
+| `4` | internal error (a Verieffect bug — please report it) |
+
+Exit codes — offline verifier (`verify_receipt.py`). Verification checks
+authenticity offline. It never re-runs probes, because probing happened when the
+claim was filed. Two modes:
+
+- **Default (authenticity only):** `0` = every receipt is authentically signed,
+  schema-valid and (in `--chain` mode) correctly chained. This says nothing about
+  the verdict — a validly signed **FAIL** receipt exits `0`, because the receipt
+  itself is genuine evidence. `1` = invalid/malformed/tampered, `--agent`
+  mismatch (single-receipt and chain mode), or an **empty chain** (zero receipts
+  prove nothing).
+- **`--require-verdict PASS|FAIL|UNVERIFIABLE`:** additionally gates on the
+  verdict (in `--chain` mode: *every* receipt's verdict must match). If the only
+  mismatches are `UNVERIFIABLE` receipts, it exits `3`; any other mismatch exits `1`.
+  Use `--require-verdict PASS` when the exit code should mean "authentic AND
+  passed". Chain mode always prints a summary line such as
+  `verdicts: 12 PASS, 1 FAIL, 0 UNVERIFIABLE`.
+- `2` = usage error, `4` = internal verifier error.
+
+### Where things live
+
+Default root is a **per-user** directory, independent of the current working
+directory: `~/.verieffect` (Windows: `%USERPROFILE%\.verieffect`). Every invocation
+therefore shares one key and one set of chains, and the private key never lands
+inside whatever repo the agent happens to run in.
+
+- `~/.verieffect/keys/verieffect_ed25519.{pem,pub}` — Ed25519 keypair, auto-generated
+  on first claim (atomically, mode 0600 on POSIX). The `.pem` (private) never leaves
+  your machine.
+- `~/.verieffect/chains/chain_<agent>_<hash8>.jsonl` — append-only receipt chain per agent.
+- Override the root with `VERIEFFECT_HOME=/some/path`.
+- **Upgrading from 0.1.0a1 to 0.2.0a1** (old default: `./data` under the CWD): if
+  `./data/keys/verieffect_ed25519.pem` exists and `~/.verieffect` doesn't (an
+  unrelated `./data/keys` folder is ignored), Verieffect prints the exact move command and
+  refuses to mint a second key until you either move the directory
+  (`mv ./data ~/.verieffect`, Windows: `move data "%USERPROFILE%\.verieffect"`) or
+  set `VERIEFFECT_HOME` to the old path. Nothing is moved automatically.
+- **Crash recovery:** if a crash tears the last chain line mid-write, new claims for
+  that agent stop with a pointer to `verieffect repair --agent <id>`. That command
+  moves the torn line to `<chain>.torn` and appends a signed `repair_note` receipt.
+  The torn receipt's verdict is lost (it may have been a FAIL), so `audit` and
+  `verify_receipt.py --chain` always print a **WARNING** listing every repair note.
+  Repair notes are not verdicts: `--require-verdict` skips them (one repair does not
+  block a strict gate forever), but a chain of *only* repair notes never meets it.
+
+## How it compares
+
+| | Verieffect | Treeship | Attestroot | Logsiegel | Veridatum |
+|---|---|---|---|---|---|
+| Queries the system of record itself (at claim time) | **Yes (probes)** | No — wraps/records | No — seals transcripts | No — chains logs | Yes |
+| Signed receipts + hash chain | Yes | Yes | Yes (Merkle + transparency log) | Yes | Ledger, not crypto-first |
+| Offline verification | Yes | Yes | Yes | Yes | No (SaaS) |
+| Deployment | Local CLI/library | Local CLI + hub | Local + countersign service | Local OSS | Hosted SaaS |
+| Scope | Technical claims (files/HTTP/DB/exit) | Agent sessions | Agent transcripts | Action trails | Business outcomes (Stripe, orders) |
+| License | MIT | Apache-2.0 | MIT (Pre-Alpha) | Apache-2.0 | Proprietary |
+
+The one-column summary: **the receipt tools above prove what the agent *recorded
+doing*; Verieffect checks whether the effect *actually landed*.** A lying agent can
+poison a recording. It can't fake what Verieffect's own probe observes. Honest
+caveat: today the claimer still picks the `target` **and** the `expect`, so a hostile
+agent could choose a check it knows will pass. The roadmap fix is operator-defined
+named checks (`verieffect check <name>`): the operator owns the target and
+expectations, and the agent only names the check. (Veridatum also checks
+the system of record, as hosted SaaS for business outcomes — see
+[Related work](#related-work-read-this-before-assuming-we-invented-it).)
+
+## How it fits together
+
+```mermaid
+flowchart LR
+    A[AI Agent] -->|"claim: I did X"| B[Verieffect CLI/library]
+    B --> C{Deterministic Probe}
+    C -->|reads| D[Real System State<br/>files · HTTP · DB · exit codes]
+    D --> C
+    C -->|PASS / FAIL / UNVERIFIABLE| E[Ed25519 Signing]
+    E --> F[Signed Receipt]
+    F --> G[Per-Agent Hash Chain<br/>append-only JSONL]
+    G --> H[Offline Verifier<br/>verify_receipt.py]
+    H --> I[Chain Audit<br/>+ external head anchor]
+```
+
+## Claim kinds
+
+| Kind | Target | Proves |
+|---|---|---|
+| `file_exists` | path | file present; size bounds / sha256 / mtime freshness / text content |
+| `http_ok` | URL | endpoint answers; status / body substring / body hash (probe makes the request itself). GET/HEAD only unless you explicitly pass `"allow_unsafe_method": true` |
+| `db_row_matches` | `sqlite:///x.db`, `x.jsonl`, `x.json` | record state: SELECT-only queries (`PRAGMA query_only` enforced), JSONL row filters, dotted-path JSON fields |
+| `exit_code` | shell command | command **run by Verieffect when the claim is filed** (not the agent's original run): exit code / stdout content. **Off by default** (`"allow_shell": true` required); runs with the verifier's full OS privileges. Use only for idempotent/read-only commands — for side-effecting work, verify the *effect* (`file_exists`/`db_row_matches`) instead |
+
+⚠️ **Trust boundary:** `http_ok` and `exit_code` act on the verifier's machine and
+network position, and every file-reading probe can read whatever the verifier can.
+Only wire claims from agents **you** operate. A claimer who can submit an `exit_code`
+claim can run commands, because the `allow_shell` / `allow_unsafe_method` gates are
+part of the claim itself. There are three known limitations: claim-payload gates,
+no path restriction (file-read oracle) and no SSRF filtering. Each must be fixed
+before the MCP server ships; see
+[Known limitations](docs/receipt-schema.md#known-limitations--fix-gated-on-the-mcp-roadmap).
+
+## Offline verification
+
+Anyone with the public key can verify receipts or whole chains — the standalone
+verifier needs nothing but `pip install cryptography`:
+
+```bash
+python verify_receipt.py --pubkey ~/.verieffect/keys/verieffect_ed25519.pub receipt.json
+python verify_receipt.py --pubkey ~/.verieffect/keys/verieffect_ed25519.pub \
+  --chain ~/.verieffect/chains/chain_my-agent_<hash8>.jsonl --agent my-agent
+
+# With an externally anchored head, also catches tail truncation/regeneration:
+python verify_receipt.py --pubkey verieffect_ed25519.pub --chain chain.jsonl \
+  --agent my-agent --expect-count 42 --expect-head <sha256-of-last-receipt>
+```
+
+`verify_receipt.py` has **zero internal imports** by design — it is the reference
+implementation of the receipt format and never raises on malformed input.
+See [`docs/receipt-schema.md`](docs/receipt-schema.md) for the full spec.
+
+### Checkpointing (anti-truncation)
+
+A forward-walk audit can't see tail truncation unless the chain head was anchored
+somewhere the operator can't quietly rewrite. `checkpoint` prints the anchor:
+
+```bash
+python -m verieffect.cli checkpoint --agent my-agent --json
+# {"agent_id": "my-agent", "count": 42, "head": "0dabf352...", "as_of": "..."}
+```
+
+Wire it into anything append-only you already have: a nightly git commit of the
+output, an OpenTimestamps stamp, a message to a channel only you control. Later:
+
+```bash
+python -m verieffect.cli audit --agent my-agent --expect-count 42 --expect-head 0dabf352...
+```
+
+**Anchor the head, not just the count.** `--expect-count` alone is not sufficient:
+`verieffect repair` replaces a torn receipt with a repair note one-for-one, so a
+clipped FAIL plus a repair keeps the count unchanged. Only `--expect-head` catches it
+(and `audit` prints the repair-note WARNING either way).
+
+### Prefer body hashes over substrings
+
+For `http_ok`, `body_contains` is a smoke test — brittle on any dynamic page (dates,
+nonces, A/B variants). `body_sha256` is the strong check for static resources; use
+substring matching only for stable markers (e.g. `"status":"ok"` in an API health
+response).
+
+## Design rules
+
+1. The probe is owned by the verifier, never the claimer. Self-reported results are
+   what we're checking, not what we trust.
+2. Anything not provable deterministically is `UNVERIFIABLE`, not `PASS`. That
+   includes an `expect` that asserts nothing (e.g. a SQL query with neither `rows`
+   nor `row_count`) and any unknown or wrongly typed `expect` key. Each probe has
+   a strict key schema; see [receipt-schema.md](docs/receipt-schema.md#expect-schemas-strict).
+3. `db_row_matches` is read-only (non-SELECT rejected, `query_only` enforced);
+   `http_ok` is read-only by default. The verifier never mutates audited systems
+   unless the claim explicitly opts in (`allow_shell` / `allow_unsafe_method` — today
+   these live in the claim payload, so they guard against accidents, not attackers).
+4. The hash chain is the audit trail — append-only; edits and mid-chain deletions are
+   detected by any offline auditor. Anchor the head hash externally (a daily commit,
+   an OpenTimestamps stamp, a posted checkpoint) to also detect truncation.
+
+## Threat model in one paragraph
+
+Verieffect catches **sloppy** agents, not **adversarial** ones: tasks that were done
+wrong, and agents whose "done" report doesn't match what actually landed. The signing key lives on the operator's
+machine, so an attacker with local code execution as the same user can forge
+receipts — no same-host scheme survives that. If your threat is a truly adversarial
+agent, run the verifier as a separate OS user/host (roadmap: hosted verifier).
+Full details in [receipt-schema.md](docs/receipt-schema.md).
+
+## Status
+
+MVP (Alpha). 200+ test checks (`python tests/test_all.py` with the venv python, or
+`pytest tests/`; CI runs ubuntu+windows × Python 3.10–3.12). Dogfooded on a 24/7
+paper-trading stack since late Oct 2026 (40+ receipts): fills, portfolio state and
+backups, plus a deliberate tamper test that breaks the chain as designed.
+Not yet battle-tested by anyone else.
+Roadmap: named operator-owned checks (`verieffect check <name>`), hosted verifier
+API, `kid` key rotation, `claimed_at`/`checked_at` split, connector library
+(Stripe/broker/Postgres), MCP server.
+
+See [examples/](examples/README.md) for runnable examples of all four probe kinds,
+[CONTRIBUTING.md](CONTRIBUTING.md) for adding probes, and
+[SECURITY.md](SECURITY.md) for vulnerability reporting.
+
+MIT licensed.