agentmeasure 0.5.0__tar.gz

agentmeasure-0.5.0/PKG-INFO

@@ -0,0 +1,389 @@
+Metadata-Version: 2.4
+Name: agentmeasure
+Version: 0.5.0
+Summary: AgentMeasure — check-up report for coding agent runs + conformance pack for telemetry fixture verification. Zero external dependencies.
+License: MIT
+Keywords: agent,codex,claude,observability,measurement,healthcheck,conformance
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# AgentMeasure Healthcheck (pre-release)
+
+**A local check-up report for your coding agent runs.**
+Point it at the logs your agent runtime already writes. It answers three
+questions with evidence — *which records are duplicated, where did the agent
+pay retries for one logical operation, and where did the same tool fail over
+and over* — and writes a terminal summary plus a local HTML report.
+
+Same honesty rules as the [conformance pack](../conformance/pack/README.md):
+every verdict is `OK / FINDING / UNPROVABLE`, and UNPROVABLE is a first-class
+result — when the logs cannot decide, that is disclosed, never zeroed.
+
+> Status: engineering preview (v0.3.0). Adapters: **Codex rollout logs** and
+> **Claude Code session logs** (`--runtime claude`; auto-detected when Codex is
+> absent). Claude v1 scope: duplicate / retry / error-run checks fully supported;
+> session token accounting is honestly **UNPROVABLE** — Claude transcripts carry
+> per-request usage only, no cumulative counter, and per-request sums are never
+> presented as session totals.
+> Distribution name decided: **`agentmeasure`** on PyPI (command of the same
+> name). PyPI publish is armed and goes live with tag `v0.3.0` once the PyPI
+> account/trusted publisher is configured — until then use the git install.
+
+## Install
+
+> **Note (2026-09-21):** PyPI still serves **0.4.0**, which crashes on the
+> first command (`NameError` in `_run_check`; `demo` / `check` / `compare`
+> all affected). The fix is in 0.4.1 and already on main. Until the 0.4.1
+> wheel lands on the index, install from git instead (option 3 below) —
+> that path runs clean.
+
+```bash
+# 1) from PyPI (live since v0.4.0 — see note above; 0.4.1 pending upload)
+pipx run agentmeasure demo
+
+# 2) install the command
+pipx install agentmeasure
+
+# 3) from git (tracks main)
+pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"
+
+# 4) zero install, straight from a repository checkout
+python3 healthcheck/agentmeasure demo
+```
+
+Python 3.9+ standard library only; zero runtime dependencies; no network code
+(a test enforces both).
+
+## Quick start
+
+```bash
+agentmeasure demo      # synthetic session — see it work, no data needed
+agentmeasure check     # your agent logs, last 7 days (Codex or Claude Code)
+agentmeasure check --all           # every local session
+agentmeasure check --runtime claude   # force the Claude Code adapter
+agentmeasure check --dir ~/.codex/sessions/2026/09/05
+agentmeasure selftest  # adapters + checks on bundled fixtures
+agentmeasure validate export.json  # check an export against its schema
+agentmeasure share run.json        # PREVIEW the sanitized summary (writes nothing)
+agentmeasure share run.json --out summary.md   # export after review
+agentmeasure --version
+```
+
+(Without installing, prefix the commands with `python3 healthcheck/agentmeasure`.)
+
+Options for `check` / `demo`: `--html PATH` (default `agentmeasure-report.html`),
+`--json PATH` (full machine-readable export), `--share PATH` (sanitized summary,
+`.md` or `.json`), `--days N`, `--save-snapshot [PATH]`, `--no-history`.
+`demo` accepts the same output paths and `--no-history`.
+
+Input filters for `check` and `compare`:
+
+```bash
+--since 2026-09-01 --until 2026-09-05   # date range (overrides --days)
+--project myrepo                        # keep sessions whose project matches
+                                        # (project = cwd basename, case-insensitive)
+```
+
+## Outcome-billing verification: prepare → recount → dispute → recovery
+
+The buyer-side bill check for per-resolution vendors (Intercom Fin, Zendesk AI
+Agents, …). Rules live in the [vendor-rules registry](../registry/vendor-rules.json)
+(single source of truth, each rule cited and confidence-graded). Four commands,
+one pipeline:
+
+```bash
+agentmeasure prepare --export native-export.csv --vendor zendesk --inspect
+agentmeasure prepare --export native-export.csv --vendor zendesk --out prepared.csv
+#   native Intercom/Zendesk export -> canonical skeleton. Judgement columns
+#   stay EMPTY for human review; native signal columns travel as hint_*
+#   cells; prov_* cells record where every value came from. No verdicts.
+
+agentmeasure recount --export prepared.csv --vendor zendesk --inspect
+agentmeasure recount --export prepared.csv --vendor zendesk --json recount.json
+agentmeasure recount --export prepared.csv --vendor zendesk --contract contract.json
+#   Tier 1: the vendor's own published rules, line by line. Findings in BOTH
+#   directions; absent evidence is UNPROVABLE, never guessed. --contract adds
+#   the buyer's outcome standard (e.g. reopen_window_hours: 72) as a separate
+#   outcome lane — renewal leverage, never netted into the claim.
+
+agentmeasure dispute --export prepared.csv --vendor zendesk --contract contract.json
+#   the negotiable pack: cover letter citing the vendor's own rule per
+#   finding + appendix + (optionally) Tier 2 under the AMS-1 standard.
+
+agentmeasure recovery --pack pack/dispute-pack.json --confirmations confirmations.csv
+#   what the vendor actually conceded: credited/paid per disputed line, with
+#   confirmation dates and evidence refs. Realized value is counted
+#   separately from the claim — an argument won is not cash.
+```
+
+`agentmeasure recount --list-vendors` shows the registry with each rule's
+source and confidence (P = primary/official, S = secondary, U = unverified).
+The browser-local recount at [the website](../website/) runs the same Tier 1
+with no upload; a cross-language parity test keeps the two from drifting.
+
+### The whole story in one artifact
+
+```bash
+agentmeasure crosscheck --export prepared.csv --vendor zendesk --ledger billing.csv
+#   your billing ledger (计费流水) against the export: charges the export
+#   doesn't flag, billed flags the ledger doesn't confirm, amounts off the
+#   published price. Disagreements are named, never resolved.
+
+agentmeasure verify --export prepared.csv --vendor zendesk --contract contract.json \
+    --ledger billing.csv --confirmations confirmations.csv --out vledger
+#   the Verified Ledger: Tier 1 + outcome lane + ledger cross-check +
+#   realized recovery in ONE document (json + md), each lane labelled
+#   claim / leverage / realized and never summed across.
+
+agentmeasure dashboard --pack vledger/verified-ledger.json --out dashboard.html
+#   offline single-file HTML: cards, tables, recovery bar. No JS, no
+#   network, printable — renders, never recomputes.
+
+agentmeasure rules-diff last-month-registry.json
+#   the monthly ritual: diff last period's rules snapshot against the
+#   packaged registry. Exit 1 = a material change (trigger, price, reopen
+#   rule) — re-verify the period under the new rule and disclose it.
+
+agentmeasure narrative --pack pack/dispute-pack.json --audience vendor \
+    --emit-prompt prompt.txt
+#   step 1 of two: the drafting prompt, built from the pack. Run YOUR OWN
+#   model on it, then:
+agentmeasure narrative --pack pack/dispute-pack.json --check-draft draft.txt --out letter.md
+#   step 2: every number in the draft is checked against the pack; any
+#   invented or rounded number refuses the draft and the rule-based
+#   template ships. The model never touches the numbers. No model? The
+#   template is the default.
+
+agentmeasure delivery --log delivery.jsonl --log-event \
+    --engagement ACME --vendor zendesk --period-index 1 --phase mapping --minutes 45
+agentmeasure delivery --log delivery.jsonl
+#   appendix-E metrics from logged phases: first look ≤10h, third period
+#   ≤1h, repeat-period reuse ≥70%. Use the mapping fingerprint `prepare`
+#   prints to track which engagements share the reusable mapping.
+```
+
+Outcome pricing beyond per-resolution support (qualified lead, completed
+case) is defined — schema first, per the BP — at
+[schemas/outcome-unit.schema.json](../schemas/outcome-unit.schema.json) with
+worked examples. The outcome **engine** now ships too:
+`agentmeasure outcomes --units units.jsonl` verifies them fail-closed, and the
+schema covers action and usage-aggregate units — one evidence plane across the
+BP's three stages.
+
+```bash
+agentmeasure crm-join --export prepared.csv --crm crm.csv --out joined.csv
+agentmeasure outcomes --units outcome-units.jsonl --json outcome-ledger.json
+agentmeasure period-compare last-month/verified-ledger.json vledger/verified-ledger.json
+agentmeasure template --export --vendor intercom --fingerprint <fp> --contract contract.json
+agentmeasure certify --pack vledger/verified-ledger.json --digest-out digest.txt
+agentmeasure incrementality --events treatment-holdout.csv
+```
+
+### Agent commerce: the receipt where the click disappears
+
+Traditional attribution starts at the click; agent commerce has none. The
+merchant-side measurement receipt (F2.1-F2.9):
+
+```bash
+agentmeasure commerce-ledger \
+    --events agent-events.jsonl --orders merchant-orders.csv \
+    --payments merchant-payments.csv --policy-version 1.0.0
+#   metric + value + evidence table; retries reconcile to 1 Operation /
+#   2 Attempts / 1 verified order; net GMV is strong; incremental is
+#   UNPROVABLE (no holdout, no claim); materiality prints the discrepancy
+#   vs the naive dashboard in currency AND percent.
+```
+
+live / replay / synthetic never mix inside one aggregation (error, not a
+blend); metric names carry their strength prefix and are linted against the
+evidence that backs them; every input is sha256-anchored. The operating layer on top:
+
+```bash
+agentmeasure pnl-trees --orders orders.csv --campaigns campaigns.csv   # Organic / Paid, ROAS in Paid only
+agentmeasure cm-ledger --orders orders.csv --costs costs.csv --period 2026-09
+agentmeasure demand-audit --measurements m.csv --thresholds t.json     # Launch / Watch / Not ready
+agentmeasure intent-taxonomy --file clusters.json                      # Operating Cell health
+agentmeasure decision-audit --policy policy.json --executions log.jsonl
+agentmeasure benchmark-export --rows brand-rows.csv --min-brands 5     # aggregates only, ever
+```
+
+ Schema:
+[schemas/commerce-profile.schema.json](../schemas/commerce-profile.schema.json);
+canonical retry vector pinned in CI at
+`conformance/vectors/commerce-retry-baseline/`.
+
+`certify` grades the artifact on the AMS-1 adoption ladder (Stage 0..3) and
+emits the deterministic digest; hand the digest to your own signing tool —
+this package never runs one. `incrementality` produces statistics about a
+difference, never a verdict: the BP's line between rule judgement and causal
+inference stays visible in the output itself.
+
+## Sharing results: preview first, export second
+
+The recommended flow never writes a shareable file until you have read it:
+
+```bash
+agentmeasure check --json run.json          # the report export embeds the summary
+agentmeasure share run.json                 # terminal PREVIEW — nothing is written
+agentmeasure share run.json --out summary.md  # export only after you reviewed it
+```
+
+`share` re-checks the summary against its whitelist before showing or writing
+anything, so a tampered or future-format export is refused rather than leaked.
+`check --share PATH` remains available as the one-step explicit alternative.
+
+## Snapshots & compare — did the change help?
+
+The re-run preview: save a snapshot, change something (a prompt, a Skill, a
+dependency), run the agent again, compare.
+
+```bash
+python3 healthcheck/agentmeasure check --save-snapshot before.json
+# …make your change and run the agent…
+python3 healthcheck/agentmeasure check --save-snapshot after.json
+python3 healthcheck/agentmeasure compare before.json after.json
+# or skip the second snapshot: compare runs a fresh check as side B
+python3 healthcheck/agentmeasure compare --all before.json
+```
+
+`compare` prints metric deltas (failed executions, retry chains, …), check
+verdict transitions (`HC-02 OK → FINDING`), and honest caveats: token totals
+are differenced only when **both** sides are provable, and version / window /
+mode mismatches are disclosed because they change what a delta is allowed to
+mean. Snapshots are local artifacts (they contain project names and session
+short-ids); the snapshot schema is versioned and validated on load.
+
+## Versioned export schemas — how other tools integrate (R8)
+
+Everything a third party needs is a JSON document with a version on it:
+
+| document | written by | schema |
+| --- | --- | --- |
+| report export | `check --json PATH` / `demo --json PATH` | `report-v1` |
+| snapshot | `--save-snapshot PATH` | `snapshot-v1` (schema field = 1) |
+| comparison | `compare --json PATH` | `compare-v1` |
+
+Reference schemas live in [`schemas/`](schemas/). Compatibility policy:
+**within a schema version, fields are only added — never renamed, retyped, or
+removed**; a consumer that meets an unknown newer version must stop and say so.
+`agentmeasure validate FILE…` checks any export against its schema, and
+[`examples/track-weekly.py`](examples/track-weekly.py) is a complete external
+consumer: it reads snapshots through the schema only, imports nothing from the
+package, and refuses future schema versions. MCP-style servers can wrap the
+same exports later; the exports, not internal modules, are the integration
+surface.
+
+## What it checks
+
+| id | check | question it answers | verdict basis |
+| --- | --- | --- | --- |
+| HC-01 | Duplicate records | byte-identical lines, repeated call/execution ids, sessions split across files, and the format's **by-design dual recording** when model-side calls and command events are both present | exact line hashes, id counts |
+| HC-02 | Retry amplification | same command failed then re-run: how many executions carried one logical operation | consecutive same-command blocks whose first attempt failed |
+| HC-03 | Tool error runs | ≥3 back-to-back failures of the same tool with no success between | consecutive failed executions, same tool kind |
+
+Plus a coverage overview: sessions, turns, executions by outcome, token
+consumption (with subset discipline), compactions, sub-agent activity, corrupt
+lines, and a **per-project breakdown** (project = cwd basename, canonical
+deduplicated counts). Every finding carries evidence — file, line number, exit
+codes — and a concrete next step.
+
+## Honest-number rules (the part most tools get wrong)
+
+- **Executions are counted from exactly one stream.** Codex Desktop can write a
+  tool call as both a `response_item` record and an `item_completed` event.
+  Summing both — or grepping the file — double-counts when both are present.
+  HC-01 discloses the observed ratio instead of hiding it.
+- **Token subsets are never added into totals.** `cached_input` ⊆ input,
+  `reasoning_output` ⊆ output in Codex logs. The report shows subsets
+  alongside, never inside, totals.
+- **Token totals use the last cumulative snapshot per session**, not sums of
+  per-event deltas (which double-count after context compaction). The aggregate
+  is marked UNPROVABLE unless every logical session has one valid snapshot;
+  missing or malformed snapshots are reported separately.
+- **Corrupt or truncated input lowers every count.** All numbers are lower bounds
+  where lines failed to parse or a file hit the safety cap; the limitation is
+  reported, never dropped.
+- **UNPROVABLE, not zero.** Sessions whose format lacks outcome events make
+  HC-02/HC-03 UNPROVABLE for that part — reported as such.
+
+## Statistics definitions (for support and accounting)
+
+| term | definition |
+| --- | --- |
+| execution | one `CommandExecution` / `McpToolCall` event; its outcome is `ok`, `failed`, or `unknown` |
+| retry chain | a maximal block of consecutive executions of the same command whose first attempt failed; one chain = one logical operation executed N times |
+| resolved chain | a chain whose last attempt succeeded |
+| tool error run | ≥3 consecutive failed executions of the same tool kind with no success or unknown in between |
+| token total | last cumulative `total_token_usage` snapshot per session, summed only when every session has a valid snapshot; otherwise UNPROVABLE |
+| project | basename of the session's recorded working directory (`session_meta.cwd`); used for the breakdown, `--project` filtering, and snapshots — never included in the share summary |
+| snapshot | versioned JSON record of one run's window, verdicts, and aggregate counters (schema 1); local artifact |
+| comparison delta | `B − A` per metric; tokens are differenced only when provable on both sides, and version/window/mode mismatches are disclosed as caveats |
+| demo run | executed on the bundled synthetic session; counted separately (`synthetic-demo`) |
+| own-data run | executed on real local logs (`own-data`) |
+| run number | local run history at `~/.agentmeasure/history.jsonl` distinguishes first vs repeated runs; delete the file to reset |
+
+## Privacy
+
+- **No network.** The package contains no network code; a test enforces this by
+  scanning every module for network imports.
+- **Raw data stays local.** Prompts, message text, and reasoning content are
+  never read into the model (only envelope types, ids, exit codes, durations,
+  and token counters).
+- **Two artifacts, two levels.** The HTML report is personal (local paths and
+  evidence metadata — raw commands are represented by hashes, and the banner
+  says so). The `--share` summary
+  is built from a fixed whitelist of aggregate counts: no prompts, paths,
+  commands, repo names, or session ids can appear in it, and a planted-string
+  test enforces this.
+- **Run history** (`~/.agentmeasure/history.jsonl`) stays on the machine and is
+  deletable.
+
+## Verified runtime support
+
+| runtime | status | verified against |
+| --- | --- | --- |
+| Codex rollout JSONL (Desktop originator) | supported | cli 0.142.3 – 0.153.0; review snapshot 54 local files, 48 sessions, ~83k lines (2026-06-30 → 2026-09-05). Sessions without `CommandExecution` events (older format) run with HC-02/HC-03 UNPROVABLE, disclosed. |
+| Codex CLI (`codex_cli_rs` originator) | expected compatible, **not yet verified** — no local samples | pending real samples |
+| Claude Code (`~/.claude/projects`) | detected, **not yet supported** | adapter ships after validation against real samples |
+
+New cli versions are handled defensively: unknown event types are accounted
+(`unknown type` counts), never silently dropped.
+
+## Exit codes
+
+| code | meaning |
+| --- | --- |
+| 0 | the check ran and produced a report (findings or not) |
+| 1 | selftest fixture mismatch (development only) |
+| 2 | input/output error: no readable logs in the window, bad `--dir`/arguments, or an output path cannot be written |
+
+## Development
+
+```bash
+python3 -m unittest discover -s healthcheck/tests   # 248 tests
+python3 healthcheck/agentmeasure selftest           # fixtures + redaction + compare + schemas
+bash healthcheck/scripts/smoke_install.sh           # clean-venv install smoke (packaged artifact)
+```
+
+The install smoke is also a CI job (Python 3.9/3.11): it builds the package,
+installs it into a fresh venv, and runs `selftest`, `demo`, `validate`, and the
+external example from the installed artifact — the repository checkout is not
+used.
+
+Python 3.9+ standard library only. Zero dependencies, by policy: if any step
+asks you to register, connect to a network, or pay — that is a bug.
+
+## Feedback
+
+- [Open an issue](https://github.com/roy-tong/AgentMeasure/issues) — include
+  the `--json` export (it contains local paths; strip them first if the log
+  location is sensitive).
+- [Email](mailto:tongroy18@gmail.com?subject=AgentMeasure%20Healthcheck%20feedback)