plumbline-harness 0.3.0__tar.gz

plumbline_harness-0.3.0/LICENSE

@@ -0,0 +1,216 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+   Copyright 2026 Bhargav Raghavendra
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

plumbline_harness-0.3.0/PKG-INFO

@@ -0,0 +1,276 @@
+Metadata-Version: 2.4
+Name: plumbline-harness
+Version: 0.3.0
+Summary: Conformance-under-perturbation testing for LLM agents.
+Author: Bhargav Raghavendra
+License: Apache-2.0
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=1.0; extra == "anthropic"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110; extra == "server"
+Requires-Dist: uvicorn>=0.27; extra == "server"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: anthropic>=1.0; extra == "dev"
+Requires-Dist: fastapi>=0.110; extra == "dev"
+Requires-Dist: uvicorn>=0.27; extra == "dev"
+Dynamic: license-file
+
+# Plumbline
+
+**Metamorphic conformance testing for LLM agents.** Declare the invariants an agent must never violate; try to break them with input changes that preserve meaning; localise every violation to a named step with a confidence interval.
+
+[![tests](https://github.com/Bhargs24/plumbline/actions/workflows/tests.yml/badge.svg)](https://github.com/Bhargs24/plumbline/actions/workflows/tests.yml)
+![python](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue)
+![evidence](https://img.shields.io/badge/evidence-2%2C082%20trajectories%20committed-informational)
+![licence](https://img.shields.io/badge/licence-Apache--2.0-lightgrey)
+
+---
+
+## 1 · The finding: a significant result that was entirely baseline artifact
+
+I built this to measure what moving an agent's control flow out of the model actually buys. The first study said: a lot. Under an injected transient tool failure, the deterministic executor reached the correct outcome **81.2%** of the time against the free-form agent's **98.2%**. A 17-point gap, p = 0.0029. Determinism looked *worse*.
+
+That result does not survive contact with how these systems are actually built. **No production finance system treats a single 503 as fatal.** Every RPA platform has a retry policy; every payments integration has backoff. The executor under test had neither.
+
+Adding one and re-running the same 768 trials settles it. (The retry study
+replays the first study's recorded model responses offline — the executor's
+single interpretation call is byte-identical by construction, so only the
+error-handling differs. 52 of the free-form agent's 768 replays had no
+recorded response to replay and are excluded as missing data; every cell's
+n is in the report's tooltips.)
+
+<img src="docs/assets/results.svg" alt="Outcome correctness by perturbation for three series with 95% Wilson intervals. Every condition sits at 100% except tool_fault, where the no-retry executor falls to 81.2% while the retry executor and the agent both hold near 100%." width="100%">
+
+| Condition | deterministic, **no retry** | deterministic, **3 retries** | free-form agent |
+|---|---|---|---|
+| baseline | 100.0% [94.3, 100] | 100.0% [94.3, 100] | 100.0% [94.3, 100] |
+| paraphrase | 100.0% [94.3, 100] | 100.0% [94.3, 100] | 100.0% [94.3, 100] |
+| distractor | 100.0% [94.3, 100] | 100.0% [94.3, 100] | 98.3% [90.9, 99.7] |
+| decoy_tools | 100.0% [94.3, 100] | 100.0% [94.3, 100] | 100.0% [89.8, 100] |
+| sampling | 100.0% [94.3, 100] | 100.0% [94.3, 100] | 100.0% [93.6, 100] |
+| **tool_fault** | **81.2%** [70.0, 88.9] | **100.0%** [94.3, 100] | 98.2% [90.6, 99.7] |
+
+```
+naive → retry :  +18.8 pts   p = 0.0005   significant
+retry → react :   −1.8 pts   p = 0.4620   NOT significant
+```
+
+**Three lines of retry logic closed the entire gap.** A well-built deterministic executor is statistically indistinguishable from a free-form agent on this task.
+
+The original headline was measuring the executor's missing error handling, not a property of determinism. It is retracted, and the naive executor is kept as a permanent control arm so the effect can be reproduced and *attributed* rather than repeated.
+
+> **What this is actually evidence for.** Not "determinism wins" and not "determinism loses". It is evidence about **benchmark construction**: a plausible, significant, well-visualised 17-point effect can be entirely an artifact of an unrealistic baseline, and no amount of statistical rigour catches that. The confidence intervals were correct. The permutation test was correct. The control condition was correct. The *baseline* was a strawman, and only domain knowledge finds that.
+>
+> Every published agent benchmark has a baseline somebody chose. This is what it looks like when that choice is the whole result.
+
+### The trace that shows it
+
+`INV-7002` is clean. Correct action: pay £4,500. One transient 503 injected into the three-way match.
+
+```
+no retry                                with retry
+─────────────────────────────────       ─────────────────────────────────
+fetch_invoice                           fetch_invoice
+match_purchase_order   ✗ 503            match_purchase_order   ✗ 503
+check_duplicate                         match_purchase_order   ✓ retried
+check_vendor_status                     check_duplicate
+flag_exception         ← WRONG          check_vendor_status
+post_audit_log                          schedule_payment £4,500 ✓
+                                        post_audit_log
+
+held a clean invoice                    paid correctly
+```
+
+📄 **[Full report](https://bhargs24.github.io/plumbline/report.html)** · 📐 **[Method and prior art](DESIGN.md)**
+
+---
+
+## 2 · Recheck the published numbers in 30 seconds
+
+All 2,082 trajectories from all three studies are committed. Everything below reads those stored traces and makes **no model calls**, so the published numbers can be rechecked for free.
+
+```bash
+git clone https://github.com/Bhargs24/plumbline && cd plumbline
+pip install -e ".[dev,server]"
+```
+
+```bash
+plumbline demo          # seeds the store from committed runs and opens the console
+```
+
+The console shows two studies side by side, every certificate, every invariant violation grouped by severity, and a trace viewer that diffs two architectures on the *same task and the same perturbation variant*.
+
+Prefer the terminal:
+
+```bash
+plumbline parity  runs/retry-study plan_execute react
+plumbline certify runs/parity-study --arm plan_execute
+plumbline show    runs/parity-study --trial tool_fault
+plumbline gate    <run_id> --min-bound 0.95     # exits non-zero for CI
+```
+
+CI recomputes the published numbers from the committed traces on every push — the certified bounds and the headline percentages are pinned in the test suite, so if the evidence stops reproducing them, the build goes red.
+
+---
+
+## 3 · Why an instrument was needed
+
+Let *A* be an agent, *x* a task input, *τ(A, x)* its trajectory and *o(τ)* the terminal state.
+
+Existing evaluation computes `score(o(τ(A, x)))` or `score(τ(A, x))` for *x* in a **fixed** set. Neither can express a claim about behaviour under *x′* ≈ *x*.
+
+Metamorphic testing supplies the missing quantifier, and needs no oracle for the correct trajectory:
+
+```
+∀ x ∈ X, ∀ T ∈ 𝒯 :   conforms(τ(A, T(x))) = conforms(τ(A, x))
+```
+
+You declare what must hold:
+
+```python
+MustCall("check_duplicate")
+Ordering("match_purchase_order", then="schedule_payment")
+CallAtMost("schedule_payment", 1)
+ArgEquals("schedule_payment", "amount_gbp", from_context="expected_amount_gbp")
+ArgEquals("flag_exception", "reason_code", from_context="primary_reason_code")
+ArgSatisfies("request_approval", approver_has_authority_for_band)
+```
+
+The last two are the ones a simpler harness cannot express. An exception filed under the **wrong routing code** goes to the wrong queue, and an approval from someone **without authority for the amount band** is, to an auditor, no approval at all. Both are real failures that "did it flag?" and "did it get approved?" cannot see.
+
+---
+
+## 4 · The domain
+
+A measuring instrument is only as interesting as what you point it at. The first domain resolves to 8 invoices and 4 boolean checks, which a capable model saturates. Discriminating between architectures needs a domain with genuine ambiguity in it.
+
+`domains/accounts_payable/` is built so failure lives where it really lives in AP — in **ambiguity**, not arithmetic. 11 tables, 16 tools, 20 invoices, 13 distinct exception classes:
+
+| | |
+|---|---|
+| **Category-dependent tolerance** | the same 2.6% price variance **passes** on a commodity line and **fails** on a services line. No arithmetic tells you which; you must consult the category |
+| **Fuzzy duplicate detection** | returns a confidence, so the agent decides what to do with 0.75 rather than a boolean |
+| Line-level three-way match | invoice against PO against goods receipt, per line |
+| FX variance | rate movement between PO date and invoice date, against tolerance |
+| Tax code validation | against the purchase order |
+| Credit notes | applied, never paid |
+| Approval matrix | three bands, with **authority checked** rather than merely recorded |
+| Duplicate vendor | a second record sharing an established vendor's bank account |
+| Sanctioned counterparty, expired PO, invoice predating its PO, unauthorised freight, no-PO invoices | |
+
+Ground truth is **derived from the system of record by the same tools the agent uses**, so the grader cannot drift from the data.
+
+> **Scope note.** The published findings above come from the first domain only. This one is built and ready to run against a live model; no model has been run against it yet, so it has no scores to report.
+
+---
+
+## 5 · Architecture
+
+<img src="docs/assets/architecture.svg" alt="Pipeline: an agent under test, or ingested OpenTelemetry spans, feeds the perturbation engine, producing trajectories, feeding conformance, consistency and equivalence analyses." width="100%">
+
+```
+src/plumbline/
+├── core/          trajectory model · typed argument comparison · NW alignment
+├── spec/          the invariant DSL, with declared severity
+├── adapters/      Claude client (trial-keyed cache) · OpenTelemetry ingest
+├── perturb/       6 transformations + semantic-equivalence guard
+├── runtime/       parallel runner · response cache · cumulative spend cap
+├── analysis/      conformance · consistency · equivalence · Wilson · permutation
+├── store/         SQLite: projects, runs, trajectories, certificates, violations
+├── server/        REST API, OTLP ingest, and a server-rendered console
+└── report/        certificate and self-contained HTML report
+domains/accounts_payable/   11 tables · 16 tools · 20-case taxonomy · policy · 3 arms
+```
+
+Three analyses that are easy to conflate and must not be:
+
+| | Question | Catches |
+|---|---|---|
+| **Conformance** | did it obey its declared rules? | an agent that is *consistently wrong* |
+| **Consistency** | same behaviour when reworded? | an agent that is *erratically right* |
+| **Equivalence** | does the replacement match the incumbent? | a migration that looks safe because outcomes match while behaviour changed |
+
+---
+
+## 6 · Design decisions worth defending
+
+**Divergence localisation uses Needleman-Wunsch.** Positional comparison of `⟨fetch, match, dup, pay⟩` against `⟨fetch, match, pay⟩` reports "index 2: expected dup, got pay" — a substitution report for what is an omission, with every later index misaligned. Scoring is match +2, mismatch −1, gap −2, chosen so one omission resolves as one gap rather than *k* cascading mismatches.
+
+**Argument comparison is typed and checked against ground truth.** `"INV-1029"` vs `"inv-1029 "` is not divergence; `4820.00` vs `48200.00` is the most expensive divergence in the system. Zero numeric tolerance by default. And comparing against *ground truth* rather than sibling runs is what catches a drift every run shares — inter-run comparison would report perfect agreement.
+
+**A failed call did not happen.** `called(n) ≜ ∃ s ∈ τ : s.name = n ∧ s.error = ∅`. A harness that treats an errored control as executed certifies a control that never ran.
+
+**Effect paths for cross-architecture comparison.** `plan_execute` logs an `interpret_request` decision `react` has no analogue for. Counting deliberation made two systems that performed identical actions score 0% alike.
+
+**The headline is a lower bound, not an average.** `min over perturbations of lower₉₅(critical conformance)`. Worst-case because averaging conceals the dangerous one; critical-only because a missing log entry and a duplicate payment are not commensurable; a bound because it should state what is defensible and should self-penalise a small *n* (a 0.95 bound needs ≈73 clean runs per condition).
+
+**Cache keys include trial identity.** Self-consistency is measured by issuing the same request *k* times. A cache keyed on request content alone serves one response to all *k* and returns 100% self-consistency as a cache artifact.
+
+---
+
+## 7 · Point it at your own agent
+
+Both conventions, detected **per span**, since production traces carry spans from several instrumentation libraries.
+
+| | OTel GenAI | OpenInference |
+|---|---|---|
+| tool name | `gen_ai.tool.name` | `tool.name` |
+| arguments | `gen_ai.tool.call.arguments` | `tool.parameters` |
+| span kind | `gen_ai.operation.name` | `openinference.span.kind` |
+
+```python
+from plumbline.adapters.otel import load_trace_file, describe_coverage
+
+trajs = load_trace_file("traces.json", task_of=lambda s: s[0]["attributes"]["invoice.id"])
+describe_coverage(trajs)
+# {'argument_comparison_available': False,
+#  'notes': ['12 of 40 tool calls carry no structured arguments, so
+#            argument-level comparison is unavailable for them.']}
+```
+
+Or over HTTP: `POST /ingest/traces?run_id=...` takes OTLP JSON directly.
+
+`describe_coverage()` reports what the traces **cannot** support. Many instrumentations record tool identity but not arguments; reporting that as perfect argument agreement would be a lie of omission.
+
+---
+
+## 8 · Position relative to prior art
+
+| Work | Resolution | Perturbation | Tool | Your agent |
+|---|---|---|---|---|
+| [Consistency as a Testable Property](https://arxiv.org/abs/2605.10516) | action type | ✅ | ❌ | ❌ |
+| [ReliabilityBench](https://arxiv.org/abs/2601.06112) | terminal state | ✅ | ❌ | ❌ |
+| [Semantic Invariance in Agentic AI](https://arxiv.org/abs/2603.13173) | response | ✅ | ❌ | ❌ |
+| [MAESTRO](https://arxiv.org/abs/2601.00481) | trace export | ❌ | ✅ | ✅ |
+| LangSmith · Braintrust · Langfuse · Arize | trace, fixed inputs | ❌ | ✅ | ✅ |
+| **Plumbline** | **action type + arguments** | ✅ | ✅ | ✅ |
+
+The delta is **resolution**. Published methods compare action-type sequences; a payment of `48200.00` against a true `4820.00` traverses an identical sequence. The closest work names this exact extension as open: *"granular trajectory similarity metrics capturing command content details beyond action type."*
+
+This is an operationalisation plus two extensions the literature names as open, not a new-science claim. `DESIGN.md` states it in full.
+
+---
+
+## 9 · Scope of the published claims
+
+**One model, one domain, one policy.** Results are from `claude-haiku-4-5` on
+the procure-to-pay domain. The claim is architectural and about measurement
+method; it is not a model ranking, and the harness is model-agnostic.
+
+**𝒯 is a chosen finite set.** Conformance under it is evidence, not proof. An
+agent is certified against the transformations someone thought to apply, and
+against the baselines someone built correctly, which is the point section 1
+makes.
+
+**Outcome equivalence is computed on ledger state**, which is exact because the
+specimen writes to a database. Agents whose output is prose need semantic
+comparison, deliberately left unimplemented rather than implemented badly.
+
+---
+
+Built by Bhargav Raghavendra · Apache-2.0