truassure 1.0.0__tar.gz

truassure-1.0.0/.gitignore

@@ -0,0 +1,231 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#   Usually these files are written by a python script from a template
+#   before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+*.lcov
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+# Pipfile.lock
+
+# uv
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+# poetry.lock
+# poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+# pdm.lock
+# pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+# pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi/*
+!.pixi/config.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule*
+celerybeat.pid
+
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs/Zensical documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#   and can be added to the global gitignore or merged into this file.  For a more nuclear
+#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
+# .idea/
+
+# Abstra
+#   Abstra is an AI-powered process automation framework.
+#   Ignore directories containing user credentials, local state, and settings.
+#   Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore that
+#   can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#   and can be added to the global gitignore or merged into this file. However, if you prefer, you
+#   could uncomment the following to ignore the entire vscode folder
+# .vscode/
+# Temporary file for partial code execution
+tempCodeRunnerFile.py
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml
+
+# --- TruAssure ---
+.env*
+!.env.example
+ta_runs/
+tests/golden/_inputs/
+*.local.yaml
+# SUT runtime data (databases, generated letters) - created by tests and local runs
+SUT/**/data/
+SUT/**/letters/
+*.db

truassure-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shan Konduru
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

truassure-1.0.0/PKG-INFO

@@ -0,0 +1,377 @@
+Metadata-Version: 2.5
+Name: truassure
+Version: 1.0.0
+Summary: TruAssure: black-box, evidence-based quality engineering for AI agents (chat, task and multi-agent).
+Author: Shan Konduru
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: jsonschema>=4.21
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.40; extra == 'anthropic'
+Provides-Extra: azure
+Requires-Dist: openai>=1.40; extra == 'azure'
+Provides-Extra: chat
+Requires-Dist: websockets>=12; extra == 'chat'
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: fastapi>=0.115; extra == 'dev'
+Requires-Dist: httpx>=0.27; extra == 'dev'
+Requires-Dist: langgraph>=0.2; extra == 'dev'
+Requires-Dist: mcp>=2.3; extra == 'dev'
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: starlette>=0.37; extra == 'dev'
+Requires-Dist: types-jsonschema>=4.21; extra == 'dev'
+Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
+Requires-Dist: uvicorn>=0.30; extra == 'dev'
+Requires-Dist: websockets>=12; extra == 'dev'
+Provides-Extra: langgraph
+Requires-Dist: langgraph>=0.2; extra == 'langgraph'
+Provides-Extra: mcp
+Requires-Dist: mcp>=2.3; extra == 'mcp'
+Provides-Extra: otel
+Requires-Dist: starlette>=0.37; extra == 'otel'
+Requires-Dist: uvicorn>=0.30; extra == 'otel'
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
+Provides-Extra: proxy
+Requires-Dist: httpx>=0.27; extra == 'proxy'
+Requires-Dist: starlette>=0.37; extra == 'proxy'
+Requires-Dist: uvicorn>=0.30; extra == 'proxy'
+Description-Content-Type: text/markdown
+
+# TruAssure: Enterprise Agentic Quality Engineering Accelerator
+
+TruAssure is a black-box, evidence-based quality engineering framework for AI agents of any kind:
+conversational assistants, coded task agents (LangGraph, OpenAI Agents SDK, custom loops), agentic
+applications and multi-agent orchestrations.
+
+For each run it answers questions without relying on the agent's own account of itself:
+
+- Did the agent understand the task?
+- Did it plan sensibly?
+- Did it take only allowed and correct actions?
+- Did it leave the environment in the right end state?
+- Did it report truthfully what it did?
+
+It also checks whether the agent behaves reliably (pass^k), robustly (input perturbations, injected faults)
+and the same way across deployments (parity).
+
+> **Status: pre-alpha, phase 2 of 6 done.**
+> - **Core engine:** episode runner, evidence store, five-stage evaluation with the evidence seams, Wilson
+>   ranges, and `ta run|show|report|gc`.
+> - **Regression gate:** TruAssure reproduces the reference verdicts on 4,221 stored chat turns.
+> - **Live runs:** it drives the real chat assistant on Docker (`SUT/`) through its chat channel.
+>
+> See [Validated on](#validated-on) for what has been checked and how.
+> [docs/planning/PLAN.md](docs/planning/PLAN.md) has the full plan and [docs/brief/BRIEF.md](docs/brief/BRIEF.md)
+> the original brief.
+
+## How it judges an agent
+
+- **Episode:** a task given to an agent in a prepared environment, run until it finishes, times out or runs
+  out of budget.
+- **Checkpoint:** a point where the episode is judged. A chat has one per user turn; a task agent usually has
+  one at the end.
+
+Each checkpoint gets a verdict at five stages, and records the first stage that failed:
+
+| # | Stage | Question |
+|---|---|---|
+| 1 | Goal | What did the agent decide the task was (an intent, a routing decision)? |
+| 2 | Plan | Were the steps, handoffs, loops, termination and approvals sensible? |
+| 3 | Action | Was every action correct, authorised and given the right arguments? |
+| 4 | End state | Is the database, file tree, git repository or SaaS system right afterwards? |
+| 5 | Output | Is the reply, artifact or pull request right, and consistent with what was actually done? |
+
+TruAssure gathers evidence from independent sources and keeps each source apart:
+
+- the agent's own trace;
+- an observer at the action boundary (an MCP or REST recording proxy, OpenTelemetry GenAI spans, or an
+  agent-to-agent tap);
+- before and after snapshots of the system of record;
+- the agent's outputs.
+
+It cross-checks these sources against each other through the evidence seams:
+
+- **SE-01** trace vs observer
+- **SE-02** authorisation vs state
+- **SE-03** silent write
+- **SE-04** phantom success
+- **SE-05** output vs action
+- **SE-06** deployment parity
+- **SE-07** identity propagation, a TruAssure extension that is not counted among the catalogue's 71 types
+
+A grader that can't reach a verdict reports *not evaluated*, which counts as a failure, never a pass.
+
+Every failure carries a structured code (for example `ACTION.FORBIDDEN_CALLED` or `SE-03.SILENT_WRITE`, listed
+in [`codes.py`](src/truassure/core/codes.py)), so severity rules, the scorecard and release gates can match on
+codes rather than message text.
+
+An agent that exposes no trace of its own can still be tested ("black-box mode"). In that mode its actions
+come from the observers only, and the goal stage reports *not evaluated*.
+
+## Layout
+
+```
+src/truassure/
+  core/        model, Protocols, catalog + JSON Schema, compiler, runner, stages/, stats, store, security,
+               runlock, report, CLI; run-level analysis: reliability (pass^k), robustness, compare (parity),
+               scorecard/ (the test-type catalogue), governance/ (defects, JUnit, packs, release gate),
+               calibration, ci (pipeline templates). Imports only stdlib, PyYAML, jsonschema
+  adapters/    drivers, observers, state, graders  (framework-specific code lives here)
+  profiles/    conversational (incl. the suite-catalog translator), task, workflow
+  testing/     fakes of every Protocol, the contract-suite harness, and a scripted agent for offline runs
+tests/         unit/  contract/  golden/  e2e/  pilots/
+pilots/        the pilot catalogs (claims/: pilot 1, desk/: pilot 2) and the grader calibration set
+tools/         check_core_imports.py (the core layering rule, an allow-list)
+               make_golden.py, make_reference_corpus.py, make_reference_runlevel.py (reference outputs
+               from the source functions, for the regression gate; dev-only)
+scripts/       setup / test / lint, smoke-chat, pilot-claims, pilot-desk, as .bat and .sh pairs
+docs/          brief/ (original brief and prompt)  planning/ (PLAN.md)
+```
+
+## Quick start
+
+**Start here:**
+
+- **Demo runbook:** [`docs/demo/TruAssure-Demo-Runbook.html`](docs/demo/TruAssure-Demo-Runbook.html), a self-contained
+  page that walks through every feature, step by step, with expected output.
+- **Client overview:** [`docs/client/TruAssure-Client-Overview.html`](docs/client/TruAssure-Client-Overview.html),
+  covering features, business value and use cases.
+- **One script for everything:** `scripts\truassure-ops.bat` (Windows) or `sh scripts/truassure-ops.sh`.
+  - `setup`: every virtual environment.
+  - `lint`, `test-all`, `test-sut all`: TruAssure's and each SUT's tests.
+  - `deploy all`, `status`, `launch all`: the SUTs on Docker Desktop.
+  - `pilots`: the live runs.
+  - `demo`: an offline tour.
+  - `console`: browse runs, reports, comparisons and gates in the browser.
+  - `mcp`: serve TruAssure to an AI assistant over MCP.
+  - `build`: build the wheel and smoke-test it in a clean venv.
+  - `help`: every command.
+- **Launchers:** `scripts\launch-console.bat` (the console over `ta_runs`), `scripts\launch-demo.bat` (the
+  offline demo, then the console over its runs) and `scripts\launch-mcp.bat` (the MCP server), each with a
+  `.sh` twin.
+
+```bat
+scripts\setup.bat          :: or: sh scripts/setup.sh
+scripts\test.bat -q        :: tests + the 95% line-and-branch coverage gate
+scripts\lint.bat           :: ruff, ruff format --check, mypy --strict, core import rule
+```
+
+The CLI is `truassure`, with `ta` as a short alias:
+
+```bat
+ta --version
+ta validate tests\fixtures\catalogs\task.yaml
+ta translate C:\myprojects\tool-calling-chatbot\qa\test_catalog.yaml --yaml -o catalog.yaml
+
+:: run a catalog against an environment (here: the built-in scripted demo agent)
+ta run tests\fixtures\catalogs\e2e_demo.yaml --env truassure.testing.scripted:demo_environment --perturbations all --repeats 3
+ta show                     :: list runs
+ta show latest              :: one run: per-case stages, result, Wilson range, reasons
+ta report latest            :: write the run's self-contained HTML report
+ta gc --days 30             :: retention: delete runs older than 30 days
+
+:: run-level analysis and governance
+ta compare A B              :: checkpoint by checkpoint; across deployments, parity against a noise floor (SE-06)
+ta defects latest           :: the defect list (severity by failure code), defects.json and .csv
+ta junit latest --flaky skip --baseline <run>   :: JUnit XML for CI; only regressions fail with a baseline
+ta scorecard latest         :: the run against the 71-type catalogue (measured only with evidence)
+ta pack claims.intake       :: a capability pack: ready / not ready
+ta gate latest --policy truassure-gate.yaml     :: the release gate; exit 1 when a rule fails
+ta calibrate pilots\calibration\harbor-consistency.yaml --grader <module:Class>   :: CE-02: Cohen's kappa
+ta init --ci github --catalog <catalog> --env <factory>   :: a pipeline that runs, reports and gates
+
+:: browse and automate
+ta console --open           :: local console on 127.0.0.1 with a one-time URL token
+ta mcp                      :: MCP server over stdio: one tool per command (install the 'mcp' extra)
+```
+
+**The report** (`ta report`) is one HTML file with everything inlined. It fetches nothing, so it works offline
+and can be attached to a ticket. Its tabs are Overview (pass rates, first failing stage, pass^k decay,
+robustness), Cases, Episodes, Scorecard, Defects, and Comparison and gate. For task and workflow episodes,
+the Episodes tab sets what the agent reported beside what the independent sources (the proxy, spans, tap and
+state diffs) recorded, with every checkpoint's verdict. Links such as `report.html#tab=episodes&case=ID` open a
+case directly. The largest corpus run's report is about 0.4 MB.
+
+**The console** (`ta console`) lists the runs in a folder and opens their reports, defect lists, scorecards and
+gate verdicts; it also compares any two runs. It listens on `127.0.0.1` (another address needs
+`--allow-remote`) and prints a URL with a random token. The first visit trades the token for an HttpOnly,
+SameSite=Strict cookie, and every request without one is refused. It reads stored runs; it never starts one.
+
+**The MCP server** (`ta mcp`) gives an AI assistant one tool per CLI command (`validate`, `translate`, `run`,
+`show`, `report`, `compare`, `defects`, `junit`, `scorecard`, `pack`, `gate`, `calibrate`, `init`, `gc`).
+Each tool's input schema is generated from the command's own arguments, so the two cannot drift; a test checks
+they match. The long-running services (`proxy`, `otel`, `a2a`, `console`, `mcp`) are not tools. Every result
+says that quoted agent evidence is data, not instructions. To add it to an MCP client:
+
+```json
+{ "mcpServers": { "truassure": { "command": "ta", "args": ["mcp"] } } }
+```
+
+Every run's summary also carries its pass^k decay and its robustness across perturbations.
+
+`--env` names a factory function (`package.module:function`) that returns an `Environment` for each worker:
+the driver for your agent, its observers, its state adapters and an optional grader. The factory module can
+sit in the current folder. Runs go to `./ta_runs`, or `$TA_RUNS_DIR`. `ta` reads settings and secrets from
+the nearest `.env`, and real environment variables always win.
+
+**Adapters available so far:**
+- drivers: the chat driver (`adapters/drivers/chat_ws.py`; install the `chat` extra) and the HTTP task driver
+  (`adapters/drivers/http_task.py`; one POST per event or task);
+- the recording MCP proxy, `ta proxy` (`adapters/observers/mcp_proxy.py`; install the `proxy` extra), and its
+  observer (`adapters/observers/proxy_log.py`), which also injects faults;
+- the agent-to-agent tap, `ta a2a` (`adapters/observers/a2a_tap.py`): an HTTP relay in front of the agents that
+  receive handoffs, or an in-process hook for a message bus. It records each handoff with the member and permissions
+  read from the identity token, and injects agent faults. The MCP proxy reads identity tokens too
+  (`--identity-header`);
+- the OpenTelemetry GenAI observer (`adapters/observers/otel_genai.py`; `execute_tool` spans from OTLP/JSON) and
+  an OTLP/HTTP receiver, `ta otel` (`adapters/observers/otel_receiver.py`; install the `otel` extra);
+- state: SQLite (`adapters/state/sqlite.py`; read-only snapshots, row diffs, SQL checks), a file tree
+  (`adapters/state/filetree.py`; content-hash diffs, file checks) and git (`adapters/state/git.py`; refs and
+  working tree, branch, commit and message checks);
+- the artifact grader (`adapters/graders/artifact_tests.py`; content checks, a Python test or a command);
+- ready live environments: the chat assistant (`truassure.profiles.conversational.live:environment`), the
+  claims agent (`truassure.profiles.task.live:environment`) and the member desk
+  (`truassure.profiles.workflow.live:environment`).
+
+Every run records the hash of `core/` it was judged with (`run.json`, `core`).
+`ta run` exits 0 when every case passes, 1 when any case fails or is flaky, 2 on bad input, and 3 when the
+environment is locked by another run. Add masking rules with `--mask LABEL=REGEX` (repeatable).
+
+## Catalogs
+
+Scenarios are written in YAML and validated by
+[`scenario.schema.json`](src/truassure/core/catalog/schema/scenario.schema.json) (JSON Schema Draft 2020-12).
+Each scenario names a profile, and each profile has its own rules:
+
+- **conversational:** has `checkpoints`, each with a `prompt`.
+- **task:** has an `event` or `task` input, plus `expect`.
+- **workflow:** adds `agents`, a `principal` and the expected `handoffs`.
+
+Keys starting with `x-` are allowed for your own extensions. See [`tests/fixtures/catalogs/`](tests/fixtures/catalogs)
+for an example of each profile.
+
+`ta translate` converts a suite catalog (suites -> cases, one chat per suite) to this format. It doesn't drop anything silently: keys it can't
+map are kept under `source.unmapped`. It never copies passwords.
+
+## Validated on
+
+| Pilot | What it proves | Status |
+|---|---|---|
+| 3 · regression gate | Same verdicts as the reference verdicts stored with the sandbox runs | **passed:** 4,221 of 4,221 turns in 40 runs (stdio, HTTP and REST transports), and every run's per-case results and Wilson ranges |
+| 1 · task agent | Generic core on an event-driven agent with no chat | **passed:** the 15-scenario catalog passes offline (20 of 20 episodes); each of 6 planted defects fires the right code at the right stage; live, it finds real agent defects (below) |
+| 2 · multi-agent | Handoffs, identity propagation, cascading failure (SS-09) | **passed:** the 12-scenario catalog passes offline (15 of 15 episodes); each of 5 planted defects fires the right code at the right stage; pilots 1 and 2 pass on the same core hash (each run records it); live, it finds real defects (below) |
+| 3 · run level (phase 5) | Same run-level results as the reference, on the same corpus | **passed:** 225 checks: pass^k, robustness, defect severities, JUnit decisions (every flaky mode, and with a baseline), every scorecard row of all 40 runs, and 19 run comparisons with their parity verdicts |
+| 1 · parity (SE-06) | Two builds of the claims agent (plain loop, LangGraph) judged against a noise floor | **passed:** judged "holds" offline, and live on Azure (15 checkpoints; the one difference was within run-to-run noise) |
+| report (phase 6) | Self-contained, offline HTML for the largest corpus run | **passed:** about 0.4 MB (limit 5 MB); nothing is fetched from the network ([`tests/golden/test_report_size.py`](tests/golden/test_report_size.py)) |
+| MCP server (phase 6) | One tool per CLI command, driven by a real MCP client | **passed:** the tool inventory and every tool's arguments equal the CLI's ([`tests/e2e/test_mcp_server.py`](tests/e2e/test_mcp_server.py)) |
+| wheel (phase 6) | `python -m build`, then the wheel alone in a clean venv | **passed** on Windows and Linux in CI ([`tools/wheel_smoke.py`](tools/wheel_smoke.py)): console scripts, packaged schema and catalogue, an offline run, report, gate and the MCP tool list |
+
+**How the regression gate works**
+- [`tools/make_reference_corpus.py`](tools/make_reference_corpus.py) turns the source project's stored
+  conversational runs into a corpus in [`tests/golden/reference/corpus`](tests/golden/reference/corpus). The
+  corpus is trimmed to what evaluation and run-level analysis read, masked, and about 1.1 MB.
+- A run is admitted only if the reference code reproduces every one of its stored verdicts from those exact
+  inputs.
+- TruAssure then has to give the same pass/fail, first failing stage, per-stage status, check counts and failure
+  codes on every turn ([`tests/golden/test_regression_gate.py`](tests/golden/test_regression_gate.py)).
+- 11 runs were excluded because they had been judged against catalog edits that were never committed. The
+  [manifest](tests/golden/reference/corpus/manifest.json) lists each one with its reason.
+- Run level: [`tools/make_reference_runlevel.py`](tools/make_reference_runlevel.py) runs the source's run-level
+  functions on the same corpus; [`tests/golden/test_runlevel_gate.py`](tests/golden/test_runlevel_gate.py) rebuilds
+  each run from TruAssure's own verdicts and must reproduce every result. The only allowed difference: types
+  TruAssure measures beyond the reference (AT-02, AT-06, AT-08, SS-03, SS-09, PO-03, CE-02) read "not run" where
+  the reference said "planned".
+
+**Live smoke run** (`scripts\smoke-chat.bat <catalog> SRI GREET PRIVACY`): TruAssure drove the chat assistant
+on Docker Desktop, with Azure OpenAI as the assistant's model.
+- 22 of 23 checkpoints passed.
+- The one failure was real agent behaviour: the assistant told a customer they had no policies without
+  looking them up first.
+- The chat assistant has no tool-traffic observer, so the trace-vs-observer seam is not checked there.
+
+**Pilot 1, the claims agent** ([`pilots/claims`](pilots/claims)): TruAssure drives the agent through its event API
+and watches it through the recording proxy, the agent's OpenTelemetry GenAI spans, the claims database and the
+letters folder.
+- Offline (the scripted model, the whole stack in-process): all 20 episodes pass (15 scenarios, plus 5 perturbed
+  variants).
+- Buggy build (`CLAIMS_DEFECTS`): six planted defects (silent write, phantom success, forbidden action, loop
+  past budget, an artifact that fails its tests, a trace that leaves calls out) each fail their scenario at the
+  planted stage with the planted code, offline. Live with Azure, the silent write and the trace omission fire
+  `SE-03` and `SE-01` (proxy and spans) the same way.
+- Live (`scripts\pilot-claims.bat`, Docker Desktop, Azure OpenAI gpt-4.1), three full runs. Every failure
+  is real agent behaviour (in the third run, which also read the spans, they agreed with the agent's trace on
+  every episode):
+  - when the approval fails (tool error) or times out, it still writes the approval letter (every run);
+  - asked to approve $2,500 on a $1,800 claim, it approves $1,800 on its own instead of refusing (every run);
+  - for a claim already under review it sometimes also sends an acknowledgement letter, where the procedure
+    says to add a note only (2 of 3 runs).
+
+**Pilot 2, the member desk** ([`pilots/desk`](pilots/desk)): TruAssure drives the triage agent, and watches the
+three agents through the A2A tap (handoffs), the MCP proxy (tool calls), both reading the identity token on the
+wire, and the desk database.
+- Offline: all 15 episodes pass (12 scenarios, plus 3 perturbed variants). Five planted defects fail their
+  scenario at the planted stage with the planted code:
+  - handoff order: `PLAN.HANDOFF_ORDER`;
+  - a failure handed on: `PLAN.FAILURE_CASCADED`;
+  - a hop for another member: `SE-07.PRINCIPAL_MISMATCH`;
+  - no permission narrowing: `SE-07.PERMISSIONS_WIDENED`;
+  - a hidden hop: `PLAN.HANDOFF_MISSING` plus `SE-01`.
+
+  Live, the three agent-side defects fire the same codes.
+- Live (`scripts\pilot-desk.bat`, Azure OpenAI gpt-4.1), two full runs. Every failure is agent behaviour:
+  - when billing fails, it still hands off to claims (SS-09) and opens no case, and triage tells the member "a
+    case has been opened" and gives the claim amount (both runs, in at least one of the two billing-failure
+    scenarios);
+  - when claims fails inside a two-part question, triage tells the member a case was opened when no agent opened
+    one (both runs);
+  - triage drops the amounts its specialists reported ("your balance and due date have been provided"): every
+    run on the two-part question, and 2 of 3 phrasings of the billing question in the first run.
+
+**Grader calibration (CE-02)** on a labelled set of 24 output-vs-action items
+([`pilots/calibration`](pilots/calibration)):
+- The deterministic rules grader scores kappa 0.75 (agreement 0.875). It misses only what needs reading for meaning:
+  a denied record described anyway, a change claimed with no action, an approved amount stated as paid.
+- The LLM grader (Azure OpenAI gpt-4.1) scored kappa 1.0, then 0.833 twice, over three runs. When it erred, it
+  missed a case "opened" with no action and policy numbers no action returned. Within a run, grading twice gave the
+  same verdicts.
+
+Also checked:
+- The ported primitives (`wilson`, `mentions`, `value_match`, `args_match`) give the same results as
+  the original source functions on the inputs in [the reference outputs](tests/golden/reference/primitives.json).
+- The reference evaluation cases, expressed in the generic model, pass, including the per-stage check counts.
+- 59 synthetic episodes (52 planted with one defect each, covering every stage check, seam and workflow check; 7
+  clean): every planted one fails at exactly the planted stage with the planted code; the clean ones pass.
+- The translator converts the full Harbor Mutual sandbox catalog (93 cases) to a catalog that passes schema validation
+  (a test that runs when the catalog is present on the machine).
+
+## Security defaults
+
+- **In place:** PII is masked before any evidence is written. The default rules cover email addresses,
+  phone numbers and secret-named fields (password, token, API key, authorization). You can add your own
+  regex rules. Masking never changes the values that verdicts are computed on. `ta gc` applies retention.
+- **Listeners:** the proxy, tap, OTel receiver and console bind to `127.0.0.1` by default; another address needs
+  `--allow-remote`. Their control endpoints need a token (`TA_PROXY_TOKEN`, `TA_OTEL_TOKEN` and the sandbox tokens; the
+  console makes its own per start). Secrets come only from the environment or `.env`, and are never printed.
+- **MCP:** tool results are marked as untrusted data, because they can quote what an agent said.
+- **Releases:** `.github/workflows/release.yml` publishes to PyPI with trusted publishing (no stored token),
+  only when a GitHub Release is published, through the `pypi` environment. Nothing has been published.
+
+## Licence
+
+MIT © 2026 Shan Konduru. All names and data in demos and fixtures belong to the fictitious company
+*Harbor Mutual*.