agentmandate 0.3.2__tar.gz → 0.4.0__tar.gz

{agentmandate-0.3.2 → agentmandate-0.4.0}/CHANGELOG.md

@@ -6,6 +6,24 @@ All notable changes to this project are documented here. The format follows
 
 ## Unreleased
 
+## 0.4.0 - 2026-07-30
+
+### Added
+
+- `mandate scenarios` exports each reachable breach as a versioned, structured
+  scenario-test skeleton. It preserves the counterexample while leaving the
+  environment, agent input, and expected control boundary for human review.
+- Reviewed scenario fields reconcile against the current reachable witnesses.
+  Disappearing paths are removed and newly reachable paths return unreviewed.
+- An evaluation-loop guide separates permitted reachability, observed agent
+  behaviour, per-call enforcement, and runtime conformance.
+
+### Changed
+
+- The README and test-obligation guide connect decision-point obligations to
+  AgentVerity and compound paths to external scenario evaluators without
+  turning AgentMandate into a runner.
+
 ## 0.3.2 - 2026-07-28
 
 ### Changed

{agentmandate-0.3.2 → agentmandate-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentmandate
-Version: 0.3.2
+Version: 0.4.0
 Summary: Compound-path and cross-release analysis of what an AI agent is permitted to do
 Project-URL: Homepage, https://github.com/mrwersa/agentmandate
 Project-URL: Repository, https://github.com/mrwersa/agentmandate
@@ -80,6 +80,8 @@ cannot:
   closed when a required control field is absent.
 - **What must the tests exercise?** `mandate obligations` turns reachable
   authority into reviewable test obligations.
+- **Which compound risks need scenarios?** `mandate scenarios` preserves each
+  counterexample as a neutral test skeleton without inventing an agent prompt.
 
 Alpha. Apache-2.0.
 
@@ -217,6 +219,7 @@ A ceiling is the maximum **cumulative** value one tool may spend against one bin
 | `mandate diff` | Effective-authority comparison of two manifests, including limits, preconditions, approvals, effects, and scope minting. `--record` emits a change record |
 | `mandate verify` | Replays recorded tool calls against the manifest and fails closed when evidence required by a declared control is missing |
 | `mandate obligations` | Derives reviewable test obligations from reachable authority, and renders reviewed ones as an [AgentVerity](https://github.com/mrwersa/agentverity) decision suite |
+| `mandate scenarios` | Exports reachable breach paths with blank environment, agent-input, and expected-control fields for human review and execution by an external evaluation harness |
 
 Every analysis command takes `--json` and exits non-zero on a finding, so they drop into CI unchanged. `scan` writes a manifest to standard output and is a one-off, not a gate.
 
@@ -241,6 +244,28 @@ For a spending tool, each trace record must carry the scope, value, currency,
 approval state, and executing principal. Missing or malformed control evidence
 does not pass as an empty value.
 
+## From authority to evaluation
+
+AgentMandate produces two different test inputs:
+
+- `obligations` names consequential decision points that reviewed bounded
+  decision tests should reach
+- `scenarios` preserves compound counterexample paths that a multi-step
+  scenario test should attempt
+
+It does not execute either test. Promptfoo, LangSmith, AgentCore Evaluations,
+pytest, or an internal harness owns behaviour and outcome grading. AgentVerity
+can qualify the repeated bounded decisions after correctness passes.
+
+```text
+reachability -> reviewed obligations and scenarios -> external evaluation
+      ^                                               |
+      |                                               v
+manifest <- reviewed production incidents <- runtime policy and traces
+```
+
+[Read the complete evaluation-loop workflow](docs/evaluation-loop.md).
+
 ## Where it fits, and what already exists
 
 This is analysis, not enforcement. It runs in CI against a manifest, it does not sit in the request path.
@@ -285,6 +310,8 @@ Search is bounded by `limits.depth`. No breach at depth 8 is not proof that none
 ## Documentation
 
 - [DESIGN.md](DESIGN.md) — the authority model, why the search is shaped this way, and what was left out
+- [docs/evaluation-loop.md](docs/evaluation-loop.md) — how authority analysis, scenario evaluation, runtime policy, and production feedback remain distinct
+- [docs/test-obligations.md](docs/test-obligations.md) — decision-point obligations and the AgentVerity bridge
 - [CONTRIBUTING.md](CONTRIBUTING.md) — branch and review workflow
 - [SECURITY.md](SECURITY.md) — reporting, and what a manifest may contain
 - [STABILITY.md](STABILITY.md) — what is guaranteed before 1.0
@@ -304,7 +331,7 @@ ruff check .
 
 ## Status
 
-Alpha, version 0.3.2. The authority model is the part most likely to change,
+Alpha, version 0.4.0. The authority model is the part most likely to change,
 because it has not yet been pointed at enough real tool graphs to know where it
 is too coarse. Issues describing a graph it models badly are the most useful
 thing you can file.

{agentmandate-0.3.2 → agentmandate-0.4.0}/README.md

@@ -46,6 +46,8 @@ cannot:
   closed when a required control field is absent.
 - **What must the tests exercise?** `mandate obligations` turns reachable
   authority into reviewable test obligations.
+- **Which compound risks need scenarios?** `mandate scenarios` preserves each
+  counterexample as a neutral test skeleton without inventing an agent prompt.
 
 Alpha. Apache-2.0.
 
@@ -183,6 +185,7 @@ A ceiling is the maximum **cumulative** value one tool may spend against one bin
 | `mandate diff` | Effective-authority comparison of two manifests, including limits, preconditions, approvals, effects, and scope minting. `--record` emits a change record |
 | `mandate verify` | Replays recorded tool calls against the manifest and fails closed when evidence required by a declared control is missing |
 | `mandate obligations` | Derives reviewable test obligations from reachable authority, and renders reviewed ones as an [AgentVerity](https://github.com/mrwersa/agentverity) decision suite |
+| `mandate scenarios` | Exports reachable breach paths with blank environment, agent-input, and expected-control fields for human review and execution by an external evaluation harness |
 
 Every analysis command takes `--json` and exits non-zero on a finding, so they drop into CI unchanged. `scan` writes a manifest to standard output and is a one-off, not a gate.
 
@@ -207,6 +210,28 @@ For a spending tool, each trace record must carry the scope, value, currency,
 approval state, and executing principal. Missing or malformed control evidence
 does not pass as an empty value.
 
+## From authority to evaluation
+
+AgentMandate produces two different test inputs:
+
+- `obligations` names consequential decision points that reviewed bounded
+  decision tests should reach
+- `scenarios` preserves compound counterexample paths that a multi-step
+  scenario test should attempt
+
+It does not execute either test. Promptfoo, LangSmith, AgentCore Evaluations,
+pytest, or an internal harness owns behaviour and outcome grading. AgentVerity
+can qualify the repeated bounded decisions after correctness passes.
+
+```text
+reachability -> reviewed obligations and scenarios -> external evaluation
+      ^                                               |
+      |                                               v
+manifest <- reviewed production incidents <- runtime policy and traces
+```
+
+[Read the complete evaluation-loop workflow](docs/evaluation-loop.md).
+
 ## Where it fits, and what already exists
 
 This is analysis, not enforcement. It runs in CI against a manifest, it does not sit in the request path.
@@ -251,6 +276,8 @@ Search is bounded by `limits.depth`. No breach at depth 8 is not proof that none
 ## Documentation
 
 - [DESIGN.md](DESIGN.md) — the authority model, why the search is shaped this way, and what was left out
+- [docs/evaluation-loop.md](docs/evaluation-loop.md) — how authority analysis, scenario evaluation, runtime policy, and production feedback remain distinct
+- [docs/test-obligations.md](docs/test-obligations.md) — decision-point obligations and the AgentVerity bridge
 - [CONTRIBUTING.md](CONTRIBUTING.md) — branch and review workflow
 - [SECURITY.md](SECURITY.md) — reporting, and what a manifest may contain
 - [STABILITY.md](STABILITY.md) — what is guaranteed before 1.0
@@ -270,7 +297,7 @@ ruff check .
 
 ## Status
 
-Alpha, version 0.3.2. The authority model is the part most likely to change,
+Alpha, version 0.4.0. The authority model is the part most likely to change,
 because it has not yet been pointed at enough real tool graphs to know where it
 is too coarse. Issues describing a graph it models badly are the most useful
 thing you can file.

{agentmandate-0.3.2 → agentmandate-0.4.0}/ROADMAP.md

@@ -18,6 +18,11 @@ Version 0.3.2 makes the existing gate dependable:
 - explicit handling of incomparable currencies and different agents
 - reviewed test obligations reconciled against current reachable authority
 
+## In progress: evaluation-loop bridge
+
+- neutral scenario skeletons derived from compound counterexample paths, with
+  application input, environment, and expected control left for human review
+
 ## Next: easier adoption
 
 1. **Framework inventory adapters.** Import tool catalogues from common agent
@@ -34,6 +39,10 @@ Version 0.3.2 makes the existing gate dependable:
 These features overlap with mature CI and security tooling deliberately. The
 new value remains the authority graph and its counterexamples.
 
+The scenario exporter is the first evaluation-loop bridge. Keep execution in
+the team's existing harness. The next adoption work should make manifests and
+traces easier to obtain, not add a bundled judge or scenario runner.
+
 ## Then: widen the model where evidence demands it
 
 Do not implement every candidate below in parallel. First collect tool graphs

{agentmandate-0.3.2 → agentmandate-0.4.0}/STABILITY.md

@@ -19,7 +19,7 @@ statement, not a waiver for silent breakage.
 Production users should pin the current minor series:
 
 ```text
-agentmandate~=0.2.0
+agentmandate~=0.4.0
 ```
 
 ## What is most likely to change

{agentmandate-0.3.2 → agentmandate-0.4.0}/agentmandate/__init__.py

@@ -13,7 +13,7 @@ against the declaration, so it is usable without another tool alongside it.
 
 from __future__ import annotations
 
-__version__ = "0.3.2"
+__version__ = "0.4.0"
 
 from .diff import Change, Delta, compare
 from .lint import Finding, check
@@ -29,6 +29,16 @@ from .obligations import (
 )
 from .reach import Authority, Breach, Step, analyse
 from .scan import Proposal, propose, render, scan_file
+from .scenarios import (
+    SCENARIOS_SCHEMA,
+    Scenario,
+    ScenarioSet,
+    ScenarioStep,
+    derive_scenarios,
+    load_scenarios,
+    reconcile_scenarios,
+    save_scenarios,
+)
 from .verify import Conformance, Observation, Violation, replay, replay_file
 
 __all__ = [
@@ -46,6 +56,10 @@ __all__ = [
     "ObligationSet",
     "Observation",
     "Proposal",
+    "SCENARIOS_SCHEMA",
+    "Scenario",
+    "ScenarioSet",
+    "ScenarioStep",
     "Step",
     "Tool",
     "Violation",
@@ -53,15 +67,19 @@ __all__ = [
     "analyse",
     "check",
     "derive",
+    "derive_scenarios",
     "compare",
     "load",
     "load_obligations",
+    "load_scenarios",
     "loads",
     "save_obligations",
+    "save_scenarios",
     "to_decision_suite",
     "propose",
     "render",
     "replay",
     "replay_file",
+    "reconcile_scenarios",
     "scan_file",
 ]

{agentmandate-0.3.2 → agentmandate-0.4.0}/agentmandate/cli.py

@@ -19,6 +19,12 @@ from .obligations import (
 )
 from .reach import analyse
 from .scan import scan_file
+from .scenarios import (
+    derive_scenarios,
+    load_scenarios,
+    reconcile_scenarios,
+    save_scenarios,
+)
 from .verify import replay_file
 
 EXIT_OK = 0
@@ -100,6 +106,26 @@ def build_parser() -> argparse.ArgumentParser:
         help="an obligations file whose decisions have been filled in",
     )
 
+    scenarios_parser = subparsers.add_parser(
+        "scenarios",
+        help="export reachable breaches as reviewed scenario-test skeletons",
+    )
+    _add_manifest(scenarios_parser)
+    scenarios_parser.add_argument("--depth", type=_positive_int, default=None)
+    scenarios_parser.add_argument(
+        "--json", action="store_true", help="machine-readable output"
+    )
+    scenarios_parser.add_argument(
+        "--reviewed",
+        default=None,
+        help="a scenario file whose application-specific fields were reviewed",
+    )
+    scenarios_parser.add_argument(
+        "--output",
+        default=None,
+        help="write the reconciled JSON scenario skeleton to this path",
+    )
+
     verify_parser = subparsers.add_parser(
         "verify", help="replay recorded calls against the manifest"
     )
@@ -171,6 +197,31 @@ def main(argv: Sequence[str] | None = None) -> int:
         _emit(obligations.to_dict(), args.json, obligations.render())
         return EXIT_FINDING if obligations.unreviewed else EXIT_OK
 
+    if args.command == "scenarios":
+        scenarios = derive_scenarios(mandate, depth=args.depth)
+        if args.reviewed:
+            try:
+                scenarios = reconcile_scenarios(
+                    scenarios,
+                    load_scenarios(args.reviewed),
+                )
+            except ValueError as exc:
+                print(f"error: {exc}", file=sys.stderr)
+                return EXIT_USAGE
+        if args.output:
+            try:
+                save_scenarios(scenarios, args.output)
+            except OSError as exc:
+                print(f"error: cannot write scenarios: {exc}", file=sys.stderr)
+                return EXIT_USAGE
+            print(
+                f"wrote {len(scenarios.scenarios)} scenario(s) to {args.output}; "
+                f"{len(scenarios.unreviewed)} need review"
+            )
+        else:
+            _emit(scenarios.to_dict(), args.json, scenarios.render())
+        return EXIT_FINDING if scenarios.scenarios else EXIT_OK
+
     if args.command == "lint":
         findings = check(mandate)
         payload = {

agentmandate-0.4.0/agentmandate/scenarios.py

@@ -0,0 +1,334 @@
+"""Turn reachable breach paths into reviewable scenario-test skeletons.
+
+``reach`` proves that a call sequence is permitted by the reviewed manifest.
+It does not prove that an agent will choose that sequence or that a deployed
+control will allow it. A scenario test is the dynamic half of that question.
+
+The exporter preserves the counterexample and leaves the application-specific
+parts blank. A human must supply the environment setup, agent input, and
+expected control boundary. The library never invents those facts.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from hashlib import sha256
+from pathlib import Path
+from typing import Any
+
+from .manifest import Mandate
+from .reach import Breach, Step, analyse
+
+SCENARIOS_SCHEMA = "agentmandate.scenarios/v1"
+
+
+def _required_string(value: Any, field: str, *, allow_blank: bool = False) -> str:
+    if not isinstance(value, str):
+        raise ValueError(f"scenario {field} must be a string")
+    if not allow_blank and not value.strip():
+        raise ValueError(f"scenario {field} must not be blank")
+    if value and not value.strip():
+        raise ValueError(f"scenario {field} must not be whitespace")
+    return value
+
+
+@dataclass(frozen=True)
+class ScenarioStep:
+    """One structured tool call from a static counterexample."""
+
+    tool: str
+    binding: str | None = None
+    spent: str | None = None
+    currency: str | None = None
+
+    @classmethod
+    def from_step(cls, step: Step) -> ScenarioStep:
+        return cls(
+            tool=step.tool,
+            binding=step.binding,
+            spent=str(step.spent) if step.spent is not None else None,
+            currency=step.currency,
+        )
+
+    @classmethod
+    def from_dict(cls, value: Any) -> ScenarioStep:
+        if not isinstance(value, dict):
+            raise ValueError("scenario step must be an object")
+        tool = _required_string(value.get("tool"), "step tool")
+        binding = value.get("binding")
+        spent = value.get("spent")
+        currency = value.get("currency")
+        for field, item in (
+            ("step binding", binding),
+            ("step spent", spent),
+            ("step currency", currency),
+        ):
+            if item is not None:
+                _required_string(item, field)
+        if (spent is None) != (currency is None):
+            raise ValueError(
+                "scenario step spent and currency must be supplied together"
+            )
+        return cls(tool=tool, binding=binding, spent=spent, currency=currency)
+
+    def to_dict(self) -> dict[str, str]:
+        payload = {"tool": self.tool}
+        if self.binding is not None:
+            payload["binding"] = self.binding
+        if self.spent is not None:
+            payload["spent"] = self.spent
+            payload["currency"] = self.currency or ""
+        return payload
+
+
+@dataclass(frozen=True)
+class Scenario:
+    """A static witness plus the application facts needed to execute it."""
+
+    kind: str
+    detail: str
+    path: tuple[ScenarioStep, ...]
+    environment: tuple[str, ...] = ()
+    agent_input: str = ""
+    expected_control: str = ""
+
+    @property
+    def identifier(self) -> str:
+        route = ">".join(step.tool for step in self.path)
+        witness = {
+            "kind": self.kind,
+            "detail": self.detail,
+            "path": [step.to_dict() for step in self.path],
+        }
+        digest = sha256(
+            json.dumps(
+                witness,
+                sort_keys=True,
+                separators=(",", ":"),
+            ).encode("utf-8")
+        ).hexdigest()[:12]
+        return f"{self.kind}:{route}:{digest}"
+
+    @property
+    def reviewed(self) -> bool:
+        return (
+            bool(self.environment)
+            and bool(self.agent_input.strip())
+            and bool(self.expected_control.strip())
+        )
+
+    @classmethod
+    def from_breach(cls, breach: Breach) -> Scenario:
+        return cls(
+            kind=breach.kind,
+            detail=breach.detail,
+            path=tuple(ScenarioStep.from_step(step) for step in breach.path),
+        )
+
+    @classmethod
+    def from_dict(cls, value: Any) -> Scenario:
+        if not isinstance(value, dict):
+            raise ValueError("scenario must be an object")
+        kind = _required_string(value.get("kind"), "kind")
+        detail = _required_string(value.get("detail"), "detail")
+        raw_path = value.get("path")
+        if not isinstance(raw_path, list) or not raw_path:
+            raise ValueError("scenario path must be a non-empty list")
+        raw_environment = value.get("environment", [])
+        if not isinstance(raw_environment, list) or any(
+            not isinstance(item, str) or not item.strip()
+            for item in raw_environment
+        ):
+            raise ValueError(
+                "scenario environment must be a list of non-blank strings"
+            )
+        scenario = cls(
+            kind=kind,
+            detail=detail,
+            path=tuple(ScenarioStep.from_dict(step) for step in raw_path),
+            environment=tuple(raw_environment),
+            agent_input=_required_string(
+                value.get("agent_input", ""), "agent_input", allow_blank=True
+            ),
+            expected_control=_required_string(
+                value.get("expected_control", ""),
+                "expected_control",
+                allow_blank=True,
+            ),
+        )
+        if "id" in value and value["id"] != scenario.identifier:
+            raise ValueError(
+                f"scenario id {value['id']!r} does not match "
+                f"{scenario.identifier!r}"
+            )
+        return scenario
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "id": self.identifier,
+            "kind": self.kind,
+            "detail": self.detail,
+            "path": [step.to_dict() for step in self.path],
+            "environment": list(self.environment),
+            "agent_input": self.agent_input,
+            "expected_control": self.expected_control,
+        }
+
+
+@dataclass(frozen=True)
+class ScenarioSet:
+    """Scenario skeletons derived from one bounded authority analysis."""
+
+    agent: str
+    depth: int
+    truncated: bool
+    scenarios: tuple[Scenario, ...]
+
+    @property
+    def unreviewed(self) -> tuple[str, ...]:
+        return tuple(
+            scenario.identifier
+            for scenario in self.scenarios
+            if not scenario.reviewed
+        )
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "schema": SCENARIOS_SCHEMA,
+            "agent": self.agent,
+            "depth": self.depth,
+            "truncated": self.truncated,
+            "scenarios": [scenario.to_dict() for scenario in self.scenarios],
+        }
+
+    @classmethod
+    def from_dict(cls, value: Any) -> ScenarioSet:
+        if not isinstance(value, dict):
+            raise ValueError("scenarios root must be an object")
+        if value.get("schema") != SCENARIOS_SCHEMA:
+            raise ValueError(
+                f"unsupported scenarios schema: {value.get('schema')!r}"
+            )
+        agent = _required_string(value.get("agent"), "agent")
+        depth = value.get("depth")
+        if isinstance(depth, bool) or not isinstance(depth, int) or depth < 1:
+            raise ValueError("scenario depth must be a positive integer")
+        truncated = value.get("truncated")
+        if not isinstance(truncated, bool):
+            raise ValueError("scenario truncated must be true or false")
+        raw_scenarios = value.get("scenarios")
+        if not isinstance(raw_scenarios, list):
+            raise ValueError("scenarios must be a list")
+        scenarios = tuple(Scenario.from_dict(item) for item in raw_scenarios)
+        identifiers = [scenario.identifier for scenario in scenarios]
+        if len(identifiers) != len(set(identifiers)):
+            raise ValueError("scenarios must not contain duplicate ids")
+        return cls(
+            agent=agent,
+            depth=depth,
+            truncated=truncated,
+            scenarios=scenarios,
+        )
+
+    def render(self) -> str:
+        lines = [f"counterexample scenarios for {self.agent}"]
+        if not self.scenarios:
+            lines.append(f"  none within depth {self.depth}")
+            return "\n".join(lines)
+        for scenario in self.scenarios:
+            lines.append(f"  {scenario.identifier}")
+            lines.append(
+                "    path: " + " -> ".join(step.tool for step in scenario.path)
+            )
+            lines.append(
+                "    environment: "
+                + (
+                    "; ".join(scenario.environment)
+                    if scenario.environment
+                    else "REVIEW: describe the test fixture and starting state"
+                )
+            )
+            lines.append(
+                "    agent input: "
+                + (
+                    scenario.agent_input
+                    or "REVIEW: write the input that should exercise this risk"
+                )
+            )
+            lines.append(
+                "    expected: "
+                + (
+                    scenario.expected_control
+                    or "REVIEW: state where the deployed control must stop it"
+                )
+            )
+        if self.truncated:
+            lines.append("")
+            lines.append(
+                f"search stopped at depth {self.depth}; deeper paths were not analysed"
+            )
+        return "\n".join(lines)
+
+
+def derive_scenarios(mandate: Mandate, depth: int | None = None) -> ScenarioSet:
+    """Preserve each reachable breach as a reviewable scenario skeleton."""
+    authority = analyse(mandate, depth=depth)
+    return ScenarioSet(
+        agent=mandate.agent,
+        depth=authority.depth,
+        truncated=authority.truncated,
+        scenarios=tuple(
+            Scenario.from_breach(breach) for breach in authority.breaches
+        ),
+    )
+
+
+def reconcile_scenarios(
+    current: ScenarioSet,
+    reviewed: ScenarioSet,
+) -> ScenarioSet:
+    """Carry human-authored fields onto witnesses that remain reachable."""
+    if current.agent != reviewed.agent:
+        raise ValueError(
+            f"cannot reconcile scenarios for {reviewed.agent!r} with "
+            f"{current.agent!r}"
+        )
+    by_id = {scenario.identifier: scenario for scenario in reviewed.scenarios}
+    reconciled = []
+    for scenario in current.scenarios:
+        earlier = by_id.get(scenario.identifier)
+        if earlier is None:
+            reconciled.append(scenario)
+            continue
+        reconciled.append(
+            Scenario(
+                kind=scenario.kind,
+                detail=scenario.detail,
+                path=scenario.path,
+                environment=earlier.environment,
+                agent_input=earlier.agent_input,
+                expected_control=earlier.expected_control,
+            )
+        )
+    return ScenarioSet(
+        agent=current.agent,
+        depth=current.depth,
+        truncated=current.truncated,
+        scenarios=tuple(reconciled),
+    )
+
+
+def load_scenarios(path: str | Path) -> ScenarioSet:
+    try:
+        value = json.loads(Path(path).read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError) as exc:
+        raise ValueError(f"cannot load scenarios: {exc}") from exc
+    return ScenarioSet.from_dict(value)
+
+
+def save_scenarios(scenarios: ScenarioSet, path: str | Path) -> None:
+    Path(path).write_text(
+        json.dumps(scenarios.to_dict(), indent=2, sort_keys=True) + "\n",
+        encoding="utf-8",
+    )