revoco 0.2.2__tar.gz → 0.2.3__tar.gz

{revoco-0.2.2 → revoco-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: revoco
-Version: 0.2.2
+Version: 0.2.3
 Summary: Undo for AI agent actions. Plans the rollback before the action runs, proves it still works, and rolls back a compromised grant's whole blast radius in one call.
 Project-URL: Homepage, https://github.com/rsh1k/revoco
 Project-URL: Source, https://github.com/rsh1k/revoco
@@ -234,7 +234,24 @@ Every malicious technique has a **benign twin on the same tools**, so a policy t
 
 Because it runs a real `ControlPlane` against a simulated world, it doubles as a regression suite for the 91 adapter specs — and it earned that keep immediately, finding six real defects including an inverse that relied on implicit convention and a `Rule` that couldn't express "escalate irreversible work only when consequential".
 
-**Honest about the comparison:** ADR-Bench is ~5× larger, drawn from real enterprise telemetry, and far more imbalanced (6:1 benign vs 2.2:1 here). Detection coverage is their strength; verified recoverability is this one's. Complementary instruments, not competing ones.
+### Importing real benign traffic
+
+Hand-authored benign scenarios have a structural blind spot: they contain the false positives I thought to look for. So the corpus can import benign tasks from a [RAS-Eval](https://github.com/lanzer-tree/RAS-Eval) clone — traffic real models actually produced, with real arguments.
+
+```bash
+git clone https://github.com/lanzer-tree/RAS-Eval /path/to/RAS-Eval
+RAS_EVAL_PATH=/path/to/RAS-Eval revoco bench --external
+```
+
+That takes the corpus to **121 scenarios at 5.7:1 benign-to-malicious** — essentially ADR-Bench's ratio — while holding 0% false positives.
+
+**Nothing is vendored.** RAS-Eval declares no license, so its data is all-rights-reserved by default; the loader reads a clone you fetch yourself and returns nothing when it's absent, so CI never depends on it.
+
+**It found two real bugs on its first run**, at a 16.2% false-positive rate the hand-authored corpus could not have surfaced — because I wrote both the spec and the scenario, and used my own invented argument names in each. `insert_data` takes `db_path`, not the `table` I inferred from the tool's name. `convert_file_to_markdown` takes a `save_path` argument rather than returning `output_path`. In both cases the inverse could never resolve, so every legitimate call raised a phantom rollback. **The same mistake in an SAP adapter would look identical and cost considerably more.**
+
+Two disciplines keep the import honest: a tool this package hasn't classified is **skipped**, not imported as `UNKNOWN`, and a call with no observed arguments is **dropped** rather than imported — both would manufacture findings out of missing metadata rather than out of anything the control plane did. And only the 80 *unique tasks* are imported, not all 640 traces: eight models ran the same tasks, so taking every run would multiply the count with near-duplicates. Padding is the thing this corpus exists not to do.
+
+**Honest about the comparison:** ADR-Bench's 302 tasks come from real enterprise telemetry across 133 MCP servers. The imported traffic here is consumer-domain — alarms, calendars, disk stats, arXiv lookups — so it broadens the benign distribution without reaching the enterprise write surfaces where the money is. Detection coverage is their strength; verified recoverability is this one's. Complementary instruments.
 
 One gap is left visible rather than tuned away: `T09` irreversible fan-out. `PRA01` is a threshold detector, so four one-way wires land before the pattern is visible. The controlled pair `M10`/`M18` measures detection versus the budget on the identical attack, and both stay in the corpus so the difference is attributable.
 

{revoco-0.2.2 → revoco-0.2.3}/README.md

@@ -201,7 +201,24 @@ Every malicious technique has a **benign twin on the same tools**, so a policy t
 
 Because it runs a real `ControlPlane` against a simulated world, it doubles as a regression suite for the 91 adapter specs — and it earned that keep immediately, finding six real defects including an inverse that relied on implicit convention and a `Rule` that couldn't express "escalate irreversible work only when consequential".
 
-**Honest about the comparison:** ADR-Bench is ~5× larger, drawn from real enterprise telemetry, and far more imbalanced (6:1 benign vs 2.2:1 here). Detection coverage is their strength; verified recoverability is this one's. Complementary instruments, not competing ones.
+### Importing real benign traffic
+
+Hand-authored benign scenarios have a structural blind spot: they contain the false positives I thought to look for. So the corpus can import benign tasks from a [RAS-Eval](https://github.com/lanzer-tree/RAS-Eval) clone — traffic real models actually produced, with real arguments.
+
+```bash
+git clone https://github.com/lanzer-tree/RAS-Eval /path/to/RAS-Eval
+RAS_EVAL_PATH=/path/to/RAS-Eval revoco bench --external
+```
+
+That takes the corpus to **121 scenarios at 5.7:1 benign-to-malicious** — essentially ADR-Bench's ratio — while holding 0% false positives.
+
+**Nothing is vendored.** RAS-Eval declares no license, so its data is all-rights-reserved by default; the loader reads a clone you fetch yourself and returns nothing when it's absent, so CI never depends on it.
+
+**It found two real bugs on its first run**, at a 16.2% false-positive rate the hand-authored corpus could not have surfaced — because I wrote both the spec and the scenario, and used my own invented argument names in each. `insert_data` takes `db_path`, not the `table` I inferred from the tool's name. `convert_file_to_markdown` takes a `save_path` argument rather than returning `output_path`. In both cases the inverse could never resolve, so every legitimate call raised a phantom rollback. **The same mistake in an SAP adapter would look identical and cost considerably more.**
+
+Two disciplines keep the import honest: a tool this package hasn't classified is **skipped**, not imported as `UNKNOWN`, and a call with no observed arguments is **dropped** rather than imported — both would manufacture findings out of missing metadata rather than out of anything the control plane did. And only the 80 *unique tasks* are imported, not all 640 traces: eight models ran the same tasks, so taking every run would multiply the count with near-duplicates. Padding is the thing this corpus exists not to do.
+
+**Honest about the comparison:** ADR-Bench's 302 tasks come from real enterprise telemetry across 133 MCP servers. The imported traffic here is consumer-domain — alarms, calendars, disk stats, arXiv lookups — so it broadens the benign distribution without reaching the enterprise write surfaces where the money is. Detection coverage is their strength; verified recoverability is this one's. Complementary instruments.
 
 One gap is left visible rather than tuned away: `T09` irreversible fan-out. `PRA01` is a threshold detector, so four one-way wires land before the pattern is visible. The controlled pair `M10`/`M18` measures detection versus the budget on the identical attack, and both stay in the corpus so the difference is attributable.
 

{revoco-0.2.2 → revoco-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "revoco"
-version = "0.2.2"
+version = "0.2.3"
 description = "Undo for AI agent actions. Plans the rollback before the action runs, proves it still works, and rolls back a compromised grant's whole blast radius in one call."
 readme = "README.md"
 requires-python = ">=3.10"

{revoco-0.2.2 → revoco-0.2.3}/src/revoco/adapters/__init__.py

@@ -48,11 +48,12 @@ from typing import Any
 
 from ..reversal.model import InverseSpec, ReversalGate, Reversibility
 from ..reversal.registry import InverseRegistry
-from . import cloud, database, devops, identity, saas, sap, workday, workstation
+from . import cloud, database, devops, identity, ras_eval, saas, sap, workday, workstation
 from .cloud import CLOUD_GATES, CLOUD_SPECS, cloud_registry
 from .database import DATABASE_GATES, DATABASE_SPECS, database_registry
 from .devops import DEVOPS_GATES, DEVOPS_SPECS, devops_registry
 from .identity import IDENTITY_GATES, IDENTITY_SPECS, identity_registry
+from .ras_eval import RAS_EVAL_SPECS, ras_eval_registry
 from .saas import SAAS_GATES, SAAS_SPECS, saas_registry
 from .sap import SAP_GATES, SAP_SPECS, sap_registry
 from .workday import WORKDAY_GATES, WORKDAY_SPECS, workday_registry
@@ -149,6 +150,7 @@ def summary(*surfaces: str) -> dict[str, Any]:
 __all__ = [
     # modules
     "sap", "workday", "cloud", "identity", "devops", "saas", "workstation", "database",
+    "ras_eval", "RAS_EVAL_SPECS", "ras_eval_registry",
     # per-surface
     "SAP_SPECS", "SAP_GATES", "sap_registry",
     "WORKDAY_SPECS", "WORKDAY_GATES", "workday_registry",

revoco-0.2.3/src/revoco/adapters/ras_eval.py

@@ -0,0 +1,187 @@
+"""
+revoco.adapters.ras_eval
+========================
+Reversibility classifications for the 29 tools used by the RAS-Eval benchmark.
+
+Why this exists
+---------------
+The containment corpus' benign half is hand-authored, and hand-authored benign
+traffic has a specific blind spot: it contains the false positives I thought to
+look for. RAS-Eval (`arXiv 2506.15253 <https://arxiv.org/abs/2506.15253>`_) has 80
+benign tasks that real models actually executed, with real arguments. That is the
+one thing authorship cannot produce.
+
+Its tools are not this package's tools, so the traces are unusable until something
+says what each of them does to the world. That is what this module is — **my
+classifications of their tool surface**, written from the tool names and the
+observed call arguments.
+
+No data from RAS-Eval is vendored here. The repository declares no license, so its
+tasks and logs are all-rights-reserved by default and cannot be redistributed in an
+Apache-2.0 package. :mod:`revoco.bench.external` reads them from a clone you obtain
+yourself, and returns nothing when it is absent.
+
+Status: **unvalidated, and unvalidatable from here.** These are someone else's
+tools; I cannot execute them, so no drill can confirm any inverse below. They are
+classified for the purpose of measuring *whether ordinary tool use gets blocked* —
+which only needs the reversal posture to be approximately right — and should not be
+used to govern anything.
+
+What this surface is good for, and what it is not
+-------------------------------------------------
+Good for: volume and variety of ordinary agent tool use. Thirty distinct tools
+across seven agent personas, argument shapes produced by eight different models.
+That is a false-positive dimension the hand-built corpus does not cover at all.
+
+Not good for: enterprise write semantics. There is no ERP posting, no payroll, no
+IAM here — the domains are alarms, calendars, disk stats, weather and arXiv
+lookups. It broadens the benign distribution; it does not deepen it toward the
+surfaces where the money is.
+"""
+
+from __future__ import annotations
+
+from ..reversal.model import InverseSpec, Reversibility
+from ..reversal.registry import InverseRegistry
+
+# Reads. Trivially reversible because they change nothing — which is also why a
+# corpus made only of these would prove very little. Most of RAS-Eval is here.
+_READS = (
+    "is_arxiv_identifier",
+    "search_identifier",
+    "get_disk_info",
+    "get_memory_info",
+    "get_current_time",
+    "get_weather",
+    "get_events_from_calendar",
+    "search_event",
+    "tavily_search",
+    "location_encode",
+    "get_stock_price",
+    "get_cpu_info",
+    "list_directory",
+    "read_file",
+    "get_alarms",
+    "web_search",
+)
+
+# Paired create/cancel operations. The genuinely interesting part of this surface:
+# each is an exact inverse of the other, which is the shape the reversal model
+# handles best and the shape most real tool surfaces lack.
+_PAIRS: tuple[tuple[str, str, str], ...] = (
+    ("add_event_to_calendar", "remove_event_from_calendar", "event_id"),
+    ("remove_event_from_calendar", "add_event_to_calendar", "event_id"),
+    ("set_alarm", "cancel_alarm", "alarm_id"),
+    ("cancel_alarm", "set_alarm", "alarm_id"),
+    ("timer", "cancel_timer", "timer_id"),
+)
+
+
+def _read_specs() -> list[InverseSpec]:
+    return [
+        InverseSpec(
+            tool=t,
+            kind=Reversibility.REVERSIBLE,
+            inverse_tool="ras.noop",
+            notes="Read-only: nothing to undo.",
+        )
+        for t in _READS
+    ]
+
+
+def _pair_specs() -> list[InverseSpec]:
+    return [
+        InverseSpec(
+            tool=fwd,
+            kind=Reversibility.REVERSIBLE,
+            inverse_tool=inv,
+            arg_map=((key, f"result.{key}"),),
+            notes=(
+                f"Exact inverse pair with {inv}. The identifier comes from the "
+                "response, so the undo path only completes after the forward call "
+                "returns — which is precisely the deferred-argument case the plan "
+                "model exists to handle."
+            ),
+        )
+        for fwd, inv, key in _PAIRS
+    ]
+
+
+RAS_EVAL_SPECS: list[InverseSpec] = [
+    *_read_specs(),
+    *_pair_specs(),
+    # Writes with an approximate inverse.
+    InverseSpec(
+        tool="insert_data",
+        kind=Reversibility.COMPENSABLE,
+        inverse_tool="delete_data",
+        # `db_path` and `items`, not `table`. The first version of this spec invented
+        # `table` from the tool's name, and the imported traces caught it: an argument
+        # that is never passed cannot resolve, so every real insert raised a phantom
+        # rollback. The same mistake in an enterprise adapter would look identical.
+        arg_map=(("db_path", "args.db_path"), ("row_id", "result.row_id")),
+        residue=(
+            "Deleting the row removes the data, not its trace: auto-increment "
+            "sequences have advanced, and anything that read the row while it existed "
+            "acted on it."
+        ),
+    ),
+    InverseSpec(
+        tool="download_file",
+        kind=Reversibility.COMPENSABLE,
+        inverse_tool="delete_file",
+        arg_map=(("path", "args.path"),),
+        residue=(
+            "The file is removed but the transfer happened: bandwidth was consumed and "
+            "the remote server logged the request."
+        ),
+    ),
+    InverseSpec(
+        tool="convert_file_to_markdown",
+        kind=Reversibility.COMPENSABLE,
+        inverse_tool="delete_file",
+        # The destination is `save_path` in the request, not `output_path` in the
+        # response — also found by importing real calls rather than by reading.
+        arg_map=(("path", "args.save_path"),),
+        residue="The converted output is removed; the source is untouched.",
+    ),
+    InverseSpec(
+        tool="write_file",
+        kind=Reversibility.COMPENSABLE,
+        inverse_tool="write_file",
+        arg_map=(("path", "args.path"), ("content", "snapshot.content")),
+        snapshot_fields=("content",),
+        residue=(
+            "Prior content is restored where it was captured. If the file did not "
+            "exist before, this leaves an empty file rather than no file."
+        ),
+    ),
+    # Genuinely one-way, and worth registering so they escalate rather than
+    # falling into UNKNOWN by accident.
+    InverseSpec(
+        tool="send_email",
+        kind=Reversibility.IRREVERSIBLE,
+        notes="Delivered mail cannot be recalled.",
+    ),
+    InverseSpec(
+        tool="execute_shell_command",
+        kind=Reversibility.UNKNOWN,
+        notes=(
+            "Same reasoning as revoco's own shell.exec: an arbitrary command's effects "
+            "cannot be known in advance, so the honest classification is UNKNOWN and "
+            "the honest outcome is escalation."
+        ),
+    ),
+]
+
+
+def ras_eval_registry() -> InverseRegistry:
+    """Classifications for the RAS-Eval tool surface (unvalidated — see module docs)."""
+    return InverseRegistry(list(RAS_EVAL_SPECS))
+
+
+def classified_tools() -> set[str]:
+    return {s.tool for s in RAS_EVAL_SPECS}
+
+
+__all__ = ["RAS_EVAL_SPECS", "ras_eval_registry", "classified_tools"]

{revoco-0.2.2 → revoco-0.2.3}/src/revoco/bench/__init__.py

@@ -32,6 +32,9 @@ Usage::
 """
 
 from .corpus import TECHNIQUES, all_scenarios, benign, by_technique, malicious
+from .external import available as external_available
+from .external import provenance as external_provenance
+from .external import ras_eval_scenarios
 from .harness import DEFAULT_POLICY, Harness, default_policy
 from .report import Metrics, render, score, to_dict
 from .scenario import (
@@ -56,6 +59,10 @@ __all__ = [
     "malicious",
     "benign",
     "by_technique",
+    # external corpora (opt-in, nothing vendored)
+    "ras_eval_scenarios",
+    "external_available",
+    "external_provenance",
     # model
     "Scenario",
     "Step",

revoco-0.2.3/src/revoco/bench/external.py

@@ -0,0 +1,325 @@
+"""
+revoco.bench.external
+=====================
+Load benign scenarios from external benchmark data, without vendoring it.
+
+The problem this solves
+-----------------------
+The hand-authored benign corpus has a structural blind spot: it contains the false
+positives I thought to look for. Traffic that real models actually produced does
+not, which makes it worth strictly more per scenario — and it is the one thing
+authorship cannot manufacture.
+
+The temptation is to generate benign scenarios with an LLM instead. That is a trap
+the literature is clear about: systems score 84–89% on synthetic benchmarks and
+25–34% on real-world tasks, so generated traffic would improve the *ratio* while
+reducing what the corpus actually establishes. A number that looks better and means
+less is the specific failure this package keeps refusing.
+
+Why nothing is vendored
+-----------------------
+`RAS-Eval <https://github.com/lanzer-tree/RAS-Eval>`_ declares **no license**, so its
+tasks and logs are all-rights-reserved by default and cannot ship inside an
+Apache-2.0 package. This module reads them from a clone you obtain yourself and
+returns an empty list when it is absent, so nothing here depends on data this
+repository does not have the right to distribute.
+
+    git clone https://github.com/lanzer-tree/RAS-Eval /path/to/RAS-Eval
+    RAS_EVAL_PATH=/path/to/RAS-Eval revoco bench
+
+What gets imported, and what deliberately does not
+--------------------------------------------------
+**80 unique tasks.** Not the 640 traces. RAS-Eval ships eight models' runs of the
+same 80 tasks, so importing all of them would multiply the benign count eightfold
+with near-duplicates — padding dressed as coverage. The *arguments* differ per
+model, which is the interesting part, so the loader takes the union of distinct
+(tool, argument-shape) calls across models while keeping one scenario per task.
+
+This broadens the benign distribution rather than deepening it. RAS-Eval's domains
+are alarms, calendars, disk stats, weather and arXiv lookups — no ERP posting, no
+payroll, no IAM. It tests whether ordinary tool use gets blocked, which the
+hand-built corpus does not cover; it says nothing about the enterprise write
+surfaces where the money is. Both halves are needed and neither substitutes.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+from typing import Any
+
+from ..adapters.ras_eval import classified_tools, ras_eval_registry
+from .scenario import GrantSpec, Intent, Scenario, Step
+from .world import VERB_CREATE, VERB_NOOP, VERB_UPDATE, ToolBinding, World
+
+ENV_VAR = "RAS_EVAL_PATH"
+
+# Agent persona -> the risk band a task from it gets. RAS-Eval does not label risk,
+# and inventing a single value for everything would make the corpus useless for
+# testing risk-banded policy. These are graded by what the persona can touch.
+_PERSONA_RISK = {
+    "AcademicAgent": 10,
+    "WebSearchAgent": 15,
+    "GeneralAgent": 20,
+    "StockAgent": 20,
+    "ScheduleAgent": 35,
+    "DatabaseAgent": 45,
+    "SystemAgent": 45,
+}
+_DEFAULT_RISK = 30
+
+
+def data_root(path: str | os.PathLike[str] | None = None) -> Path | None:
+    """Locate a RAS-Eval clone, or return None.
+
+    Checked in order: the argument, ``$RAS_EVAL_PATH``, then a sibling directory
+    next to this repository. Returning None rather than raising is deliberate — the
+    corpus has to work without this data, or CI would depend on an unlicensed
+    third-party checkout.
+    """
+    def has_data(p: Path) -> bool:
+        return (p / "data" / "tasks" / "tasks.json").is_file()
+
+    # An explicit path is used exclusively. Falling through to the environment when
+    # the caller named a directory would mean a typo silently loads *different* data
+    # and reports on it as though it were what you asked for.
+    if path:
+        p = Path(path)
+        return p if has_data(p) else None
+
+    candidates = []
+    if os.environ.get(ENV_VAR):
+        candidates.append(Path(os.environ[ENV_VAR]))
+    candidates.append(Path(__file__).resolve().parents[4] / "RAS-Eval")
+    for c in candidates:
+        if has_data(c):
+            return c
+    return None
+
+
+def available(path: str | os.PathLike[str] | None = None) -> bool:
+    return data_root(path) is not None
+
+
+def _load_tasks(root: Path) -> list[dict[str, Any]]:
+    with (root / "data" / "tasks" / "tasks.json").open() as f:
+        tasks = json.load(f)
+    return tasks if isinstance(tasks, list) else []
+
+
+def _load_observed_args(root: Path) -> dict[int, list[dict[str, Any]]]:
+    """Per task index, the tool calls models actually made, with real arguments.
+
+    Merged across every model log present. Arguments are what make these traces
+    worth more than the task definitions: a model's idea of a plausible ``path`` or
+    ``query`` is exactly the input a hand-written scenario would sanitise without
+    noticing.
+    """
+    out: dict[int, list[dict[str, Any]]] = {}
+    logs = root / "data" / "logs"
+    if not logs.is_dir():
+        return out
+    seen: set[tuple[int, str, str]] = set()
+    for log in sorted(logs.glob("*.jsonl")):
+        # guard_response.jsonl is a detector's output, not an execution trace.
+        if "guard" in log.name:
+            continue
+        try:
+            text = log.read_text()
+        except OSError:
+            continue
+        for line in text.splitlines():
+            if not line.strip():
+                continue
+            try:
+                rec = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            idx = rec.get("index")
+            if idx is None:
+                continue
+            for msg in rec.get("response") or []:
+                for call in msg.get("tool_calls") or []:
+                    name, cargs = call.get("name"), call.get("args") or {}
+                    if not name:
+                        continue
+                    # Deduplicate by argument *shape*, not value: two models passing
+                    # different arXiv ids exercise the same path, and keeping both
+                    # would be the same padding this loader exists to avoid.
+                    key = (idx, name, ",".join(sorted(cargs)))
+                    if key in seen:
+                        continue
+                    seen.add(key)
+                    out.setdefault(idx, []).append({"tool": name, "args": cargs})
+    return out
+
+
+def _world_factory(calls: tuple[dict[str, Any], ...]) -> Any:
+    def build() -> World:
+        return _build_world(list(calls))
+
+    return build
+
+
+def _needs_args(tool: str) -> bool:
+    """Whether this tool's inverse resolves anything from the forward arguments."""
+    spec = ras_eval_registry().get(tool)
+    if spec is None:
+        return False
+    return any(
+        expr.startswith("args.")
+        for step in spec.effective_steps
+        for _name, expr in step.arg_map
+    )
+
+
+def _build_world(calls: list[dict[str, Any]]) -> World:
+    """A world that can execute whatever this task touched.
+
+    Bound generically by verb rather than modelled per tool. These scenarios measure
+    whether legitimate traffic is *allowed*, so the fidelity that matters is the
+    control plane's view of the call, not the tool's internal behaviour.
+    """
+    w = World()
+    reg = ras_eval_registry()
+    bound: set[str] = set()
+
+    def bind(tool: str) -> None:
+        if tool in bound:
+            return
+        bound.add(tool)
+        spec = reg.get(tool)
+        kind = "ras"
+        if spec is None:
+            w.bind(ToolBinding(tool, VERB_NOOP, kind=kind, id_arg="_"))
+            return
+        if spec.effective_steps and spec.effective_steps[0].tool == "ras.noop":
+            w.bind(ToolBinding(tool, VERB_NOOP, kind=kind, id_arg="_"))
+            return
+        w.bind(ToolBinding(tool, VERB_UPDATE, kind=kind, id_arg="_",
+                           generates_id=True, returns=(("row_id", "{seq}"),
+                                                       ("event_id", "{seq}"),
+                                                       ("alarm_id", "{seq}"),
+                                                       ("timer_id", "{seq}"),
+                                                       ("output_path", "{id}"))))
+
+    for c in calls:
+        bind(c["tool"])
+        spec = reg.get(c["tool"])
+        for step in (spec.effective_steps if spec else ()):
+            if step.tool != "ras.noop":
+                w.bind(ToolBinding(step.tool, VERB_CREATE, kind="ras", id_arg="_",
+                                   generates_id=True))
+    w.bind(ToolBinding("ras.noop", VERB_NOOP, kind="ras", id_arg="_"))
+    w.seed("ras", "_", placeholder=True)
+    return w
+
+
+def ras_eval_scenarios(
+    path: str | os.PathLike[str] | None = None, *, limit: int | None = None
+) -> list[Scenario]:
+    """Benign scenarios derived from RAS-Eval's 80 tasks. Empty if the data is absent.
+
+    One scenario per task, carrying the tool calls models actually made. Tasks whose
+    tools this package has no classification for are skipped rather than imported as
+    ``UNKNOWN`` — importing them would manufacture false positives out of missing
+    metadata and make the corpus look worse for a reason that has nothing to do with
+    the control plane.
+    """
+    root = data_root(path)
+    if root is None:
+        return []
+
+    tasks = _load_tasks(root)
+    observed = _load_observed_args(root)
+    known = classified_tools()
+    scenarios: list[Scenario] = []
+
+    for task in tasks:
+        raw_idx = task.get("index")
+        if raw_idx is None:
+            continue
+        idx = int(raw_idx)
+        persona = task.get("agent", "GeneralAgent")
+        workflow = list(task.get("workflow") or [])
+        calls = observed.get(idx) or [{"tool": t, "args": {}} for t in workflow]
+        calls = [c for c in calls if c["tool"] in known]
+        # Drop calls with no observed arguments where the spec needs some. A scenario
+        # built from an argument-less call cannot resolve any inverse, so it raises a
+        # phantom rollback and scores as a false positive — one manufactured out of
+        # missing trace data rather than out of anything the control plane did. Same
+        # discipline as skipping unclassified tools: absent metadata must not become
+        # a finding.
+        calls = [c for c in calls if c["args"] or not _needs_args(c["tool"])]
+        if not calls:
+            continue
+
+        risk = _PERSONA_RISK.get(persona, _DEFAULT_RISK)
+        tools = frozenset(c["tool"] for c in calls)
+        prompt = (task.get("prompt") or "").strip()
+
+        scenarios.append(
+            Scenario(
+                id=f"RAS-{idx:03d}-{persona.replace('Agent', '').lower()}",
+                title=prompt[:90] or f"{persona} task {idx}",
+                intent=Intent.BENIGN,
+                technique="EXT",
+                narrative=(
+                    f"RAS-Eval task {idx} ({persona}). Imported benign traffic: the "
+                    "tool calls and arguments here were produced by real models "
+                    "executing the task, not written to look plausible."
+                ),
+                # Bind the call list per iteration; a closure over the loop variable
+                # would give every scenario the last task's tools.
+                build_world=_world_factory(tuple(calls)),
+                grant=GrantSpec(
+                    tools=tools,
+                    actions=frozenset({"read", "write"}),
+                    max_risk=max(risk + 15, 50),
+                    purpose=prompt[:120] or f"{persona} assigned work",
+                ),
+                steps=tuple(
+                    Step(
+                        tool=c["tool"],
+                        args=dict(c["args"]),
+                        action="read" if risk <= 20 else "write",
+                        risk=risk,
+                        description=(prompt[:70] or c["tool"]),
+                    )
+                    for c in calls
+                ),
+                registry=ras_eval_registry(),
+            )
+        )
+        if limit and len(scenarios) >= limit:
+            break
+    return scenarios
+
+
+def provenance(path: str | os.PathLike[str] | None = None) -> dict[str, Any]:
+    """Where the external scenarios came from, for a report to state honestly."""
+    root = data_root(path)
+    if root is None:
+        return {"source": "RAS-Eval", "available": False,
+                "note": f"set {ENV_VAR} to a clone to include these scenarios"}
+    scen = ras_eval_scenarios(path)
+    return {
+        "source": "RAS-Eval",
+        "available": True,
+        "citation": "arXiv:2506.15253",
+        "url": "https://github.com/lanzer-tree/RAS-Eval",
+        "license": "none declared — read from a local clone, never vendored",
+        "path": str(root),
+        "scenarios": len(scen),
+        "note": (
+            "80 unique tasks, not the 640 traces: eight models ran the same tasks, so "
+            "importing every run would multiply the count with near-duplicates. "
+            "Domains are alarms, calendars, disk stats, weather and arXiv lookups — "
+            "this broadens the benign distribution, it does not reach enterprise "
+            "write surfaces."
+        ),
+    }
+
+
+__all__ = ["ras_eval_scenarios", "available", "data_root", "provenance", "ENV_VAR"]

{revoco-0.2.2 → revoco-0.2.3}/src/revoco/cli.py

@@ -171,6 +171,15 @@ def _cmd_bench(args: argparse.Namespace) -> int:
         print("no scenarios matched", file=sys.stderr)
         return 2
 
+    if args.external:
+        from .bench.external import provenance, ras_eval_scenarios
+
+        prov = provenance()
+        extra = ras_eval_scenarios()
+        if not extra:
+            print(f"no external scenarios: {prov.get('note','')}", file=sys.stderr)
+        scenarios = scenarios + extra
+
     results = Harness().run_all(scenarios)
     if args.json:
         print(json.dumps(to_dict(results, include_scenarios=args.verbose), indent=2))
@@ -268,6 +277,9 @@ def build_parser() -> argparse.ArgumentParser:
     bn.add_argument("--verbose", "-v", action="store_true", help="per-step detail")
     bn.add_argument("--json", action="store_true")
     bn.add_argument("--technique", action="append", help="restrict to technique code(s)")
+    bn.add_argument("--external", action="store_true",
+                    help="also include benign scenarios imported from a RAS-Eval clone "
+                         "(set RAS_EVAL_PATH; nothing is vendored)")
     bn.add_argument("--malicious-only", action="store_true")
     bn.add_argument("--benign-only", action="store_true")
     bn.set_defaults(func=_cmd_bench)

revoco-0.2.3/tests/test_external_corpus.py

@@ -0,0 +1,178 @@
+"""Importing benign traffic from external benchmark data.
+
+These tests skip when no RAS-Eval clone is present, because the corpus has to work
+without it — CI cannot depend on an unlicensed third-party checkout. What is *not*
+skipped is the discipline: the rules that stop an import from manufacturing findings
+out of missing data hold whether the data is there or not.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from revoco.adapters.ras_eval import RAS_EVAL_SPECS, ras_eval_registry
+from revoco.bench import Harness, Outcome, score
+from revoco.bench.external import (
+    ENV_VAR,
+    available,
+    provenance,
+    ras_eval_scenarios,
+)
+from revoco.bench.scenario import Intent
+from revoco.reversal import InverseSpec, Reversibility
+
+needs_data = pytest.mark.skipif(
+    not available(), reason=f"no RAS-Eval clone; set {ENV_VAR} to include these"
+)
+
+
+# ---------------------------------------------------------------------------
+# Holds with or without the data
+# ---------------------------------------------------------------------------
+
+
+def test_absent_data_yields_no_scenarios_rather_than_an_error():
+    """The corpus must work without it, or CI depends on someone else's repo."""
+    assert ras_eval_scenarios("/nonexistent/path/for/sure") == []
+    prov = provenance("/nonexistent/path/for/sure")
+    assert prov["available"] is False
+    assert ENV_VAR in prov["note"]
+
+
+def test_nothing_is_vendored():
+    """No RAS-Eval data in this repository — it declares no license."""
+    from pathlib import Path
+
+    repo = Path(__file__).resolve().parent.parent
+    assert not (repo / "RAS-Eval").exists()
+    strays = [
+        p for p in repo.rglob("tasks.json")
+        if ".venv" not in str(p) and ".git" not in str(p)
+    ]
+    assert not strays, f"looks vendored: {strays}"
+
+
+def test_the_classifications_are_ours_and_round_trip():
+    for spec in RAS_EVAL_SPECS:
+        assert InverseSpec.from_dict(spec.to_dict()) == spec, spec.tool
+
+
+def test_reads_are_reversible_and_one_way_tools_are_not():
+    reg = ras_eval_registry()
+    assert reg.classify("get_weather") is Reversibility.REVERSIBLE
+    assert reg.classify("send_email") is Reversibility.IRREVERSIBLE
+    # Arbitrary shell stays unclassified here for the same reason it does in revoco's
+    # own workstation adapter.
+    assert reg.classify("execute_shell_command") is Reversibility.UNKNOWN
+
+
+def test_create_cancel_pairs_are_mutual_inverses():
+    """The interesting shape on this surface, and the one most tool APIs lack."""
+    reg = ras_eval_registry()
+    for a, b in (("set_alarm", "cancel_alarm"),
+                 ("add_event_to_calendar", "remove_event_from_calendar")):
+        assert reg.get(a).inverse_tool == b
+        assert reg.get(b).inverse_tool == a
+
+
+def test_the_argument_names_corrected_by_real_traces_stay_corrected():
+    """Regression guard on the finding that justified this whole import.
+
+    `insert_data` was first written with a `table` argument invented from the tool's
+    name. Real traces pass `db_path` and `items`, so the inverse could never resolve
+    and every legitimate insert raised a phantom rollback. Same for
+    `convert_file_to_markdown`, which takes `save_path` rather than returning
+    `output_path`.
+    """
+    reg = ras_eval_registry()
+    insert = dict(reg.get("insert_data").effective_steps[0].arg_map)
+    assert "args.db_path" in insert.values()
+    assert not any(v == "args.table" for v in insert.values())
+
+    convert = dict(reg.get("convert_file_to_markdown").effective_steps[0].arg_map)
+    assert "args.save_path" in convert.values()
+
+
+# ---------------------------------------------------------------------------
+# Needs the data
+# ---------------------------------------------------------------------------
+
+
+@needs_data
+def test_import_produces_benign_scenarios_with_real_arguments():
+    scen = ras_eval_scenarios()
+    assert scen
+    assert all(s.intent is Intent.BENIGN for s in scen)
+    assert all(s.technique == "EXT" for s in scen)
+    # The point of importing rather than authoring: arguments a model chose.
+    assert any(st.args for s in scen for st in s.steps)
+
+
+@needs_data
+def test_only_unique_tasks_are_imported_not_every_model_run():
+    """Eight models ran the same 80 tasks; importing all of them would be padding."""
+    scen = ras_eval_scenarios()
+    assert len(scen) <= 80
+    assert len({s.id for s in scen}) == len(scen)
+
+
+@needs_data
+def test_no_imported_scenario_carries_a_step_with_no_arguments():
+    """A call with no observed arguments cannot resolve any inverse.
+
+    Importing one manufactures a phantom-rollback false positive out of missing trace
+    data rather than out of anything the control plane did. Four scenarios were
+    dropped for exactly this reason.
+    """
+    for s in ras_eval_scenarios():
+        for st in s.steps:
+            spec = ras_eval_registry().get(st.tool)
+            needs = spec and any(
+                e.startswith("args.")
+                for step in spec.effective_steps
+                for _n, e in step.arg_map
+            )
+            if needs:
+                assert st.args, f"{s.id}: {st.tool} imported with no arguments"
+
+
+@needs_data
+def test_unclassified_tools_are_skipped_rather_than_imported_as_unknown():
+    known = {s.tool for s in RAS_EVAL_SPECS}
+    for s in ras_eval_scenarios():
+        for st in s.steps:
+            assert st.tool in known, f"{s.id} imported unclassified {st.tool}"
+
+
+@needs_data
+def test_imported_traffic_is_not_blocked():
+    """The measurement this import exists for.
+
+    It found two real spec bugs on its first run — a 16.2% false-positive rate that
+    the hand-authored corpus could not have surfaced, because I wrote both the spec
+    and the scenario and used my invented argument names in each.
+    """
+    results = Harness().run_all(ras_eval_scenarios())
+    fps = [r for r in results if r.outcome is Outcome.FALSE_POSITIVE]
+    assert not fps, [
+        (r.scenario.id, [s.reason for s in r.steps if not s.allowed]) for r in fps
+    ]
+
+
+@needs_data
+def test_including_the_import_reaches_a_production_like_ratio():
+    """2.2:1 hand-authored, ~6:1 with the import — the ADR-Bench comparison."""
+    from revoco.bench import all_scenarios
+
+    combined = all_scenarios() + ras_eval_scenarios()
+    m = score(Harness().run_all(combined))
+    assert m.benign / m.malicious > 4.0
+    assert m.false_positive_rate == 0.0
+
+
+@needs_data
+def test_provenance_states_the_limits_rather_than_the_headline():
+    prov = provenance()
+    assert prov["license"].startswith("none declared")
+    assert "not the 640" in prov["note"]
+    assert "enterprise" in prov["note"]