revoco 0.2.2__tar.gz → 0.3.0__tar.gz

{revoco-0.2.2 → revoco-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: revoco
-Version: 0.2.2
+Version: 0.3.0
 Summary: Undo for AI agent actions. Plans the rollback before the action runs, proves it still works, and rolls back a compromised grant's whole blast radius in one call.
 Project-URL: Homepage, https://github.com/rsh1k/revoco
 Project-URL: Source, https://github.com/rsh1k/revoco
@@ -212,6 +212,33 @@ It keeps five states apart, and the distinctions carry the value: a window that
 
 ---
 
+## Persistence
+
+```python
+from revoco import ControlPlane
+from revoco.store import SqliteStore
+
+store = SqliteStore("/var/lib/revoco/revoco.db")
+cp = ControlPlane(store=store, ...)
+
+report = store.startup_report()      # what this restart means
+print(report.notes)
+```
+
+Four decisions, each with a reason:
+
+**WAL with `synchronous=FULL`.** SQLite's defaults are ambiguous enough that you can't rely on them, and `NORMAL` doesn't survive power loss. For an evidence store that isn't a trade-off — a ledger that silently drops its last entries is precisely the truncation case a self-contained hash chain **cannot** detect, because the remaining prefix still verifies.
+
+**Append-only by triggers, not permissions.** SQLite has no `GRANT`, so the INSERT-only grant you'd use in Postgres isn't available. `BEFORE UPDATE`/`BEFORE DELETE` triggers that `RAISE(ABORT)` are the mechanism, and they bind *any* client — including a `sqlite3` shell, not just this code. Worth stating because the Postgres instinct silently produces no protection here.
+
+**The ledger append and journal write are one transaction.** The correctness question persistence actually raises. Written separately, a crash between them leaves the journal claiming a plan the ledger never recorded — and afterwards nothing can tell you which is true. Also: entries are prepared, persisted, *then* added to the in-memory chain, so the in-memory head can never outrun the durable one.
+
+**No fake freshness after downtime.** The tempting fix for "every proof is stale after an outage" is to stop counting downtime against staleness. That's wrong — an ERP upgrade during the outage is exactly when a spec silently becomes a confident wrong rollback, and a proof that survived on a technicality is a phantom rollback with a certificate. So downtime counts, `startup_report()` says how many proofs went stale and for how long, and `due()` puts them at the front of the queue. Visible and remediated fast beats invisible and assumed good.
+
+`startup_report()` also distinguishes a **broken chain from a restart** — a clean restart loses nothing, so verification failure is never an artefact of restarting and says so in as many words.
+
+---
+
 ## The containment benchmark
 
 ```bash
@@ -234,7 +261,24 @@ Every malicious technique has a **benign twin on the same tools**, so a policy t
 
 Because it runs a real `ControlPlane` against a simulated world, it doubles as a regression suite for the 91 adapter specs — and it earned that keep immediately, finding six real defects including an inverse that relied on implicit convention and a `Rule` that couldn't express "escalate irreversible work only when consequential".
 
-**Honest about the comparison:** ADR-Bench is ~5× larger, drawn from real enterprise telemetry, and far more imbalanced (6:1 benign vs 2.2:1 here). Detection coverage is their strength; verified recoverability is this one's. Complementary instruments, not competing ones.
+### Importing real benign traffic
+
+Hand-authored benign scenarios have a structural blind spot: they contain the false positives I thought to look for. So the corpus can import benign tasks from a [RAS-Eval](https://github.com/lanzer-tree/RAS-Eval) clone — traffic real models actually produced, with real arguments.
+
+```bash
+git clone https://github.com/lanzer-tree/RAS-Eval /path/to/RAS-Eval
+RAS_EVAL_PATH=/path/to/RAS-Eval revoco bench --external
+```
+
+That takes the corpus to **121 scenarios at 5.7:1 benign-to-malicious** — essentially ADR-Bench's ratio — while holding 0% false positives.
+
+**Nothing is vendored.** RAS-Eval declares no license, so its data is all-rights-reserved by default; the loader reads a clone you fetch yourself and returns nothing when it's absent, so CI never depends on it.
+
+**It found two real bugs on its first run**, at a 16.2% false-positive rate the hand-authored corpus could not have surfaced — because I wrote both the spec and the scenario, and used my own invented argument names in each. `insert_data` takes `db_path`, not the `table` I inferred from the tool's name. `convert_file_to_markdown` takes a `save_path` argument rather than returning `output_path`. In both cases the inverse could never resolve, so every legitimate call raised a phantom rollback. **The same mistake in an SAP adapter would look identical and cost considerably more.**
+
+Two disciplines keep the import honest: a tool this package hasn't classified is **skipped**, not imported as `UNKNOWN`, and a call with no observed arguments is **dropped** rather than imported — both would manufacture findings out of missing metadata rather than out of anything the control plane did. And only the 80 *unique tasks* are imported, not all 640 traces: eight models ran the same tasks, so taking every run would multiply the count with near-duplicates. Padding is the thing this corpus exists not to do.
+
+**Honest about the comparison:** ADR-Bench's 302 tasks come from real enterprise telemetry across 133 MCP servers. The imported traffic here is consumer-domain — alarms, calendars, disk stats, arXiv lookups — so it broadens the benign distribution without reaching the enterprise write surfaces where the money is. Detection coverage is their strength; verified recoverability is this one's. Complementary instruments.
 
 One gap is left visible rather than tuned away: `T09` irreversible fan-out. `PRA01` is a threshold detector, so four one-way wires land before the pattern is visible. The controlled pair `M10`/`M18` measures detection versus the budget on the identical attack, and both stay in the corpus so the difference is attributable.
 
@@ -314,7 +358,7 @@ This is a **working foundation**, published so it can be read, run, and extended
 - **Intent-drift detection is lexical overlap.** It flags divergence; it does not establish intent.
 - **A hash chain does not detect truncation.** Edits, reorders, and interior deletions break verification; dropping the most recent entries leaves a valid prefix. Anchor the head hash externally — `Ledger.checkpoint()` gives you the value to publish.
 - **In-memory stores are single-process.** `InMemorySessionStore.would_exceed` followed by `commit` is not atomic, so two concurrent calls can both pass a check only one should. A shared store must make that pair atomic.
-- **Nothing persists yet.** The ledger, the reversal journal, and the drill register are all in memory, so they reset on restart. Three consequences worth knowing: the horizon forgets undo windows that are still open, `RecoverabilityRegister`'s freshness window means nothing across deploys, and a restart loses the evidence chain rather than breaking it — which is a different failure from tampering and currently indistinguishable from it. Persisting the ledger needs WAL with `synchronous=FULL` and append-only enforced by triggers (SQLite has no `GRANT`), the ledger append and journal write in one transaction, and a startup grace period so a long outage does not mass-degrade every proof to `IRREVERSIBLE` and block legitimate work.
+- **Persistence is opt-in and single-writer.** Pass `store=SqliteStore(path)` and the ledger, journal and drill history survive a restart; omit it and everything is in memory, which is fine for a test and wrong for anything real. The hash chain is single-writer by construction, so two processes sharing one file will collide on sequence numbers — the store raises rather than overwriting, but it does not coordinate. Postgres with advisory locks is the shape for multi-replica; the interface is small enough to swap.
 - **Control mappings are a self-assessment aid, not a certification** or a legal opinion. NIST AI RMF is voluntary; EU AI Act conformity is assessed against a quality-management system of which logging is one clause.
 
 Every place needing production hardening is marked `# HARDENING:` in the source. Search for it before deploying.

{revoco-0.2.2 → revoco-0.3.0}/README.md

@@ -179,6 +179,33 @@ It keeps five states apart, and the distinctions carry the value: a window that
 
 ---
 
+## Persistence
+
+```python
+from revoco import ControlPlane
+from revoco.store import SqliteStore
+
+store = SqliteStore("/var/lib/revoco/revoco.db")
+cp = ControlPlane(store=store, ...)
+
+report = store.startup_report()      # what this restart means
+print(report.notes)
+```
+
+Four decisions, each with a reason:
+
+**WAL with `synchronous=FULL`.** SQLite's defaults are ambiguous enough that you can't rely on them, and `NORMAL` doesn't survive power loss. For an evidence store that isn't a trade-off — a ledger that silently drops its last entries is precisely the truncation case a self-contained hash chain **cannot** detect, because the remaining prefix still verifies.
+
+**Append-only by triggers, not permissions.** SQLite has no `GRANT`, so the INSERT-only grant you'd use in Postgres isn't available. `BEFORE UPDATE`/`BEFORE DELETE` triggers that `RAISE(ABORT)` are the mechanism, and they bind *any* client — including a `sqlite3` shell, not just this code. Worth stating because the Postgres instinct silently produces no protection here.
+
+**The ledger append and journal write are one transaction.** The correctness question persistence actually raises. Written separately, a crash between them leaves the journal claiming a plan the ledger never recorded — and afterwards nothing can tell you which is true. Also: entries are prepared, persisted, *then* added to the in-memory chain, so the in-memory head can never outrun the durable one.
+
+**No fake freshness after downtime.** The tempting fix for "every proof is stale after an outage" is to stop counting downtime against staleness. That's wrong — an ERP upgrade during the outage is exactly when a spec silently becomes a confident wrong rollback, and a proof that survived on a technicality is a phantom rollback with a certificate. So downtime counts, `startup_report()` says how many proofs went stale and for how long, and `due()` puts them at the front of the queue. Visible and remediated fast beats invisible and assumed good.
+
+`startup_report()` also distinguishes a **broken chain from a restart** — a clean restart loses nothing, so verification failure is never an artefact of restarting and says so in as many words.
+
+---
+
 ## The containment benchmark
 
 ```bash
@@ -201,7 +228,24 @@ Every malicious technique has a **benign twin on the same tools**, so a policy t
 
 Because it runs a real `ControlPlane` against a simulated world, it doubles as a regression suite for the 91 adapter specs — and it earned that keep immediately, finding six real defects including an inverse that relied on implicit convention and a `Rule` that couldn't express "escalate irreversible work only when consequential".
 
-**Honest about the comparison:** ADR-Bench is ~5× larger, drawn from real enterprise telemetry, and far more imbalanced (6:1 benign vs 2.2:1 here). Detection coverage is their strength; verified recoverability is this one's. Complementary instruments, not competing ones.
+### Importing real benign traffic
+
+Hand-authored benign scenarios have a structural blind spot: they contain the false positives I thought to look for. So the corpus can import benign tasks from a [RAS-Eval](https://github.com/lanzer-tree/RAS-Eval) clone — traffic real models actually produced, with real arguments.
+
+```bash
+git clone https://github.com/lanzer-tree/RAS-Eval /path/to/RAS-Eval
+RAS_EVAL_PATH=/path/to/RAS-Eval revoco bench --external
+```
+
+That takes the corpus to **121 scenarios at 5.7:1 benign-to-malicious** — essentially ADR-Bench's ratio — while holding 0% false positives.
+
+**Nothing is vendored.** RAS-Eval declares no license, so its data is all-rights-reserved by default; the loader reads a clone you fetch yourself and returns nothing when it's absent, so CI never depends on it.
+
+**It found two real bugs on its first run**, at a 16.2% false-positive rate the hand-authored corpus could not have surfaced — because I wrote both the spec and the scenario, and used my own invented argument names in each. `insert_data` takes `db_path`, not the `table` I inferred from the tool's name. `convert_file_to_markdown` takes a `save_path` argument rather than returning `output_path`. In both cases the inverse could never resolve, so every legitimate call raised a phantom rollback. **The same mistake in an SAP adapter would look identical and cost considerably more.**
+
+Two disciplines keep the import honest: a tool this package hasn't classified is **skipped**, not imported as `UNKNOWN`, and a call with no observed arguments is **dropped** rather than imported — both would manufacture findings out of missing metadata rather than out of anything the control plane did. And only the 80 *unique tasks* are imported, not all 640 traces: eight models ran the same tasks, so taking every run would multiply the count with near-duplicates. Padding is the thing this corpus exists not to do.
+
+**Honest about the comparison:** ADR-Bench's 302 tasks come from real enterprise telemetry across 133 MCP servers. The imported traffic here is consumer-domain — alarms, calendars, disk stats, arXiv lookups — so it broadens the benign distribution without reaching the enterprise write surfaces where the money is. Detection coverage is their strength; verified recoverability is this one's. Complementary instruments.
 
 One gap is left visible rather than tuned away: `T09` irreversible fan-out. `PRA01` is a threshold detector, so four one-way wires land before the pattern is visible. The controlled pair `M10`/`M18` measures detection versus the budget on the identical attack, and both stay in the corpus so the difference is attributable.
 
@@ -281,7 +325,7 @@ This is a **working foundation**, published so it can be read, run, and extended
 - **Intent-drift detection is lexical overlap.** It flags divergence; it does not establish intent.
 - **A hash chain does not detect truncation.** Edits, reorders, and interior deletions break verification; dropping the most recent entries leaves a valid prefix. Anchor the head hash externally — `Ledger.checkpoint()` gives you the value to publish.
 - **In-memory stores are single-process.** `InMemorySessionStore.would_exceed` followed by `commit` is not atomic, so two concurrent calls can both pass a check only one should. A shared store must make that pair atomic.
-- **Nothing persists yet.** The ledger, the reversal journal, and the drill register are all in memory, so they reset on restart. Three consequences worth knowing: the horizon forgets undo windows that are still open, `RecoverabilityRegister`'s freshness window means nothing across deploys, and a restart loses the evidence chain rather than breaking it — which is a different failure from tampering and currently indistinguishable from it. Persisting the ledger needs WAL with `synchronous=FULL` and append-only enforced by triggers (SQLite has no `GRANT`), the ledger append and journal write in one transaction, and a startup grace period so a long outage does not mass-degrade every proof to `IRREVERSIBLE` and block legitimate work.
+- **Persistence is opt-in and single-writer.** Pass `store=SqliteStore(path)` and the ledger, journal and drill history survive a restart; omit it and everything is in memory, which is fine for a test and wrong for anything real. The hash chain is single-writer by construction, so two processes sharing one file will collide on sequence numbers — the store raises rather than overwriting, but it does not coordinate. Postgres with advisory locks is the shape for multi-replica; the interface is small enough to swap.
 - **Control mappings are a self-assessment aid, not a certification** or a legal opinion. NIST AI RMF is voluntary; EU AI Act conformity is assessed against a quality-management system of which logging is one clause.
 
 Every place needing production hardening is marked `# HARDENING:` in the source. Search for it before deploying.

{revoco-0.2.2 → revoco-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "revoco"
-version = "0.2.2"
+version = "0.3.0"
 description = "Undo for AI agent actions. Plans the rollback before the action runs, proves it still works, and rolls back a compromised grant's whole blast radius in one call."
 readme = "README.md"
 requires-python = ">=3.10"

{revoco-0.2.2 → revoco-0.3.0}/src/revoco/adapters/__init__.py

@@ -48,11 +48,12 @@ from typing import Any
 
 from ..reversal.model import InverseSpec, ReversalGate, Reversibility
 from ..reversal.registry import InverseRegistry
-from . import cloud, database, devops, identity, saas, sap, workday, workstation
+from . import cloud, database, devops, identity, ras_eval, saas, sap, workday, workstation
 from .cloud import CLOUD_GATES, CLOUD_SPECS, cloud_registry
 from .database import DATABASE_GATES, DATABASE_SPECS, database_registry
 from .devops import DEVOPS_GATES, DEVOPS_SPECS, devops_registry
 from .identity import IDENTITY_GATES, IDENTITY_SPECS, identity_registry
+from .ras_eval import RAS_EVAL_SPECS, ras_eval_registry
 from .saas import SAAS_GATES, SAAS_SPECS, saas_registry
 from .sap import SAP_GATES, SAP_SPECS, sap_registry
 from .workday import WORKDAY_GATES, WORKDAY_SPECS, workday_registry
@@ -149,6 +150,7 @@ def summary(*surfaces: str) -> dict[str, Any]:
 __all__ = [
     # modules
     "sap", "workday", "cloud", "identity", "devops", "saas", "workstation", "database",
+    "ras_eval", "RAS_EVAL_SPECS", "ras_eval_registry",
     # per-surface
     "SAP_SPECS", "SAP_GATES", "sap_registry",
     "WORKDAY_SPECS", "WORKDAY_GATES", "workday_registry",

revoco-0.3.0/src/revoco/adapters/ras_eval.py

@@ -0,0 +1,187 @@
+"""
+revoco.adapters.ras_eval
+========================
+Reversibility classifications for the 29 tools used by the RAS-Eval benchmark.
+
+Why this exists
+---------------
+The containment corpus' benign half is hand-authored, and hand-authored benign
+traffic has a specific blind spot: it contains the false positives I thought to
+look for. RAS-Eval (`arXiv 2506.15253 <https://arxiv.org/abs/2506.15253>`_) has 80
+benign tasks that real models actually executed, with real arguments. That is the
+one thing authorship cannot produce.
+
+Its tools are not this package's tools, so the traces are unusable until something
+says what each of them does to the world. That is what this module is — **my
+classifications of their tool surface**, written from the tool names and the
+observed call arguments.
+
+No data from RAS-Eval is vendored here. The repository declares no license, so its
+tasks and logs are all-rights-reserved by default and cannot be redistributed in an
+Apache-2.0 package. :mod:`revoco.bench.external` reads them from a clone you obtain
+yourself, and returns nothing when it is absent.
+
+Status: **unvalidated, and unvalidatable from here.** These are someone else's
+tools; I cannot execute them, so no drill can confirm any inverse below. They are
+classified for the purpose of measuring *whether ordinary tool use gets blocked* —
+which only needs the reversal posture to be approximately right — and should not be
+used to govern anything.
+
+What this surface is good for, and what it is not
+-------------------------------------------------
+Good for: volume and variety of ordinary agent tool use. Thirty distinct tools
+across seven agent personas, argument shapes produced by eight different models.
+That is a false-positive dimension the hand-built corpus does not cover at all.
+
+Not good for: enterprise write semantics. There is no ERP posting, no payroll, no
+IAM here — the domains are alarms, calendars, disk stats, weather and arXiv
+lookups. It broadens the benign distribution; it does not deepen it toward the
+surfaces where the money is.
+"""
+
+from __future__ import annotations
+
+from ..reversal.model import InverseSpec, Reversibility
+from ..reversal.registry import InverseRegistry
+
+# Reads. Trivially reversible because they change nothing — which is also why a
+# corpus made only of these would prove very little. Most of RAS-Eval is here.
+_READS = (
+    "is_arxiv_identifier",
+    "search_identifier",
+    "get_disk_info",
+    "get_memory_info",
+    "get_current_time",
+    "get_weather",
+    "get_events_from_calendar",
+    "search_event",
+    "tavily_search",
+    "location_encode",
+    "get_stock_price",
+    "get_cpu_info",
+    "list_directory",
+    "read_file",
+    "get_alarms",
+    "web_search",
+)
+
+# Paired create/cancel operations. The genuinely interesting part of this surface:
+# each is an exact inverse of the other, which is the shape the reversal model
+# handles best and the shape most real tool surfaces lack.
+_PAIRS: tuple[tuple[str, str, str], ...] = (
+    ("add_event_to_calendar", "remove_event_from_calendar", "event_id"),
+    ("remove_event_from_calendar", "add_event_to_calendar", "event_id"),
+    ("set_alarm", "cancel_alarm", "alarm_id"),
+    ("cancel_alarm", "set_alarm", "alarm_id"),
+    ("timer", "cancel_timer", "timer_id"),
+)
+
+
+def _read_specs() -> list[InverseSpec]:
+    return [
+        InverseSpec(
+            tool=t,
+            kind=Reversibility.REVERSIBLE,
+            inverse_tool="ras.noop",
+            notes="Read-only: nothing to undo.",
+        )
+        for t in _READS
+    ]
+
+
+def _pair_specs() -> list[InverseSpec]:
+    return [
+        InverseSpec(
+            tool=fwd,
+            kind=Reversibility.REVERSIBLE,
+            inverse_tool=inv,
+            arg_map=((key, f"result.{key}"),),
+            notes=(
+                f"Exact inverse pair with {inv}. The identifier comes from the "
+                "response, so the undo path only completes after the forward call "
+                "returns — which is precisely the deferred-argument case the plan "
+                "model exists to handle."
+            ),
+        )
+        for fwd, inv, key in _PAIRS
+    ]
+
+
+RAS_EVAL_SPECS: list[InverseSpec] = [
+    *_read_specs(),
+    *_pair_specs(),
+    # Writes with an approximate inverse.
+    InverseSpec(
+        tool="insert_data",
+        kind=Reversibility.COMPENSABLE,
+        inverse_tool="delete_data",
+        # `db_path` and `items`, not `table`. The first version of this spec invented
+        # `table` from the tool's name, and the imported traces caught it: an argument
+        # that is never passed cannot resolve, so every real insert raised a phantom
+        # rollback. The same mistake in an enterprise adapter would look identical.
+        arg_map=(("db_path", "args.db_path"), ("row_id", "result.row_id")),
+        residue=(
+            "Deleting the row removes the data, not its trace: auto-increment "
+            "sequences have advanced, and anything that read the row while it existed "
+            "acted on it."
+        ),
+    ),
+    InverseSpec(
+        tool="download_file",
+        kind=Reversibility.COMPENSABLE,
+        inverse_tool="delete_file",
+        arg_map=(("path", "args.path"),),
+        residue=(
+            "The file is removed but the transfer happened: bandwidth was consumed and "
+            "the remote server logged the request."
+        ),
+    ),
+    InverseSpec(
+        tool="convert_file_to_markdown",
+        kind=Reversibility.COMPENSABLE,
+        inverse_tool="delete_file",
+        # The destination is `save_path` in the request, not `output_path` in the
+        # response — also found by importing real calls rather than by reading.
+        arg_map=(("path", "args.save_path"),),
+        residue="The converted output is removed; the source is untouched.",
+    ),
+    InverseSpec(
+        tool="write_file",
+        kind=Reversibility.COMPENSABLE,
+        inverse_tool="write_file",
+        arg_map=(("path", "args.path"), ("content", "snapshot.content")),
+        snapshot_fields=("content",),
+        residue=(
+            "Prior content is restored where it was captured. If the file did not "
+            "exist before, this leaves an empty file rather than no file."
+        ),
+    ),
+    # Genuinely one-way, and worth registering so they escalate rather than
+    # falling into UNKNOWN by accident.
+    InverseSpec(
+        tool="send_email",
+        kind=Reversibility.IRREVERSIBLE,
+        notes="Delivered mail cannot be recalled.",
+    ),
+    InverseSpec(
+        tool="execute_shell_command",
+        kind=Reversibility.UNKNOWN,
+        notes=(
+            "Same reasoning as revoco's own shell.exec: an arbitrary command's effects "
+            "cannot be known in advance, so the honest classification is UNKNOWN and "
+            "the honest outcome is escalation."
+        ),
+    ),
+]
+
+
+def ras_eval_registry() -> InverseRegistry:
+    """Classifications for the RAS-Eval tool surface (unvalidated — see module docs)."""
+    return InverseRegistry(list(RAS_EVAL_SPECS))
+
+
+def classified_tools() -> set[str]:
+    return {s.tool for s in RAS_EVAL_SPECS}
+
+
+__all__ = ["RAS_EVAL_SPECS", "ras_eval_registry", "classified_tools"]

{revoco-0.2.2 → revoco-0.3.0}/src/revoco/bench/__init__.py

@@ -32,6 +32,9 @@ Usage::
 """
 
 from .corpus import TECHNIQUES, all_scenarios, benign, by_technique, malicious
+from .external import available as external_available
+from .external import provenance as external_provenance
+from .external import ras_eval_scenarios
 from .harness import DEFAULT_POLICY, Harness, default_policy
 from .report import Metrics, render, score, to_dict
 from .scenario import (
@@ -56,6 +59,10 @@ __all__ = [
     "malicious",
     "benign",
     "by_technique",
+    # external corpora (opt-in, nothing vendored)
+    "ras_eval_scenarios",
+    "external_available",
+    "external_provenance",
     # model
     "Scenario",
     "Step",