evalshift 1.2.2__tar.gz → 1.3.0__tar.gz

{evalshift-1.2.2 → evalshift-1.3.0}/CHANGELOG.md

@@ -7,6 +7,108 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.3.0] - 2026-10-09
+
+### Added
+
+- `captures.store` in `evalshift.yaml`: name the object store the capture SDK
+  ships captures to — `s3://<bucket>/<prefix>`, `gs://<bucket>/<prefix>` or
+  `az://<account>/<container>/<prefix>` — and `capture sync` and `capture list`
+  mirror new captures and toolset sidecars from it into `.evalshift/` before
+  they run, with no flags. Only objects missing locally are downloaded
+  (captures are immutable, toolsets content-addressed), captures already
+  promoted are skipped so `capture clean` never causes a re-download, and
+  every file is written atomically. New `capture fetch` does only that step;
+  `--since 30m|24h|7d|<ISO>` limits a fetch to recently written captures and
+  `--offline` skips it. Keys from a bucket are validated against the SDK's
+  layout and anything else is ignored as malformed — including a suite
+  directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with
+  `.` or a space, and a key with a trailing newline — and no destination can
+  land outside `.evalshift/`; folder-marker keys ending in `/` (the S3
+  console's "Create folder", ADLS directories) are skipped without being
+  counted. A list or download failure exits 1 with a hint naming the
+  provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC
+  role, `gcloud auth application-default login`, `az login`); a local write
+  failure names the file and asks you to check disk space and permissions on
+  its directory instead. Extras:
+  `evalshift[s3]`, `[gcs]`, `[azure]` (azure-storage-blob and azure-identity);
+  a missing one exits 1 naming the `pip install` to run. Credentials come from
+  each provider's default chain and are rejected in the URI: a value with `@`
+  or `?` fails to load, naming the accepted forms. No rejected value is echoed
+  back (it may be a pasted key or connection string); at most its scheme is
+  named.
+  `capture list --json` keeps stdout pure JSON (the fetch line and warnings
+  go to stderr). `doctor` reports whether the store is installed (`fail`, exit 1,
+  when the extra is missing) and reachable (`warn` when it cannot be listed).
+  Without `captures.store` nothing changes; a `sync` or `list` whose
+  `evalshift.yaml` cannot be read or does not validate skips the fetch,
+  warning only when the file has a top-level `captures:` key. Older CLIs
+  reject the `captures:` key, so bump the CLI (and a pinned CI action's
+  `evalshift-version`) before adding it.
+- `evaluators.trace_invariants`: hand-written rules over tool-call traces —
+  `forbidden`, `required`, `order`, `call_count` (`min_calls` and/or
+  `max_calls` per tool; equal bounds = an exact count) and
+  `arguments` (JSON Schema) — checked on both sides of every pair, on replayed
+  traces or (`traces: imported`) on imported ones. A rule the target breaks
+  counts whatever the source did. Its `applies_to` is enforced: a prompt
+  outside the globs is never checked, and `evaluate` warns, naming the entry
+  and its `applies_to`, when the globs matched nothing in the run (so a typo
+  is not a silent clean run). A `traces: imported` entry on a run without
+  `traces.jsonl` fails `evaluate` with an error naming that family.
+- `migration_policy.max_invariant_violations` (default `0`, per-slice
+  overridable): distinct examples on which the target broke a blocking trace
+  rule, so an example that broke rules in two entries counts once. A breach
+  fails the run even when the suite is too small for statistics, and a
+  slice whose own budget breaches reads `fail` however few comparisons it has.
+  `analyze` adds a recommendation when the source broke the same rules the
+  target did: review the rule with its owner rather than raising the budget.
+  Pushed bundles' `decision.policy` now always carries the field; hosted
+  EvalShift accepts and stores it since 2026-10-08 (it never re-evaluates it —
+  a run that carries its own policy is answered with the CLI's decision).
+- The HTML report lists every broken trace rule in a "Trace rules broken"
+  panel, with its owner and whether the source broke it too — advisory
+  (`blocking: false`) entries included, though only blocking ones count
+  toward the budget; those rows are tagged "advisory". `report.json` carries
+  the same rows under a new top-level `invariant_violations` key, each with
+  the entry's `blocking` flag. The "Overall, by evaluator" table tags every
+  advisory (`blocking: false`) evaluator's row the same way and says what a
+  trace-rule row measures, and the "Top
+  regression causes" panel counts `INVARIANT_VIOLATION` as the budget does —
+  distinct examples with a blocking target violation — so the narrative never
+  quotes a figure the verdict did not use.
+- `evalshift init` scaffolds `max_invariant_violations: 0` in every
+  `--profile` policy block, beside the other budgets. Older CLIs reject the
+  field, so bump the CI action's `evalshift-version` pin to this release
+  before committing a freshly scaffolded config.
+- `capture sync` keeps a managed entry's hand-written
+  `evaluators.trace_invariants` block when it regenerates the entry. The
+  marker comment still says hand edits are overwritten; this block is the one
+  exception.
+- `push` now uploads each score row's one-line `explanation` and, for
+  `trace_invariants` rows, the rules each side broke (`violations.source` /
+  `violations.target`: rule id, type, tool, round, detail, and sample on
+  repeated-sampling runs). Hosted EvalShift shows them on the example detail
+  and on share links instead of only the violation count. `rules_checked`,
+  `owner` and `failure_categories` are not repeated on the score row (they
+  already ship in `evaluator_config` / `decision`). Requires a hosted service
+  that accepts the two fields (hosted EvalShift was updated before this
+  release); nothing else in the bundle changed.
+
+### Changed
+
+- The report's per-prompt Executive summary — and the header's mean score Δ,
+  which averages its rows — now picks each prompt's worst severity with ties
+  broken toward the most negative effect size (the order the hero panel and
+  the narrative already used) instead of whichever evaluator `analyze` listed
+  first, and skips advisory (`blocking: false`) evaluators while the prompt
+  has a gating one. An advisory rule the target satisfied by calling no tools
+  at all could otherwise read as the prompt's "+9% improved" above a failed
+  verdict.
+- Upgrading moves the resume `config_hash`: the new config fields are part of
+  every config's canonical dump, so a run left in progress by an older CLI
+  cannot be `--resume`d after upgrading (it fails with the usual config-hash
+  mismatch). Start a fresh run; cached responses are reused.
+
 ## [1.2.2] - 2026-10-05
 
 ### Fixed

{evalshift-1.2.2 → evalshift-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalshift
-Version: 1.2.2
+Version: 1.3.0
 Summary: Run your prompts on two LLMs and find out, with statistical confidence, what regressed.
 Project-URL: Homepage, https://github.com/evalshift/evalshift-cli
 Project-URL: Documentation, https://github.com/evalshift/evalshift-cli#readme
@@ -41,7 +41,14 @@ Requires-Dist: scipy>=1.12
 Requires-Dist: sqlalchemy>=2.0
 Requires-Dist: tiktoken>=0.7
 Requires-Dist: typer>=0.12
+Provides-Extra: azure
+Requires-Dist: azure-identity>=1.15; extra == 'azure'
+Requires-Dist: azure-storage-blob>=12.19; extra == 'azure'
 Provides-Extra: dev
+Requires-Dist: azure-identity>=1.15; extra == 'dev'
+Requires-Dist: azure-storage-blob>=12.19; extra == 'dev'
+Requires-Dist: boto3>=1.34; extra == 'dev'
+Requires-Dist: google-cloud-storage>=2.14; extra == 'dev'
 Requires-Dist: mypy>=1.10; extra == 'dev'
 Requires-Dist: pre-commit>=3.7; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
@@ -51,6 +58,10 @@ Requires-Dist: respx>=0.21; extra == 'dev'
 Requires-Dist: ruff==0.16.4; extra == 'dev'
 Requires-Dist: types-jsonschema>=4.22; extra == 'dev'
 Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
+Provides-Extra: gcs
+Requires-Dist: google-cloud-storage>=2.14; extra == 'gcs'
+Provides-Extra: s3
+Requires-Dist: boto3>=1.34; extra == 's3'
 Description-Content-Type: text/markdown
 
 # EvalShift
@@ -318,7 +329,11 @@ shape, `fail-on` modes, and baseline behavior.
 
 Migrating an agent (a prompt that uses tools)? EvalShift detects
 regressions in *which* tools the new model calls, *what* arguments it
-passes, and in what order and parallelism within a response. By default
+passes, and in what order and parallelism within a response. Hand-written
+`trace_invariants` rules (auth before a charge, a tool that must never be
+called, at most one charge, argument bounds) hold *both* models to a
+contract whatever the source did, gated by
+`migration_policy.max_invariant_violations`. By default
 each example is one model call scored against the first recorded round;
 promote with `--rounds all` to replay every recorded round teacher-forced,
 with the recorded tool results fed back and each round scored on its own.
@@ -344,7 +359,10 @@ assets, works offline) has:
 
 * **Migration verdict** — the policy decision up top: which budgets failed, the
   top regression causes, and the recommendation.
-* **Executive summary** — one row per prompt with a severity badge.
+* **Executive summary** — one row per prompt with a severity badge, drawn from
+  blocking evaluators (advisory ones only when nothing else scored the prompt).
+* **Trace rules broken** — every hand-written trace rule the target broke,
+  with its owner and whether the source broke it too; advisory rows are tagged.
 * **What changed, in plain language** — the verdict, the economics and the
   behavioural drift explained by `defaults.insights_model`. Every figure in it
   is copied from the computed statistics, never generated. Needs a provider
@@ -358,7 +376,8 @@ assets, works offline) has:
 
 Your prompts and suite stay local for `doctor`, `run`, `evaluate`, `analyze`,
 and `report`. The only outbound calls in local mode are to the LLM providers you
-configure — any provider LiteLLM supports, called with your own API keys.
+configure — any provider LiteLLM supports, called with your own API keys — and,
+if you set `captures.store`, reads from your own capture bucket.
 Anthropic, DeepSeek, Google and OpenAI ids additionally get a curated pricing and
 capability entry; everything else is passed through with the provider inferred
 from the id.

{evalshift-1.2.2 → evalshift-1.3.0}/README.md

@@ -263,7 +263,11 @@ shape, `fail-on` modes, and baseline behavior.
 
 Migrating an agent (a prompt that uses tools)? EvalShift detects
 regressions in *which* tools the new model calls, *what* arguments it
-passes, and in what order and parallelism within a response. By default
+passes, and in what order and parallelism within a response. Hand-written
+`trace_invariants` rules (auth before a charge, a tool that must never be
+called, at most one charge, argument bounds) hold *both* models to a
+contract whatever the source did, gated by
+`migration_policy.max_invariant_violations`. By default
 each example is one model call scored against the first recorded round;
 promote with `--rounds all` to replay every recorded round teacher-forced,
 with the recorded tool results fed back and each round scored on its own.
@@ -289,7 +293,10 @@ assets, works offline) has:
 
 * **Migration verdict** — the policy decision up top: which budgets failed, the
   top regression causes, and the recommendation.
-* **Executive summary** — one row per prompt with a severity badge.
+* **Executive summary** — one row per prompt with a severity badge, drawn from
+  blocking evaluators (advisory ones only when nothing else scored the prompt).
+* **Trace rules broken** — every hand-written trace rule the target broke,
+  with its owner and whether the source broke it too; advisory rows are tagged.
 * **What changed, in plain language** — the verdict, the economics and the
   behavioural drift explained by `defaults.insights_model`. Every figure in it
   is copied from the computed statistics, never generated. Needs a provider
@@ -303,7 +310,8 @@ assets, works offline) has:
 
 Your prompts and suite stay local for `doctor`, `run`, `evaluate`, `analyze`,
 and `report`. The only outbound calls in local mode are to the LLM providers you
-configure — any provider LiteLLM supports, called with your own API keys.
+configure — any provider LiteLLM supports, called with your own API keys — and,
+if you set `captures.store`, reads from your own capture bucket.
 Anthropic, DeepSeek, Google and OpenAI ids additionally get a curated pricing and
 capability entry; everything else is passed through with the provider inferred
 from the id.

{evalshift-1.2.2 → evalshift-1.3.0}/docs/agents.md

@@ -22,8 +22,9 @@ The killer scenario it catches:
 
 * **`ToolTrace` data model**: provider-agnostic, populated from
   Anthropic / OpenAI / Gemini responses.
-* **Three new evaluators**: `tool_selection`, `tool_arguments`,
-  `tool_trace_structure`.
+* **Four new evaluators**: `tool_selection`, `tool_arguments`,
+  `tool_trace_structure`, and hand-written `trace_invariants` (both sides
+  judged against your rules, not against each other).
 * **Suite extension**: optional `expected_tools`, `expected_tool_count`,
   `expected_no_tools`, `expected_parallel` per example, plus a required
   `toolset_ref` or inline `tools` — see [Suite ground truth](#suite-ground-truth).
@@ -85,7 +86,9 @@ defaults:
 
 evaluators:
   # Top-level: what every suite is scored with. Tool evaluators do not belong
-  # here — `capture sync` writes them per suite (see below).
+  # here — `capture sync` writes them per suite (see below). Hand-written
+  # `trace_invariants` are the exception: top level is fine, and sync keeps
+  # them on the managed entries it rewrites.
   semantic:
     embedding_model: gemini/gemini-embedding-001
     blocking: false
@@ -123,7 +126,11 @@ inherited. See
 Hand edits inside the markers are regenerated away on the next sync. To keep
 them — a `severity_floor: high` on `routing`, a per-field strategy — set
 `managed: false` on that suite's entry; sync then prints what it would have
-written instead of writing it.
+written instead of writing it. The one exception is a hand-written
+`evaluators.trace_invariants` block: no capture can derive a team's rules, so
+sync carries it into the regenerated entry verbatim, even though the marker
+comment says hand edits are overwritten. See
+[Configuration → `managed`](configuration.md#managed).
 
 > **Note:** `structural.length` is intentionally **not** in the
 > scaffolded config. Agent runs frequently produce empty `final_text`
@@ -437,6 +444,7 @@ together and `run` replays each one with its recorded history prefix. See
 | `tool_selection`       | You care about *which* tools fire (most common).        |
 | `tool_arguments`       | You care about *what* the model passes to each tool.    |
 | `tool_trace_structure` | You care about call counts, parallelism, or refusals.   |
+| `trace_invariants`     | A rule must hold whatever the source did (auth before a charge, at most one charge, argument bounds). |
 
 You can run multiple at once. Each becomes an independent comparison
 in `analysis.json`, with the existing Benjamini-Hochberg correction
@@ -460,6 +468,18 @@ It scores the same recall, counting duplicates (two expected `archive_project`
 calls need two actual ones) and ignoring extra calls, without penalising a
 permutation. Call counts are `tool_trace_structure`'s job, not this one's.
 
+When an order genuinely matters — authenticate before charging — don't rely on
+the recording's order to say so. Write it as an `order` rule under
+[`trace_invariants`](configuration.md#evaluatorstrace_invariants), which checks
+both models against the rule rather than against each other, and keep
+`expected_set` for the fan-out.
+
+`trace_invariants` is also the rule-based counterpart to `agent_trace`'s
+`dangerous_tools`: that one flags a target that made *more* dangerous calls
+than the source, so a dangerous call both made passes; a `forbidden` or
+`call_count` rule fails the target whatever the source did. It reads replayed
+traces by default and imported ones with `traces: imported`.
+
 ### Scoring free-text arguments
 
 Free-text tool arguments — search queries, titles, descriptions — will never
@@ -583,6 +603,13 @@ identically is not a ✗ — it is the same pair before and after, and what
 is wrong is the ground truth. Look for **Ground truth missed by both**
 in the evaluator table when that happens.
 
+A **Trace rules broken** panel appears when the target broke a hand-written
+[`trace_invariants`](configuration.md#evaluatorstrace_invariants) rule: every
+broken rule with its owner and whether the source broke it too, whatever the
+delta, advisory (`blocking: false`) rows tagged **advisory**. `report.json`
+carries the same rows under `invariant_violations`, and the evaluator table
+tags an advisory entry's row the same way.
+
 ## Troubleshooting
 
 * **A suite reports tool evaluator scores you didn't expect on some rows** —

{evalshift-1.2.2 → evalshift-1.3.0}/docs/configuration.md

@@ -21,6 +21,7 @@ evaluators: {...}         # optional (but at least one is needed for `evaluate`)
 suites: {...}             # optional named suites for `run --suite-name`, each with
                           # its own optional `evaluators:` block
 retention: {...}          # optional run-history pruning policy
+captures: {...}           # optional, see below
 ```
 
 Unknown keys are rejected (`extra: forbid` everywhere) so typos fail
@@ -126,6 +127,7 @@ migration_policy:
   min_equivalence_rate: 0.75
   max_tool_argument_drift: 0.20
   max_tool_divergence: 0.20  # share of pairs where the target routed elsewhere
+  max_invariant_violations: 0  # examples where the target broke a blocking trace rule
   tool_argument_drift_floor: 0.9   # below this a call counts as drifted
   max_cost_increase: 0.30    # target may cost up to 30% more
   max_latency_increase: 0.30 # …and be up to 30% slower
@@ -196,6 +198,25 @@ means the target called a tool the source did not, or skipped one it did.
 On a teacher-forced multi-round replay the row's score is the mean over the
 replayed rounds, so an example counts as diverged if **any** round diverged.
 
+`max_invariant_violations` is a **count**, not a rate: the number of examples
+on which the target broke at least one rule of a blocking
+[`trace_invariants`](#evaluatorstrace_invariants) evaluator, judged against the
+rules alone — a rule the source broke too still counts against the target. The
+default `0` fails on the first broken rule. Examples are counted distinctly by
+`(prompt_id, example_id)`: every `trace_invariants` entry writes its own row, and
+an example that broke rules in two entries counts once, over a denominator of
+the distinct examples with a blocking row. A row averaged over
+`samples_per_example` repeats counts when any one sample broke a rule. Because
+a rule is an assertion the team wrote down rather than a statistic, a breach is
+always conclusive: no interval softens it, it carries no `1/n` granularity
+warning, and it turns `inconclusive` and `conditional_pass` into `fail` even
+when the suite is too small for the paired tests. The same holds per slice: a
+slice whose own `max_invariant_violations` breaches reads `fail` however few
+comparisons it has. The budget's row appears only in scopes that scored at least
+one blocking `trace_invariants` row, so a project with no trace rules (or only
+advisory ones) sees no new budget row. It is overridable per slice like the
+others.
+
 A `tool_selection.conformance` row where **both** models missed the recorded
 ground truth by the same margin is excluded from every policy rate: its zero
 delta is a shared failure, not evidence the migration is safe. Such rows are
@@ -263,6 +284,16 @@ Two behaviours to know:
   `conclusive: false` on a scope that scored zero records — their
   `0/0` default looks clean but measures nothing.
 
+* **`max_invariant_violations` is conclusive by construction.** It counts
+  distinct examples, so it has no interval and no "too small to be sure": a breach
+  fails the run (or the slice) outright, overriding `inconclusive` and
+  `conditional_pass`, and the verdict's `reason` names the scope and the
+  count. Only blocking `trace_invariants` evaluators count toward it. When the
+  source broke the same rule on the same example, `recommendations` adds a
+  line saying so — not an exemption, but a hint that either both models break
+  the rule or the rule no longer matches the toolset, which its owner should
+  review.
+
 * **`fail_on_dropped_params` gates constraints, not scores.** Default
   `false`. A promoted capture can pin generation parameters
   (`response_format`, `tool_choice`, `parallel_tool_calls`, `top_p`, a
@@ -312,6 +343,8 @@ the snapshot each run pushed. `policy` is `null` when no `migration_policy` is
 configured, and on a `migration_decision.json` written before this field
 existed. See [What `push` sends](hosted.md#what-push-sends-block-by-block) for
 the upload contract and the notices `push` prints around a policy-less run.
+Because every budget is resolved, the snapshot carries `max_invariant_violations`
+(at its default `0`) even for a config that never mentions trace rules.
 
 ## `prompts`
 
@@ -404,8 +437,9 @@ bad generations fall back to deterministic templated prose.
 
 ## `evaluators`
 
-Seven sub-keys, all optional — `structural`, `semantic`, `tool_selection`,
-`tool_arguments`, `tool_trace_structure`, `agent_trace` and `llm_judge`, each
+Eight sub-keys, all optional — `structural`, `semantic`, `tool_selection`,
+`tool_arguments`, `tool_trace_structure`, `agent_trace`, `trace_invariants` and
+`llm_judge`, each
 documented below. **At least one evaluator must be configured for
 `evalshift evaluate` to do anything.**
 
@@ -471,7 +505,7 @@ A list. Each entry has:
 | `name`            | string | (required)    | Identifier surfaced in reports.  |
 | `conformance`     | enum   | `expected`    | Ground-truth axis: `expected` / `expected_set` / `off`. `expected_set` is `expected` made order-insensitive: multiset recall of `example.expected_tools` names. Use it when the expected calls are a parallel fan-out whose order carries no meaning. |
 | `divergence`      | enum   | `set`         | Target-vs-source axis: `set` (Jaccard on tool names) / `exact` (sequence equality) / `first` (first call only) / `off`. |
-| `applies_to`      | list   | `["*"]`       | Glob list of prompt ids. |
+| `applies_to`      | list   | `["*"]`       | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). |
 | `severity_floor`  | enum   | `null`        | If set, surfaces in metadata so the analysis layer can floor severity. |
 
 The two axes are independent and each writes **its own record**, under
@@ -502,7 +536,7 @@ deprecated, and a config still carrying one fails validation.
 | Field                   | Type   | Default | Description |
 | ----------------------- | ------ | ------- | ----------- |
 | `name`                  | string | (required) | Identifier. |
-| `applies_to`            | list   | `["*"]` | Glob list. |
+| `applies_to`            | list   | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). |
 | `against`               | enum   | `source` | What arguments are compared to: `source` (drift from the source model) or `expected` (correctness against `expected_tools[].arguments`, scored on both sides). |
 | `strategies`            | dict   | `{}`    | Per-field strategy overrides (`exact`/`subset`/`numeric`/`semantic`/`auto`). |
 | `default_strategy`      | enum   | `auto`  | Strategy for fields `strategies` does not name. `auto` is the ladder below; `exact` restores byte-equality scoring. |
@@ -578,7 +612,7 @@ unaffected: it counts calls whose *target* score fell below
 | Field                 | Type   | Default | Description |
 | --------------------- | ------ | ------- | ----------- |
 | `name`                | string | (required) | Identifier. |
-| `applies_to`          | list   | `["*"]` | Glob list. |
+| `applies_to`          | list   | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). |
 | `check_call_count`    | bool   | `true`  | Score the number of tool calls. |
 | `check_parallelism`   | bool   | `true`  | Score parallel-vs-sequential alignment. |
 | `check_refusals`      | bool   | `true`  | Score refusal alignment; mismatches force `severity_floor: high`. |
@@ -593,7 +627,7 @@ the normal `raw.jsonl` model-call artifact.
 | Field                        | Type   | Default | Description |
 | ---------------------------- | ------ | ------- | ----------- |
 | `name`                       | string | required | Identifier surfaced in scores and reports. |
-| `applies_to`                 | list   | `["*"]` | Glob list of prompt ids. |
+| `applies_to`                 | list   | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). |
 | `check_tool_order`           | bool   | `true`  | Compare source and target tool-call order. |
 | `check_arguments`            | bool   | `true`  | Compare arguments for same-name matched tool calls. |
 | `check_missing_verification` | bool   | `true`  | Check dangerous tools have an earlier verification tool. |
@@ -610,6 +644,136 @@ evaluators:
       dangerous_tools: ["issue_refund"]
 ```
 
+### `evaluators.trace_invariants`
+
+A list. Each entry is a set of hand-written rules that every in-scope tool-call
+trace must satisfy, checked on **both** sides of every pair and judged against
+the rules, not against the source: a rule the source also broke still counts
+against the target. See [Evaluators → Trace invariants](evaluators.md#trace-invariants)
+for when to reach for it.
+
+Write it in the top-level `evaluators:` block (outside the `capture sync`
+managed region) or in a suite's own `evaluators:` override. `capture sync`
+keeps a managed suite entry's `trace_invariants` when it regenerates that
+entry — see [`managed`](#managed).
+
+```yaml
+evaluators:
+  trace_invariants:
+    - name: payments_contract
+      owner: "@payments-team"     # optional; echoed onto every record and the report
+      applies_to: ["checkout-*"]  # prompt-id globs; default ["*"]
+      blocking: true              # default; false = reported, never gates
+      traces: replayed            # default; `imported` checks `evalshift traces import` traces
+      rules:
+        - id: auth-before-charge
+          type: order
+          before: authenticate
+          after: charge_card
+        - id: no-legacy-refund
+          type: forbidden
+          tools: [refund_v1]
+        - id: one-charge
+          type: call_count
+          tool: charge_card
+          max_calls: 1              # min_calls, max_calls or both; both equal = exact
+        - id: must-book
+          type: required
+          tools: [create_booking]
+        - id: sane-amount
+          type: arguments
+          tool: charge_card
+          json_schema:
+            type: object
+            required: [amount]
+            properties:
+              amount: {type: number, minimum: 0.01, maximum: 5000}
+
+migration_policy:
+  max_invariant_violations: 0     # default; examples where the target broke any blocking rule
+```
+
+| Field        | Type   | Default      | Description |
+| ------------ | ------ | ------------ | ----------- |
+| `name`       | string | (required)   | Identifier surfaced in scores and reports. |
+| `applies_to` | list   | `["*"]`      | Glob list of prompt ids. **Enforced here**, unlike on the other families: a prompt outside these globs is never checked, gets no row and is never counted. If the globs match no prompt in a run (with `traces: imported`, no imported trace pair), `evaluate` prints a warning naming the entry and its `applies_to`, because none of its rules were checked (`compare` scores quietly and does not print it); matching at least one prompt stays silent. |
+| `blocking`   | bool   | `true`       | `true` counts target violations toward [`max_invariant_violations`](#migration_policy); `false` still checks and reports them, but never gates. |
+| `traces`     | string | `replayed`   | `replayed` checks the tool calls `evalshift run` replayed (a pair whose replay recorded no tool trace gets no row). `imported` checks the traces brought in with [`evalshift traces import`](traces.md); `evaluate` fails on a run with no `traces.jsonl`, naming `trace_invariants (traces: imported)` in the error. |
+| `owner`      | string | `null`       | Free text (a team or a handle). Echoed onto every record and into the report, so a violation names who to ask. Nothing enforces it. |
+| `rules`      | list   | (required)   | At least one rule. Every rule has an `id` (unique within the entry; it names violations in reports) and a `type`. |
+
+The unit of checking is a **response**: one model turn, plus the tool calls
+already in its **context** before it answered. On a replayed trace there is one
+response per replayed round, and round *k*'s context is the tool calls on the
+example's `history` assistant turns plus the *recorded* calls of the rounds the
+teacher-forced replay fed back before it; the model's own earlier rounds are
+never context, because the replay never feeds them back. An imported trace is
+one response with an empty context — every call in it was the agent's own.
+Context calls can never *be* violations, since the model under test did not
+make them, but they can satisfy or count toward a rule:
+
+| `type`       | Fields | Violated when | Context role |
+| ------------ | ------ | ------------- | ------------ |
+| `forbidden`  | `tools` (list) | the model calls any tool in `tools` (one violation per call) | none |
+| `required`   | `tools` (list) | some tool in `tools` is called in no response by the model | none: a call only in the context does not satisfy it. Use `call_count` `min_calls` when the conversation may already contain the call |
+| `order`      | `before`, `after` (two different tools) | the model calls `after` and no `before` precedes it, either in the context or earlier in the same response | a context `before` satisfies it |
+| `call_count` | `tool`, `min_calls` and/or `max_calls` (at least one; `max_calls` ≥ `min_calls`) | `max_calls`: a model call to `tool` takes the running count past it, counting context calls first (one violation per excess call). `min_calls`: at the end of the last response, context calls plus the model's calls to `tool` number fewer than it (one violation). Both set and equal is exact cardinality | context calls count, toward both bounds |
+| `arguments`  | `tool`, `json_schema` | a model call to `tool` has arguments that fail `json_schema` (Draft 2020-12, itself validated when the config loads) | none |
+
+"Earlier in the same response" means emitted earlier; imported traces are
+ordered by `sequence_index`, so the two modes agree.
+
+**Scoring.** Each side scores the share of rules it broke zero times —
+`(rules − rules broken) / rules` — so a rule broken three times costs the same
+as one broken once, and `delta = target − source`. The record's kind is
+`trace_invariants`; its metadata carries `rules_checked`, `source_violations`
+and `target_violations` (each a list of `{rule_id, rule_type, tool,
+round_index, detail}`), `owner` when set, and the failure category
+`INVARIANT_VIOLATION` when the target broke anything. The report's **Top
+regression causes** panel (and `failure_categories` in
+`migration_decision.json`) counts that category the way the budget does —
+distinct examples on which the target broke a *blocking* rule — so an advisory
+violation, or a second entry broken on the same example, never inflates it
+past the figure the verdict used. With `samples_per_example` above 1 the
+example's row keeps every scored sample's violations, each tagged with its
+`sample` ordinal. The HTML report's **Trace rules broken** panel and
+`report.json`'s `invariant_violations` list every rule the target broke,
+whatever the delta — including those of advisory (`blocking: false`) entries,
+which never count toward the budget — with the owner and whether the source
+broke it too. Each row carries the entry's `blocking` flag, and the panel tags
+`blocking: false` rows **advisory**. The **Overall, by evaluator** table tags
+every advisory (`blocking: false`) evaluator's row the same way, and every
+`trace_invariants` row there also says what it measures: each side judged
+against the entry's rules, not against the other side. The
+per-prompt **Executive summary** — and the header's mean score Δ, which
+averages its rows — is drawn from blocking evaluators only (worst severity
+first, ties to the most negative effect size), falling back to advisory ones
+on a prompt that has nothing else, so an advisory rule the target satisfied by
+calling no tools cannot read as the prompt's improvement above a failed
+verdict.
+
+Four things to know:
+
+- **`required` and `min_calls` are not vacuous.** `order`, `arguments` and a
+  `call_count` with only `max_calls` say nothing about a trace that never calls
+  the tool. `required` and `call_count` with `min_calls` fail on every in-scope
+  trace that lacks the calls. A single-shot replay sees only the first
+  response, so scope them with `applies_to`, or use them with
+  `capture sync --rounds all` or with imported traces.
+- **`required` counts the model's calls; `call_count` counts the
+  conversation's.** `required` asks whether the model under test made the
+  call. `call_count` asks whether the conversation contains it, so a recorded
+  history that already called `charge_card` satisfies `min_calls: 1` and
+  leaves `max_calls: 1` no room.
+- **Teacher-forced rounds see recorded context.** Round *k* is judged with the
+  recorded rounds before it as context, not the model's own earlier rounds.
+  That applies to `min_calls` too: it is judged on the last response's context
+  plus that response's own calls.
+- **Malformed arguments fail any schema that requires a field.** Arguments the
+  model emitted as unparseable JSON are recorded as
+  `{"_raw": …, "_parse_error": true}`, which fails any `json_schema` that
+  requires a field.
+
 ### `evaluators.llm_judge`
 
 A list of pairwise judges. Each entry has:
@@ -769,8 +933,8 @@ suites:
 
 The block takes the same families as the top-level `evaluators:` — `structural`,
 `semantic`, `llm_judge`, `tool_selection`, `tool_arguments`,
-`tool_trace_structure`, `agent_trace` — and resolution is **family-level
-replacement**:
+`tool_trace_structure`, `agent_trace`, `trace_invariants` — and resolution is
+**family-level replacement**:
 
 - A family the suite does **not** mention is inherited from the top level.
 - A family it **does** mention replaces the top-level one wholesale. There is no
@@ -787,9 +951,15 @@ a name that has no `suites:` entry, resolves to the top-level block.
 
 ### `managed`
 
-`capture sync` regenerates a managed suite's entire entry — `path` and
+`capture sync` regenerates a managed suite's entry — `path` and
 `evaluators` — from what that suite's captures contain, so hand edits inside the
-marker-delimited region are overwritten. Set `managed: false` to freeze an entry:
+marker-delimited region are overwritten. The one exception is
+`evaluators.trace_invariants`: no capture can derive a team's rules, so sync
+copies a managed entry's existing `trace_invariants` block into the entry it
+regenerates, verbatim. The marker comment above the region still says hand edits
+are overwritten; that text cannot change (sync finds the region by matching it
+exactly), so this paragraph is where the exception is written down. Every other
+hand edit is overwritten. Set `managed: false` to freeze an entry:
 
 ```yaml
 suites:
@@ -844,6 +1014,50 @@ evalshift runs clean --suite main_chat    # restrict to one suite
 `--keep` beats `EVALSHIFT_MAX_RUNS`, which beats the config value. `runs clean` works even without a
 valid `evalshift.yaml` (it falls back to the defaults), so it's always available for disk cleanup.
 
+## `captures`
+
+Where the capture SDK's recordings live when they are not on this machine. Omit the block
+entirely for the default: captures are read from `.evalshift/captures/` (or `EVALSHIFT_DIR`)
+on local disk and nothing here is needed.
+
+Production agents on Fargate, Lambda or Kubernetes lose their disk when they stop, so the
+SDK can ship captures to an object store you own instead (`EVALSHIFT_SINK=<uri>` in the
+agent's environment — see [Capture SDK](sdk.md)). Name the same store here and the CLI
+pulls new captures down before it promotes them, with no flags to remember:
+
+```yaml
+captures:
+  store: s3://acme-evals/support-agent
+```
+
+| Field   | Type          | Default | Description |
+| ------- | ------------- | ------- | ----------- |
+| `store` | string (URI)  | (none)  | Object store the SDK writes to. `capture sync` and `capture list` fetch new captures and toolset sidecars from it into the local directory first; `capture fetch` does only that step. |
+
+CLI versions before this release reject the `captures:` key (the config is strict), so bump
+the CLI — and a pinned GitHub Action's `evalshift-version` — before adding it.
+
+Accepted URI forms — the same grammar the SDK uses:
+
+| Form | Store | Install | Credentials |
+| --- | --- | --- | --- |
+| `s3://<bucket>/<prefix>` | Amazon S3; MinIO, Cloudflare R2, Backblaze B2 via boto3's `AWS_ENDPOINT_URL` | `pip install "evalshift[s3]"` | IAM role in CI (OIDC), `aws sso login` locally |
+| `gs://<bucket>/<prefix>` | Google Cloud Storage | `pip install "evalshift[gcs]"` | Workload Identity in CI, `gcloud auth application-default login` locally |
+| `az://<account>/<container>/<prefix>` | Azure Blob Storage | `pip install "evalshift[azure]"` | Managed Identity / federated credential in CI, `az login` locally |
+
+`<prefix>` is optional. Credentials never go in the URI: a value containing `@` or `?` fails
+to load, naming the accepted forms. A rejected value is never echoed back (it may be a pasted
+key or connection string); the error names at most its scheme. Under the prefix the layout is exactly the local one —
+`captures/<suite>/cap_<hex>.json` and `toolsets/<hex>.json` — so the bucket is a mirror of
+`.evalshift/`, not a different format.
+
+What the fetch does: it lists the bucket, downloads only objects missing locally (captures
+are immutable and toolsets content-addressed, so "exists" means "current"), skips captures
+already promoted into a suite (so `capture clean` never causes a re-download), and writes
+each file atomically. `--since 24h` (or `7d`, `30m`, an ISO date) limits a fetch to recently
+written captures; `--offline` skips it and works with what is already local. Retention in the bucket
+is yours to set with a lifecycle rule; the SDK's `max_captures` does not apply there.
+
 ## Capture lifecycle
 
 The companion `evalshift-sdk` package records real agent runs to
@@ -852,11 +1066,16 @@ the base). The CLI consumes them via the `capture` commands:
 
 ```shell
 evalshift capture list                          # see what the SDK recorded
+evalshift capture fetch                         # (captures.store only) mirror new captures locally
 evalshift capture sync --input-var query        # promote every capture + wire suites:
 evalshift run --suite-name support_agent --yes  # score a candidate model against it
 evalshift capture clean                         # prune already-promoted captures
 ```
 
+With [`captures.store`](#captures) configured, `list` and `sync` run the same fetch first
+and print one summary line (`fetched 12 capture(s) and 1 toolset(s) from s3://…`); pass
+`--offline` to skip it. `promote`, `diff` and `clean` operate on the local mirror only.
+
 `capture clean [<suite>]` deletes promoted captures by default (`--promoted`);
 `--all` deletes every capture, promoted or not. It never touches promoted
 suites, then offers to sweep toolset sidecars no surviving capture or suite
@@ -883,7 +1102,9 @@ key on evaluator names across runs, so regenerating a suite must not rename what
 it already wired. `structural` is deliberately not derived: nothing in a capture
 says what shape an answer must have. Sync regenerates only the suites it just
 promoted and carries every other entry in the region forward verbatim, and
-`managed: false` freezes an entry entirely (see [`suites`](#suites)).
+`managed: false` freezes an entry entirely (see [`suites`](#suites)). A
+regenerated entry keeps its hand-written `evaluators.trace_invariants` (see
+[`managed`](#managed)).
 Captures with no recorded events are skipped, and captures whose replayed
 content duplicates an already-promoted case (or an earlier capture in the
 same run) are skipped too — duplicate examples inflate *n* and corrupt the