@openwop/openwop-conformance 2.46.26 → 2.46.28

package/CHANGELOG.md

@@ -1,3 +1,15 @@
+## [2.46.28] — 2026-10-26 — RFC 0244 G2 closed; RFC 0251 and the §F legs
+
+- **New scenario `v2-eval-regression-baseline` (RFC 0251)** over two fixtures sharing one `suiteId` (`conformance-eval-regression-base` / `-candidate`), whose paired scores are equal while the aggregates differ by 1/3: `0251.shape` (server-free), `baseline-echoed`, `baseline-refused`, `baseline-uniform` (needs `OPENWOP_TEST_TENANT_B_API_KEY`, else `blocked`), `fields-refused`, and RFC 0244 §F `0244.regression-paired` and `regression-no-false-verdict`. Judges in `lib/eval-regression-witness.ts`.
+- **Host impact:** opt-in. The legs need `agents.evalSuite.baselineSelection` and both fixtures listed; no host does yet, so they record `inapplicable`.
+- **RFC 0244's G2 closes** on openwop-app's certified cut `fc9f666ed` (the three counter legs and `0244.efficiency-tool-calls` `executed-pass`), so the packed `spec/v1/gaps.json` changes.
+
+## [2.46.27] — 2026-10-25 — RFC 0244 never-gated and trial-isolation legs
+
+- **New leg `0244.efficiency-never-gated` in `v2-eval-trajectory-trials`**, on the new fixture `conformance-eval-efficiency-ungated`: tasks `lean` (one scripted call) and `busy` (eight) share an output and an expectation, so `busy` must pass with `lean`'s score and the suite must pass at `passScore` 1. `blocked` when the calls never ran or `lean` fails. Judge `judgeNeverGated` in `lib/eval-efficiency-witness.ts`.
+- **New leg `0244.trial-memory-isolated` (RFC 0244 G3)**, on the new fixture `conformance-eval-memory-isolation` (`trials.count: 2`, gated on `agents.evalSuite.trials`), through the conformance-reserved memory tool `conformance.eval:memory` (contract in `fixtures.md`). Each trial must read the `memorySeed` entry, and its first read of `trial-note` must find nothing before its own write. Judge in `lib/eval-memory-witness.ts`; a tool whose writes never read back records `blocked`.
+- **Host impact:** none until a host lists a new fixture in discovery `fixtures`. Listing the memory fixture opts the host in to the memory-tool contract.
+
 ## [2.46.26] — 2026-10-25 — RFC 0244 model-usage and cost legs
 
 - **New leg `0244.efficiency-model-usage` in `v2-eval-trajectory-trials`**, on the existing trajectory and trials runs. Per-task `modelCallCount`, `inputTokens` and `outputTokens`, times the trial count, must sum to the agent's `provider.usage` inside its `agent.invocation.*` brackets (judge and delegate calls excluded); without brackets they must not exceed the run's usage.

package/README.md

@@ -137,7 +137,7 @@ Each scenario file records its disposition and assertion count in `<report-dir>/
 
 ## What's Covered
 
-The current suite has 661 scenario files under `src/scenarios/`. By target major ([`scenario-majors.json`](./scenario-majors.json)): 221 files run against a v2 host and 455 against a v1 host; 15 of them run at both.
+The current suite has 662 scenario files under `src/scenarios/`. By target major ([`scenario-majors.json`](./scenario-majors.json)): 222 files run against a v2 host and 455 against a v1 host; 15 of them run at both.
 
 The v2 scenarios (`src/scenarios/v2-*.test.ts`) cover, by area:
 
@@ -166,7 +166,7 @@ Three kinds of scenario share the tree:
 - **Host** checks drive a live host over HTTP.
 - **Gated** checks run only when the host advertises the family, profile or fixture they need, and otherwise record `inapplicable` or `blocked` with a reason. `--require-behavior` turns an advertised-but-unwitnessable behaviour into a failure.
 
-Current source tree: 661 scenario files. [`coverage.md`](./coverage.md) maps each spec document and every OpenAPI operation to its scenarios. [`CHANGELOG.md`](./CHANGELOG.md) records when each scenario landed.
+Current source tree: 662 scenario files. [`coverage.md`](./coverage.md) maps each spec document and every OpenAPI operation to its scenarios. [`CHANGELOG.md`](./CHANGELOG.md) records when each scenario landed.
 
 ## How the 2.x suite is built
 

package/dist/spec-artifacts.lock.json

@@ -1,5 +1,5 @@
 {
   "package": "@openwop/spec-artifacts",
-  "version": "2.46.26",
-  "stampSha256": "e53058ead4546c46dc23498974219d081a05193fdb5dec16db93a221d023e0d9"
+  "version": "2.46.28",
+  "stampSha256": "6ae1b50a4dd848cc709f5ead006df2ddf0e3499d91c1eb2dad44c7fcf71b287f"
 }

package/fixtures/eval-suites/conformance-eval-efficiency-ungated.json

@@ -0,0 +1,97 @@
+{
+  "suiteId": "core.openwop.evals.conformance-eval-efficiency-ungated",
+  "version": "1.0.0",
+  "targetAgentId": "core.conformance.agents.scripted",
+  "modes": [
+    "golden"
+  ],
+  "thresholds": {
+    "passScore": 1
+  },
+  "tasks": [
+    {
+      "taskId": "lean",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-1"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "done"
+        }
+      }
+    },
+    {
+      "taskId": "busy",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-1"
+            }
+          },
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-2"
+            }
+          },
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-3"
+            }
+          },
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-4"
+            }
+          },
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-5"
+            }
+          },
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-6"
+            }
+          },
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-7"
+            }
+          },
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-8"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "done"
+        }
+      }
+    }
+  ]
+}

package/fixtures/eval-suites/conformance-eval-memory-isolation.json

@@ -0,0 +1,71 @@
+{
+  "suiteId": "core.openwop.evals.conformance-eval-memory-isolation",
+  "version": "1.0.0",
+  "targetAgentId": "core.conformance.agents.scripted",
+  "modes": [
+    "golden"
+  ],
+  "trials": {
+    "count": 2,
+    "k": 1,
+    "passRule": "all"
+  },
+  "tasks": [
+    {
+      "taskId": "write-then-read",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:memory",
+            "arguments": {
+              "op": "read",
+              "id": "seeded"
+            }
+          },
+          {
+            "tool": "conformance.eval:memory",
+            "arguments": {
+              "op": "read",
+              "id": "trial-note"
+            }
+          },
+          {
+            "tool": "conformance.eval:memory",
+            "arguments": {
+              "op": "write",
+              "id": "trial-note",
+              "content": "written"
+            }
+          },
+          {
+            "tool": "conformance.eval:memory",
+            "arguments": {
+              "op": "read",
+              "id": "trial-note"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "done"
+        }
+      },
+      "fixtures": {
+        "memorySeed": [
+          {
+            "id": "seeded",
+            "content": "seed",
+            "tags": [
+              "conformance"
+            ],
+            "createdAt": "2026-01-01T00:00:00Z"
+          }
+        ]
+      }
+    }
+  ]
+}

package/fixtures/eval-suites/conformance-eval-regression-base.json

@@ -0,0 +1,74 @@
+{
+  "suiteId": "core.openwop.evals.conformance-eval-regression",
+  "version": "1.0.0",
+  "targetAgentId": "core.conformance.agents.scripted",
+  "modes": [
+    "golden",
+    "regression"
+  ],
+  "tasks": [
+    {
+      "taskId": "base-only",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-1"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "never-in-output"
+        }
+      }
+    },
+    {
+      "taskId": "shared-a",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-1"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "done"
+        }
+      }
+    },
+    {
+      "taskId": "shared-b",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-1"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "done"
+        }
+      }
+    }
+  ]
+}

package/fixtures/eval-suites/conformance-eval-regression-candidate.json

@@ -0,0 +1,74 @@
+{
+  "suiteId": "core.openwop.evals.conformance-eval-regression",
+  "version": "1.1.0",
+  "targetAgentId": "core.conformance.agents.scripted",
+  "modes": [
+    "golden",
+    "regression"
+  ],
+  "tasks": [
+    {
+      "taskId": "shared-a",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-1"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "done"
+        }
+      }
+    },
+    {
+      "taskId": "shared-b",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-1"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "done"
+        }
+      }
+    },
+    {
+      "taskId": "candidate-only",
+      "input": {
+        "script": [
+          {
+            "tool": "conformance.eval:lookup",
+            "arguments": {
+              "orderId": "o-1"
+            }
+          }
+        ],
+        "output": "done"
+      },
+      "expected": {
+        "kind": "golden",
+        "match": {
+          "strategy": "contains",
+          "value": "done"
+        }
+      }
+    }
+  ]
+}

package/fixtures.md

@@ -77,6 +77,10 @@ All fixtures MUST advertise:
 | Eval — Fixture Error | `conformance-eval-fixture-error` | RFC 0245 — `onMiss: "fail"`; one task calls the same tool twice, served first by an `isError: true` entry, then by a success (`v2-eval-trajectory-trials`) | eval run `completed`; the first call is recorded as a tool error | ≤ 30s |
 | Eval — Counter (live) | `conformance-eval-counter-live` | RFC 0244 §D, G2 — `onMiss: "live"`; one task makes four `conformance.eval:counter` calls that match no entry, so each runs the live counter (`v2-eval-trajectory-trials`) | eval run `completed`; four live `{key, count}` outcomes | ≤ 30s |
 | Eval — Counter (miss) | `conformance-eval-counter-miss` | RFC 0244 §D, G2 — `onMiss: "fail"`; three `inc S` calls against two exact entries, then an unmatched `inc F` (`v2-eval-trajectory-trials`) | eval run `completed`; `{served: 1}`, `{served: 2}`, then two tool errors | ≤ 30s |
+| Eval — Efficiency Ungated | `conformance-eval-efficiency-ungated` | RFC 0244 §C — two golden tasks with one output, `lean` (1 call) and `busy` (8 calls), at `passScore: 1` (`v2-eval-trajectory-trials`) | eval run `completed`; both tasks and the suite pass | ≤ 30s |
+| Eval — Memory Isolation | `conformance-eval-memory-isolation` | RFC 0244 §B, G3 — `trials.count: 2`; one task seeds `seeded`, then reads it, reads `trial-note`, writes it and reads it back through `conformance.eval:memory` (`v2-eval-trajectory-trials`) | eval run `completed`; each trial reads the seed and finds no `trial-note` before its own write | ≤ 30s |
+| Eval — Regression (base) | `conformance-eval-regression-base` | RFC 0251 — the baseline run: suite 1.0.0, `base-only` fails, `shared-a` and `shared-b` pass (`v2-eval-regression-baseline`) | eval run `completed`; `aggregateScore` 2/3 | ≤ 30s |
+| Eval — Regression (candidate) | `conformance-eval-regression-candidate` | RFC 0251, RFC 0244 §F — the same `suiteId` at 1.1.0: the two shared tasks unchanged, plus `candidate-only` (passes), run against the base run (`v2-eval-regression-baseline`) | eval run `completed`; `aggregateScore` 1; `scoreDelta` 1/3; `pairedTaskCount` 2; verdict neither `improved` nor `regressed` | ≤ 30s |
 | Dispatch Fan-Out Waves | `conformance-dispatch-fanout-waves` | RFC 0243 §A.3 — a parallel `core.dispatch` (`maxConcurrency: 2`, `wait-all`/`collect`) whose supervisor's `mockDispatchPlan` names three `conformance-delay` children. At most `min(2, maxFanOut)` run at once and none is dropped. Gated on `agents.fanOut` admitting `wait-all` with `collect`. | `completed` | ≤ 30s |
 | Dispatch Per-Item Input | `conformance-dispatch-per-item` | RFC 0243 §A.4 — a sequential `core.dispatch` whose decision dispatches `conformance-dispatch-input-mapping-child` twice with `nextWorkerInputs` `item-a`, `item-b`. At `agents.fanOut.perItemInput: true` each child receives its item; otherwise the dispatch node fails `validation_error` and no child is dispatched. | `completed`, or `failed` without `perItemInput` | ≤ 30s |
 | Dispatch Per-Item Length Mismatch | `conformance-dispatch-per-item-mismatch` | `orchestrator-decision.schema.json` `nextWorkerInputs` — one item for two children. Every dispatch host fails the dispatch node `validation_error` and dispatches no child. | `failed` | ≤ 30s |
@@ -1048,7 +1052,7 @@ The `fixtures/eval-suites/` sub-directory holds `AgentEvalSuite` documents (`../
 Every task targets **the scripted agent `core.conformance.agents.scripted`**. Its contract:
 
 - `input.script` is an ordered list of `{ tool, arguments? }`. The agent makes exactly those tool calls, in order, each emitting `agent.tool-called` with `agentId: "core.conformance.agents.scripted"`, `toolName: tool` and `inputs: arguments`.
-- Each call goes through the host's eval tool boundary. In a task with `fixtures.toolResponses`, the RFC 0244 §D selection and `onMiss` rules apply. Otherwise the host answers the call with its own `arguments`. No scripted call reaches a live tool or an external effect, except a miss on the counter tool under `onMiss: "live"` (below).
+- Each call goes through the host's eval tool boundary. In a task with `fixtures.toolResponses`, the RFC 0244 §D selection and `onMiss` rules apply. Otherwise the host answers the call with its own `arguments`. No scripted call reaches a live tool or an external effect, except a miss on the counter tool under `onMiss: "live"` and a call to the memory tool (both below).
 - After the last call the agent decides, and its output is `input.output`.
 
 | Fixture | Facet | What the suite asserts |
@@ -1061,6 +1065,10 @@ Every task targets **the scripted agent `core.conformance.agents.scripted`**. It
 | `conformance-eval-fixture-error` | `agents.evalSuite.fixtureErrors` + `agents.evalSuite.fixtureMatching` | RFC 0245: two `agent.tool-called` and two `agent.tool-returned` for `conformance.eval:lookup`, paired by `callId`; the first return carries `error` with non-empty `code` and `message` and no `outcome` (`status: "error"` under `toolHooks.prePostEvents`), the second carries no `error` (`outcome` is optional; not `status: "error"` under tool hooks); `recovers` reports `fixtureMissCount: 0`. Without `fixtureErrors`: `400 validation_error`. That the agent saw an error and no live tool ran is not observable here (RFC 0245 G1, RFC 0244 G2) |
 | `conformance-eval-counter-live` | `agents.evalSuite.fixtureMatching` | run twice, before and after `conformance-eval-counter-miss`: `fixtureMissCount` > 0; the two live `inc L` counts are consecutive; the second run's first `L` is the first run's last plus one (else `blocked`), and its `F` and `S` reads equal the first run's |
 | `conformance-eval-counter-miss` | `agents.evalSuite.fixtureMatching` | the three `inc S` calls return `{served: 1}`, `{served: 2}`, then a tool error; `inc F` returns a tool error. Neither miss moves the live counter |
+| `conformance-eval-efficiency-ungated` | `agents.evalSuite` | RFC 0244 §C never-gated: with all nine scripted calls in the log and `lean` passing, `busy` passes with `lean`'s score and the suite passes. A host MUST NOT fail either on `efficiency`, reported or not |
+| `conformance-eval-memory-isolation` | `agents.evalSuite.trials` | RFC 0244 §B isolation: both reads of `seeded` return `seed`; of the four `trial-note` reads, two return `written` (each trial's read-back) and two return `null` (each trial's first read). A later trial reading `written` first saw an earlier trial's write. Fewer than two read-backs record `blocked` |
+| `conformance-eval-regression-base` | `agents.evalSuite.baselineSelection` + `regression` in `modes` | run first with `evalModes: [golden]`; the baseline of the candidate run |
+| `conformance-eval-regression-candidate` | `agents.evalSuite.baselineSelection` + `regression` in `modes` (+ `regressionVerdict` for the §F legs) | run with `evalModes: [golden, regression]` and the base run as `baselineRunId`: both carriers name it, `scoreDelta` 1/3, `pairedTaskCount` 2, verdict neither `improved` nor `regressed`. Both fixtures share one `suiteId`, so a host's suite store resolves them by fixture id |
 
 **The counter tool `conformance.eval:counter` is conformance-RESERVED** (the pattern of `core.conformance.side-effect`). A host that lists either counter fixture in discovery `fixtures` MUST, at its eval tool boundary:
 
@@ -1070,6 +1078,14 @@ Every task targets **the scripted agent `core.conformance.agents.scripted`**. It
 
 The counter has no external effect, so serving it is safe on a production tenant. Two concurrent cuts on one tenant make the persistence check fail; the legs then record `blocked`, not a failure.
 
+**The memory tool `conformance.eval:memory` is conformance-RESERVED.** A host that lists `conformance-eval-memory-isolation` in discovery `fixtures` MUST, at its eval tool boundary:
+
+- answer every call to it from the agent's memory for the current trial: the store the task's `fixtures.memorySeed` seeds (RFC 0004 `MemoryAdapter` entries) and the agent's own memory writes land in. It is never served from `toolResponses` and never counted as a fixture miss.
+- answer `{op: "write", id, content}` by storing the entry and returning `{id}`, and `{op: "read", id}` by returning `{id, content}`, with `content: null` when the trial's memory holds no entry `id`;
+- record `outcome` on `agent.tool-returned` for every memory call.
+
+The tool reaches only the trial's eval memory, so it has no external effect. A host that answers it from a store other than the one its trials really use proves nothing about §B isolation, and the contract forbids it.
+
 ## OAuth provider fixtures
 
 RFC 0199 adds `provider.issuer`, `provider.pkce` and the two issuer-less providers (`issuerlessProviders[]`) the distinct-redirect-URI leg needs; the endpoints a host actually talks to are supplied at run time by the `oauth/authorize-start` seam, which points them at the suite's authorization-server double (`src/lib/oauth-as-double.ts`).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@openwop/openwop-conformance",
-  "version": "2.46.26",
+  "version": "2.46.28",
   "description": "Black-box conformance suite for OpenWOP hosts (v2 current; v1 scenarios retained for the frozen v1 tree).",
   "repository": {
     "type": "git",
@@ -56,6 +56,6 @@
     "@openwop/spec-artifacts": "file:../spec-artifacts"
   },
   "peerDependencies": {
-    "@openwop/spec-artifacts": "2.46.26"
+    "@openwop/spec-artifacts": "2.46.28"
   }
 }