bitfab-cli 0.2.422 → 0.2.424

package/dist/index.js

@@ -28787,12 +28787,12 @@ var labelEvidenceShape = external_exports.preprocess(parseJsonString, external_e
 var saveAgentLabels = {
   name: "save_agent_labels",
   title: "Save Agent Labels",
-  description: "Set, skip, or archive the agent's pass/fail verdict on one or more traces (`labelSource=\"agent\"`). Use this AFTER you have read the traces with get_traces and decided yourself whether each one looks like a pass, a fail, or genuinely cannot be judged. To set a verdict, pass `label` (true=PASS, false=FAIL) and `annotation` (your reasoning, shown to the human reviewer in the labeling UI). Add `evidence` as `{ spanId, text }` entries to ground that reasoning in spans on the evaluated trace. Optionally pass `confidence` (`VeryLow|Low|Medium|High|VeryHigh`) to record how confident you are - surfaced in the labeling UI so reviewers can prioritize low-confidence verdicts. To explicitly skip a trace you cannot decide on (instead of leaving it unlabeled), pass `skip: true` and omit label. That records an explicit skip so coverage checks know you intentionally did not verdict it, and `annotation` still applies: pass it to say why the verdict was withheld (the check never ran, the tape was punctured, the run never terminated), which is the only place a reader learns that. A skip with no annotation clears the note the previous verdict carried, so a stale reason never sits under a withheld verdict. To clear a previously-set agent verdict (e.g., you changed your mind or labeled the wrong trace), pass `archive: true` and omit label/annotation. New verdicts start unapproved (`approvedAt=null`); once a human approves one, it joins the validated dataset (queryable via `search_traces` with `validated: true`). Archiving is non-destructive - the row is hidden from all reads but kept for audit, and you can immediately re-label the trace from scratch. For replay results, key each label by the replay item's `originalTraceId` (the original trace it was replayed from; `sourceTraceId` is accepted as a deprecated alias) plus the top-level `experimentId` instead of a `traceId`: the server resolves it to the replay trace via lineage, so you never need a server-generated replay trace id. When the experiment ran each trace more than once, also pass the item's `attempt` so each attempt gets its own verdict. Returns an agent-readable summary with one parseable effective label line per updated trace, keyed by the id you supplied (`originalTraceId` for replay verdicts, otherwise `traceId`), so command callers can verify persistence. Before judging a replay, call get_trace_assertions on the original trace ids: an assertion says what the user asked this specific case to do, so the verdict is measured against that rather than a guess. When an assertion's target cannot be found on the trace you are judging, the check never ran, so pass `skip: true` for that trace rather than recording a FAIL. Pass `assertionId` to score ONE assertion on the trace (the `[ID: ...]` get_trace_assertions prints), one entry per assertion, all in the same call. Omit it for the trace's single whole-trace verdict. get_trace_labels reads these back one line per assertion, keyed by the same assertionId, so a per-assertion write is verifiable per assertion.",
+  description: "Set, skip, or archive the agent's pass/fail verdict on one or more of a trace's approved assertions (`labelSource=\"agent\"`). Every verdict and skip scores ONE assertion and must carry its `assertionId` (the `[ID: ...]` get_trace_assertions prints). Labels are only ever written on assertions, never directly on a trace: a trace with no approved assertions gets no verdict, so add assertions first (save_trace_assertions) or report your judgment in the conversation. Use this AFTER you have read the traces with get_traces and decided yourself whether each one looks like a pass, a fail, or genuinely cannot be judged. To set a verdict, pass `label` (true=PASS, false=FAIL) and `annotation` (your reasoning, shown to the human reviewer in the labeling UI). Add `evidence` as `{ spanId, text }` entries to ground that reasoning in spans on the evaluated trace. Optionally pass `confidence` (`VeryLow|Low|Medium|High|VeryHigh`) to record how confident you are - surfaced in the labeling UI so reviewers can prioritize low-confidence verdicts. To explicitly skip an assertion you cannot decide on (instead of leaving it unlabeled), pass `skip: true` and omit label. That records an explicit skip so coverage checks know you intentionally did not verdict it, and `annotation` still applies: pass it to say why the verdict was withheld (the check never ran, the tape was punctured, the run never terminated), which is the only place a reader learns that. A skip with no annotation clears the note the previous verdict carried, so a stale reason never sits under a withheld verdict. To clear a previously-set agent verdict (e.g., you changed your mind or labeled the wrong trace), pass `archive: true` and omit label/annotation; an archive may omit `assertionId` to clear an old whole-trace agent verdict. A verdict counts as soon as it is written; there is no approval step, and a person can override it with their own label. Archiving is non-destructive - the row is hidden from all reads but kept for audit, and you can immediately re-label the trace from scratch. For replay results, key each label by the replay item's `originalTraceId` (the original trace it was replayed from; `sourceTraceId` is accepted as a deprecated alias) plus the top-level `experimentId` instead of a `traceId`: the server resolves it to the replay trace via lineage, so you never need a server-generated replay trace id. When the experiment ran each trace more than once, also pass the item's `attempt` so each attempt gets its own verdict. Returns an agent-readable summary with one parseable effective label line per entry you sent (so a trace scored on several assertions gets one line per assertion), keyed by the id you supplied (`originalTraceId` for replay verdicts, otherwise `traceId`), so command callers can verify persistence. Before judging a replay, call get_trace_assertions on the original trace ids: an assertion says what the user asked this specific case to do, so the verdict is measured against that rather than a guess. When an assertion's target cannot be found on the trace you are judging, the check never ran, so pass `skip: true` for that assertion rather than recording a FAIL. Send one entry per assertion, all in the same call. get_trace_labels reads these back one line per assertion, keyed by the same assertionId, so a per-assertion write is verifiable per assertion.",
   inputSchema: {
     experimentId: external_exports.uuid().optional().describe("The experiment ID. Required when any label targets a trace by originalTraceId (replay verdicts); ignored otherwise."),
     labels: external_exports.preprocess(parseJsonString, external_exports.array(external_exports.object({
       traceId: external_exports.uuid().optional().describe("The trace ID to update. Provide this OR originalTraceId, not both."),
-      assertionId: external_exports.uuid().optional().describe("The assertion this verdict scores, taken from the `[ID: ...]` get_trace_assertions prints. Omit it to set the trace's whole-trace verdict, of which there is one. A replay may name an assertion it inherited from the original trace it was replayed from."),
+      assertionId: external_exports.uuid().optional().describe("The assertion this verdict scores, taken from the `[ID: ...]` get_trace_assertions prints. Required unless `archive` is true: labels are never written directly on a trace. A replay may name an assertion it inherited from the original trace it was replayed from."),
       originalTraceId: external_exports.uuid().optional().describe("For replay verdicts: the original trace ID the replay item was replayed from (the replay item's originalTraceId). The server resolves it to the replay trace via lineage, so you never need a server replay trace id. Requires the top-level experimentId. Provide this OR traceId, not both."),
       sourceTraceId: external_exports.uuid().optional().meta({ deprecated: true }).describe("Deprecated alias for originalTraceId. Prefer originalTraceId; accepted for back-compat."),
       attempt: external_exports.number().int().min(0).optional().describe("For replay verdicts on an experiment that ran each trace more than once: which attempt of originalTraceId this verdict is for (0-based, the replay item's attempt). Defaults to 0, so single-attempt callers never need it."),
@@ -28802,22 +28802,22 @@ var saveAgentLabels = {
       annotation: external_exports.string().min(1).optional().describe("Your reasoning, shown to the human reviewer in the labeling UI. Required when archive and skip are both false/omitted. Optional alongside `skip: true`, where it says why the verdict was withheld."),
       confidence: external_exports.enum(["VeryLow", "Low", "Medium", "High", "VeryHigh"]).optional().describe("How confident you are in this verdict: VeryLow / Low / Medium / High / VeryHigh. Surfaced as a chip in the labeling UI; low-confidence labels are prioritized for human review. Optional; only meaningful when paired with a label."),
       evidence: labelEvidenceShape.nullable().optional().describe("Span-by-span evidence for this verdict. Omit it to preserve existing evidence. Pass null to clear existing evidence.")
-    })).min(1).max(200)).describe("One entry per assertion on traces that have assertions, otherwise one whole-trace entry (1-200)")
+    })).min(1).max(200)).describe("One entry per approved assertion, each carrying its assertionId; an archive entry may omit it (1-200)")
   }
 };
 var saveHumanLabels = {
   name: "save_human_labels",
   title: "Save Human Labels",
-  description: "Record one or more human-authored pass/fail verdicts that are VALIDATED immediately (`labelSource=\"human\"`, no approval step). Unlike save_agent_labels (which writes agent suggestions that start unapproved), labels set here join the validated dataset the instant they are written and satisfy `search_traces` with `validated: true`. Pass `label` (true=PASS, false=FAIL), `annotation` (the reasoning behind the verdict), and optionally `evidence` as `{ spanId, text }` entries grounded in the evaluated trace. Optionally pass `confidence` (`VeryLow|Low|Medium|High|VeryHigh`). USE ONLY when a human has explicitly decided the verdict, for example saving a known production bug as a test case. This records a verdict on a run that already happened; to record what a trace should do on its next replay, use save_trace_assertions. Do NOT use this for the agent's own first-pass guesses on traces awaiting human review; use save_agent_labels for those so they go through the normal approve/edit loop. Pass `assertionId` when the human judged ONE assertion (the `[ID: ...]` get_trace_assertions prints), one entry per assertion. Omit it for the trace's single whole-trace verdict. get_trace_labels reads these back one line per assertion, keyed by the same assertionId, so a per-assertion write is verifiable per assertion.",
+  description: "Record one or more human-authored pass/fail verdicts that are VALIDATED immediately (`labelSource=\"human\"`) on one or more of a trace's approved assertions. Unlike save_agent_labels (the agent's own verdicts), labels set here join the validated dataset the instant they are written and satisfy `search_traces` with `validated: true`. Pass `label` (true=PASS, false=FAIL), `annotation` (the reasoning behind the verdict), and optionally `evidence` as `{ spanId, text }` entries grounded in the evaluated trace. Optionally pass `confidence` (`VeryLow|Low|Medium|High|VeryHigh`). USE ONLY when a human has explicitly decided the verdict, for example saving a known production bug as a test case. This records a verdict on a run that already happened; to record what a trace should do on its next replay, use save_trace_assertions. Do NOT use this for the agent's own first-pass guesses; use save_agent_labels for those. Every entry must carry the `assertionId` of the ONE assertion the human judged (the `[ID: ...]` get_trace_assertions prints), one entry per assertion: labels are never written directly on a trace, so a trace with no approved assertions needs assertions first (save_trace_assertions). get_trace_labels reads these back one line per assertion, keyed by the same assertionId, so a per-assertion write is verifiable per assertion.",
   inputSchema: {
     labels: external_exports.preprocess(parseJsonString, external_exports.array(external_exports.object({
       traceId: external_exports.uuid().describe("The trace ID to label"),
-      assertionId: external_exports.uuid().optional().describe("The assertion this verdict scores, taken from the `[ID: ...]` get_trace_assertions prints. Omit it to set the trace's whole-trace verdict, of which there is one."),
+      assertionId: external_exports.uuid().describe("The assertion this verdict scores, taken from the `[ID: ...]` get_trace_assertions prints. Required: labels are never written directly on a trace."),
       label: external_exports.boolean().describe("true for PASS, false for FAIL. Required."),
       annotation: external_exports.string().min(1).describe("The reasoning behind the verdict. Required."),
       confidence: external_exports.enum(["VeryLow", "Low", "Medium", "High", "VeryHigh"]).optional().describe("How confident the verdict is: VeryLow / Low / Medium / High / VeryHigh. Optional."),
       evidence: labelEvidenceShape.nullable().optional().describe("Span-by-span evidence for this verdict. Omit it to preserve existing evidence. Pass null to clear existing evidence.")
-    })).min(1).max(200)).describe("One entry per assertion on traces that have assertions, otherwise one whole-trace entry (1-200)")
+    })).min(1).max(200)).describe("One entry per approved assertion, each carrying its assertionId (1-200)")
   }
 };
 var assertionTargetShape = external_exports.discriminatedUnion("kind", [
@@ -28886,7 +28886,7 @@ var generateTraceAssertions = {
 var saveTraceAssertions = {
   name: "save_trace_assertions",
   title: "Save Trace Assertions",
-  description: `Record what SHOULD happen when this trace is replayed. Bitfab stores two kinds of label on a trace. A VERDICT says how a run that already happened turned out, and that is save_agent_labels / save_human_labels. An ASSERTION, which is what this tool writes, says what a correct run looks like for THIS specific input, carries no pass/fail of its own, and is checked against a later replay once a reviewer approves it. A trace holds at most one verdict per author and any number of assertions. Call it when the user describes the right answer for a particular case, for example "this booking should have picked the 6am flight, not the 9am". This is not a verdict on a run that already happened, which is save_agent_labels, and it is not a check that applies to every trace of a function, which is save_grader. Pass \`assertion\` plus optional \`passCriteria\` / \`failCriteria\`, the same trio save_grader takes, so an assertion that proves out across many traces can later be promoted into a grader with no rewriting. \`humanNote\` stores people-only context on the assertion; preserve and return it, but never use it to judge whether a replay passed. \`assigneeEmail\` names the member who should review it, which routes the assertion to that person without deciding anything, so it never changes the approval state. Pass an entry's \`id\` to edit an existing assertion, or omit it to add a new one, so two callers adding different assertions to one trace never overwrite each other. What this tool writes is saved awaiting review, so nothing checks it yet; a reviewer approves it, in Bitfab or with \`approvalState\` on a later call here, and from then on every replay of the trace is judged against it. \`approvalState\` decides an assertion that already exists, so deciding one is \`{ id, approvalState }\` with nothing else: an omitted \`assertion\` keeps the current wording. It is refused on a new assertion, so the call that invents a claim is never the call that puts it into force. One decision covers the assertion and the category it is filed under, so set \`category_assertion_id\` and \`approvalState\` in the same entry when you are deciding both; recategorizing on a later call without deciding returns it to not reviewed, since the approval covered the old category. What it gates is whether replays check the assertion at all, so decide from the assertion and the spans it cites rather than from memory of having written it. Rewording an assertion, or changing its criteria, target, or category, returns it to not reviewed unless the same call says otherwise. Call get_trace_assertions first to find the assertion to edit and to avoid creating a duplicate, and read them back before judging a replay, where it lists the approved ones by default. Pass it \`approvalStates: ["pending"]\` to read back drafts awaiting review, including the ones you just wrote. Up to ${MAX_ASSERTIONS_PER_REQUEST} per call.`,
+  description: `Record what SHOULD happen when this trace is replayed. Bitfab stores two kinds of label on a trace. A VERDICT says how a run that already happened turned out, and that is save_agent_labels / save_human_labels. An ASSERTION, which is what this tool writes, says what a correct run looks like for THIS specific input, carries no pass/fail of its own, and is checked against a later replay once a reviewer approves it. A trace holds any number of assertions, and each verdict is recorded on one of them. Call it when the user describes the right answer for a particular case, for example "this booking should have picked the 6am flight, not the 9am". This is not a verdict on a run that already happened, which is save_agent_labels, and it is not a check that applies to every trace of a function, which is save_grader. Pass \`assertion\` plus optional \`passCriteria\` / \`failCriteria\`, the same trio save_grader takes, so an assertion that proves out across many traces can later be promoted into a grader with no rewriting. \`humanNote\` stores people-only context on the assertion; preserve and return it, but never use it to judge whether a replay passed. \`assigneeEmail\` names the member who should review it, which routes the assertion to that person without deciding anything, so it never changes the approval state. Pass an entry's \`id\` to edit an existing assertion, or omit it to add a new one, so two callers adding different assertions to one trace never overwrite each other. What this tool writes is saved awaiting review, so nothing checks it yet; a reviewer approves it, in Bitfab or with \`approvalState\` on a later call here, and from then on every replay of the trace is judged against it. \`approvalState\` decides an assertion that already exists, so deciding one is \`{ id, approvalState }\` with nothing else: an omitted \`assertion\` keeps the current wording. It is refused on a new assertion, so the call that invents a claim is never the call that puts it into force. One decision covers the assertion and the category it is filed under, so set \`category_assertion_id\` and \`approvalState\` in the same entry when you are deciding both; recategorizing on a later call without deciding returns it to not reviewed, since the approval covered the old category. What it gates is whether replays check the assertion at all, so decide from the assertion and the spans it cites rather than from memory of having written it. Rewording an assertion, or changing its criteria, target, or category, returns it to not reviewed unless the same call says otherwise. Call get_trace_assertions first to find the assertion to edit and to avoid creating a duplicate, and read them back before judging a replay, where it lists the approved ones by default. Pass it \`approvalStates: ["pending"]\` to read back drafts awaiting review, including the ones you just wrote. Up to ${MAX_ASSERTIONS_PER_REQUEST} per call.`,
   inputSchema: {
     traceId: external_exports.uuid().describe("The ORIGINAL trace to attach assertions to. A replay trace id is refused, and the error names the original to retry with, because a replay reads its original's assertions automatically."),
     assertions: external_exports.preprocess(parseJsonString, external_exports.array(external_exports.object({
@@ -29882,10 +29882,10 @@ var semver3 = __toESM(require_semver2(), 1);
 
 // ../bitfab-plugin-lib/dist/bakedSdkVersions.js
 var BAKED_SDK_VERSIONS = {
-  typescript: "0.64.7",
-  python: "0.64.7",
-  ruby: "0.64.7",
-  go: "0.64.6"
+  typescript: "0.64.9",
+  python: "0.64.9",
+  ruby: "0.64.9",
+  go: "0.64.8"
 };
 
 // ../bitfab-plugin-lib/dist/installedSdk.js
@@ -33847,15 +33847,19 @@ var FAN_OUT_JUDGING = `{{#claude}}**Scale the judging with fan-out when there ar
 Make each subagent prompt fully self-contained: its batch's per-item payloads (each item carries its own artifacts, enumerated below), the fixed rubric, and any shared context you gathered once (so no subagent re-derives it or touches the repo). Tell it to return one verdict entry per item in the exact shape this step persists, and nothing else.
 
 **Then collect and persist once.** Wait for every batch, concatenate their verdict arrays into the single full set covering all items, and make the one batched persist call this step already describes, unchanged. Fan-out changes only how you produce the verdicts, never how they are stored or routed: same call, same shape, same buckets, same downstream steps.{{/claude}}{{^claude}}**Judge serially.** This editor doesn't use subagents for judging, so judge every item yourself, inline in this agent, as described above, regardless of how many there are.{{/claude}}`;
-var PER_ASSERTION_VERDICTS = `**Read each trace's assertions first, then write one verdict per assertion.** An assertion says what a correct run looks like for that one case, and it is checked against the trace under evaluation. That is what the {{tool:getTraceAssertions}} call in this step is for, and one call covers up to 100 trace IDs, so make it once and before you judge anything. Each assertion comes back on its own line under its trace as \`[ID: <uuid>] checks <target>: <assertion>\`, with its pass and fail criteria in parentheses when it has them. It may also carry a Human note so an agent can preserve or edit context intended for people. **Never use a Human note as evidence for a verdict; assess only the assertion, its pass/fail criteria, and the evaluated trace.** Hold that \`[ID: <uuid>]\` value, it is the \`assertionId\` every verdict on that assertion carries. **Only assertions a person has approved come back, because only those are checked on a replay.** A trace whose assertions are all still awaiting review comes back as "no approved assertions", the same wording a trace with none gets: judge it as a whole trace and write no per-assertion verdict for it.
+var PER_ASSERTION_VERDICTS = `**Read each trace's assertions first, then write one verdict per assertion.** An assertion says what a correct run looks like for that one case, and it is checked against the trace under evaluation. That is what the {{tool:getTraceAssertions}} call in this step is for, and one call covers up to 100 trace IDs, so make it once and before you judge anything. Each assertion comes back on its own line under its trace as \`[ID: <uuid>] checks <target>: <assertion>\`, with its pass and fail criteria in parentheses when it has them. It may also carry a Human note so an agent can preserve or edit context intended for people. **Never use a Human note as evidence for a verdict; assess only the assertion, its pass/fail criteria, and the evaluated trace.** Hold that \`[ID: <uuid>]\` value, it is the \`assertionId\` every verdict on that assertion carries. **Only assertions a person has approved come back, because only those are checked on a replay.** A trace whose assertions are all still awaiting review comes back as "no approved assertions", the same wording a trace with none gets.
 
-- **A trace that has assertions:** send one {{tool:saveAgentLabels}} entry per assertion, each carrying that assertion's \`assertionId\`, its own \`label\` (\`true\` for passed, \`false\` for failed), and its own \`annotation\`, which is the reasoning for that one assertion and nothing else. Judge each assertion on its own evidence. **Write no whole-trace verdict for that trace.** The trace verdict is derived from the per-assertion labels, so a whole-trace entry alongside them would compete with the rows it is derived from.
-- **A trace with no assertions:** nothing changes. Write the one whole-trace verdict this step has always written, \`{ traceId, label, annotation }\` with no \`assertionId\`.
+- **A trace that has assertions:** send one {{tool:saveAgentLabels}} entry per assertion, each carrying that assertion's \`assertionId\`, its own \`label\` (\`true\` for passed, \`false\` for failed), and its own \`annotation\`, which is the reasoning for that one assertion and nothing else. Judge each assertion on its own evidence. The trace verdict is derived from the per-assertion labels.
+- **A trace with no approved assertions:** write **no** verdict for it. Labels are only ever written on assertions, never directly on a trace, and {{tool:saveAgentLabels}} refuses an entry without an \`assertionId\`. Tell the user which traces have no approved assertions yet, so they can add assertions (and approve them) before those traces can be labeled.
 - **An assertion whose target cannot be resolved on the trace:** skip it, never fail it. Send \`{ traceId, assertionId, skip: true }\` and no \`label\`. A missing target means the check never ran, so recording passed or failed there is a lie the reviewer cannot see. Its sibling assertions on the same trace are still verdicted normally.
 
 **Never flatten the assertions into one entry.** No \`N/M assertion(s) passed\` tally, no \`[PASS]\` or \`[FAIL]\` lines inside an annotation, no paragraph standing in for the rows. One boolean cannot say which assertion failed, and prose cannot be filtered, counted, or trended. Each annotation covers exactly the assertion its entry names.
 
-Per-assertion entries and whole-trace verdicts travel together in the same {{tool:saveAgentLabels}} call, so batch them into as few calls as the tool's per-call limit allows.`;
+Send every trace's per-assertion entries in the same {{tool:saveAgentLabels}} call, batched into as few calls as the tool's per-call limit allows.`;
+var FIX_SCENARIO_LABEL = `**Labels are only ever written on assertions, never directly on a trace**, so the validated label goes on an assertion of the **original** trace:
+   - Call {{tool:getTraceAssertions}} with the original trace ID. If an approved assertion already states the correct behavior this bug violates, use its \`assertionId\`.
+   - Otherwise create one from the failure annotation you wrote in Phase Fix \`resolve\`: call {{tool:saveTraceAssertions}} with one entry whose \`assertion\` states the correct behavior (with \`failCriteria\` naming what the original output got wrong), then call {{tool:saveTraceAssertions}} again with \`{ id, approvalState: "approved" }\` for the id it returned. A new assertion is saved awaiting review and a label can only score an approved one, and the user has just asked to keep this scenario as a test, so approve it here and tell the user in one line that you added and approved the assertion (they can edit or reject it in Bitfab).
+   - Then call {{tool:saveHumanLabels}} once with \`[{ traceId, assertionId, label: false, annotation }]\` on the **original** trace. This records a **validated human verdict**, so the trace counts as a real regression test the instant it's written. Use \`label: false\` because the original output was the bug; the fix is what now turns it green on replay. (On the rare occasion the user is adding an already-*passing* trace to guard against future regressions, use \`label: true\`.)`;
 var REPLAY_SAFETY_CHECK = `**Mandatory pre-run replay safety check.** Complete this before executing replay for the first time, and re-run it whenever the registry entry, replay root, span boundaries, dispatch model, or mock strategy changes. Do not discover unsafe coverage by running replay: a successful email, payment, queue publish, or database write has already caused the damage.
 
 1. Read the replay registry entry and require an explicit recorded-output strategy: normally \`mock: "marked"\` / \`mock="marked"\`; \`all\` is allowed only when every matched recorded child is intentionally frozen. Never accept \`none\` for a path with unsafe external actions.
@@ -33924,6 +33928,7 @@ var assistantFlow = Flow.parse({
     getTraceAssertions: { kind: "mcp", name: "get_trace_assertions" },
     saveAgentLabels: { kind: "mcp", name: "save_agent_labels" },
     saveHumanLabels: { kind: "mcp", name: "save_human_labels" },
+    saveTraceAssertions: { kind: "mcp", name: "save_trace_assertions" },
     listDatasets: { kind: "mcp", name: "list_datasets" },
     saveDataset: { kind: "mcp", name: "save_dataset" },
     addTracesToDataset: { kind: "mcp", name: "add_traces_to_dataset" },
@@ -33940,7 +33945,7 @@ var assistantFlow = Flow.parse({
   },
   intro: `Use the local plugin MCP tools ({{tool:listFunctions}}, {{tool:searchTraces}}, {{tool:getTraces}}, {{tool:getTraceLabels}}, {{tool:getTraceAssertions}}, {{tool:saveAgentLabels}}, {{tool:listDatasets}}, {{tool:saveDataset}}, {{tool:addTracesToDataset}}, {{tool:removeTracesFromDataset}}, {{tool:listExperiments}}, {{tool:listExperimentTraces}}, {{tool:getReplayStatus}}) to find what's failing in a traced function, build a dataset of labeled traces, and iterate on the code/prompts using replay until pass rates improve.
 
-**MCP tools:** This skill uses \`list_trace_functions\`, \`search_traces\`, \`get_traces\`, \`get_trace_labels\`, \`get_trace_assertions\`, \`save_agent_labels\`, \`save_human_labels\`, \`list_datasets\`, \`save_dataset\`, \`add_traces_to_dataset\`, \`remove_traces_from_dataset\`, \`list_experiments\`, \`list_experiment_traces\`, and \`get_replay_status\` from the **local plugin MCP server** (bundled with this plugin){{#claude}}. Do NOT use the remote Bitfab MCP tools (\`mcp__Simforge__*\` or \`mcp__Bitfab__*\`), use only the \`mcp__plugin_bitfab_Bitfab__*\` variants.{{/claude}}{{^claude}}, exposed under the \`mcp__Bitfab__*\` prefix.{{/claude}}
+**MCP tools:** This skill uses \`list_trace_functions\`, \`search_traces\`, \`get_traces\`, \`get_trace_labels\`, \`get_trace_assertions\`, \`save_trace_assertions\`, \`save_agent_labels\`, \`save_human_labels\`, \`list_datasets\`, \`save_dataset\`, \`add_traces_to_dataset\`, \`remove_traces_from_dataset\`, \`list_experiments\`, \`list_experiment_traces\`, and \`get_replay_status\` from the **local plugin MCP server** (bundled with this plugin){{#claude}}. Do NOT use the remote Bitfab MCP tools (\`mcp__Simforge__*\` or \`mcp__Bitfab__*\`), use only the \`mcp__plugin_bitfab_Bitfab__*\` variants.{{/claude}}{{^claude}}, exposed under the \`mcp__Bitfab__*\` prefix.{{/claude}}
 
 {{#ask-user-question}}**Always use** \`AskUserQuestion\` **when asking questions, reporting results, or presenting choices** (two exceptions, each a report with no decision to hand the user: the \`benchmark\` scorecard is printed as Markdown tables directly in chat, not via \`AskUserQuestion\`, since tables don't render inside that UI, see Phase Benchmark; and \`share-results\` reports the experiment outcome and then routes silently by the already-approved plan, see Phase 5, so it is a plain chat report and never an \`AskUserQuestion\`, using one there would re-introduce the post-experiment prompt this flow deliberately removed). Never print a question as text and wait. Rules:{{/ask-user-question}}{{^ask-user-question}}**When the flow branches with options for the user, always present them clearly and wait for the user's answer before proceeding.** Number or letter the options so the user can pick by reference. **Two steps are reports, not option branches, and must NOT wait:** the \`benchmark\` scorecard prints its tables directly in chat, and \`share-results\` reports the experiment outcome then routes silently by the already-approved plan (see Phase 5) via auto-evaluated \`when\` conditions with no options to present, waiting there would re-introduce the post-experiment prompt this flow removes. Rules:{{/ask-user-question}}
 
@@ -34616,7 +34621,7 @@ Hold the trace ID, function key, code path, the failure annotation, and the trac
     {
       id: "add-trace",
       title: "Phase Add: Attach a Trace to a Dataset",
-      intro: "Reached only from `add-trace` mode. This is the lightweight path: attach one or more existing traces to a dataset (picking or creating one), then stop. No labeling, no diagnosis, no experiments, and no browser interaction. It does still run the `readTracesBatched` CLI command to resolve trace function keys. The traces attach **raw**; the user labels and approves them later wherever they review datasets.",
+      intro: "Reached only from `add-trace` mode. This is the lightweight path: attach one or more existing traces to a dataset (picking or creating one), then stop. No labeling, no diagnosis, no experiments, and no browser interaction. It does still run the `readTracesBatched` CLI command to resolve trace function keys. The traces attach **raw**; the user labels their assertions later wherever they review datasets.",
       stepStyle: "list",
       steps: [
         {
@@ -34680,7 +34685,7 @@ Then tell the user in one line, identifying the dataset by its \`datasetId\` (al
     {
       id: "dataset",
       title: "Phase 3: Pick a Dataset and Label Traces",
-      intro: `A **dataset** is the named bucket of labeled traces an experiment replays against. This phase picks (or creates) one for the trace function, labels candidate traces, attaches them to the dataset, then hands off to the per-dataset review page where the user approves labels and can ask the agent to add or remove traces.
+      intro: `A **dataset** is the named bucket of labeled traces an experiment replays against. This phase picks (or creates) one for the trace function, labels candidate traces, attaches them to the dataset, then hands off to the per-dataset review page where the user records their own verdicts and can ask the agent to add or remove traces.
 
 In \`dataset\` mode this phase is the entry point, Phase 1 (function picker) and Phase 2 (instrumentation/replay verification) are skipped, so the trace function key comes from the argument. Before calling any MCP tools, grep the codebase for the key (e.g. \`grep -r "<traceFunctionKey>" --include="*.ts" --include="*.tsx" --include="*.py" --include="*.rb" --include="*.go" --include="*.baml"\`) and note the file path, every later step ("Label them yourself", and Phase 4 "Read the code" in \`wizard\` mode) needs it.`,
       stepStyle: "list",
@@ -34718,7 +34723,7 @@ In \`dataset\` mode this phase is the entry point, Phase 1 (function picker) and
           branches: [
             {
               when: "the dataset already has traces (search returned results)",
-              description: `The dataset is not empty. Tell the user the linked dataset page contains the existing traces, where they can review, approve, or edit labels. Then go straight to waiting for their review. Do NOT ask how to source new candidates or offer to find more traces. The user should review what's already in the dataset first; they can request more traces in chat if needed.`,
+              description: `The dataset is not empty. Tell the user the linked dataset page contains the existing traces, where they can review labels and record their own verdict on any assertion. Then go straight to waiting for their review. Do NOT ask how to source new candidates or offer to find more traces. The user should review what's already in the dataset first; they can request more traces in chat if needed.`,
               next: "dataset/await-event"
             },
             {
@@ -34736,7 +34741,7 @@ In \`dataset\` mode this phase is the entry point, Phase 1 (function picker) and
           body: `**Ask how to source candidate traces.** Before searching, decide *where* the candidate traces come from. Three real options:
 
 1. **Define new criteria**: agent searches unlabeled traces shaped by what the user wants to surface. Best when the user has a hypothesis or a specific failure pattern in mind.
-2. **Reuse existing labels for this function**: pull traces that already have a validated human or approved-agent label (from any prior dataset on this function) and seed the new dataset with them. Best when the user wants to hill-climb off prior labeling work, same labels, different cut, add more later.
+2. **Reuse existing labels for this function**: pull traces that already have a validated (human) label (from any prior dataset on this function) and seed the new dataset with them. Best when the user wants to hill-climb off prior labeling work, same labels, different cut, add more later.
 3. **Open / you decide**: agent samples broadly with no hypothesis, ignoring prior labels for the search shape. Best for discovery passes.
 
 **Probe for prior label volume first** so the recommendation is grounded. Call {{tool:searchTraces}} with \`traceFunctionKey: <key>\`, \`validated: true\`, \`limit: 50\` to see roughly how many validated labels already exist for this function. Note the count, you'll need it for the recommendation and for option 2.
@@ -34787,7 +34792,7 @@ Hold the chosen mode in working context, the next steps branch on it.
             "ask"
           ],
           title: "Seed dataset from existing validated labels",
-          body: `**Seed dataset from existing validated labels.** Reachable only when the user picked Option B in \`ask-search-mode\`. Pull traces that already have a validated label (human-authored, or agent-authored and human-approved) for this function, attach them to the picked dataset, and route on whether the user also wants to add more.
+          body: `**Seed dataset from existing validated labels.** Reachable only when the user picked Option B in \`ask-search-mode\`. Pull traces that already have a validated (human-authored) label for this function, attach them to the picked dataset, and route on whether the user also wants to add more.
 
 1. Call {{tool:searchTraces}} with \`traceFunctionKey: <key>\`, \`validated: true\`, and a generous \`limit\` (50 is the cap). Both \`labelResult: true\` and \`labelResult: false\` matter, failures are the hill-climbing signal, but passes anchor the regression boundary. If 50 isn't enough to cover the function's labeled history, run a second call with \`labelResult: false\` only to bias toward fails first, then a third with \`labelResult: true\`. De-dupe trace IDs across calls.
 2. Call {{tool:getTraceLabels}} on the resulting trace IDs so the labels, annotations, and saved evidence are in working context. Don't re-label them, these are already validated. {{tool:getTraceLabels}} returns only label + annotation + evidence + approved (no span content) and takes up to 100 IDs per call, so the deduped set from step 1 is usually a single call; if it deduped more than 100 IDs, split them into chunks of 100 and call {{tool:getTraceLabels}} once per chunk (sequential is fine, the responses are small).
@@ -34849,13 +34854,13 @@ Hold the user's answer (the chosen option **and** any free-text detail) in worki
 
 {{branches}}
 
-Recommend Option A, an agent first pass turns the labeling page into a quick approve/edit review. But respect the user's choice: if they pick B, do **not** call {{tool:saveAgentLabels}} for any of these candidates. They want to label from scratch in the labeling page, with no agent verdicts pre-filled. If no new candidate traces were found in the previous step, skip this question and continue.`,
+Recommend Option A, an agent first pass means the user reviews verdicts that are already filled in: saving each one on the labeling page, whether they keep your pick or change it, records it as their own validated label. But respect the user's choice: if they pick B, do **not** call {{tool:saveAgentLabels}} for any of these candidates. They want to label from scratch in the labeling page, with no agent verdicts pre-filled. If no new candidate traces were found in the previous step, skip this question and continue.`,
           branches: [
             {
               option: {
                 letter: "A",
-                label: "Agent labels first, I approve / edit",
-                description: "agent makes a first pass; you approve or edit each verdict in the labeling page"
+                label: "Agent labels first, I override",
+                description: "agent records a verdict on each assertion; you keep or change each one on the labeling page, and saving it makes it your validated label"
               },
               recommended: true,
               next: "dataset/label-self"
@@ -34864,7 +34869,7 @@ Recommend Option A, an agent first pass turns the labeling page into a quick app
               option: {
                 letter: "B",
                 label: "I'll label them manually",
-                description: "no agent verdicts; you label every trace from scratch in the labeling page"
+                description: "no agent verdicts; you label each assertion yourself on the labeling page"
               },
               next: "dataset/attach-to-dataset"
             }
@@ -34883,19 +34888,19 @@ Recommend Option A, an agent first pass turns the labeling page into a quick app
           commandCalls: ["readTracesBatched"],
           title: "Agent first pass: label them yourself before opening the labeling page",
           emit: "Labeling traces",
-          body: `**Agent first pass: label them yourself before opening the labeling page**: Reachable only when the user picked Option A in the previous step. **You** label the approved candidate traces so the labeling page becomes an approve/edit review instead of a blank labeling session. Run \`{{command:readTracesBatched}} <trace-id...> --scope full\` **once** with all the approved trace IDs. It fans the reads out in parallel batches of 10 and writes the combined result to a temp file; the command prints \`{"status":"ok","outputFile":"..."}\` as JSON, so \`Read\` that \`outputFile\`. **Use this command here, not the \`get_traces\` MCP tool directly:** \`get_traces\` caps at 10 IDs, so calling it for the approved set would re-introduce the serial per-batch fan-out \`readTracesBatched\` exists to replace. Then call {{tool:getTraceAssertions}} with the same trace IDs, so you know what each trace was asked to do before you judge it. Read each trace's inputs / output / spans yourself, and decide whether it passed or failed, one decision per assertion on a trace that has assertions and one whole-trace decision on a trace that has none (the per-assertion block below fixes both shapes). **Ground your judgment in the codebase, not just the trace text.** Before you start labeling, read the instrumented function in the user's source (located in Phase 2 in \`wizard\` mode, or via the grep step in this phase's intro in \`dataset\` mode) and any nearby code that explains intent, comments, docstrings, README sections, related tests, BAML files, so you know what the function is *supposed* to do and what "good" looks like for it. Apply the same context to every trace: does this output achieve the function's goal as expressed in the code? Does it match the patterns in the already-validated traces? **{{#claude}}First decide how you'll produce the verdicts, judge serially yourself or fan the judging out per the fan-out block immediately below, based on how many candidates there are. Do not persist yet.{{/claude}}{{^claude}}Judge each candidate as the block immediately below describes. Do not persist yet.{{/claude}}** However you produce them, the verdicts then land in a single {{tool:saveAgentLabels}} call, one entry per assertion on the traces that have assertions and one whole-trace entry on the traces that have none, in the shapes the per-assertion block below fixes. **Every entry carries its own \`label\` (true for passed, false for failed) and its own \`annotation\` (a one-or-two-sentence explanation written for the human reviewer, ideally referencing what the code is trying to do).** Commit to a verdict, if you genuinely cannot decide, you didn't read the trace or the code carefully enough. The labels you save here start unapproved; they only become part of the validated dataset once a human approves them in the labeling page.
+          body: `**Agent first pass: label them yourself before opening the labeling page**: Reachable only when the user picked Option A in the previous step. **You** label the approved candidate traces' assertions so the labeling page opens with your verdicts already in place, and the user saves each one, keeping or changing your pick, which records it as their own validated label. Run \`{{command:readTracesBatched}} <trace-id...> --scope full\` **once** with all the approved trace IDs. It fans the reads out in parallel batches of 10 and writes the combined result to a temp file; the command prints \`{"status":"ok","outputFile":"..."}\` as JSON, so \`Read\` that \`outputFile\`. **Use this command here, not the \`get_traces\` MCP tool directly:** \`get_traces\` caps at 10 IDs, so calling it for the approved set would re-introduce the serial per-batch fan-out \`readTracesBatched\` exists to replace. Then call {{tool:getTraceAssertions}} with the same trace IDs, so you know what each trace was asked to do before you judge it. Read each trace's inputs / output / spans yourself, and decide whether it passed or failed, one decision per assertion on each trace that has approved assertions (the per-assertion block below fixes the shape). A trace with no approved assertions gets no verdict; tell the user it needs assertions before it can be labeled. **Ground your judgment in the codebase, not just the trace text.** Before you start labeling, read the instrumented function in the user's source (located in Phase 2 in \`wizard\` mode, or via the grep step in this phase's intro in \`dataset\` mode) and any nearby code that explains intent, comments, docstrings, README sections, related tests, BAML files, so you know what the function is *supposed* to do and what "good" looks like for it. Apply the same context to every trace: does this output achieve the function's goal as expressed in the code? Does it match the patterns in the already-validated traces? **{{#claude}}First decide how you'll produce the verdicts, judge serially yourself or fan the judging out per the fan-out block immediately below, based on how many candidates there are. Do not persist yet.{{/claude}}{{^claude}}Judge each candidate as the block immediately below describes. Do not persist yet.{{/claude}}** However you produce them, the verdicts then land in a single {{tool:saveAgentLabels}} call, one entry per assertion on the traces that have approved assertions, in the shape the per-assertion block below fixes. **Every entry carries its own \`label\` (true for passed, false for failed) and its own \`annotation\` (a one-or-two-sentence explanation written for the human reviewer, ideally referencing what the code is trying to do).** Commit to a verdict, if you genuinely cannot decide, you didn't read the trace or the code carefully enough. The labels you save here count as soon as they are written; there is no approval step. The user reviews them on the dataset page and records their own verdict on any assertion they disagree with, and only those human verdicts count as validated.
 
 ${PER_ASSERTION_VERDICTS}
 
 ${FAN_OUT_JUDGING}
 
-**Per-item inputs for this step (however you produce the verdicts):** each candidate trace's own artifacts, its input and output (from the \`{{command:readTracesBatched}} --scope full\` load above), its assertions with their ids (from the {{tool:getTraceAssertions}} call above), judged against the shared context, what the function is supposed to do and what "good" looks like for it (the instrumented function and the nearby intent you read above, plus the patterns from the already-validated traces). Each verdict is passed or failed plus its one-or-two-sentence annotation, in the \`{ traceId, assertionId, label, annotation }\` shape for a trace's assertions or the \`{ traceId, label, annotation }\` shape for a trace with none. {{#claude}}When you fan out, each subagent's prompt carries its batch's trace inputs/outputs and that batch's assertions plus that shared context, and returns its batch's entries in those same shapes, one per assertion wherever the trace has assertions, which you concatenate across batches. {{/claude}}However the verdicts are produced, you make the single {{tool:saveAgentLabels}} call (one array, all traces) just as described above. The labels still land unapproved for human review in the labeling page no matter how they were produced.
+**Per-item inputs for this step (however you produce the verdicts):** each candidate trace's own artifacts, its input and output (from the \`{{command:readTracesBatched}} --scope full\` load above), its assertions with their ids (from the {{tool:getTraceAssertions}} call above), judged against the shared context, what the function is supposed to do and what "good" looks like for it (the instrumented function and the nearby intent you read above, plus the patterns from the already-validated traces). Each verdict is passed or failed plus its one-or-two-sentence annotation, in the \`{ traceId, assertionId, label, annotation }\` shape, one per assertion. A trace with no approved assertions produces no entry. {{#claude}}When you fan out, each subagent's prompt carries its batch's trace inputs/outputs and that batch's assertions plus that shared context, and returns its batch's entries in that same shape, one per assertion, which you concatenate across batches. {{/claude}}However the verdicts are produced, you make the single {{tool:saveAgentLabels}} call (one array, all traces) just as described above. The labels count as written no matter how they were produced, and the user can override any of them with their own verdict on the dataset page. Only verdicts the user saves count as validated, so tell the user to save the verdicts they agree with too, not just change the ones they don't: the dataset confirm step needs at least one failing verdict the user saved.
 
 **The cross-trace failure-pattern synthesis stays separate.** Phase 4 (\`understand-failures\`) is a deliberate join: it reads all the labels at once so the holistic "these N traces fail the same way" view is never lost. Per-trace labeling here is mechanical and independent, {{#claude}}and when you fan out, each subagent sees only its own batch, so {{/claude}}do not fold cross-trace synthesis into the labeling, that recognition is Phase 4's job on the full set.
 
-> \u{1F6A8} **HARD RULE, DO NOT SKIP (agent-first mode only):** When the user picked Option A, you MUST call {{tool:saveAgentLabels}} with verdicts for every approved trace BEFORE sharing the labeling page link. Sending the user into an agent-first review with no pre-labeled verdicts is a process violation. (In manual mode this step is unreachable, and the rule does not apply.)
+> \u{1F6A8} **HARD RULE, DO NOT SKIP (agent-first mode only):** When the user picked Option A, you MUST call {{tool:saveAgentLabels}} with a verdict for every assertion on every approved trace that has assertions BEFORE sharing the labeling page link, and tell the user which approved traces have no approved assertions yet and so cannot be labeled until assertions are added. Sending the user into an agent-first review without the verdicts you could have written is a process violation. If none of the approved traces has approved assertions, make no {{tool:saveAgentLabels}} call (an empty batch is refused), tell the user none of these traces can be labeled until they have approved assertions, and share the labeling page link as usual. (In manual mode this step is unreachable, and the rule does not apply.)
 
-> **Made a mistake?** If you realize a verdict was wrong (e.g., you mislabeled a trace or want to re-evaluate), call {{tool:saveAgentLabels}} again with \`{ traceId, archive: true }\` for those traces, or \`{ traceId, assertionId, archive: true }\` to archive one assertion's verdict without touching its siblings. The previous label is hidden (kept for audit), and you can re-label the trace from scratch with another \`save_agent_labels\` call.`,
+> **Made a mistake?** If you realize a verdict was wrong (e.g., you mislabeled a trace or want to re-evaluate), call {{tool:saveAgentLabels}} again with \`{ traceId, assertionId, archive: true }\` to archive one assertion's verdict without touching its siblings. The previous label is hidden (kept for audit), and you can re-label that assertion with another \`save_agent_labels\` call.`,
           next: "dataset/attach-to-dataset"
         },
         {
@@ -34990,7 +34995,7 @@ Then return to the await-event step. If the user wants more edits, they'll ask a
 - **\`costRun\` is false (the common case):** call {{tool:getTraceLabels}} with all of them. It returns only the verdict fields (label + annotation + evidence + approved, no span content) and takes up to 100 IDs per call, so for almost every dataset this is a single call. Only if the dataset has more than 100 traces, split the IDs into chunks of 100 and call {{tool:getTraceLabels}} once per chunk; the responses are small, so plain sequential calls are fine.
 - **\`costRun\` is set:** run \`{{command:readTracesBatched}} <trace-id...> --scope full\` **once** with all the trace IDs instead, because the Phase 5 cost-delta step reads each original trace's recorded token usage from this load and {{tool:getTraceLabels}} does not carry it. The command fans the \`scope: "full"\` reads out in parallel batches of 10 and writes the combined result to a temp file; it prints \`{"status":"ok","outputFile":"..."}\` as JSON, so \`Read\` that \`outputFile\`. **Use this command here, not the \`get_traces\` MCP tool directly:** \`get_traces\` caps at 10 IDs, so calling it for the dataset would re-introduce the serial per-batch fan-out \`readTracesBatched\` exists to replace.
 
-**Re-read them fresh here even if you read them earlier in this phase:** the Bitfab labeling review persists human approvals and label/annotation edits to the DB, so cached context from the find / label steps can be stale. This is the working set for confirm + every Phase 5 experiment.`,
+**Re-read them fresh here even if you read them earlier in this phase:** the Bitfab labeling page persists the user's own verdicts and annotation edits to the DB, so cached context from the find / label steps can be stale. This is the working set for confirm + every Phase 5 experiment.`,
           next: "dataset/confirm-dataset"
         },
         {
@@ -34998,12 +35003,17 @@ Then return to the await-event step. If the user wants more edits, they'll ask a
           kind: "branch",
           toolCalls: ["searchTraces", "ask"],
           title: "Confirm the dataset",
-          body: `**Confirm the dataset**: Present the dataset {{presentMode}}: each entry showing (trace ID, label, annotation summary). The dataset must contain at least one **validated failing label**: i.e. at least one trace where a human either authored or approved a \`false\` label. To check, call {{tool:searchTraces}} restricted to the dataset trace IDs with \`validated: true\` and \`labelResult: false\`. Two outcomes:
+          body: `**Confirm the dataset**: Present the dataset {{presentMode}}: each entry showing (trace ID, label, annotation summary). The dataset must contain at least one **validated failing label**: i.e. at least one trace where a person recorded a \`false\` verdict on one of its assertions. To check, call {{tool:searchTraces}} restricted to the dataset trace IDs with \`validated: true\` and \`labelResult: false\`. Two outcomes:
 
 {{whens}}
 
-Unapproved agent labels do **not** satisfy this requirement by design, \`validated: true\` excludes them.`,
+Agent labels do **not** satisfy this requirement by design, \`validated: true\` counts labels a person wrote or approved.`,
           branches: [
+            {
+              when: "check fails, but the agent verdicts include failing ones the user has not saved yet",
+              description: "ask the user to save (keep) the failing verdicts they agree with on the dataset page, then wait for their review and re-check",
+              next: "dataset/await-event"
+            },
             {
               when: "check fails (no validated failing label, search returns nothing)",
               description: "tell the user and loop back to find or label more unlabeled traces",
@@ -35696,20 +35706,20 @@ Hold the verdicts in working context for the final report, the \`share-results\`
           emit: "Evaluating results",
           body: `**Evaluate against labels & annotations.** Score only items where \`item.error\` is unset. Items with \`item.error\` set are unreplayable (already classified) and go in their own bucket, never pass, fail, or regression.
 
-> \u{1F6A8} **MANDATORY, NOT A JUDGMENT CALL:** Reaching this step means verdicts can be persisted (by lineage \`originalTraceId\` or direct \`traceId\`, per \`check-verdict-persistence\`), so you MUST derive **and persist** a verdict (or one of the three explicit non-verdicts below) for **every** completed (\`item.error\` unset) replay item. \`item.error\` unset means "score this item against its criteria", it does NOT mean "it passed". Persisting is not optional, you do not get to decide a run "isn't worth scoring", "looks like noise", or "would pollute the dataset" and jump to the report with an empty scorecard. There is no "this whole run isn't meaningful, so skip labeling" path: that path does not exist, and \`verify-replay-labels\` will route you back if the persisted set doesn't cover every completed item.
+> \u{1F6A8} **MANDATORY, NOT A JUDGMENT CALL:** Reaching this step means verdicts can be persisted (by lineage \`originalTraceId\` or direct \`traceId\`, per \`check-verdict-persistence\`), so you MUST derive a verdict (or one of the three explicit non-verdicts below) for **every** completed (\`item.error\` unset) replay item, and persist one entry per assertion for every completed item whose original has approved assertions. **Agents never write whole-trace verdicts:** an item whose original has no approved assertions is judged in the chat report only, is not persisted, and is left out of the \`expected*\` list. If no completed item's original has approved assertions, there is nothing to persist: make no \`persistReplayLabels.js\` call and say so in the report. \`item.error\` unset means "score this item against its criteria", it does NOT mean "it passed". Persisting is not optional, you do not get to decide a run "isn't worth scoring", "looks like noise", or "would pollute the dataset" and jump to the report with an empty scorecard. There is no "this whole run isn't meaningful, so skip labeling" path: that path does not exist, and \`verify-replay-labels\` will route you back if the persisted set doesn't cover every completed item whose original has approved assertions.
 
 **The verdict is decided by the output against its criteria, never by your opinion of how informative, useful, or production-like the run is.** Environmental doubts, a mock/stub model, no real gateway, flaky infra, an output that "looks like noise", a worry that the labels would pollute the dataset, are **caveats only**: write them into the \`annotation\` and lower the \`confidence\` (e.g. \`Low\` / \`VeryLow\`). They never suppress a verdict and never skip this step. A low-confidence verdict with a caveat is still a verdict and still gets persisted.
 
 **Finalize the live evaluation loop; do not start from scratch.** If \`replay-against-dataset\` produced a non-empty run \`events.jsonl\`, read it before judging and reuse any prep, unreplayable buckets, verdicts, and persisted original trace IDs already produced during the running replay. Each progress event's \`item.originalTraceId\` is the original trace id: use it both to identify which dataset item settled and as the key each verdict is persisted by (item files carry it under the same \`originalTraceId\` field, plus a deprecated \`sourceTraceId\` alias). Enriched progress rows carry \`item.itemPath\`; the item file has the full input, replay output, original output, and metadata needed to judge during replay. Progress-only rows are only a trigger/prep signal. If the file is missing or empty, continue from the final complete event; older scripts remain valid and simply do not get incremental evaluation. The final verdict still comes from the completed replay item files / experiment trace data, and persistence is still keyed by the **original trace** id.
 
-When possible, evaluate and persist in small completed batches rather than waiting to judge every item at once: every time a progress event gives you a replay item ref with an original trace id, full item file, and its original label/annotation, read the item file, derive that item's verdict, and append it to the pending verdict set. Once a batch has complete coverage for its expected original trace ids, call \`persistReplayLabels.js\` for that batch. If the current SDK only exposes replay outputs at the end, use progress during the run for preparation and failed-item bucketing, then do the first persist immediately after this run's \`complete\` row appears. At the end of the replay, run the same coverage check over the complete row's item refs; any item not already persisted must be judged and persisted before continuing. This keeps Bitfab's experiment view filling in as early as the available data allows while preserving the same final correctness checks.
+When possible, evaluate and persist in small completed batches rather than waiting to judge every item at once: every time a progress event gives you a replay item ref with an original trace id, full item file, and its original label/annotation, read the item file, derive that item's verdict, and append it to the pending verdict set. Once a batch has complete coverage for its expected original trace ids (only items whose originals have approved assertions), call \`persistReplayLabels.js\` for that batch. If the current SDK only exposes replay outputs at the end, use progress during the run for preparation and failed-item bucketing, then do the first persist immediately after this run's \`complete\` row appears. At the end of the replay, run the same coverage check over the complete row's item refs; any item whose original has approved assertions and is not already persisted must be judged and persisted before continuing, and every other completed item is judged in the chat report. This keeps Bitfab's experiment view filling in as early as the available data allows while preserving the same final correctness checks.
 
 For each completed (non-errored) replay item, derive a verdict by comparing the replay's new output against the original trace's label and annotation (from Phase 3 in \`wizard\`/\`dataset\` modes, loaded by \`pick-dataset\` in Phase 5 Setup in \`experiment\` and \`benchmark\` modes; in \`fix\` mode's single-trace pass, the failure annotation held from Phase Fix \`resolve\` is the criterion, and on a later full-dataset run the added trace plus any pre-existing dataset siblings loaded by \`fix-add-to-dataset\`):
 
 - **fail**-labeled original: does the replay's new output address the annotation? If yes \u2192 \`label: true\` (PASS). If no \u2192 \`label: false\` (FAIL). Use the annotation as the acceptance criterion.
 - **pass**-labeled original: preserved \u2192 \`label: true\` (PASS). regressed \u2192 \`label: false\` (FAIL).
 
-**The ONLY permitted non-verdicts for a completed item are these three enumerated skip cases.** Each is per-item and must be accounted for explicitly with a \`{skip: true}\` entry. Nothing else, and never "the whole run isn't worth scoring", justifies omitting a completed item:
+**The ONLY permitted non-verdicts for a completed item are these three enumerated skip cases.** Each is per-item and must be accounted for explicitly: with a \`{skip: true}\` entry per assertion when the original has approved assertions, or as a skipped row in the chat report when it has none. Nothing else, and never "the whole run isn't worth scoring", justifies omitting a completed item:
 
 - **Unlabeled original** (no validated or agent label, possible in \`benchmark\` mode, where the dataset only needs \u22651 trace regardless of label mix): there is no acceptance criterion to score against, so do NOT pass/fail it. Mark it \`skip: true\` and note "unlabeled, no expected result", it counts toward \`S\` (skipped) and is excluded from \`scorable\`, never pass or fail.
 - **Fail-labeled original with an unusable annotation** (the label exists and says FAIL, but the annotation is empty or contentless, e.g. "bad", "wrong", "fix this", so it states no criterion for what "fixed" looks like): the fail label tells you the direction but not what to check the new output against. This skip is narrow and applies ONLY when BOTH hold: (a) the original is **fail**-labeled (a pass-labeled original never needs the annotation, you score it by preserved-vs-regressed against the original output, so this case does not apply), and (b) you **cannot reconstruct the defect** from the code, the function's intent, or the original's own output. If you can recover what was wrong from any of those, you have a criterion, so verdict it (a thin-but-usable annotation like "returns null instead of the email" is NOT unusable). Only when the criterion is genuinely unrecoverable: \`skip: true\` and note "fail label, annotation states no checkable criterion". This is not an escape for annotations you simply find terse or inconvenient.
@@ -35728,7 +35738,7 @@ For each completed (non-errored) replay item, derive a verdict by comparing the
 
 Keep re-replays to the minimum: only items that fail step 1, one replay each, never a full baseline arm across the dataset. If even step 2 errors or yields no usage, report that item's delta as uncomputable ("no baseline, original did not complete cleanly") instead of guessing.
 
-**The verdict you produce here lands on the REPLAY trace (not the original).** You key each verdict by the run's persist key (from \`check-verdict-persistence\`): in lineage keying, the original's \`originalTraceId\`, which the server resolves through its lineage to this run's replay trace; in direct keying, the server replay \`traceId\` directly. Either way the label lands on the replay trace, which is what makes "did this fix actually pass on replay?" queryable across iterations.
+**The per-assertion verdicts you persist here land on the REPLAY trace (not the original).** Verdicts on items whose originals have no approved assertions stay in the chat report. You key each persisted verdict by the run's persist key (from \`check-verdict-persistence\`): in lineage keying, the original's \`originalTraceId\`, which the server resolves through its lineage to this run's replay trace; in direct keying, the server replay \`traceId\` directly. Either way the label lands on the replay trace, which is what makes "did this fix actually pass on replay?" queryable across iterations.
 
 ${FAN_OUT_JUDGING}
 
@@ -35748,9 +35758,9 @@ ${FAN_OUT_JUDGING}
   "experimentId": "<experimentId>",
   "expectedOriginalTraceIds": ["<originalTraceId1>", "<originalTraceId2>", "..."],
   "verdicts": [
-    { "originalTraceId": "<originalTraceId1>", "label": true, "annotation": "Now returns the missing field; original annotation said it was empty.", "confidence": "High" },
-    { "originalTraceId": "<originalTraceId2>", "label": false, "annotation": "Output still hallucinates a tool argument.", "confidence": "VeryHigh" },
-    { "originalTraceId": "<originalTraceId3>", "skip": true }
+    { "originalTraceId": "<originalTraceId1>", "assertionId": "<assertionId1>", "label": true, "annotation": "Now returns the missing field this assertion checks for.", "confidence": "High" },
+    { "originalTraceId": "<originalTraceId2>", "assertionId": "<assertionId2>", "label": false, "annotation": "Output still hallucinates a tool argument.", "confidence": "VeryHigh" },
+    { "originalTraceId": "<originalTraceId3>", "assertionId": "<assertionId3>", "skip": true, "annotation": "The span this assertion targets is missing on the replay." }
   ]
 }
 \`\`\`
@@ -35761,38 +35771,37 @@ ${FAN_OUT_JUDGING}
 {
   "expectedTraceIds": ["<serverReplayTraceId1>", "<serverReplayTraceId2>", "..."],
   "verdicts": [
-    { "traceId": "<serverReplayTraceId1>", "label": true, "annotation": "Now returns the missing field.", "confidence": "High" },
-    { "traceId": "<serverReplayTraceId2>", "skip": true }
+    { "traceId": "<serverReplayTraceId1>", "assertionId": "<assertionId1>", "label": true, "annotation": "Now returns the missing field.", "confidence": "High" },
+    { "traceId": "<serverReplayTraceId2>", "assertionId": "<assertionId2>", "skip": true }
   ]
 }
 \`\`\`
 
-In lineage keying, \`experimentId\` is this replay run's experiment ID (from the progress rows or the final \`ReplayResult\`). The server uses it to resolve each original trace to its replay trace within this run. The \`expected*\` list MUST be the full set of ids covered by this call's batch (and across all batches, every completed \`item.error\`-unset replay item must be persisted exactly once, no fewer, per the mandatory-coverage rule above). For the final end-of-run call, use only the ids not already successfully persisted by an earlier batch. \`verdicts\` MUST cover every id in the \`expected*\` list, keyed by the same id field as that list. A trace with no assertions gets exactly one entry, either a \`{label, annotation, confidence?}\` verdict or a \`{skip: true}\` explicit skip (skips allowed only for the three enumerated skip cases above, never for an environmental doubt). A trace that has assertions gets one entry per assertion instead, in the shapes the per-assertion block below fixes. \`confidence\` is optional but recommended (\`VeryLow|Low|Medium|High|VeryHigh\`). It surfaces in the labeling UI so reviewers can prioritize low-confidence verdicts. If any expected id gets no entry at all, the script returns \`status: "missing-coverage"\` and the verify step routes you back to fill the gaps.
+In lineage keying, \`experimentId\` is this replay run's experiment ID (from the progress rows or the final \`ReplayResult\`). The server uses it to resolve each original trace to its replay trace within this run. The \`expected*\` list MUST be the full set of ids covered by this call's batch (and across all batches, every completed \`item.error\`-unset replay item whose original has approved assertions must be persisted exactly once, no fewer, per the mandatory-coverage rule above; items with no approved assertions are never persisted). For the final end-of-run call, use only the ids not already successfully persisted by an earlier batch. \`verdicts\` MUST cover every id in the \`expected*\` list, keyed by the same id field as that list. The \`expected*\` list holds only items whose originals have approved assertions, and each of those gets one entry per assertion, every entry carrying its \`assertionId\`: either a \`{label, annotation, confidence?}\` verdict or a \`{skip: true, annotation?}\` explicit skip (skips allowed only for the three enumerated skip cases above or a missing assertion target, never for an environmental doubt). The script rejects an entry without an \`assertionId\` with \`status: "invalid-input"\`. \`confidence\` is optional but recommended (\`VeryLow|Low|Medium|High|VeryHigh\`). It surfaces in the labeling UI so reviewers can prioritize low-confidence verdicts. If any expected id gets no entry at all, the script returns \`status: "missing-coverage"\` and the verify step routes you back to fill the gaps.
 
-**A replay inherits the original trace's assertions, so score them one at a time.** Before you judge this batch, call {{tool:getTraceAssertions}} once with the batch's **original** trace ids (one call covers up to 100 ids, so make it once and before you judge anything). Each assertion comes back under its trace as \`[ID: <uuid>] checks <target>: <assertion>\`, and that \`[ID: <uuid>]\` value is the \`assertionId\` its verdict carries. Only assertions a person has approved are listed, since only those are checked on a replay. A trace that comes back "no approved assertions" has nothing for you to score one at a time.
+**A replay inherits the original trace's assertions, so score them one at a time.** Before you judge this batch, call {{tool:getTraceAssertions}} once with the batch's **original** trace ids (one call covers up to 100 ids, so make it once and before you judge anything). Each assertion comes back under its trace as \`[ID: <uuid>] checks <target>: <assertion>\`, and that \`[ID: <uuid>]\` value is the \`assertionId\` its verdict carries. Only assertions a person has approved are listed, since only those are checked on a replay. A trace that comes back "no approved assertions" gets no persisted entry: judge it in the chat report only.
 
 **Never use a Human note as evidence for a verdict; assess only the assertion, its pass/fail criteria, and the evaluated trace.** The note is returned only so you can preserve or edit context intended for people.
 
-- **The original had assertions:** write one entry per assertion, each carrying that assertion's \`assertionId\`, its own \`label\`, and its own \`annotation\` covering that one assertion and nothing else. **Write no whole-trace entry for that trace.** The trace verdict is derived from the per-assertion rows, so an entry beside them contradicts the rows it comes from, and the script rejects the whole file with \`status: "invalid-input"\`.
-- **The original had none:** nothing changes. Write the single whole-trace entry with no \`assertionId\`, exactly as before.
+- **The original had assertions:** write one entry per assertion, each carrying that assertion's \`assertionId\`, its own \`label\`, and its own \`annotation\` covering that one assertion and nothing else. The trace verdict is derived from the per-assertion rows.
+- **The original had none:** write no entry and leave it out of the \`expected*\` list. Agents do not write whole-trace verdicts, so its verdict is reported in the chat only.
 - **An assertion whose target cannot be found on the trace you are judging:** \`{ "assertionId": "<uuid>", "skip": true }\` alongside the id key, never a FAIL. A missing target means the check never ran. Its sibling assertions on the same trace are still verdicted normally.
 
-Per-assertion and whole-trace entries travel in the same file and the same call, so a mixed batch is one \`verdicts\` array. Lineage keying, one trace with assertions and one without:
+Lineage keying, one trace with three assertions (a second trace with no assertions is simply absent from the file):
 
 \`\`\`json
 {
   "experimentId": "<experimentId>",
-  "expectedOriginalTraceIds": ["<originalTraceId1>", "<originalTraceId2>"],
+  "expectedOriginalTraceIds": ["<originalTraceId1>"],
   "verdicts": [
     { "originalTraceId": "<originalTraceId1>", "assertionId": "<assertionId1>", "label": true, "annotation": "The rebooked leg now names the carrier this assertion asks for.", "confidence": "High" },
     { "originalTraceId": "<originalTraceId1>", "assertionId": "<assertionId2>", "label": false, "annotation": "The seat preference is still dropped." },
-    { "originalTraceId": "<originalTraceId1>", "assertionId": "<assertionId3>", "skip": true },
-    { "originalTraceId": "<originalTraceId2>", "label": true, "annotation": "No assertions on this trace, and the output addresses the original annotation." }
+    { "originalTraceId": "<originalTraceId1>", "assertionId": "<assertionId3>", "skip": true }
   ]
 }
 \`\`\`
 
-**The script's coverage check is per trace, not per assertion.** It cannot see which traces have assertions, so it accepts either shape and only reports a trace that got no entry at all. That makes the per-assertion count yours to hold. A trace with six assertions and one entry clears the script's check and still loses five verdicts. Count the assertions you read from {{tool:getTraceAssertions}} against the entries you wrote for that trace before you run the script.
+**The script's coverage check is per trace, not per assertion.** It cannot see how many assertions each trace has, so it only reports a trace that got no entry at all. That makes the per-assertion count yours to hold. A trace with six assertions and one entry clears the script's check and still loses five verdicts. Count the assertions you read from {{tool:getTraceAssertions}} against the entries you wrote for that trace before you run the script.
 
 3. Run the script:
 
@@ -35815,12 +35824,12 @@ Per-assertion and whole-trace entries travel in the same file and the same call,
           title: "Verify replay labels persisted",
           body: `**Verify replay labels persisted.** Route on the \`status\` field of the JSON the script printed in \`evaluate-results\`. The script is the deterministic check, if it didn't return \`ok\`, the agent's verdicts are NOT yet on the replay traces and the experiment delta will be wrong on the next iteration.
 
-**Nothing reaches the report without this: \`share-results\` and the benchmark scorecard are unreachable without a \`status: "ok"\` here.** The result carries \`idKind\` (\`originalTraceId\` for lineage keying, \`traceId\` for direct keying) and a \`persistedIds\` list under that key. \`status: "ok"\` only proves the script's \`expected*\` list was all verdicted; it does NOT prove you handed the script every completed item. Before you treat \`ok\` as a pass, **cross-check the coverage yourself**: \`persistedIds\` (plus the \`item.error\` unreplayable bucket) MUST account for every item in the replay output. If any completed (\`item.error\` unset) replay item is missing from the persisted set, you under-scoped the \`expected*\` list (most often by deciding the run "wasn't worth scoring") and the scorecard would be silently empty or partial: go back to \`evaluate-results\`, rebuild the verdicts file covering ALL completed items, and re-persist before routing onward.
+**Nothing reaches the report without this: \`share-results\` and the benchmark scorecard are unreachable without a \`status: "ok"\` here, unless no completed item's original has approved assertions, in which case nothing was persisted and there is nothing to verify.** The result carries \`idKind\` (\`originalTraceId\` for lineage keying, \`traceId\` for direct keying) and a \`persistedIds\` list under that key. \`status: "ok"\` only proves the script's \`expected*\` list was all verdicted; it does NOT prove you handed the script every completed item. Before you treat \`ok\` as a pass, **cross-check the coverage yourself**: \`persistedIds\` (plus the \`item.error\` unreplayable bucket and the items whose originals have no approved assertions, which are reported in chat only) MUST account for every item in the replay output. If any completed (\`item.error\` unset) replay item whose original has approved assertions is missing from the persisted set, you under-scoped the \`expected*\` list (most often by deciding the run "wasn't worth scoring") and the scorecard would be silently empty or partial: go back to \`evaluate-results\`, rebuild the verdicts file covering ALL completed items that have approved assertions, and re-persist before routing onward.
 
 {{whens}}`,
           branches: [
             {
-              when: '`status: "ok"` AND `persistedIds` (plus the unreplayable `item.error` bucket) account for every completed replay item',
+              when: '`status: "ok"` AND `persistedIds` (plus the unreplayable `item.error` bucket and the items with no approved assertions) account for every completed replay item, or no completed item had approved assertions so nothing was persisted',
               description: "labels are persisted on the replay traces and the verdicts file is gone. In `benchmark` mode continue to the benchmark scorecard (a terminal report, no iteration); in all other modes continue to share-results (experiments page was already opened before evaluation)",
               next: {
                 byMode: {
@@ -35837,18 +35846,18 @@ Per-assertion and whole-trace entries travel in the same file and the same call,
               }
             },
             {
-              when: '`status: "ok"` but a completed (`item.error` unset) replay item is absent from `persistedIds` (your own coverage cross-check failed: the `expected*` list was under-scoped)',
-              description: "the script passed only because you handed it a truncated `expected*` list; the report would be silently empty or partial. Go back to `evaluate-results`, rebuild the verdicts file so the `expected*` list is the full set of completed replay items, and re-persist. Do NOT proceed to the report",
+              when: '`status: "ok"` but a completed (`item.error` unset) replay item whose original has approved assertions is absent from `persistedIds` (your own coverage cross-check failed: the `expected*` list was under-scoped)',
+              description: "the script passed only because you handed it a truncated `expected*` list; the report would be silently empty or partial. Go back to `evaluate-results`, rebuild the verdicts file so the `expected*` list is the full set of completed replay items whose originals have approved assertions, and re-persist. Do NOT proceed to the report",
               next: "iterate/evaluate-results"
             },
             {
               when: '`status: "missing-coverage"` (script returned a non-empty `missingIds` array)',
-              description: "you under-verdicted. Read the traces behind the missing ids, which are keyed by the run's `idKind` (in lineage mode each `missingIds` entry is the `originalTraceId` of the trace replayed from; in direct mode it is the replay trace's own server id): run `{{command:readTracesBatched}} <trace-id...> --scope full` and `Read` its `outputFile` if you didn't already, since you are persisting pass/fail verdicts on their span content. Decide each one (PASS / FAIL with annotation, or `skip: true` if genuinely ambiguous), write a NEW verdicts file at the same path covering ALL the originally expected ids (the script needs the full `expected*` list each call, not just the gaps), and re-run the script. Loop back here with the new result",
+              description: "you under-verdicted. Read the traces behind the missing ids, which are keyed by the run's `idKind` (in lineage mode each `missingIds` entry is the `originalTraceId` of the trace replayed from; in direct mode it is the replay trace's own server id): run `{{command:readTracesBatched}} <trace-id...> --scope full` and `Read` its `outputFile` if you didn't already, since you are persisting pass/fail verdicts on their span content. Decide each of their assertions (PASS / FAIL with annotation, or `skip: true` if genuinely ambiguous, every entry carrying its `assertionId`), write a NEW verdicts file at the same path covering ALL the originally expected ids (the script needs the full `expected*` list each call, not just the gaps), and re-run the script. Loop back here with the new result",
               next: "iterate/verify-replay-labels"
             },
             {
               when: '`status: "invalid-input"` (malformed verdicts JSON or missing fields)',
-              description: "the verdicts file you wrote doesn't match either accepted shape. Read the script's `message` field, fix the JSON (most common: missing annotation on a non-skip entry; a lineage file missing `experimentId` or with an empty `expectedOriginalTraceIds`; a verdict keyed by the wrong id field for the file's mode), and re-run the script. Loop back here",
+              description: "the verdicts file you wrote doesn't match either accepted shape. Read the script's `message` field, fix the JSON (most common: an entry missing its `assertionId`; missing annotation on a non-skip entry; a lineage file missing `experimentId` or with an empty `expectedOriginalTraceIds`; a verdict keyed by the wrong id field for the file's mode), and re-run the script. Loop back here",
               next: "iterate/verify-replay-labels"
             },
             {
@@ -35888,12 +35897,12 @@ Route on the replay scope and the targeted trace's result. This split matters: i
               next: "iterate/share-results"
             },
             {
-              when: "the targeted single-trace replay verdict is PASS (`label: true`)",
+              when: "the targeted single-trace replay verdict is PASS (every approved assertion passes, or, when the trace has none, your in-chat judgment is a pass)",
               description: "the target trace went green. Now add it to a dataset, then ask whether to re-run the full dataset, inspect the before/after, keep iterating, or stop",
               next: "iterate/fix-add-to-dataset"
             },
             {
-              when: "the targeted single-trace replay verdict is FAIL (`label: false`), skipped, unreplayable, shape-incompatible, or could not be persisted because the item carried no persist key (neither the lineage `originalTraceId` nor a direct server `traceId`)",
+              when: "the targeted single-trace replay verdict is FAIL (any assertion fails, or the in-chat judgment fails), skipped, unreplayable, shape-incompatible, or could not be persisted because the item carried no persist key (neither the lineage `originalTraceId` nor a direct server `traceId`)",
               description: "the fix did not prove the target trace green this pass. If `fixAddedToDataset` is not set, nothing is in a dataset yet; if it IS set (an earlier pass already added the trace to the dataset and a later iteration regressed it), the saved dataset trace is still there and this iteration just regressed it. Ask whether to inspect the before/after, keep iterating, or stop",
               next: "iterate/fix-single-trace-failed"
             }
@@ -35905,6 +35914,8 @@ Route on the replay scope and the targeted trace's result. This split matters: i
           toolCalls: [
             "read",
             "getTraces",
+            "getTraceAssertions",
+            "saveTraceAssertions",
             "saveHumanLabels",
             "listDatasets",
             "saveDataset",
@@ -35929,7 +35940,7 @@ So the prompt is **two** options when none exist yet (Create a new dataset \xB7
 
 Once a dataset is chosen (existing or newly created), save the scenario, in order:
 
-1. **Label the original trace as a validated fail.** Using the failure annotation you wrote in Phase Fix \`resolve\` (what the original output got wrong and what correct behavior is), call {{tool:saveHumanLabels}} once with \`[{ traceId, label: false, annotation }]\` on the **original** trace. This records a **validated human verdict** (not an unapproved agent suggestion), so the trace counts as a real regression test the instant it's written, with no Bitfab approval step. Use \`label: false\` because the original output was the bug; the fix is what now turns it green on replay. (On the rare occasion the user is adding an already-*passing* trace to guard against future regressions, use \`label: true\`.)
+1. **Label the original trace as a validated fail.** ${FIX_SCENARIO_LABEL}
 2. **Attach the trace.** Call {{tool:addTracesToDataset}} with the \`datasetId\` and \`[traceId]\` (idempotent, a safe no-op if already present). **Confirm the attach landed**: the trace is in the dataset when it was added now or was already present. If the tool reports it skipped the trace (e.g. a scope mismatch), do **not** claim it was added or set \`fixAddedToDataset\`, re-pick a dataset scoped to this function and attach again before continuing.
 3. **Record whether the dataset has siblings, and load their labels if so.** Note whether the dataset already held other traces besides the one you just attached, and hold that as \`datasetHasSiblings\` (the next step uses it to decide whether re-running the whole dataset is meaningful). If there **are** siblings, the user may next choose to re-run the full dataset, so run \`{{command:readTracesBatched}} <trace-id...> --scope full\` over the dataset's trace IDs and \`Read\` the \`outputFile\` it prints, so every sibling's validated label + annotation is in context for that experiment. **Skip the load** when you just created the dataset or it held only this trace (no siblings).
 
@@ -36070,6 +36081,8 @@ When the user picks **Show in Bitfab**, open it with \`fixSingleTraceExperimentG
           id: "fix-save-unfixed",
           kind: "action",
           toolCalls: [
+            "getTraceAssertions",
+            "saveTraceAssertions",
             "saveHumanLabels",
             "listDatasets",
             "saveDataset",
@@ -36084,7 +36097,7 @@ When the user picks **Show in Bitfab**, open it with \`fixSingleTraceExperimentG
 
 Otherwise add it now:
 
-1. **Label the original trace as a validated fail** using the failure annotation from Phase Fix \`resolve\`: call {{tool:saveHumanLabels}} once with \`[{ traceId, label: false, annotation }]\` on the **original** trace.
+1. **Label the original trace as a validated fail** using the failure annotation from Phase Fix \`resolve\`. ${FIX_SCENARIO_LABEL}
 2. **Pick or create a dataset:** if the user named a dataset ID, use it (first confirm it's scoped to this function via {{tool:listDatasets}}; if it belongs to a different function, {{tool:addTracesToDataset}} would silently skip every trace, so {{askVerb}} and offer a correctly-scoped dataset instead). Otherwise call {{tool:listDatasets}} with the function key and ask where to save, **keeping the prompt to at most four options, never one per dataset**: **none exist** \u2192 create silently; **exactly one** \u2192 offer "Add to \`<name>\`" (recommended) and "Create a new dataset"; **two or more** \u2192 offer, in this order, "Add to \`<recommended>\`" (the most recently used dataset with traces), "Choose a different dataset" (which then presents a second {{presentMode}} listing existing datasets, most recently used first, so the user can route to a specific one), then "Create a new dataset". Hold the \`datasetId\`.
 3. **Attach the trace** with {{tool:addTracesToDataset}} (\`datasetId\`, \`[traceId]\`; idempotent). **Confirm the attach landed**: the trace is saved when it was added now or was already present. If the tool reports it skipped the trace (for example, the dataset is scoped to a different function or the trace is not in this org), do **not** claim save and do **not** set \`fixAddedToDataset\`; re-pick a dataset scoped to this function and attach again before stopping.
 
@@ -36239,7 +36252,7 @@ This is a single-trace, in-chat path: run the replay directly, no progress-bar w
 
 There is no verdict to persist for an errored item. Offer a retry only after the diagnosed cause is addressed, or offer to stop.
 
-**If the replay completed**, call {{tool:getTraceAssertions}} with the ORIGINAL trace id first. An assertion says what the user asked this one case to do, and the replay inherits the original's assertions, so it is what the new output is measured against. Each one comes back as \`[ID: <uuid>] checks <target>: <assertion>\`, and that \`[ID: <uuid>]\` value is the \`assertionId\` its verdict carries. Only assertions a person has approved are listed, since only those are checked on a replay. "no approved assertions" means the trace has nothing to measure against, and everything below reads exactly as it always has.
+**If the replay completed**, call {{tool:getTraceAssertions}} with the ORIGINAL trace id first. An assertion says what the user asked this one case to do, and the replay inherits the original's assertions, so it is what the new output is measured against. Each one comes back as \`[ID: <uuid>] checks <target>: <assertion>\`, and that \`[ID: <uuid>]\` value is the \`assertionId\` its verdict carries. Only assertions a person has approved are listed, since only those are checked on a replay. "no approved assertions" means the one-line verdict below stays in the chat and nothing is persisted.
 
 **Never use a Human note as evidence for a verdict; assess only the assertion, its pass/fail criteria, and the evaluated trace.** The note is returned only so you can preserve or edit context intended for people.
 
@@ -36249,20 +36262,11 @@ Then compare the new output against the original trace's assertions, label, and
 - Original was **pass**: preserved \u2192 "**Pass**: output unchanged in quality." regressed \u2192 "**Regressed**: was passing, now <what broke>."
 - No label on the original: show a short before/after diff and summarize whether it looks better.
 
-**Then persist that verdict onto the replay trace.** The pass/fail you just reported is a local label, save it so it survives the session and lands on the replay trace, exactly like the full replay path. This is not optional when persistence is possible: displaying a verdict and dropping it is the bug this step exists to prevent.
-
-- **If the completed item's trace id is non-null** (its value is already the SERVER replay trace id, per the \`run\` step): persist against it directly, no \`get_replay_status\` call. Write a one-entry verdicts file to an absolute path under \`<repoRoot>/.bitfab/tmp/\` (\`<repoRoot>\` = \`git rev-parse --show-toplevel\`; create the dir if missing) and run the persist script. **The verdicts file keys are the command's fixed camelCase contract (\`expectedTraceIds\`, \`traceId\`) regardless of the SDK language, its VALUE is the server trace id you held from the replay output (\`traceId\` in TS, \`trace_id\` in Python/Ruby):**
+**Then, when the original has approved assertions, persist one verdict per assertion onto the replay trace.** Save them so they survive the session and land on the replay trace, exactly like the full replay path. This is not optional when persistence is possible. **When the original has no approved assertions, persist nothing:** agents do not write whole-trace verdicts, so the one-line verdict stays in the chat.
 
-  \`\`\`json
-  {
-    "expectedTraceIds": ["<server-trace-id>"],
-    "verdicts": [
-      { "traceId": "<server-trace-id>", "label": true, "annotation": "<the same one-line reason you reported above>", "confidence": "High" }
-    ]
-  }
-  \`\`\`
+- **If the completed item's trace id is non-null** (its value is already the SERVER replay trace id, per the \`run\` step): persist against it directly, no \`get_replay_status\` call. Write a verdicts file to an absolute path under \`<repoRoot>/.bitfab/tmp/\` (\`<repoRoot>\` = \`git rev-parse --show-toplevel\`; create the dir if missing) and run the persist script. **The verdicts file keys are the command's fixed camelCase contract (\`expectedTraceIds\`, \`traceId\`) regardless of the SDK language, its VALUE is the server trace id you held from the replay output (\`traceId\` in TS, \`trace_id\` in Python/Ruby):**
 
-  **If the original had assertions**, that one entry becomes one entry per assertion instead, each carrying its \`assertionId\`, its own \`label\`, and its own \`annotation\` for that assertion alone, and the file carries **no** whole-trace entry for the trace. The trace verdict is derived from the per-assertion rows, so sending both makes the script reject the file with \`status: "invalid-input"\`. An assertion whose target cannot be found on the replay trace gets \`{ "traceId": "<server-trace-id>", "assertionId": "<uuid>", "skip": true }\`, never a FAIL, and its siblings are still verdicted:
+  The file carries one entry per assertion, each carrying its \`assertionId\`, its own \`label\`, and its own \`annotation\` for that assertion alone. The script rejects an entry without an \`assertionId\` with \`status: "invalid-input"\`. An assertion whose target cannot be found on the replay trace gets \`{ "traceId": "<server-trace-id>", "assertionId": "<uuid>", "skip": true }\`, never a FAIL, and its siblings are still verdicted:
 
   \`\`\`json
   {
@@ -36278,9 +36282,9 @@ Then compare the new output against the original trace's assertions, label, and
   {{command:persistReplayLabels}} <repoRoot>/.bitfab/tmp/verdicts-<experiment-id>.json
   \`\`\`
 
-  \`label\` is \`true\` for Pass, \`false\` for Still-failing / Regressed. Read the script's single JSON status line: \`ok\` means the verdict is now on the replay trace, add "\xB7 saved" to your one-line report.
+  Each entry's \`label\` is \`true\` when that assertion passes on the replay and \`false\` when it fails. Read the script's single JSON status line: \`ok\` means the verdicts are now on the replay trace, add "\xB7 saved" to your one-line report.
   - **If the completed item's trace id is \`null\`** (old server/SDK that returns no server-trace-id mapping, from the \`run\` step's note): persistence is impossible. Keep the verdict in-chat only and tell the user once: "This replay didn't return a server trace ID, so the verdict can't be saved. Upgrade the SDK/server." Don't block the flow on it.
-- **No-label original with no assertions either** (you showed a before/after diff, no pass/fail): there's no verdict to persist, just report the diff. An unlabeled original that HAS assertions is not this case, the assertions are the criteria, so score them one per assertion and persist them.
+- **Original with no approved assertions:** there's nothing to persist, just report the one-line verdict (or, for a no-label original, the before/after diff). An unlabeled original that HAS assertions is not this case, the assertions are the criteria, so score them one per assertion and persist them.
 
 {{branches}}`,
           branches: [
@@ -36660,7 +36664,7 @@ Each replay pushes a temporary bitfab-replay/<UUID> branch. For each push-trigge
 
 Verify with the replay command the user already runs. First add --cloud --cloud-preview, which pushes nothing and lists the changed and new files the snapshot would carry; review them for anything that should not leave the machine, since filename screening is not a secret scanner, and have the user gitignore it. Once the workflow is merged and secrets are set, and remote runs are authorized, run the same command with --cloud --dry-run: a real run that replays nothing but receives the secrets, stops on any empty secret, runs the --check command, and resolves the traces. Then, only when a replay is authorized, run with --cloud alone.
 
-From an agent session always add --cloud-detach, since a replay can run for hours, and follow it with --cloud-watch <UUID> (prints the replay's progress and result summary and exits with its code), --cloud-status, --cloud-cancel, or --cloud-cleanup. --cloud-timeout <minutes> stops one replay early and keeps finished traces. Report separately whether the workflow is written, secrets are set, the workflow is merged, the preview was reviewed, the dry run passed, and a real replay completed.`,
+From an agent session always add --cloud-detach, since a replay can run for hours, and follow it with --cloud-watch <UUID> (prints the replay's progress and result summary and exits with its code), --cloud-status, --cloud-cancel, or --cloud-cleanup. --cloud-timeout <minutes> stops one replay early and keeps finished traces. --cloud-env NAME=VALUE (repeatable) sets a non-secret environment variable for one replay; secrets go in the workflow's env and --cloud-secrets. Report separately whether the workflow is written, secrets are set, the workflow is merged, the preview was reviewed, the dry run passed, and a real replay completed.`,
           next: "cleanup/finish"
         }
       ]

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bitfab-cli",
-  "version": "0.2.422",
+  "version": "0.2.424",
   "description": "Install and configure the Bitfab plugin in Claude Code, Codex, or Cursor, or the Bitfab skill pack in Amp (Alpha).",
   "type": "module",
   "main": "dist/index.js",