bitfab-cli 0.2.428 → 0.2.429

package/dist/index.js

@@ -35729,14 +35729,14 @@ For each completed (non-errored) replay item, derive a verdict by comparing the
 
 **When \`costRun\` is set, also capture the token delta for each completed item.** The run's goal is fewer tokens, so a verdict is not finished without the cost direction. **First make sure each original's recorded token usage is actually in your context.** It normally arrives with the dataset load, but if build-dataset loaded this dataset via {{tool:getTraceLabels}} (\`costRun\` was false at build time and only flipped on afterward), that load carried labels but no token usage: read the dataset's original trace IDs once by running \`{{command:readTracesBatched}} <trace-id...> --scope summary\` (\`summary\` is enough for the token totals) and \`Read\` the \`outputFile\` it prints, before computing deltas. This is a cheap read, not a re-replay, so do it before the missing-usage recovery ladder below. Then compare each original trace's recorded token usage against the replay item's \`tokens\` from the replay output: per item, record input and output tokens baseline (original) vs new (replay) and the % change. Carry the per-item deltas plus the dataset totals (baseline vs new input/output tokens and overall % change) forward for the report. A pass that costs more tokens is a cost regression even when the output is still correct: flag it rather than burying it under a green verdict. (If the replay output reports only total tokens, report the total delta; if it splits input and output, report both.) **Compute the delta on the basis the run carries (\`costBasis\`), so the chat numbers match the page lens.** On \`costBasis = uncached\`, subtract each side's cached input first: the per-side figure is \`(input - cached) + output\`, taking \`cached\` from the original's recorded usage and from the replay item's \`tokens\` (which carries \`input\`, \`output\`, and \`cached\`), and label the reported figures "uncached" so they read the same as the cyan/orange page trend. An item with no cached tokens then equals \`input + output\` on its own, exactly as the page shows it. On \`costBasis = all\` (the default), report \`input + output\` with no subtraction. Carry the per-item figures forward already on the chosen basis so \`share-results\` and the scorecard inherit it without re-deciding.
 
-**Never run a separate baseline replay arm.** The baseline is the original traces' recorded usage, full stop. There is exactly one replay per item (the changed-code arm). Do not replay the original/unchanged code to "establish a baseline"; the original trace already is that measurement. The only exception is the missing-usage recovery below, and even that re-runs only the affected items, never the whole dataset.
+**Never run a separate baseline experiment.** The baseline is the original traces' recorded usage, full stop. There is exactly one replay per item (the changed-code experiment). Do not replay the original/unchanged code to "establish a baseline"; the original trace already is that measurement. The only exception is the missing-usage recovery below, and even that re-runs only the affected items, never the whole dataset.
 
 **Handle originals that aren't a usable baseline.** An original trace is not a trustworthy baseline if it errored, has a failed or incomplete status, or recorded no token usage. In any of those cases its numbers don't reflect a clean run of the unchanged code. Re-replaying the unchanged code is expensive and is the LAST resort, never the first move. For each such item, walk this order and stop at the first step that yields a baseline:
 
 1. **Reuse a recorded run.** Check whether any other already-recorded trace or prior replay of the same item ran the unchanged code cleanly and has usage (e.g. an earlier good trace for the same input, or a prior baseline from an earlier iteration). If so, use it. No new run.
 2. **Backfill only the truly unrecoverable items.** Only if no clean recorded usage exists anywhere for that item, re-replay the unchanged code for that one item.
 
-Keep re-replays to the minimum: only items that fail step 1, one replay each, never a full baseline arm across the dataset. If even step 2 errors or yields no usage, report that item's delta as uncomputable ("no baseline, original did not complete cleanly") instead of guessing.
+Keep re-replays to the minimum: only items that fail step 1, one replay each, never a full baseline experiment across the dataset. If even step 2 errors or yields no usage, report that item's delta as uncomputable ("no baseline, original did not complete cleanly") instead of guessing.
 
 **The per-assertion verdicts you persist here land on the REPLAY trace (not the original).** Verdicts on items whose originals have no approved assertions stay in the chat report. You key each persisted verdict by the run's persist key (from \`check-verdict-persistence\`): in lineage keying, the original's \`originalTraceId\`, which the server resolves through its lineage to this run's replay trace; in direct keying, the server replay \`traceId\` directly. Either way the label lands on the replay trace, which is what makes "did this fix actually pass on replay?" queryable across iterations.
 
@@ -36411,7 +36411,7 @@ Use \u2705 for pass-verdict rows (fixed, still-passing), \u274C for fail-verdict
 
 **If running in text-only mode** (trace IDs were unavailable): append a one-line note under the tables that persistent results require upgrading to \`@bitfab/sdk\` 0.13.5+.
 
-**When \`costRun\` is set, add token cost to both tables.** In Table 1, add \`Input tokens | base \u2192 new (\xB1X%)\` and \`Output tokens | base \u2192 new (\xB1Y%)\` rows (total tokens only if the replay output does not split them). In Table 2, add a \`Tokens\` column showing each trace's \`original \u2192 replay\` total and % change. For a cost benchmark the headline is the token delta, not the pass rate alone: report both. Baseline tokens come from each original trace's recorded usage. For originals that errored, failed, or recorded no usage, apply the same cheapest-first recovery as the evaluate step (reuse a clean recorded run, else a one-off per-item backfill replay of the unchanged code), never a dataset-wide baseline arm; new tokens come from the replay output. Where no baseline could be recovered, show the cell as "no baseline" rather than a fabricated delta. Use the run's basis (\`costBasis\`), matching the page lens: on \`uncached\`, the Input row and the Tokens column are the uncached figures (\`(input - cached) + output\`, \`input - cached\` for the input row) and the table labels the row/column "uncached"; on \`all\`, raw \`input + output\`. Don't mix bases within a scorecard.
+**When \`costRun\` is set, add token cost to both tables.** In Table 1, add \`Input tokens | base \u2192 new (\xB1X%)\` and \`Output tokens | base \u2192 new (\xB1Y%)\` rows (total tokens only if the replay output does not split them). In Table 2, add a \`Tokens\` column showing each trace's \`original \u2192 replay\` total and % change. For a cost benchmark the headline is the token delta, not the pass rate alone: report both. Baseline tokens come from each original trace's recorded usage. For originals that errored, failed, or recorded no usage, apply the same cheapest-first recovery as the evaluate step (reuse a clean recorded run, else a one-off per-item backfill replay of the unchanged code), never a dataset-wide baseline experiment; new tokens come from the replay output. Where no baseline could be recovered, show the cell as "no baseline" rather than a fabricated delta. Use the run's basis (\`costBasis\`), matching the page lens: on \`uncached\`, the Input row and the Tokens column are the uncached figures (\`(input - cached) + output\`, \`input - cached\` for the input row) and the table labels the row/column "uncached"; on \`all\`, raw \`input + output\`. Don't mix bases within a scorecard.
 
 This is a terminal step. Report the scorecard and stop. Do not offer to iterate or make changes (the user can run \`{{cmd}}assistant experiment <key>\` separately if they want to fix failures).`,
           next: "cleanup/finish"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bitfab-cli",
-  "version": "0.2.428",
+  "version": "0.2.429",
   "description": "Install and configure the Bitfab plugin in Claude Code, Codex, or Cursor, or the Bitfab skill pack in Amp (Alpha).",
   "type": "module",
   "main": "dist/index.js",