bitfab-cli 0.2.307 → 0.2.309

package/dist/index.js

@@ -26936,10 +26936,10 @@ var semver3 = __toESM(require_semver2(), 1);
 
 // ../bitfab-plugin-lib/dist/bakedSdkVersions.js
 var BAKED_SDK_VERSIONS = {
-  typescript: "0.44.0",
-  python: "0.42.3",
-  ruby: "0.42.0",
-  go: "0.41.0"
+  typescript: "0.44.1",
+  python: "0.44.1",
+  ruby: "0.43.1",
+  go: "0.41.1"
 };
 
 // ../bitfab-plugin-lib/dist/installedSdk.js
@@ -32827,9 +32827,9 @@ cd <project-dir> && {{command:replayProgress}} --label <pipeline-name> --run-dir
 
 3. **Run the live replay/evaluation loop while the replay is still running.** Use the run directory you passed with \`--run-dir\` and poll \`<run-dir>/events.jsonl\` alongside the human output. Each \`type: "progress"\` JSONL row is a normalized SDK item-finish event; \`item.originalTraceId\` is the **original trace id, the key verdicts are persisted by** (every item file carries it under the same \`originalTraceId\` field, plus a deprecated \`sourceTraceId\` alias). On modern SDKs, \`item.traceId\` may already hold the server replay trace ID after the per-item flush; it is null only when that flush/readback could not confirm the ID, and the final persistence barrier/result fills it when possible. \`item.replayTraceId\` is only set by older SDKs. Treat a new successful progress row as "this original trace's replay item finished" and start evaluation for that item as soon as the row carries \`item.itemPath\` and an \`item.originalTraceId\`: read \`item.itemPath\` for the full input, replay output, original output, and metadata. Failed progress rows become unreplayable candidates immediately and should be carried forward with their error string. If the row is progress-only (old SDK/current basic reporter: no \`item.itemPath\`), do the cheap prep only and defer judging to the final \`type: "complete"\` row's item refs. If the JSONL file is missing or empty (old SDK, old replay script, unwired \`onItemFinish\` / \`on_item_finish\`, or an unwritable log path), do not block or treat it as a replay failure: continue from the final complete event exactly as older scripts did from the old result file.
 
-   For every item you can judge during the run, key the verdict by its **original trace id** (the \`originalTraceId\` field in each item file, the same \`item.originalTraceId\` on progress rows), the original trace the item was replayed from, which the server resolves to this run's replay trace, no local-to-server id mapping step. You need the run's \`testRunId\` to persist (enriched progress rows may carry \`event.testRunId\`; older scripts may only reveal it in the final \`ReplayResult\`). As soon as a small batch of judged items has its original trace ids and you know the \`testRunId\`, persist that batch with \`{{command:persistReplayLabels}}\`. Keep a set of original trace IDs already persisted so the final reconciliation never double-writes a verdict.
+   For every item you can judge during the run, key the verdict by its **original trace id** (the \`originalTraceId\` field in each item file, the same \`item.originalTraceId\` on progress rows), the original trace the item was replayed from, which the server resolves to this run's replay trace, no local-to-server id mapping step. You need the run's \`testRunId\` to persist (enriched progress rows may carry \`event.testRunId\`; older scripts may only reveal it in the final \`ReplayResult\`). As soon as a small batch of judged items has its original trace ids and you know the \`testRunId\`, persist that batch with \`{{command:persistReplayLabels}}\`. Keep a set of original trace IDs already persisted so the final reconciliation never double-writes a verdict. A verdict judged live was judged before the item's outlines existed (\`originalTraceOutline\` and \`traceOutline\` are \`null\` on progress item files), so also keep the set of original trace ids judged without outlines; step 4 re-checks those against the final item files.
 
-4. **When the background command finishes, read the final \`type: "complete"\` row from this run's \`events.jsonl\`**. Its \`result\` carries run metadata (\`testRunId\`, \`testRunUrl\`, \`itemCount\`) and its \`items\` array carries item refs. Read each needed \`items[].itemPath\` for the full replay item (trace ID, duration, tokens, model, and full original/new outputs). Read from the **files**, not from the captured command output, which the harness truncates in the middle.
+4. **When the background command finishes, read the final \`type: "complete"\` row from this run's \`events.jsonl\`**. Its \`result\` carries run metadata (\`testRunId\`, \`testRunUrl\`, \`itemCount\`) and its \`items\` array carries item refs. Read each needed \`items[].itemPath\` for the full replay item (trace ID, duration, tokens, model, full original/new outputs, and the two trace outlines). Read from the **files**, not from the captured command output, which the harness truncates in the middle. On current SDKs each completed item also carries \`originalTraceOutline\` and \`traceOutline\`: the original and the replayed trace's span tree (span names, types, nesting, order, durations, tokens, model, errors, and whether each span was mocked) with no inputs or outputs. The SDK fills both in at completion from the server, so they are \`null\` on live progress rows and on older SDKs. Use them to judge execution shape (did the replay call the same tools in the same order, did a span that used to run get mocked or skipped, did a child span error) straight from the item file, instead of reading both traces back from the server for that comparison. **Re-check every item you judged live:** its verdict was derived before the outlines existed, so re-read its final item file (the complete row's write fills both outlines) and run the execution-shape comparison now. When that comparison changes the verdict, include the corrected \`{ originalTraceId, label, annotation }\` in the next \`{{command:persistReplayLabels}}\` batch: a new agent verdict for the same original trace replaces the earlier one in place, so this correction is the one case where re-persisting an already-persisted id is right. When it does not change the verdict, leave the persisted verdict alone.
 
 **Before running: verify the replay script prints the full original and new output values AND at least one verdict persist key (\`item.originalTraceId\` or \`item.traceId\`) for every item** (not just lengths, counts, hashes, or truncated previews) so the run's \`items/*.json\` files carry them. Modern items may carry both: prefer lineage persistence by \`item.originalTraceId\` (older SDKs may print it under the deprecated \`sourceTraceId\` alias), and fall back to the server replay \`item.traceId\` only when original lineage is absent. After an item's replay trace is flushed, its \`item.traceId\` is available to lifecycle callbacks and in the final result. The oldest SDKs have no \`item.originalTraceId\` and persist by \`item.traceId\`. If the script prints neither, fix it first; the Replay Output Contract and example script live in the SDK reference at \`https://docs.bitfab.ai/<language>-sdk.md\`. Subagents can't evaluate an improvement from \`5 \u2192 7 (+2)\`, and an item that carries no persist key blocks verdict persistence for that item.
 
@@ -33050,7 +33050,7 @@ When the server reports \`completed\` with \`traceCount > 0\`, call {{tool:listE
 
 If \`replay-against-dataset\` already consumed a non-empty run \`events.jsonl\`, reuse that progress-derived work and any verdicts already produced from item files referenced by progress rows. Successful progress events identify which original traces finished, and failed events already define unreplayable candidates. Do not reclassify those failures as output regressions. If the progress file is missing or empty, fall back to the completed replay output; this is expected for old SDKs or replay scripts without \`onItemFinish\` / \`on_item_finish\`. Since no persist key is available on this path, verdicts can't be persisted and stay in working context only.
 
-For each completed (non-errored) replay item, derive a verdict by comparing the replay's new output against the original trace's label and annotation:
+For each completed (non-errored) replay item, derive a verdict by comparing the replay's new output against the original trace's label and annotation, and when the item file carries \`originalTraceOutline\` and \`traceOutline\`, also compare the two span trees so a replay that produced the right text by skipping a required tool call, erroring in a child span, or leaning on a mocked span that used to run real code does not pass on output alone:
 
 - **fail**-labeled original: does the replay's new output address the annotation? If yes, mark as PASS. If no, mark as FAIL.
 - **pass**-labeled original: preserved means PASS, regressed means FAIL.
@@ -33084,11 +33084,11 @@ Hold the verdicts in working context for the final report, the \`share-results\`
 
 **The verdict is decided by the output against its criteria, never by your opinion of how informative, useful, or production-like the run is.** Environmental doubts, a mock/stub model, no real gateway, flaky infra, an output that "looks like noise", a worry that the labels would pollute the dataset, are **caveats only**: write them into the \`annotation\` and lower the \`confidence\` (e.g. \`Low\` / \`VeryLow\`). They never suppress a verdict and never skip this step. A low-confidence verdict with a caveat is still a verdict and still gets persisted.
 
-**Finalize the live evaluation loop; do not start from scratch.** If \`replay-against-dataset\` produced a non-empty run \`events.jsonl\`, read it before judging and reuse any prep, unreplayable buckets, verdicts, and persisted original trace IDs already produced during the running replay. Each progress event's \`item.originalTraceId\` is the original trace id: use it both to identify which dataset item settled and as the key each verdict is persisted by (item files carry it under the same \`originalTraceId\` field, plus a deprecated \`sourceTraceId\` alias). Enriched progress rows carry \`item.itemPath\`; the item file has the full input, replay output, original output, and metadata needed to judge during replay. Progress-only rows are only a trigger/prep signal. If the file is missing or empty, continue from the final complete event; older scripts remain valid and simply do not get incremental evaluation. The final verdict still comes from the completed replay item files / experiment trace data, and persistence is still keyed by the **original trace** id.
+**Finalize the live evaluation loop; do not start from scratch.** If \`replay-against-dataset\` produced a non-empty run \`events.jsonl\`, read it before judging and reuse any prep, unreplayable buckets, verdicts, and persisted original trace IDs already produced during the running replay. Each progress event's \`item.originalTraceId\` is the original trace id: use it both to identify which dataset item settled and as the key each verdict is persisted by (item files carry it under the same \`originalTraceId\` field, plus a deprecated \`sourceTraceId\` alias). Enriched progress rows carry \`item.itemPath\`; the item file has the full input, replay output, original output, and metadata needed to judge during replay (its \`originalTraceOutline\` and \`traceOutline\` are still \`null\` mid-run and are filled by the final write, so an execution-shape comparison waits for the complete row). Progress-only rows are only a trigger/prep signal. If the file is missing or empty, continue from the final complete event; older scripts remain valid and simply do not get incremental evaluation. The final verdict still comes from the completed replay item files / experiment trace data, and persistence is still keyed by the **original trace** id.
 
-When possible, evaluate and persist in small completed batches rather than waiting to judge every item at once: every time a progress event gives you a replay item ref with an original trace id, full item file, and its original label/annotation, read the item file, derive that item's verdict, and append it to the pending verdict set. Once a batch has complete coverage for its expected original trace ids, call \`persistReplayLabels.js\` for that batch. If the current replay script/SDK only exposes replay outputs at the end, use progress during the run for preparation and failed-item bucketing, then do the first persist immediately after this run's \`complete\` row appears. At the end of the replay, run the same coverage check over the complete row's item refs; any item not already persisted must be judged and persisted before continuing. This keeps Studio's experiment view filling in as early as the available data allows while preserving the same final correctness checks.
+When possible, evaluate and persist in small completed batches rather than waiting to judge every item at once: every time a progress event gives you a replay item ref with an original trace id, full item file, and its original label/annotation, read the item file, derive that item's verdict, and append it to the pending verdict set. Once a batch has complete coverage for its expected original trace ids, call \`persistReplayLabels.js\` for that batch. If the current replay script/SDK only exposes replay outputs at the end, use progress during the run for preparation and failed-item bucketing, then do the first persist immediately after this run's \`complete\` row appears. At the end of the replay, run the same coverage check over the complete row's item refs; any item not already persisted must be judged and persisted before continuing, and any item judged live before its outlines existed must be re-checked against the final item file's \`originalTraceOutline\` and \`traceOutline\`, re-persisting only when that comparison changes the verdict. This keeps Studio's experiment view filling in as early as the available data allows while preserving the same final correctness checks.
 
-For each completed (non-errored) replay item, derive a verdict by comparing the replay's new output against the original trace's label and annotation (from Phase 3 in \`wizard\`/\`dataset\` modes, loaded by \`pick-dataset\` in Phase 5 Setup in \`experiment\` and \`benchmark\` modes; in \`fix\` mode's single-trace pass, the failure annotation held from Phase Fix \`resolve\` is the criterion, and on a later full-dataset run the added trace plus any pre-existing dataset siblings loaded by \`fix-add-to-dataset\`):
+For each completed (non-errored) replay item, derive a verdict by comparing the replay's new output against the original trace's label and annotation, and when the item file carries \`originalTraceOutline\` and \`traceOutline\`, also compare the two span trees so a replay that produced the right text by skipping a required tool call, erroring in a child span, or leaning on a mocked span that used to run real code does not pass on output alone (from Phase 3 in \`wizard\`/\`dataset\` modes, loaded by \`pick-dataset\` in Phase 5 Setup in \`experiment\` and \`benchmark\` modes; in \`fix\` mode's single-trace pass, the failure annotation held from Phase Fix \`resolve\` is the criterion, and on a later full-dataset run the added trace plus any pre-existing dataset siblings loaded by \`fix-add-to-dataset\`):
 
 - **fail**-labeled original: does the replay's new output address the annotation? If yes \u2192 \`label: true\` (PASS). If no \u2192 \`label: false\` (FAIL). Use the annotation as the acceptance criterion.
 - **pass**-labeled original: preserved \u2192 \`label: true\` (PASS). regressed \u2192 \`label: false\` (FAIL).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bitfab-cli",
-  "version": "0.2.307",
+  "version": "0.2.309",
   "description": "Install and configure the Bitfab plugin in Claude Code, Codex, or Cursor, or the Bitfab skill pack in Amp (Alpha).",
   "type": "module",
   "main": "dist/index.js",