bitfab-cli 0.2.494 → 0.2.495
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/index.js +9 -8
- package/package.json +1 -1
package/dist/index.js
CHANGED
|
@@ -31139,10 +31139,10 @@ var semver2 = __toESM(require_semver2(), 1);
|
|
|
31139
31139
|
|
|
31140
31140
|
// ../bitfab-plugin-lib/dist/bakedSdkVersions.js
|
|
31141
31141
|
var BAKED_SDK_VERSIONS = {
|
|
31142
|
-
typescript: "0.
|
|
31143
|
-
python: "0.
|
|
31144
|
-
ruby: "0.
|
|
31145
|
-
go: "0.
|
|
31142
|
+
typescript: "0.70.0",
|
|
31143
|
+
python: "0.70.0",
|
|
31144
|
+
ruby: "0.69.0",
|
|
31145
|
+
go: "0.68.0"
|
|
31146
31146
|
};
|
|
31147
31147
|
|
|
31148
31148
|
// ../bitfab-plugin-lib/dist/installedSdk.js
|
|
@@ -34751,7 +34751,7 @@ var REGISTRY_PLACEMENT_RULES = [
|
|
|
34751
34751
|
"**Rebuild what the recording cannot keep.** When the root takes live objects as arguments (tools with their functions, clients, callbacks), the recording keeps only their data. Rebuild them in the entry's input adapter (`adaptInputs` / `adapt_inputs` / `AdaptInputs`) by calling the same code production uses. If production builds them inline, extract that code into an exported function that production and the registry both call. That edits production code, so confirm it first. Show the user the function being extracted, the file it moves to, and each call site that changes, state that the logic moves unchanged, and wait for a yes before editing. Present it as a small visibility change, not an architecture change. If the user declines, stop and say that replay cannot rebuild these arguments without it.",
|
|
34752
34752
|
"**Rebuilt tools run for real unless marked.** A tool call returns its recorded output during replay only inside a marked span (`mockOnReplay` / `mock_on_replay` / `WithMockOnReplay`). Tool calls recorded by a framework handler, such as the Claude Agent SDK handler's tool spans, are records only, so those tools run for real. Before any replay, list every tool the registry wires in and put each one that writes, sends, charges, or calls a paid API behind a marked span. If a tool cannot be marked because it runs where replay cannot reach it, stop and tell the user that replaying this workflow runs that tool for real.",
|
|
34753
34753
|
"**Experiment settings are not registry entries.** Set a model swap or any other experiment setting through an environment variable the app already reads. `optionsFactory` / `options_factory` only changes replay options such as mocking, never how the app builds the root. Never add a second registry entry with the setting written into it.",
|
|
34754
|
-
"**Check the wiring before calling it done.** Run `--dry-run` first. The first real replay of an unchanged trace (after the replay safety check) should take the same tool path as the original. A tool the original called that the replay skips means production wiring is still missing."
|
|
34754
|
+
"**Check the wiring before calling it done.** Run `--dry-run` first. Every replay command, dry runs included, names its code change, and a wiring check has none, so pass `--code-change skip`. The first real replay of an unchanged trace (after the replay safety check) should take the same tool path as the original. A tool the original called that the replay skips means production wiring is still missing."
|
|
34755
34755
|
];
|
|
34756
34756
|
function bulletList(rules, indent) {
|
|
34757
34757
|
return rules.map((rule) => `${indent}- ${rule}`).join("\n");
|
|
@@ -36031,7 +36031,7 @@ The SDK-installed \`bitfab-replay\` command lets the team regression-test any re
|
|
|
36031
36031
|
title: "Create or update replay registry",
|
|
36032
36032
|
body: `**Create the replay registry module** adapted to this codebase. Ground it in the docs, not memory: fetch \`https://docs.bitfab.ai/reference/<language>.md\` for the canonical registry shape, then \`https://docs.bitfab.ai/<language>-sdk.md\` for the module template, the executable command, and the worked example under "When the traced function needs the app's dependencies". Quote the exact registry export convention and command in your plan.
|
|
36033
36033
|
|
|
36034
|
-
The project file holds only app imports and bootstrap, registry entries, and per-function defaults (\`adaptInputs\` / \`adapt_inputs\`, mock overrides, database branching). The SDK-installed \`bitfab-replay\` executable loads it through \`--registry\` and owns everything else: every flag (\`--limit\`, \`--trace-ids\`, \`--dataset-ids\`, \`--mock\`, and the rest), progress callbacks, code-change
|
|
36034
|
+
The project file holds only app imports and bootstrap, registry entries, and per-function defaults (\`adaptInputs\` / \`adapt_inputs\`, mock overrides, database branching). The SDK-installed \`bitfab-replay\` executable loads it through \`--registry\` and owns everything else: every flag (\`--limit\`, \`--trace-ids\`, \`--dataset-ids\`, \`--mock\`, the required \`--code-change\`, and the rest), progress callbacks, code-change capture, the stderr summary, stdout JSON, and \`BITFAB_REPLAY_RESULT_PATH\`. The registry never invokes a replay runner and never parses, forwards, or prints any of that. The one exception is Go, whose registry command ends by passing \`os.Args[1:]\` to \`bitfab.RunReplayCLI\`, which then owns all of it. The non-negotiables are:
|
|
36035
36035
|
{{registryPlacementRules}}
|
|
36036
36036
|
- **Register the production root itself, with the app's own client.** For a decorated root, register the decorated function directly: its trace function key comes from the decorator, and a plain closure around it (for example \`(x) => fn(x)\`) carries no key and nests a duplicate root. For a TypeScript decorated method, register \`instance.method.bind(instance)\`, which keeps its key. For Python class methods, register \`Class.method\` or a bound \`instance.method\`; for Ruby, the production \`receiver\` and \`method_name\`; for Go, the production function or method value with \`TraceFunctionKey\`, since a plain Go function carries no key. Import the \`Bitfab\` client the instrumentation uses; never construct a second one. Never register a stub or identity function; reshape old inputs with the entry's input adapter instead.
|
|
36037
36037
|
- **Replay root parity (hard rule):** for keys with a decorated or manually wrapped root function, the function stored in the registry must be the exact same exported top-level traced wrapper that production/runtime calls to create the root span. You must do this unless it is genuinely impossible in the host app; inconvenience, extra refactoring, an inline wrapper, or needing to move code is not impossible. If production creates that wrapper inline inside a route, job, handler, callback, or local file scope, extract it into the nearest appropriate service/module, export it, and update both production and replay to import and call that same symbol. Do not replay a convenient inner helper unless that exact helper is also the production root traced wrapper. Avoid duplicate semantic wrappers split across production and replay with names like \`runX\`, \`processX\`, or \`generateX\`. For handler-instrumented keys with explicit-key replay, verify parity against the production keyed handler/run-wrapper entrypoint that re-invokes the same framework entrypoint production calls. If exported-symbol parity is impossible, stop and document the concrete blocker.
|
|
@@ -36231,7 +36231,7 @@ Leave the live request path untouched: only the replayed function reads the bran
|
|
|
36231
36231
|
|
|
36232
36232
|
1. Resolve the exact fresh trace ID. Before triggering the instrumented function, call {{tool:searchTraces}} with \`{ traceFunctionKey: "<key>", limit: 10 }\` and retain the returned IDs. Record the current timestamp, then run the instrumented function once (or have the user trigger it). Poll the same search once or twice and compare the returned IDs with the before set. Select the ID only when exactly one new root trace has a timestamp after the recorded time. If there are zero new IDs, stop and report that no fresh trace arrived; if concurrent traffic produces more than one candidate, stop and ask the user for the intended trace ID. \`search_traces\` does not return trace inputs, so never guess based on an assumed input match. Confirm the exact ID is eligible by calling {{tool:searchTraces}} with \`{ traceFunctionKey: "<key>", traceIds: ["<fresh-trace-id>"], hasDbSnapshot: true, limit: 1 }\`. If it is absent, do not replay it: report that the fresh trace did not capture a snapshot and diagnose SDK freshness/database connection first.
|
|
36233
36233
|
2. **Mandatory replay-safety check, before any replay command.** Read the replay registry module, the real production root it imports, and every reachable external-action span. Inventory database writes, outbound mutations, queue publishes, email, payments, filesystem writes, and similar unsafe actions. Confirm each one is behind a manual replay-mockable descendant selected by the registry entry's actual strategy and has a serializable recorded output; \`mock: "marked"\` requires \`mockOnReplay\` on that boundary. Confirm the boundary runs in the same replay context: Python worker threads require \`trace_across_threads=True\`; Python async-generator spans cannot be mocked; Ruby child threads, pre-created consumers, and processes do not inherit replay interception; TypeScript synchronous selected spans cannot use lazy \`marked\` output and may use \`all\` only when freezing every matched child is acceptable. Also reject unsafe import-time/module initialization, because it runs before replay interception. Confirm the replay call requests \`dbBranch\` / \`db_branch\`. If any unsafe action is unselected, unmockable, outside context, or uncertain, **do not run the smoke test**; report the exact blocker and required boundary change. Mocking, not the app environment, is the safety boundary.
|
|
36234
|
-
3. Run the SDK-installed replay command against only the verified ID using \`--trace-ids <fresh-trace-id>\` (for example, \`pnpm with-env bitfab-replay --registry scripts/replayRegistry.ts <pipeline> --trace-ids <fresh-trace-id>\`, \`poetry run bitfab-replay --registry scripts/replay_registry.py <pipeline> --trace-ids <fresh-trace-id>\`, or \`bundle exec bitfab-replay --registry scripts/replay_registry.rb <pipeline> --trace-ids <fresh-trace-id>\`, with the app's normal environment loader). Never substitute \`--limit 1\`.
|
|
36234
|
+
3. Run the SDK-installed replay command against only the verified ID using \`--trace-ids <fresh-trace-id>\` (for example, \`pnpm with-env bitfab-replay --registry scripts/replayRegistry.ts <pipeline> --code-change skip --trace-ids <fresh-trace-id>\`, \`poetry run bitfab-replay --registry scripts/replay_registry.py <pipeline> --code-change skip --trace-ids <fresh-trace-id>\`, or \`bundle exec bitfab-replay --registry scripts/replay_registry.rb <pipeline> --code-change skip --trace-ids <fresh-trace-id>\`, with the app's normal environment loader). Pass \`--code-change skip\`: this run checks the instrumentation, not a code change. Never substitute \`--limit 1\`.
|
|
36235
36235
|
4. Confirm the branch was injected: inside the replayed function, \`getCurrentReplayBranch()\` (TypeScript), \`get_current_replay_branch()\` (Python), or \`Bitfab.current_replay_branch\` (Ruby) must be non-null. Compare its \`databaseUrl\` / \`database_url\` host and database with the app's normal \`DATABASE_URL\`; they should differ. Print the experiment URL from the replay output (\`experimentUrl\` in TypeScript, \`experiment_url\` in Python/Ruby; older SDKs print \`testRunUrl\` / \`test_run_url\`) so the user can open the experiment.
|
|
36236
36236
|
|
|
36237
36237
|
If the branch accessor is null for a freshly captured trace, check that the source database is connected, that the trace actually carries a snapshot reference, and that the SDK supports always-on capture (upgrade with \`{{cmd}}update\` when needed). Re-check the dashboard Database section in step {{step:db-snapshot/connect-db}}; there is no separate replay-environment active flag.
|
|
@@ -37087,8 +37087,9 @@ Replaying traces against the current code:
|
|
|
37087
37087
|
- A replay runs a trace's historical inputs through the current code. Use it to check a fix, compare prompt or model changes, or test a dataset before release. It runs through the \`bitfab-replay\` command that the Bitfab SDK installs. These tools do not start replays.
|
|
37088
37088
|
- It needs the project's replay registry module. Look for \`scripts/replayRegistry.*\`, \`scripts/replay_registry.*\`, or \`cmd/registry/main.go\`. If none exists, \`${platform2.setupHint}\` creates one.
|
|
37089
37089
|
- Run it from the project directory with the project's package runner. TypeScript: \`pnpm exec bitfab-replay --registry <registry-path> <trace-function-key> --dataset-id <dataset-id>\`. Python: \`uv run bitfab-replay ...\` or \`poetry run bitfab-replay ...\`. Ruby: \`bundle exec bitfab-replay ...\`. Go: \`go run ./cmd/registry <trace-function-key> ...\`.
|
|
37090
|
+
- Every replay needs \`--code-change\`: \`working-tree\` (uncommitted), \`last-branch\` (vs parent branch), \`default-branch\` (vs the default branch), \`skip\` (baseline), or \`manual\` with \`--code-change-file <path>\`. Ask if unclear.
|
|
37090
37091
|
- Pass \`--trace-ids <id>,<id>\` instead of \`--dataset-id\` to replay specific traces. Pass \`--name <name>\` to name the run, and \`--attempts <n>\` to replay each trace more than once when results vary. Pass \`--cloud\` to run it on GitHub Actions once \`${platform2.setupHint}\` has set up cloud replay. Run the command with \`--help\` for every option.
|
|
37091
|
-
- Each replay creates an experiment and prints its ID. ${getReplayStatus.name} reports its progress. If it stopped early and is resumable, resume it with the same
|
|
37092
|
+
- Each replay creates an experiment and prints its ID. ${getReplayStatus.name} reports its progress. If it stopped early and is resumable, resume it with the same command plus \`--resume <experiment-id>\`. Leave out \`--dataset-id\` and \`--trace-ids\`, because a resume reuses the experiment's own traces.
|
|
37092
37093
|
|
|
37093
37094
|
Recording a fresh run of a trace:
|
|
37094
37095
|
- \`bitfab-seed --registry <registry-path> <trace-function-key> --from-trace <trace-id>\` runs the function again on a trace's inputs and records the result under the same trace ID. It runs side effects for real, so warn the user first. It does not create an experiment.
|
package/package.json
CHANGED