ablation-workbench 0.2.0__tar.gz

ablation_workbench-0.2.0/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.py[cod]
+.venv/
+.DS_Store
+dist/
+build/
+*.egg-info/
+.ablation/
+experiments/
+node_modules/
+.pytest_cache/
+.coverage

ablation_workbench-0.2.0/CHANGELOG.md

@@ -0,0 +1,86 @@
+# Changelog
+
+<!-- version list -->
+
+## v0.2.0 (2026-10-28)
+
+### Bug Fixes
+
+- Preserve legacy review anchors with fixed baseline
+  ([#2](https://github.com/yiminspace/ablation/pull/2),
+  [`3b76cd0`](https://github.com/yiminspace/ablation/commit/3b76cd09a53277fb384136ac88bc51e38b20326f))
+
+- Resolve exact external logs for replay evidence
+  ([#1](https://github.com/yiminspace/ablation/pull/1),
+  [`149fccf`](https://github.com/yiminspace/ablation/commit/149fccfa5d4444ee16771bd2c64f8134e12bc2be))
+
+- 保留旧版评论并校验导入分组和长 diff ([#3](https://github.com/yiminspace/ablation/pull/3),
+  [`dcdf8c9`](https://github.com/yiminspace/ablation/commit/dcdf8c969f96186e17ce96a617f6bb42927a8dea))
+
+- 等待外部日志入库后自动关联测试记录 ([#5](https://github.com/yiminspace/ablation/pull/5),
+  [`d76308e`](https://github.com/yiminspace/ablation/commit/d76308e431d61a8e8dba81fdbaa2a12466856e80))
+
+### Documentation
+
+- Require replay log association before handoff
+  ([#8](https://github.com/yiminspace/ablation/pull/8),
+  [`1f839ba`](https://github.com/yiminspace/ablation/commit/1f839ba70a61fbd190a3612778897fdcdc80ede2))
+
+- 对齐 ablation 中英文 skill 指引 ([#6](https://github.com/yiminspace/ablation/pull/6),
+  [`a771914`](https://github.com/yiminspace/ablation/commit/a771914a49c1ec3cc62b03e73df2033cf0b1e8ba))
+
+- 补齐中文版 skill 元数据和对应结构 ([#7](https://github.com/yiminspace/ablation/pull/7),
+  [`31bf4d9`](https://github.com/yiminspace/ablation/commit/31bf4d9ed9cc7bd5859cf2df9e3afe29a007182f))
+
+### Features
+
+- Add complete replay history and detail drawers
+  ([#9](https://github.com/yiminspace/ablation/pull/9),
+  [`c6b849e`](https://github.com/yiminspace/ablation/commit/c6b849e5ab28bc62c0836d4dcedeb4ee00d1535a))
+
+- Simplify original request reviews and source links
+  ([#2](https://github.com/yiminspace/ablation/pull/2),
+  [`3b76cd0`](https://github.com/yiminspace/ablation/commit/3b76cd09a53277fb384136ac88bc51e38b20326f))
+
+- Simplify request review and link rationale to source
+  ([#2](https://github.com/yiminspace/ablation/pull/2),
+  [`3b76cd0`](https://github.com/yiminspace/ablation/commit/3b76cd09a53277fb384136ac88bc51e38b20326f))
+
+- 在实验标题下实时展示测试日志 ([#4](https://github.com/yiminspace/ablation/pull/4),
+  [`8905d67`](https://github.com/yiminspace/ablation/commit/8905d67e8ace99f023985c15605e3ea30e3e1b31))
+
+- 对齐 Quarry 的自动发布与版本更新流程 ([#10](https://github.com/yiminspace/ablation/pull/10),
+  [`52949cd`](https://github.com/yiminspace/ablation/commit/52949cd16c67051e1adb57a20066eaed6b5042a6))
+
+- 按需求分组审阅提示词改动并保留原文 diff ([#3](https://github.com/yiminspace/ablation/pull/3),
+  [`dcdf8c9`](https://github.com/yiminspace/ablation/commit/dcdf8c969f96186e17ce96a617f6bb42927a8dea))
+
+
+## Unreleased
+
+- Align releases and update discovery with Quarry: automatically version main commits, publish validated tagged packages to PyPI, and show periodic GUI update notifications with explicit upgrade commands and release notes.
+
+- Group related prompt edits by requirement, with rationale, expected effects and GitHub source links; keep original line diffs and comment anchors available in both views.
+
+- Show only candidate prompt changes in the default diff, with decoded text, line and word differences, collapsed context, without duplicate summaries or replay status; show modification rationale directly below the diff. Keep input history and scenario changes in the full request evidence.
+
+- Add optional platform-neutral replay tracing headers, response ID/link extraction, and independently authenticated log lookups.
+- Save external evidence with each run and display log links in the GUI without treating association failure as model failure.
+
+
+## 0.1.0 — development preview
+
+Initial open-source preview, distributed through GitHub Releases. This preview did not publish to PyPI or enable registry updates.
+
+- Import request files, JSON logs, and HTTP captures into portable experiments, with configurable source endpoints and field mappings.
+- Preserve source evidence, normalization notes, candidates, synthetic scenarios, model outputs, and assertion results.
+- Review prompt, input, request, response, and check differences in a local GitHub-style interface.
+- Focus the initial view on prompt changes and rationale, with detailed evidence and comparison settings available on demand.
+- Submit anchored comments, copy a portable handoff, or dispatch a configured local agent.
+- Keep explanations in the current review and retain candidate history for requested changes.
+- Preserve historical failure counts and let follow-up questions reference previously reviewed responses after another replay.
+- Reject credential-bearing request patches and source URL queries, and validate imported experiment bundles before saving.
+- Record oversized replay results as execution errors and terminate remaining Agent child processes on cancellation.
+- Provide cached update discovery, explicit package-manager upgrades, and offline release notes.
+
+Replays call a configured model endpoint. Returned tool calls are recorded but never executed. Agent execution is tested with local stubs; no real-model behavior or desktop-chat wakeup is claimed.

ablation_workbench-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,23 @@
+# Contributing
+
+Use Python 3.11 or newer. Install the source into an isolated environment:
+
+```sh
+python -m venv .venv
+.venv/bin/python -m pip install -e .
+PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v
+```
+
+Keep inference and agent tests offline: use local HTTP fixtures and child-process stubs. Do not check in captured company logs, credentials, personal filesystem paths, or model outputs containing private data. Synthetic demo results must remain visibly labelled.
+
+CLI and GUI share the experiment engine. Changes to experiment formats need a migration or explicit version rejection, and a regression test using an older fixture. Comments bind to the reviewed content; explanation-only reviews must not silently create candidates or replays.
+
+Use conventional commit titles (`feat:`, `fix:`, `docs:`, `test:`, `chore:`) and describe observable behavior and validation in pull requests. Update `CHANGELOG.md` for user-visible changes.
+
+## Releases
+
+Main commits use conventional commit titles. Semantic Release automatically updates the package version and changelog, creates the matching tag and GitHub Release, and dispatches the PyPI publication workflow, following Quarry's release pipeline.
+
+Publication validates that exact tag through CI, builds the bundled UI and Python distributions, and rejects a package/tag mismatch before uploading. Manual dispatch remains available for an exact tag or a one-off forced version bump. Before the first publication, configure the `ablation-workbench` PyPI trusted publisher for `yiminspace/ablation`, `release.yml`, and the GitHub `pypi` environment. A local build does not publish anything.
+
+Installed GUIs check PyPI in the background and show the upgrade command and release notes. They never install an update automatically. Source/editable installs skip checks; `ABLATION_UPDATE_CHECK=0` disables them. Restart the GUI after upgrading. Package upgrades do not install or synchronize personal skills. Release verification details are in [the release checklist](docs/release-review.md).

ablation_workbench-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ablation contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ablation_workbench-0.2.0/PKG-INFO

@@ -0,0 +1,217 @@
+Metadata-Version: 2.5
+Name: ablation-workbench
+Version: 0.2.0
+Summary: A local CLI and review GUI for reproducible prompt experiments
+Project-URL: Homepage, https://github.com/yiminspace/ablation
+Project-URL: Repository, https://github.com/yiminspace/ablation
+Project-URL: Issues, https://github.com/yiminspace/ablation/issues
+Project-URL: Changelog, https://github.com/yiminspace/ablation/blob/main/CHANGELOG.md
+Author: Ablation contributors
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# Ablation
+
+A local prompt debugging workbench. Import a model request or captured log, change a prompt, replay comparable cases, and review the evidence with GitHub-style diffs and inline discussions.
+
+The CLI and browser GUI share one experiment store. The optional Agent skill uses conversation context to find the source; users can simply ask **“Use ablation to investigate this request.”** Import request files, exported logs, or HTTP captures, and configure field mappings for your source format.
+
+[中文说明](README.zh-CN.md) · [Interface contract](docs/implementation-contract.md) · [Contributing](CONTRIBUTING.md) · [MIT license](LICENSE)
+
+## Install
+
+Supports macOS and Linux with Python 3.11 or later. The GUI is included; end users do not need Node.js. Versioned releases are distributed through PyPI and [GitHub Releases](https://github.com/yiminspace/ablation/releases). The first PyPI publication requires the repository trusted publisher described in the release checklist.
+
+```sh
+pipx install ablation-workbench
+ablation --version
+ablation demo --open
+```
+
+Alternatively, `uv tool install ablation-workbench`, or download the wheel from GitHub Releases and run `uv tool install /path/to/ablation_workbench-0.1.0-py3-none-any.whl`. For development use `python -m venv .venv` followed by `.venv/bin/python -m pip install -e .`.
+
+The demo uses synthetic inputs and makes no model requests. Its initial output panel is explicitly untested. It includes an over-specific candidate and a more general candidate to demonstrate comparisons without presenting fabricated performance claims.
+
+## From a request to an experiment
+
+Keep connection configuration and experiments in a workspace:
+
+```sh
+ablation --workspace ./experiments init
+ablation --workspace ./experiments config profile model \
+  --base-url https://model.example.invalid \
+  --api-key-env MODEL_API_KEY
+
+ablation --workspace ./experiments import request.json --format request
+ablation --workspace ./experiments gui
+```
+
+The example addresses are placeholders. Replace them with your own endpoints and supply credentials through the named environment variables. Configuration stores variable names, not secret values. Source retrieval and model inference use separate endpoint and credential settings.
+
+Choose an explicit import format:
+
+| Format | Input |
+| --- | --- |
+| `request` (default) | A Responses API or Chat Completions request object. Use `--response response.json` to attach a historical response. |
+| `capture` | A JSON envelope containing `request` and optionally `response`; an HTTP envelope can place each JSON body in its `body` field. |
+| `log` | A JSON log with request fields in `params`, optionally overridden by top-level `model`, `input`, `messages`, and `tools`; `response` is optional. Use mapping settings for other shapes. |
+
+You can upload JSON through the GUI or pass a file or `-` for stdin to `ablation import`. Original evidence is retained separately from the replay request. Normalization notes and the source diff disclose reconstruction; missing mapped fields are errors rather than silently invented values.
+
+For a log shaped as `{"payload":{"request":{...},"response":{...}}}`, configure JSON Pointers to those objects. The same mapping applies to a local export or a remote JSON API:
+
+```sh
+ablation --workspace ./experiments config profile logs \
+  --source-base-url https://logs.example.invalid \
+  --source-path '/records/{ref}' \
+  --source-request-pointer /payload/request \
+  --source-response-pointer /payload/response \
+  --token-env LOGS_TOKEN
+
+ablation --workspace ./experiments import source-log.json --format log --profile logs
+ablation --workspace ./experiments import record-123 --ref --format log --profile logs
+```
+
+Omit `--source-response-pointer` when your export has no response. To map individual request fields, use `--source-fields-json '{"model":"/metadata/model","input":"/payload/messages"}'`; each pointer starts at the original source root and explicitly overrides that top-level request field. Fields in the selected request object or `params` remain intact unless explicitly overridden; other source fields remain in the original evidence.
+
+Remote imports require an explicit `--source-base-url`. Supply a full JSON URL on the same origin, or a reference resolved by the configured `--source-path` template. `--ref` forces a positional value to be treated as a reference instead of a local file. Source authentication uses either `--token-env` or the pair `--source-user-env` / `--source-password-env`. It never falls back to the inference API key. A webpage sharing link is not necessarily a JSON endpoint; export its data with an existing tool, then import that JSON. Replay can independently use the `model` profile.
+
+## Change, replay, compare
+
+Candidates are immutable JSON patches with a rationale. Keep the explanation concise: why the change helps, then where it belongs and what it affects. Add verified GitHub source links using Markdown; identify unconfirmed locations or origins explicitly. Scenarios independently describe changed inputs and their expected behavior. Every candidate is derived from the fixed source; prior model responses never become the next baseline automatically.
+
+```json
+[
+  {
+    "op": "replace",
+    "path": "/instructions",
+    "value": "Summarize the supplied evidence in three concise bullets. State when a requested fact is missing."
+  }
+]
+```
+
+Save that as `candidate.json`. Pass explanation text with `--rationale`, or use `--rationale-file rationale.md` to preserve multiline UTF-8 text and source links. Both store the same rationale displayed below the GUI diff. Then:
+
+```sh
+ablation --workspace ./experiments candidate EXPERIMENT_ID \
+  --label 'Concise evidence summary' --patch candidate.json \
+  --rationale 'Specify output structure and how to handle missing information.'
+
+ablation --workspace ./experiments run EXPERIMENT_ID \
+  --candidate baseline --candidate CANDIDATE_ID \
+  --scenario original --repeats 3 --profile model
+
+ablation --workspace ./experiments diff EXPERIMENT_ID \
+  --candidate CANDIDATE_ID --artifact response
+```
+
+The GUI offers the same candidate/scenario creation and replay controls. Runs save the actual request, response, request hash, timing and assertion results. Compare prompt, scenario input, full request, response, checks, and source reconstruction. HTTP failures are separate from failed assertions. A run without checks is **unscored**, not passing. Repetition counts are not independent scenario counts or statistical reliability claims.
+
+The initial view fixes the imported original request on the left and shows the modified request on the right. Both sides retain the original JSON structure and field order. Instructions, tool descriptions, and input-message changes appear together, with unchanged context collapsed. Older versions and candidate/scenario creation live in **运行记录**. The main view contains only actual changes and model outputs.
+
+The summary counts historical runs for the selected versions and scenario, so a later passing replay does not hide earlier failures. The response diff shows the latest completed outputs; inspect individual runs for earlier evidence.
+
+The sidebar starts collapsed and lists saved experiment sessions by title and full session ID, with search and replay counts. Each session retains all replay batches, scenarios, repeats, outcomes, and log links. The initial view prioritizes the diff, with the complete batch history on the later **运行记录** tab; **运行记录** lists all runs compactly. Clicking anywhere on a batch row opens a narrow, full-height right drawer with Messages, checks, and raw JSON views. A selector switches between the runs in that batch. Its timeline includes recorded inputs, tool calls/results, screenshot thumbnails that open at full size, and the new model output. Opening a run positions the timeline at its highlighted new output; a locate button returns to it. The timeline adapts Bifrost's Apache-2.0 components; the pinned source, modifications, and license ship with the web assets. External log links remain available for provenance. The GUI uses Voyage components with GitHub as the default theme. The top-right standard VoyageToolbar provides light/dark and theme selection. Its feedback and language placeholders remain disabled until those product capabilities are configured. All styles and the toolbar are bundled locally for offline use; `npm run build:ui` synchronizes the pinned assets and `npm run check:ui` verifies them.
+
+Replay invokes a single model request. Returned tool calls are inspected as data and never executed. A model response does not prove that an external action occurred.
+
+## Optional external log tracking
+
+Replay profiles may configure independent observers through `config profile NAME --observers-json observers.json`. The file is an array, so a request passing through a proxy and a model gateway can retain both records. No tracking headers or external lookups are enabled by default.
+
+```json
+[
+  {
+    "name": "Gateway logs",
+    "headers": {"X-Trace-Id": "{run_id}"},
+    "record_id_header": "X-Request-Id",
+    "url_template": "https://logs.example.invalid/records/{record_id}"
+  },
+  {
+    "name": "Capture proxy",
+    "record_id_pointer": "/capture/entry_id",
+    "url_pointer": "/capture/view_url"
+  }
+]
+```
+
+These are synthetic examples; select header names, response fields, and URL routes documented by your service. `headers` templates accept `{experiment_id}`, `{batch_id}`, and `{run_id}`. They cannot override authentication or transport headers. Response extraction supports JSON Pointers (`record_id_pointer`, `url_pointer`) and case-insensitive HTTP headers (`record_id_header`, `url_header`). Use `request_id_pointer` or `request_id_header` for transport correlation IDs that are not external log IDs; they remain separate and can be used as `{request_id}` in lookup templates. `url_template` and optional `batch_url_template` also accept `{record_id}`, `{response_id}`, `{start_time}`, and `{end_time}` (Unix seconds bounding this run); substituted values are URL-encoded. Escape literal JSON braces in templates as `{{` and `}}`.
+
+For Bifrost, inject a unique run metadata header and configure an exact lookup against that metadata. Extract the actual log ID and construct its `selected_log` detail URL; do not use the transport `x-request-id` as the log ID. A metadata-filter search URL is only a list entry point. A TapeProxy target can extract an entry reference from its documented response header or API. Platform names are labels, not built-in routing: another service uses the same mappings without changing the replay engine.
+
+An optional `lookup` reads a separate JSON log API after inference. It requires `source_base_url`, a relative `source_path` template, `match_pointer`, and `match_value`. Optional `records_pointer` selects the record array; `record_id_pointer` and `url_pointer` extract the uniquely matched record. Lookup authentication uses its own `token_env` or `source_user_env` / `source_password_env`, never the inference API key. For example:
+
+```json
+{
+  "name": "Separate log service",
+  "headers": {"X-Trace-Id": "{run_id}"},
+  "lookup": {
+    "source_base_url": "https://logs.example.invalid",
+    "source_path": "/api/logs?trace={run_id}",
+    "token_env": "LOG_READ_TOKEN",
+    "records_pointer": "/records",
+    "match_pointer": "/trace_id",
+    "match_value": "{run_id}",
+    "record_id_pointer": "/id"
+  },
+  "url_template": "https://logs.example.invalid/records/{record_id}"
+}
+```
+
+A configured lookup is authoritative even when response extraction returns an ID or URL. Each lookup request has a 5-second timeout, requires one exact match, and follows no redirects. For delayed ingestion, set lookup `attempts` (1–5, default 1) and `interval_seconds` (0–5, default 1); only an empty exact-match result is retried, while ambiguous matches and request failures stop immediately. Missing or failed association is recorded as `unavailable` without changing the model result or assertion score. `available` means a record ID or link was obtained from the configured response or matching API record; `linked` means only a search/view URL was constructed, not that a log exists or is accessible.
+
+Runs retain local tracking IDs, the injected non-secret headers, selected response headers, an observer configuration snapshot, and `external_evidence` records with `platform`, `status`, and optional `request_id`, `record_id`, `url`, `batch_url`, or `error`. The GUI displays links in run history and details; CLI JSON and exported bundles retain the same evidence. Profiles without observers still retain the model response ID and standard request-ID headers when supplied. Historical runs remain unchanged and import without the new optional fields.
+
+## Review and continue
+
+Click a diff line's **+**, add a comment, then finish the review. Comments retain the reviewed version, scenario, location and before/after snippets.
+
+Use **继续追问** to ask about an existing comment. The follow-up preserves its original snapshot even if a newer replay has changed the response. Unsaved comments on changed responses still require reconciliation before submission.
+
+- **复制到对话** (Copy to conversation) saves the review and copies a portable handoff. Paste it into the current Agent conversation. The handoff includes commands for writing an answer back to the experiment.
+- **交给 Agent** (Send to Agent) invokes the configured local Agent and records a job. Configure Codex once with `ablation agent configure --backend codex --cwd /path/to/project`. Other Agents can use a trusted argv command that receives the handoff on stdin: `--backend command --command-json '["executable","argument"]'`.
+
+**追问解释** (Comment) requests explanation only. Its replies appear below the reviewed change and do not produce a new candidate or replay. **请求修改** (Request changes) authorizes a new candidate and the requested verification within the experiment. Applying changes to a business repository remains an explicit user task for the Agent; this tool does not guess which source file generated a request field.
+
+```sh
+ablation review text EXPERIMENT_ID REVIEW_ID
+ablation review reply EXPERIMENT_ID REVIEW_ID --comment-index 0 \
+  --text 'This change makes the output format explicit while keeping the same source evidence.'
+ablation review dispatch EXPERIMENT_ID REVIEW_ID
+```
+
+CLI dispatch waits until completion; Ctrl-C requests cancellation. The GUI shows progress, errors and cancellation. A completed Agent process is not a passing replay: inspect the resulting experiment evidence. The runner manages its own CLI sessions; it does not claim to wake an existing desktop chat.
+
+## Portable skill
+
+Copy `skills/ablation` into your Agent's supported skill directory. The skill supplies context selection, experiment design and result interpretation; the executable supplies mechanics and also works without a skill. Skill installation and synchronization are not performed automatically.
+
+## Persistence and updates
+
+The default workspace is `~/.local/share/ablation`; override it with `--workspace` or `ABLATION_WORKSPACE`. Experiments and review replies persist on disk. `ablation export EXPERIMENT_ID --out experiment.json` and `ablation import-bundle experiment.json` move experiments between workspaces; imports get new IDs and retain provenance.
+
+Software versions, experiment schema versions and candidate IDs are separate. Historical request/response records remain unchanged when software updates. Credentials remain in your environment. Exported evidence can contain prompt content and model responses; review the bundle before sharing it.
+
+The GUI checks PyPI in the background, caches checks for 24 hours, and displays a newer version with an installation-aware upgrade command and release-note link. It does not install updates automatically. Editable/source-checkout installations are skipped; set `ABLATION_UPDATE_CHECK=0` to disable checks. `ablation update-check` reports availability, and an unavailable registry is not proof that the installed version is current. `ablation upgrade` shows the installation-aware plan; `ablation upgrade --yes` remains an explicit CLI operation and refuses to run while jobs are active. Restart the local GUI after upgrading. The changelog is bundled for offline viewing.
+
+CLI stdout is JSON except for human review handoffs, changelogs and help. Errors go to stderr. Exit codes: `0` completed command (including unscored replay), `2` invalid input/configuration, `3` replay assertions failed, `4` execution or transport failure, `130` cancelled. Inspect saved per-run evidence even when the command returns nonzero.
+
+## Development checks
+
+```sh
+PYTHONPATH=src python -m unittest discover -s tests -v
+python -m pip wheel --no-deps . -w dist
+npm ci
+npx playwright install chromium
+npm run test:browser
+```
+
+Tests use synthetic fixtures and local source, model, and Agent substitutes. They do not establish compatibility with a particular live data service, model account, or desktop chat integration. Browser verification and release review details belong in `docs/release-review.md`.

ablation_workbench-0.2.0/README.md

@@ -0,0 +1,197 @@
+# Ablation
+
+A local prompt debugging workbench. Import a model request or captured log, change a prompt, replay comparable cases, and review the evidence with GitHub-style diffs and inline discussions.
+
+The CLI and browser GUI share one experiment store. The optional Agent skill uses conversation context to find the source; users can simply ask **“Use ablation to investigate this request.”** Import request files, exported logs, or HTTP captures, and configure field mappings for your source format.
+
+[中文说明](README.zh-CN.md) · [Interface contract](docs/implementation-contract.md) · [Contributing](CONTRIBUTING.md) · [MIT license](LICENSE)
+
+## Install
+
+Supports macOS and Linux with Python 3.11 or later. The GUI is included; end users do not need Node.js. Versioned releases are distributed through PyPI and [GitHub Releases](https://github.com/yiminspace/ablation/releases). The first PyPI publication requires the repository trusted publisher described in the release checklist.
+
+```sh
+pipx install ablation-workbench
+ablation --version
+ablation demo --open
+```
+
+Alternatively, `uv tool install ablation-workbench`, or download the wheel from GitHub Releases and run `uv tool install /path/to/ablation_workbench-0.1.0-py3-none-any.whl`. For development use `python -m venv .venv` followed by `.venv/bin/python -m pip install -e .`.
+
+The demo uses synthetic inputs and makes no model requests. Its initial output panel is explicitly untested. It includes an over-specific candidate and a more general candidate to demonstrate comparisons without presenting fabricated performance claims.
+
+## From a request to an experiment
+
+Keep connection configuration and experiments in a workspace:
+
+```sh
+ablation --workspace ./experiments init
+ablation --workspace ./experiments config profile model \
+  --base-url https://model.example.invalid \
+  --api-key-env MODEL_API_KEY
+
+ablation --workspace ./experiments import request.json --format request
+ablation --workspace ./experiments gui
+```
+
+The example addresses are placeholders. Replace them with your own endpoints and supply credentials through the named environment variables. Configuration stores variable names, not secret values. Source retrieval and model inference use separate endpoint and credential settings.
+
+Choose an explicit import format:
+
+| Format | Input |
+| --- | --- |
+| `request` (default) | A Responses API or Chat Completions request object. Use `--response response.json` to attach a historical response. |
+| `capture` | A JSON envelope containing `request` and optionally `response`; an HTTP envelope can place each JSON body in its `body` field. |
+| `log` | A JSON log with request fields in `params`, optionally overridden by top-level `model`, `input`, `messages`, and `tools`; `response` is optional. Use mapping settings for other shapes. |
+
+You can upload JSON through the GUI or pass a file or `-` for stdin to `ablation import`. Original evidence is retained separately from the replay request. Normalization notes and the source diff disclose reconstruction; missing mapped fields are errors rather than silently invented values.
+
+For a log shaped as `{"payload":{"request":{...},"response":{...}}}`, configure JSON Pointers to those objects. The same mapping applies to a local export or a remote JSON API:
+
+```sh
+ablation --workspace ./experiments config profile logs \
+  --source-base-url https://logs.example.invalid \
+  --source-path '/records/{ref}' \
+  --source-request-pointer /payload/request \
+  --source-response-pointer /payload/response \
+  --token-env LOGS_TOKEN
+
+ablation --workspace ./experiments import source-log.json --format log --profile logs
+ablation --workspace ./experiments import record-123 --ref --format log --profile logs
+```
+
+Omit `--source-response-pointer` when your export has no response. To map individual request fields, use `--source-fields-json '{"model":"/metadata/model","input":"/payload/messages"}'`; each pointer starts at the original source root and explicitly overrides that top-level request field. Fields in the selected request object or `params` remain intact unless explicitly overridden; other source fields remain in the original evidence.
+
+Remote imports require an explicit `--source-base-url`. Supply a full JSON URL on the same origin, or a reference resolved by the configured `--source-path` template. `--ref` forces a positional value to be treated as a reference instead of a local file. Source authentication uses either `--token-env` or the pair `--source-user-env` / `--source-password-env`. It never falls back to the inference API key. A webpage sharing link is not necessarily a JSON endpoint; export its data with an existing tool, then import that JSON. Replay can independently use the `model` profile.
+
+## Change, replay, compare
+
+Candidates are immutable JSON patches with a rationale. Keep the explanation concise: why the change helps, then where it belongs and what it affects. Add verified GitHub source links using Markdown; identify unconfirmed locations or origins explicitly. Scenarios independently describe changed inputs and their expected behavior. Every candidate is derived from the fixed source; prior model responses never become the next baseline automatically.
+
+```json
+[
+  {
+    "op": "replace",
+    "path": "/instructions",
+    "value": "Summarize the supplied evidence in three concise bullets. State when a requested fact is missing."
+  }
+]
+```
+
+Save that as `candidate.json`. Pass explanation text with `--rationale`, or use `--rationale-file rationale.md` to preserve multiline UTF-8 text and source links. Both store the same rationale displayed below the GUI diff. Then:
+
+```sh
+ablation --workspace ./experiments candidate EXPERIMENT_ID \
+  --label 'Concise evidence summary' --patch candidate.json \
+  --rationale 'Specify output structure and how to handle missing information.'
+
+ablation --workspace ./experiments run EXPERIMENT_ID \
+  --candidate baseline --candidate CANDIDATE_ID \
+  --scenario original --repeats 3 --profile model
+
+ablation --workspace ./experiments diff EXPERIMENT_ID \
+  --candidate CANDIDATE_ID --artifact response
+```
+
+The GUI offers the same candidate/scenario creation and replay controls. Runs save the actual request, response, request hash, timing and assertion results. Compare prompt, scenario input, full request, response, checks, and source reconstruction. HTTP failures are separate from failed assertions. A run without checks is **unscored**, not passing. Repetition counts are not independent scenario counts or statistical reliability claims.
+
+The initial view fixes the imported original request on the left and shows the modified request on the right. Both sides retain the original JSON structure and field order. Instructions, tool descriptions, and input-message changes appear together, with unchanged context collapsed. Older versions and candidate/scenario creation live in **运行记录**. The main view contains only actual changes and model outputs.
+
+The summary counts historical runs for the selected versions and scenario, so a later passing replay does not hide earlier failures. The response diff shows the latest completed outputs; inspect individual runs for earlier evidence.
+
+The sidebar starts collapsed and lists saved experiment sessions by title and full session ID, with search and replay counts. Each session retains all replay batches, scenarios, repeats, outcomes, and log links. The initial view prioritizes the diff, with the complete batch history on the later **运行记录** tab; **运行记录** lists all runs compactly. Clicking anywhere on a batch row opens a narrow, full-height right drawer with Messages, checks, and raw JSON views. A selector switches between the runs in that batch. Its timeline includes recorded inputs, tool calls/results, screenshot thumbnails that open at full size, and the new model output. Opening a run positions the timeline at its highlighted new output; a locate button returns to it. The timeline adapts Bifrost's Apache-2.0 components; the pinned source, modifications, and license ship with the web assets. External log links remain available for provenance. The GUI uses Voyage components with GitHub as the default theme. The top-right standard VoyageToolbar provides light/dark and theme selection. Its feedback and language placeholders remain disabled until those product capabilities are configured. All styles and the toolbar are bundled locally for offline use; `npm run build:ui` synchronizes the pinned assets and `npm run check:ui` verifies them.
+
+Replay invokes a single model request. Returned tool calls are inspected as data and never executed. A model response does not prove that an external action occurred.
+
+## Optional external log tracking
+
+Replay profiles may configure independent observers through `config profile NAME --observers-json observers.json`. The file is an array, so a request passing through a proxy and a model gateway can retain both records. No tracking headers or external lookups are enabled by default.
+
+```json
+[
+  {
+    "name": "Gateway logs",
+    "headers": {"X-Trace-Id": "{run_id}"},
+    "record_id_header": "X-Request-Id",
+    "url_template": "https://logs.example.invalid/records/{record_id}"
+  },
+  {
+    "name": "Capture proxy",
+    "record_id_pointer": "/capture/entry_id",
+    "url_pointer": "/capture/view_url"
+  }
+]
+```
+
+These are synthetic examples; select header names, response fields, and URL routes documented by your service. `headers` templates accept `{experiment_id}`, `{batch_id}`, and `{run_id}`. They cannot override authentication or transport headers. Response extraction supports JSON Pointers (`record_id_pointer`, `url_pointer`) and case-insensitive HTTP headers (`record_id_header`, `url_header`). Use `request_id_pointer` or `request_id_header` for transport correlation IDs that are not external log IDs; they remain separate and can be used as `{request_id}` in lookup templates. `url_template` and optional `batch_url_template` also accept `{record_id}`, `{response_id}`, `{start_time}`, and `{end_time}` (Unix seconds bounding this run); substituted values are URL-encoded. Escape literal JSON braces in templates as `{{` and `}}`.
+
+For Bifrost, inject a unique run metadata header and configure an exact lookup against that metadata. Extract the actual log ID and construct its `selected_log` detail URL; do not use the transport `x-request-id` as the log ID. A metadata-filter search URL is only a list entry point. A TapeProxy target can extract an entry reference from its documented response header or API. Platform names are labels, not built-in routing: another service uses the same mappings without changing the replay engine.
+
+An optional `lookup` reads a separate JSON log API after inference. It requires `source_base_url`, a relative `source_path` template, `match_pointer`, and `match_value`. Optional `records_pointer` selects the record array; `record_id_pointer` and `url_pointer` extract the uniquely matched record. Lookup authentication uses its own `token_env` or `source_user_env` / `source_password_env`, never the inference API key. For example:
+
+```json
+{
+  "name": "Separate log service",
+  "headers": {"X-Trace-Id": "{run_id}"},
+  "lookup": {
+    "source_base_url": "https://logs.example.invalid",
+    "source_path": "/api/logs?trace={run_id}",
+    "token_env": "LOG_READ_TOKEN",
+    "records_pointer": "/records",
+    "match_pointer": "/trace_id",
+    "match_value": "{run_id}",
+    "record_id_pointer": "/id"
+  },
+  "url_template": "https://logs.example.invalid/records/{record_id}"
+}
+```
+
+A configured lookup is authoritative even when response extraction returns an ID or URL. Each lookup request has a 5-second timeout, requires one exact match, and follows no redirects. For delayed ingestion, set lookup `attempts` (1–5, default 1) and `interval_seconds` (0–5, default 1); only an empty exact-match result is retried, while ambiguous matches and request failures stop immediately. Missing or failed association is recorded as `unavailable` without changing the model result or assertion score. `available` means a record ID or link was obtained from the configured response or matching API record; `linked` means only a search/view URL was constructed, not that a log exists or is accessible.
+
+Runs retain local tracking IDs, the injected non-secret headers, selected response headers, an observer configuration snapshot, and `external_evidence` records with `platform`, `status`, and optional `request_id`, `record_id`, `url`, `batch_url`, or `error`. The GUI displays links in run history and details; CLI JSON and exported bundles retain the same evidence. Profiles without observers still retain the model response ID and standard request-ID headers when supplied. Historical runs remain unchanged and import without the new optional fields.
+
+## Review and continue
+
+Click a diff line's **+**, add a comment, then finish the review. Comments retain the reviewed version, scenario, location and before/after snippets.
+
+Use **继续追问** to ask about an existing comment. The follow-up preserves its original snapshot even if a newer replay has changed the response. Unsaved comments on changed responses still require reconciliation before submission.
+
+- **复制到对话** (Copy to conversation) saves the review and copies a portable handoff. Paste it into the current Agent conversation. The handoff includes commands for writing an answer back to the experiment.
+- **交给 Agent** (Send to Agent) invokes the configured local Agent and records a job. Configure Codex once with `ablation agent configure --backend codex --cwd /path/to/project`. Other Agents can use a trusted argv command that receives the handoff on stdin: `--backend command --command-json '["executable","argument"]'`.
+
+**追问解释** (Comment) requests explanation only. Its replies appear below the reviewed change and do not produce a new candidate or replay. **请求修改** (Request changes) authorizes a new candidate and the requested verification within the experiment. Applying changes to a business repository remains an explicit user task for the Agent; this tool does not guess which source file generated a request field.
+
+```sh
+ablation review text EXPERIMENT_ID REVIEW_ID
+ablation review reply EXPERIMENT_ID REVIEW_ID --comment-index 0 \
+  --text 'This change makes the output format explicit while keeping the same source evidence.'
+ablation review dispatch EXPERIMENT_ID REVIEW_ID
+```
+
+CLI dispatch waits until completion; Ctrl-C requests cancellation. The GUI shows progress, errors and cancellation. A completed Agent process is not a passing replay: inspect the resulting experiment evidence. The runner manages its own CLI sessions; it does not claim to wake an existing desktop chat.
+
+## Portable skill
+
+Copy `skills/ablation` into your Agent's supported skill directory. The skill supplies context selection, experiment design and result interpretation; the executable supplies mechanics and also works without a skill. Skill installation and synchronization are not performed automatically.
+
+## Persistence and updates
+
+The default workspace is `~/.local/share/ablation`; override it with `--workspace` or `ABLATION_WORKSPACE`. Experiments and review replies persist on disk. `ablation export EXPERIMENT_ID --out experiment.json` and `ablation import-bundle experiment.json` move experiments between workspaces; imports get new IDs and retain provenance.
+
+Software versions, experiment schema versions and candidate IDs are separate. Historical request/response records remain unchanged when software updates. Credentials remain in your environment. Exported evidence can contain prompt content and model responses; review the bundle before sharing it.
+
+The GUI checks PyPI in the background, caches checks for 24 hours, and displays a newer version with an installation-aware upgrade command and release-note link. It does not install updates automatically. Editable/source-checkout installations are skipped; set `ABLATION_UPDATE_CHECK=0` to disable checks. `ablation update-check` reports availability, and an unavailable registry is not proof that the installed version is current. `ablation upgrade` shows the installation-aware plan; `ablation upgrade --yes` remains an explicit CLI operation and refuses to run while jobs are active. Restart the local GUI after upgrading. The changelog is bundled for offline viewing.
+
+CLI stdout is JSON except for human review handoffs, changelogs and help. Errors go to stderr. Exit codes: `0` completed command (including unscored replay), `2` invalid input/configuration, `3` replay assertions failed, `4` execution or transport failure, `130` cancelled. Inspect saved per-run evidence even when the command returns nonzero.
+
+## Development checks
+
+```sh
+PYTHONPATH=src python -m unittest discover -s tests -v
+python -m pip wheel --no-deps . -w dist
+npm ci
+npx playwright install chromium
+npm run test:browser
+```
+
+Tests use synthetic fixtures and local source, model, and Agent substitutes. They do not establish compatibility with a particular live data service, model account, or desktop chat integration. Browser verification and release review details belong in `docs/release-review.md`.