agent-tune-kit 2.2.0__tar.gz → 2.2.1__tar.gz

{agent_tune_kit-2.2.0 → agent_tune_kit-2.2.1}/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-tune-kit",
-  "version": "2.2.0",
+  "version": "2.2.1",
   "description": "Local Codex plugin for iterative Agent tuning with guided Skills, reusable runner templates, versioned results, and static validation.",
   "author": {
     "name": "hustyichi",

agent_tune_kit-2.2.1/PKG-INFO

@@ -0,0 +1,199 @@
+Metadata-Version: 2.5
+Name: agent-tune-kit
+Version: 2.2.1
+Summary: Local Codex plugin for iterative Agent tuning with guided Skills, reusable runner templates, versioned results, and static validation.
+Project-URL: Homepage, https://github.com/hustyichi/agent-tune-kit
+Project-URL: Repository, https://github.com/hustyichi/agent-tune-kit
+Author-email: hustyichi <hustyichi@163.com>
+License: UNLICENSED
+Keywords: agent-tune-kit,codex-skills,evaluation,local-plugin,reporting
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# Agent Tune Kit
+
+English | [简体中文](README.md)
+
+[![PyPI](https://img.shields.io/pypi/v/agent-tune-kit.svg)](https://pypi.org/project/agent-tune-kit/)
+
+Agent Tune Kit is a **local Codex plugin** that helps you evaluate an existing Agent, identify failures, and tune prompts, code, parameters, or tool configuration from the results.
+
+Use it to prepare evaluation data, test your Agent in batches, review failures and reports, then check whether your changes help in the next evaluation. Each round keeps its results and tuning notes for review and comparison.
+
+## Before You Start
+
+- A local Agent project that Codex can read and edit, such as a chatbot, tool-using Agent, or RAG Agent.
+- Evaluation data, preferably CSV; 5 to 20 examples are enough to start. If you have no dataset yet, start with business examples.
+- A basis for judging results, such as expected answers, labels, or acceptance criteria.
+
+## Install and Enable
+
+Install ATK and its bundled Skills in your terminal:
+
+```sh
+uv tool install agent-tune-kit
+atk install
+```
+
+In Codex, enter `/plugins`, select and enable `Agent Tune Kit`, then restart Codex and open a new session.
+
+<details>
+<summary>Other installation options: pipx or local source</summary>
+
+With pipx:
+
+```sh
+pipx install agent-tune-kit
+atk install
+```
+
+From local source: update your chosen branch, then run these commands from the **ATK repository root**. Rerun them after source updates; changes are reinstalled even when the version is unchanged.
+
+```sh
+uv tool install --reinstall .
+atk install
+```
+
+</details>
+
+Finish active ATK tasks before reinstalling, and open a new session afterward to load updates. If installation fails, run `atk status` and see the [installation and cache notes](docs/skill-template-pack-usage.md#local-plugin-install-and-smoke). A successful check does not confirm that the plugin is enabled in your current session.
+
+## Complete Your First Evaluation and Tuning Round
+
+**Open your own Agent project in Codex**, then send the Skill instructions below. Installation commands run in a terminal; `$atk-…` instructions belong in your Codex conversation.
+
+### 1. Prepare Data as Needed
+
+Skip to the next step if you already have suitable data. To organize a dataset or fill gaps in evaluation criteria:
+
+```text
+$atk-dataset Prepare data/eval.csv for evaluation. Preserve existing fields and answers; fill only grounded gaps in the judgment criteria.
+```
+
+You can also provide business examples or request a correction, such as changing the expected result for atk_id 7. Fixed column names and exported feedback files are not required. Answers without trustworthy support remain gaps rather than being invented.
+
+Use `$atk-visualize-dataset` when you want a local page for browsing and reviewing data, then give feedback to `$atk-dataset`. Data preparation does not run the Agent.
+
+### 2. Connect Your Agent
+
+Tell Codex where your Agent entry point and evaluation data are:
+
+```text
+$atk-init My Agent entry point is scripts/agent.py and the evaluation dataset is data/eval.csv
+```
+
+Codex reads the project, prepares the adapter, and organizes a runnable dataset. If your Agent is already connected through a compatible adapter, go straight to evaluation.
+
+Evaluation appends five reserved output columns: `agent_output`, `output_status`, `error_type`, `error_message`, and `log_path`. Managed V1/V2 adapters and `atk eval --check` reject name collisions before calling the Agent or allocating a result version; the error lists the dataset path and conflicting columns. For example, after confirming the change, rename a business `output_status` field to `review_status` and update adapter and judgment-rule references. ATK never renames fields or overwrites source values automatically; data preparation may still preserve these business columns unchanged. Old standalone runners need a separate check or migration to the shared engine.
+
+### 3. Evaluate, Diagnose, and Read the Report
+
+Send these instructions in order:
+
+```text
+$atk-eval
+$atk-diagnose
+$atk-report
+```
+
+| Instruction | What you get |
+| --- | --- |
+| `$atk-eval` | Actual Agent outputs and execution records for each sample |
+| `$atk-diagnose` | Passed, failed, or unknown judgments with reasons and an initial investigation of failures |
+| `$atk-report` | Main issues, cause analysis, and tuning recommendations |
+
+Use `$atk-visualize-failures` for a local offline page to search, filter, and review individual failures.
+
+V2 shows start/completion progress by default, counting executions and scored samples separately for grouped cases. Progress goes to stderr so `--json` stdout remains valid JSON. Use `--no-progress` to suppress it; `--check` emits no execution progress.
+
+**Unknown does not count as passed.** Missing answers, criteria, or valid judgment records cannot establish that the Agent behaved correctly.
+
+### 4. Tune from the Results
+
+```text
+$atk-optimize
+```
+
+Codex uses the report and failures to edit your Agent's prompts, code, parameters, or tool configuration. It records why it made changes and what to verify next. Save your current code with Git before tuning so you can compare and restore it.
+
+### Workflow
+
+![Agent Tune Kit workflow: prepare data and connect your Agent as needed, then repeat evaluation, diagnosis, reporting, and tuning. Dataset and failure browsers are optional.](docs/assets/architecture.en.svg)
+
+## Review Results and Verify Improvement
+
+Each round saves results under `.atk/results/vN/` in your Agent project. Start with `report.md` for the main findings, use `failure_cases.html` to review individual failures, and read `tuning_plan.md` for the changes and their rationale.
+
+After tuning, you can first rerun the previous round's confirmed failures:
+
+```text
+$atk-eval --only-failures
+$atk-diagnose
+$atk-report
+```
+
+A new evaluation does not overwrite existing nonempty result directories. From the second round onward, when prior tuning notes exist and conditions are comparable, the report identifies target issues as resolved, partially resolved, unresolved, or impossible to judge.
+
+**Rerunning failures does not measure the full-dataset pass rate.** To check effects on other samples, run `$atk-eval` again, then diagnose and report for a full regression check. Changing data or evaluation criteria alone does not prove Agent improvement.
+
+Execution summaries and `atk results inspect` expose `technical_failures` and `technical_failure_ids`, separately from business judgments. They count reported `output_status=error` rows; a V2 mapper must explicitly translate failures embedded in business outputs. ATK does not guess domain field meanings. Technical errors do not automatically become business failures or enter `--only-failures`.
+
+Use `$atk-eval --ids 7 9` to revisit unknown or selected samples. Managed V1/V2 filter IDs in canonical dataset order before offset/limit; grouped adapters retain full case context. Missing IDs stop execution, and `--ids` is mutually exclusive with `--only-failures`. A selected rerun is not a full regression run.
+
+### Complete Output Structure
+
+This tree lists persistent evaluation outputs and optional files produced by the latest version. They appear as the corresponding steps and configuration require, rather than all being created at installation.
+
+```text
+.atk/
+├── context.md                 # Optional: agreed goals, standards, and human feedback
+├── datasets/
+│   ├── dataset.csv            # Prepared data with sample IDs (atk_id)
+│   ├── dataset.html           # Optional dataset browser
+│   └── dataset_review.csv     # Optional: human-exported feedback placed here
+├── runner/
+│   ├── eval_runner.py         # Agent adapter
+│   └── failure_rule.py        # Project judgment rule, generated as needed
+└── results/
+    ├── v1/
+    │   ├── eval_results.csv    # Actual outputs and execution status per sample
+    │   ├── run.json            # Round status and evidence index
+    │   ├── run-events.jsonl    # As needed: events outside individual executions
+    │   ├── executions/
+    │   │   └── <execution-id>/ # One execution; may serve multiple samples
+    │   │       ├── execution.json
+    │   │       ├── input.json
+    │   │       ├── output.json
+    │   │       ├── context/
+    │   │       │   ├── samples.json
+    │   │       │   ├── environment.json
+    │   │       │   └── snapshot-<N>
+    │   │       ├── events.jsonl
+    │   │       ├── app.log
+    │   │       └── artifacts/  # Additional saved artifacts, as needed
+    │   ├── judgments.csv       # Per-sample judgments and reasons
+    │   ├── judgment-history/   # Previous judgments saved on reassessment
+    │   │   └── <hash>.csv
+    │   ├── failure_cases.csv   # Confirmed failures
+    │   ├── failure_cases.html  # Optional failure browser
+    │   ├── report.md           # Analysis and cross-round validation
+    │   └── tuning_plan.md      # Tuning changes and verification targets
+    └── v2/
+        └── ...                # Later rounds follow the applicable layout
+```
+
+| File or directory | When and why to use it |
+| --- | --- |
+| `datasets/dataset.csv`, `dataset.html` | Review inputs and expected results. The browser is optional; exported review CSVs can also be imported from Downloads |
+| `context.md` | Optionally keep agreed goals, behavior standards, and feedback for future tuning; no manual run statistics are needed |
+| `runner/` | Connect or adjust Agent invocation; Skills use these files during routine evaluation |
+| `eval_results.csv` | Read source fields, actual `agent_output`, execution status `output_status`, and `log_path` |
+| `judgments.csv`, `failure_cases.csv` | Read individual judgments and confirmed failures. Unknowns stay outside the failure set; reassessments preserve old records in `judgment-history/` |
+| `report.md`, `failure_cases.html`, `tuning_plan.md` | Read analysis, review failures, inspect changes, and choose the next verification targets |
+| `run.json`, `executions/`, `run-events.jsonl` | Investigate a run using saved status, inputs, outputs, context, events, and additional artifacts |
+
+Within each execution, `context/` holds relevant samples, environment details, and snapshots such as the adapter file. `app.log` is a view of captured logs and may be empty. Detailed model and tool traces depend on project instrumentation; calls are not all recorded automatically.
+
+Failed or interrupted executions may leave only partial files; a directory does not prove completion. Evidence-only imports may have no `eval_results.csv` and cannot be scored directly. Execution evidence can contain project data, so confirm retention and redaction needs during setup.

agent_tune_kit-2.2.1/README.en.md

@@ -0,0 +1,185 @@
+# Agent Tune Kit
+
+English | [简体中文](README.md)
+
+[![PyPI](https://img.shields.io/pypi/v/agent-tune-kit.svg)](https://pypi.org/project/agent-tune-kit/)
+
+Agent Tune Kit is a **local Codex plugin** that helps you evaluate an existing Agent, identify failures, and tune prompts, code, parameters, or tool configuration from the results.
+
+Use it to prepare evaluation data, test your Agent in batches, review failures and reports, then check whether your changes help in the next evaluation. Each round keeps its results and tuning notes for review and comparison.
+
+## Before You Start
+
+- A local Agent project that Codex can read and edit, such as a chatbot, tool-using Agent, or RAG Agent.
+- Evaluation data, preferably CSV; 5 to 20 examples are enough to start. If you have no dataset yet, start with business examples.
+- A basis for judging results, such as expected answers, labels, or acceptance criteria.
+
+## Install and Enable
+
+Install ATK and its bundled Skills in your terminal:
+
+```sh
+uv tool install agent-tune-kit
+atk install
+```
+
+In Codex, enter `/plugins`, select and enable `Agent Tune Kit`, then restart Codex and open a new session.
+
+<details>
+<summary>Other installation options: pipx or local source</summary>
+
+With pipx:
+
+```sh
+pipx install agent-tune-kit
+atk install
+```
+
+From local source: update your chosen branch, then run these commands from the **ATK repository root**. Rerun them after source updates; changes are reinstalled even when the version is unchanged.
+
+```sh
+uv tool install --reinstall .
+atk install
+```
+
+</details>
+
+Finish active ATK tasks before reinstalling, and open a new session afterward to load updates. If installation fails, run `atk status` and see the [installation and cache notes](docs/skill-template-pack-usage.md#local-plugin-install-and-smoke). A successful check does not confirm that the plugin is enabled in your current session.
+
+## Complete Your First Evaluation and Tuning Round
+
+**Open your own Agent project in Codex**, then send the Skill instructions below. Installation commands run in a terminal; `$atk-…` instructions belong in your Codex conversation.
+
+### 1. Prepare Data as Needed
+
+Skip to the next step if you already have suitable data. To organize a dataset or fill gaps in evaluation criteria:
+
+```text
+$atk-dataset Prepare data/eval.csv for evaluation. Preserve existing fields and answers; fill only grounded gaps in the judgment criteria.
+```
+
+You can also provide business examples or request a correction, such as changing the expected result for atk_id 7. Fixed column names and exported feedback files are not required. Answers without trustworthy support remain gaps rather than being invented.
+
+Use `$atk-visualize-dataset` when you want a local page for browsing and reviewing data, then give feedback to `$atk-dataset`. Data preparation does not run the Agent.
+
+### 2. Connect Your Agent
+
+Tell Codex where your Agent entry point and evaluation data are:
+
+```text
+$atk-init My Agent entry point is scripts/agent.py and the evaluation dataset is data/eval.csv
+```
+
+Codex reads the project, prepares the adapter, and organizes a runnable dataset. If your Agent is already connected through a compatible adapter, go straight to evaluation.
+
+Evaluation appends five reserved output columns: `agent_output`, `output_status`, `error_type`, `error_message`, and `log_path`. Managed V1/V2 adapters and `atk eval --check` reject name collisions before calling the Agent or allocating a result version; the error lists the dataset path and conflicting columns. For example, after confirming the change, rename a business `output_status` field to `review_status` and update adapter and judgment-rule references. ATK never renames fields or overwrites source values automatically; data preparation may still preserve these business columns unchanged. Old standalone runners need a separate check or migration to the shared engine.
+
+### 3. Evaluate, Diagnose, and Read the Report
+
+Send these instructions in order:
+
+```text
+$atk-eval
+$atk-diagnose
+$atk-report
+```
+
+| Instruction | What you get |
+| --- | --- |
+| `$atk-eval` | Actual Agent outputs and execution records for each sample |
+| `$atk-diagnose` | Passed, failed, or unknown judgments with reasons and an initial investigation of failures |
+| `$atk-report` | Main issues, cause analysis, and tuning recommendations |
+
+Use `$atk-visualize-failures` for a local offline page to search, filter, and review individual failures.
+
+V2 shows start/completion progress by default, counting executions and scored samples separately for grouped cases. Progress goes to stderr so `--json` stdout remains valid JSON. Use `--no-progress` to suppress it; `--check` emits no execution progress.
+
+**Unknown does not count as passed.** Missing answers, criteria, or valid judgment records cannot establish that the Agent behaved correctly.
+
+### 4. Tune from the Results
+
+```text
+$atk-optimize
+```
+
+Codex uses the report and failures to edit your Agent's prompts, code, parameters, or tool configuration. It records why it made changes and what to verify next. Save your current code with Git before tuning so you can compare and restore it.
+
+### Workflow
+
+![Agent Tune Kit workflow: prepare data and connect your Agent as needed, then repeat evaluation, diagnosis, reporting, and tuning. Dataset and failure browsers are optional.](docs/assets/architecture.en.svg)
+
+## Review Results and Verify Improvement
+
+Each round saves results under `.atk/results/vN/` in your Agent project. Start with `report.md` for the main findings, use `failure_cases.html` to review individual failures, and read `tuning_plan.md` for the changes and their rationale.
+
+After tuning, you can first rerun the previous round's confirmed failures:
+
+```text
+$atk-eval --only-failures
+$atk-diagnose
+$atk-report
+```
+
+A new evaluation does not overwrite existing nonempty result directories. From the second round onward, when prior tuning notes exist and conditions are comparable, the report identifies target issues as resolved, partially resolved, unresolved, or impossible to judge.
+
+**Rerunning failures does not measure the full-dataset pass rate.** To check effects on other samples, run `$atk-eval` again, then diagnose and report for a full regression check. Changing data or evaluation criteria alone does not prove Agent improvement.
+
+Execution summaries and `atk results inspect` expose `technical_failures` and `technical_failure_ids`, separately from business judgments. They count reported `output_status=error` rows; a V2 mapper must explicitly translate failures embedded in business outputs. ATK does not guess domain field meanings. Technical errors do not automatically become business failures or enter `--only-failures`.
+
+Use `$atk-eval --ids 7 9` to revisit unknown or selected samples. Managed V1/V2 filter IDs in canonical dataset order before offset/limit; grouped adapters retain full case context. Missing IDs stop execution, and `--ids` is mutually exclusive with `--only-failures`. A selected rerun is not a full regression run.
+
+### Complete Output Structure
+
+This tree lists persistent evaluation outputs and optional files produced by the latest version. They appear as the corresponding steps and configuration require, rather than all being created at installation.
+
+```text
+.atk/
+├── context.md                 # Optional: agreed goals, standards, and human feedback
+├── datasets/
+│   ├── dataset.csv            # Prepared data with sample IDs (atk_id)
+│   ├── dataset.html           # Optional dataset browser
+│   └── dataset_review.csv     # Optional: human-exported feedback placed here
+├── runner/
+│   ├── eval_runner.py         # Agent adapter
+│   └── failure_rule.py        # Project judgment rule, generated as needed
+└── results/
+    ├── v1/
+    │   ├── eval_results.csv    # Actual outputs and execution status per sample
+    │   ├── run.json            # Round status and evidence index
+    │   ├── run-events.jsonl    # As needed: events outside individual executions
+    │   ├── executions/
+    │   │   └── <execution-id>/ # One execution; may serve multiple samples
+    │   │       ├── execution.json
+    │   │       ├── input.json
+    │   │       ├── output.json
+    │   │       ├── context/
+    │   │       │   ├── samples.json
+    │   │       │   ├── environment.json
+    │   │       │   └── snapshot-<N>
+    │   │       ├── events.jsonl
+    │   │       ├── app.log
+    │   │       └── artifacts/  # Additional saved artifacts, as needed
+    │   ├── judgments.csv       # Per-sample judgments and reasons
+    │   ├── judgment-history/   # Previous judgments saved on reassessment
+    │   │   └── <hash>.csv
+    │   ├── failure_cases.csv   # Confirmed failures
+    │   ├── failure_cases.html  # Optional failure browser
+    │   ├── report.md           # Analysis and cross-round validation
+    │   └── tuning_plan.md      # Tuning changes and verification targets
+    └── v2/
+        └── ...                # Later rounds follow the applicable layout
+```
+
+| File or directory | When and why to use it |
+| --- | --- |
+| `datasets/dataset.csv`, `dataset.html` | Review inputs and expected results. The browser is optional; exported review CSVs can also be imported from Downloads |
+| `context.md` | Optionally keep agreed goals, behavior standards, and feedback for future tuning; no manual run statistics are needed |
+| `runner/` | Connect or adjust Agent invocation; Skills use these files during routine evaluation |
+| `eval_results.csv` | Read source fields, actual `agent_output`, execution status `output_status`, and `log_path` |
+| `judgments.csv`, `failure_cases.csv` | Read individual judgments and confirmed failures. Unknowns stay outside the failure set; reassessments preserve old records in `judgment-history/` |
+| `report.md`, `failure_cases.html`, `tuning_plan.md` | Read analysis, review failures, inspect changes, and choose the next verification targets |
+| `run.json`, `executions/`, `run-events.jsonl` | Investigate a run using saved status, inputs, outputs, context, events, and additional artifacts |
+
+Within each execution, `context/` holds relevant samples, environment details, and snapshots such as the adapter file. `app.log` is a view of captured logs and may be empty. Detailed model and tool traces depend on project instrumentation; calls are not all recorded automatically.
+
+Failed or interrupted executions may leave only partial files; a directory does not prove completion. Evidence-only imports may have no `eval_results.csv` and cannot be scored directly. Execution evidence can contain project data, so confirm retention and redaction needs during setup.

agent_tune_kit-2.2.1/README.md

@@ -0,0 +1,185 @@
+# Agent Tune Kit
+
+简体中文 | [English](README.en.md)
+
+[![PyPI](https://img.shields.io/pypi/v/agent-tune-kit.svg)](https://pypi.org/project/agent-tune-kit/)
+
+Agent Tune Kit 是一个**本地 Codex 插件**，帮助你评测已有 Agent、找出失败原因，并根据结果调优 prompt、代码、参数或工具配置。
+
+你可以用它整理评测数据、批量测试 Agent、查看失败样本和分析报告，再通过下一轮评测验证改动是否有效。每轮结果和调优记录都会保留，便于回看与比较。
+
+## 开始前准备什么
+
+- 一个 Codex 可以读取和修改的本地 Agent 项目，例如聊天机器人、工具调用 Agent 或 RAG Agent。
+- 一份评测数据，推荐 CSV，5 到 20 条样例就能开始；没有现成数据也可以先提供业务示例。
+- 判断结果好坏的依据，例如期望答案、标签或可验收的行为标准。
+
+## 安装与启用
+
+在终端安装 ATK 和配套 Skills：
+
+```sh
+uv tool install agent-tune-kit
+atk install
+```
+
+在 Codex 中输入 `/plugins`，选择并启用 `Agent Tune Kit`，然后重启 Codex 并新建会话。
+
+<details>
+<summary>其他安装方式：pipx 或本地源码</summary>
+
+使用 pipx：
+
+```sh
+pipx install agent-tune-kit
+atk install
+```
+
+使用本地源码：更新所需分支的代码后，在 **ATK 仓库根目录**运行以下命令；后续更新源码时重新执行即可，同版本改动也会重新安装。
+
+```sh
+uv tool install --reinstall .
+atk install
+```
+
+</details>
+
+重装前请先结束正在运行的 ATK 任务，安装后新建会话以加载更新。安装有问题时可运行 `atk status` 检查，并参考[安装与缓存说明](docs/skill-template-pack-usage.md#local-plugin-install-and-smoke)；检查通过不代表插件已在当前会话启用。
+
+## 完成第一轮评测与调优
+
+**在 Codex 中打开你自己的 Agent 项目**，再发送下面的 Skill 指令。安装命令在终端运行，`$atk-…` 指令在 Codex 对话中使用。
+
+### 1. 准备数据（按需）
+
+已有合适的数据可以直接进入下一步。需要整理数据或补充评测标准时：
+
+```text
+$atk-dataset 使用 data/eval.csv 准备评测数据，保留原字段和已有答案，只补充缺失且有依据的判断标准。
+```
+
+也可以提供业务示例，或直接提出修订，例如“修改 atk_id 为 7 的预期结果”。无需固定列名，也无需先导出反馈文件。缺少可信依据的答案会保留为空缺，不会编造。
+
+需要人工浏览和复核时，可使用 `$atk-visualize-dataset` 生成本地数据浏览页，再把反馈交给 `$atk-dataset`。准备数据不会运行 Agent。
+
+### 2. 接入 Agent
+
+告诉 Codex 你的 Agent 入口和评测数据位置：
+
+```text
+$atk-init 我的 Agent 入口是 scripts/agent.py，评测数据是 data/eval.csv
+```
+
+Codex 会读取项目、准备接入文件并整理可运行的数据集。已经完成接入且仍兼容时，可直接开始评测。
+
+评测会追加 `agent_output`、`output_status`、`error_type`、`error_message`、`log_path` 五个保留输出列。共享引擎的 V1/V2 接入及 `atk eval --check` 都会在调用 Agent、创建结果版本前检查重名，报错会列出数据集路径和冲突列。例如，业务中的 `output_status` 可在确认后改为 `review_status`，并同步适配器和判定规则的字段引用。ATK 不会自动改名或覆盖原值；数据准备阶段仍可原样保留这些业务字段。旧的自管 runner 需单独核对或迁移到共享引擎。
+
+### 3. 评测、诊断并查看报告
+
+依次发送：
+
+```text
+$atk-eval
+$atk-diagnose
+$atk-report
+```
+
+| 指令 | 你会得到什么 |
+| --- | --- |
+| `$atk-eval` | 每条样本的 Agent 实际输出和执行记录 |
+| `$atk-diagnose` | 通过、失败、未判定的结果及理由，以及失败原因的初步定位 |
+| `$atk-report` | 本轮主要问题、原因分析和调优建议 |
+
+需要逐条搜索、筛选和复核失败样本时，使用 `$atk-visualize-failures` 生成本地离线浏览页。
+
+V2 执行时默认显示开始和完成进度，分别统计执行项与评分样本，避免把一个多样本病例算成一条。进度写到 stderr，`--json` 的 stdout 保持完整 JSON；可用 `--no-progress` 关闭进度，`--check` 不输出执行进度。
+
+**未判定不算通过。** 缺少答案、判断依据或有效判定记录时，不能据此认为 Agent 表现正确。
+
+### 4. 根据结果调优
+
+```text
+$atk-optimize
+```
+
+Codex 会根据报告和失败样本修改 Agent 的 prompt、代码、参数或工具配置，并记录改动原因和下一轮需要验证的问题。建议在调优前用 Git 保存当前代码，方便比较和恢复。
+
+### 使用流程
+
+![Agent Tune Kit 使用流程：按需准备数据并接入 Agent，再重复评测、诊断、报告和调优。数据与失败样本浏览均为可选。](docs/assets/architecture.svg)
+
+## 查看结果与验证改善
+
+每轮结果保存在 Agent 项目的 `.atk/results/vN/` 下。优先查看 `report.md` 了解主要问题，用 `failure_cases.html` 复核具体失败，用 `tuning_plan.md` 查看调优改动及其原因。
+
+调优后，可以先重跑上一轮已确认的失败样本：
+
+```text
+$atk-eval --only-failures
+$atk-diagnose
+$atk-report
+```
+
+已有非空结果目录不会被下一轮评测覆盖。从第二轮开始，如果上一轮有调优记录且实验条件可比，报告会说明目标问题已解决、部分解决、未解决，还是无法判断。
+
+**只重跑失败样本不能代表全量通过率。** 需要确认改动是否影响其他样本时，再运行 `$atk-eval` 并继续诊断、报告，完成全量回归。修改数据或评测标准本身，也不能证明 Agent 改善。
+
+超时等技术失败与业务判定分开统计：运行摘要和 `atk results inspect` 提供 `technical_failures` 与 `technical_failure_ids`。它们只统计适配器已上报的 `output_status=error`；业务输出内部的失败需要 V2 mapper 显式映射，默认不会猜测字段含义。技术失败不会自动写成业务 failed，也不会自动进入 `--only-failures`。
+
+要补测 unknown 或指定样本，可用 `$atk-eval --ids 7 9`。共享 V1/V2 按 canonical 数据集顺序筛选 ID，再应用 offset/limit；分组适配器仍保留完整病例上下文。未知 ID 会停止，`--ids` 与 `--only-failures` 不能同时使用；局部补测不代表完整回归。
+
+### 完整输出结构
+
+下面列出最新版本评测流程中持续保存的正式产物及可选文件。它们随对应步骤和配置生成，不会在安装后一次性出现。
+
+```text
+.atk/
+├── context.md                 # 可选：已确认的调优目标、标准和人工反馈
+├── datasets/
+│   ├── dataset.csv            # 整理后的评测数据，包含样本标识 atk_id
+│   ├── dataset.html           # 可选：数据集浏览页
+│   └── dataset_review.csv     # 可选：人工导出并放回此处的复核反馈
+├── runner/
+│   ├── eval_runner.py         # Agent 接入文件
+│   └── failure_rule.py        # 按需生成：项目判定规则
+└── results/
+    ├── v1/
+    │   ├── eval_results.csv    # 每条样本的实际输出与执行状态
+    │   ├── run.json            # 本轮执行状态与证据索引
+    │   ├── run-events.jsonl    # 按需生成：未归属到单次执行的事件
+    │   ├── executions/
+    │   │   └── <execution-id>/ # 一次执行；可能关联多条样本
+    │   │       ├── execution.json
+    │   │       ├── input.json
+    │   │       ├── output.json
+    │   │       ├── context/
+    │   │       │   ├── samples.json
+    │   │       │   ├── environment.json
+    │   │       │   └── snapshot-<N>
+    │   │       ├── events.jsonl
+    │   │       ├── app.log
+    │   │       └── artifacts/  # 按需生成：额外保存的产物
+    │   ├── judgments.csv       # 每条样本的判定及理由
+    │   ├── judgment-history/   # 重判时保留的历史判定
+    │   │   └── <hash>.csv
+    │   ├── failure_cases.csv   # 已确认失败的样本
+    │   ├── failure_cases.html  # 可选：失败样本浏览页
+    │   ├── report.md           # 本轮分析与跨轮验证
+    │   └── tuning_plan.md      # 本轮调优改动及验证目标
+    └── v2/
+        └── ...                # 后续轮次沿用相应结构
+```
+
+| 文件或目录 | 什么时候看、用来做什么 |
+| --- | --- |
+| `datasets/dataset.csv`、`dataset.html` | 检查测试输入和期望结果；浏览页按需生成，导出的复核 CSV 也可以从下载目录导入 |
+| `context.md` | 可选保存已经确认的目标、行为标准和反馈，供后续调优复用；不需要手工维护运行统计 |
+| `runner/` | 接入或调整 Agent 调用方式时使用，日常评测由 Skills 调用 |
+| `eval_results.csv` | 查看样本原始字段、实际输出 `agent_output`、执行状态 `output_status` 和日志位置 `log_path` |
+| `judgments.csv`、`failure_cases.csv` | 查看逐条判定和确认失败的样本；未判定不进入失败集，重判时旧记录保存在 `judgment-history/` |
+| `report.md`、`failure_cases.html`、`tuning_plan.md` | 阅读分析、复核失败、检查改动，再确定下一轮验证目标 |
+| `run.json`、`executions/`、`run-events.jsonl` | 需要排查某次运行时查看；保存状态、输入输出、上下文、事件和额外产物 |
+
+执行目录中的 `context/` 保存相关样本、运行环境和接入文件等快照；`app.log` 是已采集日志的浏览视图，可能为空。模型和工具调用的详细轨迹取决于项目是否接入采集，并非所有调用都会自动记录。
+
+执行失败或中断时，可能只有部分文件；有目录不等于运行完成。仅导入证据的轮次可能没有 `eval_results.csv`，也不能直接用于评分。执行证据可能包含项目数据，接入时应确认需要保留或脱敏的内容。

{agent_tune_kit-2.2.0 → agent_tune_kit-2.2.1}/agent_tune_kit/__init__.py

@@ -1,3 +1,3 @@
 """Agent Tune Kit package."""
 
-__version__ = "2.2.0"
+__version__ = "2.2.1"

{agent_tune_kit-2.2.0 → agent_tune_kit-2.2.1}/agent_tune_kit/cli/evaluation.py

@@ -7,7 +7,9 @@ def register(subparsers):
     parser.add_argument("--limit", type=int)
     parser.add_argument("--offset", type=int, default=0)
     parser.add_argument("--concurrency", type=int, default=1)
-    parser.add_argument("--only-failures", action="store_true")
+    selection = parser.add_mutually_exclusive_group()
+    selection.add_argument("--only-failures", action="store_true")
+    selection.add_argument("--ids", nargs="+", help="select canonical samples by stable atk_id before offset/limit")
     parser.add_argument("--no-progress", action="store_true")
     parser.add_argument("--python")
     parser.add_argument("--json", action="store_true")
@@ -17,6 +19,6 @@ def execute(args):
     return run_evaluation(
         **{
             key: getattr(args, key)
-            for key in ["check", "limit", "offset", "concurrency", "only_failures", "no_progress", "python"]
+            for key in ["check", "limit", "offset", "concurrency", "only_failures", "ids", "no_progress", "python"]
         }
     )