kc-beta 0.1.2 → 0.3.0

package/template/skills/zh/meta-meta/skill-to-workflow/SKILL.md

@@ -3,273 +3,152 @@ name: skill-to-workflow
 description: Distill a proven verification skill into a Python workflow with worker LLM prompts. Use when a rule skill has been tested and reaches the SKILL_ACCURACY threshold defined in .env. Covers the decision of what to implement as code vs LLM calls, prompt engineering for small context windows, model tier selection and progressive downgrade, and testing workflows against the coding agent's own results as ground truth. Also use when optimizing existing workflows for cost or speed.
 ---
 
-# 技能到工作流的蒸馏
+# Skill to Workflow
 
-## 蒸馏的本质
+The skill is the ground truth. The workflow is a cheaper, faster approximation. Your job is to make the approximation as good as the original while being as cheap as possible.
 
-技能（Skill）是标准答案。它由你这个编程智能体直接执行，调用最强的模型、拥有完整的上下文、不计成本地追求准确率。
+## Engineering Goal
 
-工作流（Workflow）是技能的廉价近似。它由 Python 代码驱动，调用更小、更便宜的Worker LLM（执行模型），在有限的上下文窗口内完成核查。
+Optimize the full chain: **shortest workflow** (fewest nodes) → **smallest model per node** (cheapest tier that meets accuracy) → **shortest prompt per model** (minimum tokens). This is the engineering objective — not prompt template sophistication or framework compliance.
 
-蒸馏的目标：**在成本大幅降低的前提下，尽可能逼近技能的准确率。**
+## When to Start
 
-这不是翻译。你不是把 SKILL.md 翻译成 Python。你是在重新设计核查流程，让一个能力更弱的执行者也能做对。
+A skill is ready for workflow distillation when:
+- It has been tested on all documents in Samples/.
+- Its accuracy meets or exceeds the SKILL_ACCURACY threshold in `.env`.
+- Edge cases are documented in the skill's `assets/corner_cases.json`.
+- You understand the rule well enough to explain exactly how you verify it.
 
-## 启动蒸馏的前提条件
+If any of these are not true, go back and iterate on the skill first.
 
-必须同时满足以下条件才能启动蒸馏：
+## The Distillation Decision
 
-1. **技能测试准确率达标**：在 `assets/samples.json` 和 `assets/corner_cases.json` 上的准确率 ≥ `.env` 中的 `SKILL_ACCURACY` 阈值
-2. **边界案例已充分记录**：至少覆盖了已知的主要例外情形
-3. **判定逻辑已稳定**：最近两轮迭代没有对核心判定逻辑做出修改
+For each step in your skill-based verification process, ask:
 
-如果技能本身还在频繁迭代，不要急于蒸馏。等它稳定下来。
+### Can this be done with regex or Python? (Cost: zero)
+- Date extraction with known formats → regex
+- Numeric comparison against threshold → Python arithmetic
+- Chinese numeral conversion → Python lookup table
+- Format validation (ID numbers, codes) → regex
+- Table cell extraction from structured markdown → string manipulation
 
-## 蒸馏决策：代码还是模型调用
+If yes, write it as code. These are free, fast, and deterministic.
 
-这是蒸馏过程中最关键的决策。原则很简单：
+### Does this require language understanding? (Cost: worker LLM call)
+- Finding the relevant section in a document → LLM
+- Extracting an entity described in natural language → LLM
+- Judging semantic adequacy ("adequate risk disclosure") → LLM
+- Resolving ambiguous references → LLM
 
-### 用 Python 代码实现（零成本）
+If yes, design a worker LLM prompt. Use the smallest model tier that maintains accuracy.
 
-- 日期比较、金额计算、税率换算
-- 正则匹配（发票号码格式、统一社会信用代码校验）
-- 字段存在性检查
-- 格式标准化（大小写、日期格式转换）
-- 枚举值校验（货币代码、国别代码）
-- 数学运算（价税合计 = 不含税金额 × (1 + 税率)）
+### The hybrid approach (most common)
+Most rules are a mix: regex extracts the number, Python compares it to the threshold, LLM handles the exceptional cases. Design the workflow as a pipeline where cheap steps run first and expensive steps run only when needed.
 
-### 用Worker LLM调用实现（有成本）
+## Workflow Structure
 
-- 从非结构化文本中提取关键信息
-- 理解自然语言描述的业务含义
-- 判断两段文字的语义是否一致
-- 识别和解析复杂的表格结构
-- 分类判断（如：该笔费用属于哪个科目）
-
-### 混合方案（推荐）
-
-大多数核查规则的最优实现是混合方案：
-
-```
-1. Python 预处理（格式化、提取结构化字段）
-2. LLM 调用（语义理解、非结构化信息提取）
-3. Python 后处理（逻辑判断、计算、格式化输出）
-```
-
-把 LLM 调用夹在中间，用代码限制它的输入范围和输出格式。这样既能利用 LLM 的语义能力，又能用代码保证确定性。
-
-## 工作流文件结构
+A workflow is a Python file (or small set of files) in `workflows/`:
 
 ```
-workflows/R001-invoice-date-validity/
-├── workflow_v1.py        # 主流程代码
-├── prompts/              # Worker LLM的提示词模板
-│   ├── extract_dates.md  # 日期提取提示词
-│   └── judge_validity.md # 有效性判断提示词
-├── config.json           # 配置（模型层级、参数）
-└── CHANGELOG.md          # 变更记录
+workflows/
+  rule_001_capital_adequacy/
+    workflow_v1.py        # The main workflow script
+    prompts/
+      extract.txt         # Worker LLM prompt for extraction
+      judge.txt           # Worker LLM prompt for judgment (if needed)
+    config.json           # Model assignments, thresholds
 ```
 
-### workflow_v1.py 的结构要求
+The workflow file should have a clear entry point:
 
 ```python
-"""
-R001 - 发票日期有效性核查工作流
-蒸馏自: rule-skills/R001-invoice-date-validity/
-技能准确率: 95%
-蒸馏日期: 2025-04-01
-"""
-
-import json
-import os
-from pathlib import Path
-
-def run_verification(document_data: dict, config: dict) -> dict:
+def verify(document_text: str, config: dict) -> dict:
     """
-    工作流入口函数。
-    
-    Args:
-        document_data: 待核查的单据数据
-        config: 运行时配置（模型选择、API地址等）
-    
     Returns:
-        标准核查结果字典
+        {
+            "rule_id": "R001",
+            "result": "pass" | "fail" | "missing" | "error",
+            "extracted_value": ...,
+            "confidence": 0.0-1.0,
+            "comment": "..." (only when fail),
+            "model_used": "...",
+            "llm_calls": int,
+            "llm_tokens": int
+        }
     """
-    # 步骤1: 预处理（纯代码）
-    # 步骤2: LLM提取（如需要）
-    # 步骤3: 逻辑判断（纯代码）
-    # 步骤4: 格式化输出
-    pass
-```
-
-入口函数必须是 `run_verification`，签名固定。这样质量监控和批量处理可以统一调度。
-
-### config.json
-
-```json
-{
-  "rule_id": "R001",
-  "rule_name": "发票日期有效性",
-  "distilled_from": "rule-skills/R001-invoice-date-validity/",
-  "version": "v1",
-  "model_tier": "TIER3",
-  "llm_steps": ["extract_dates"],
-  "code_steps": ["normalize_format", "compare_dates", "format_output"],
-  "estimated_cost_per_doc": 0.002,
-  "api_base_url": "${API_BASE_URL}",
-  "api_key": "${API_KEY}"
-}
-```
-
-## Worker LLM的提示词工程
-
-Worker LLM不是你。它的上下文窗口更小，推理能力更弱，对业务背景一无所知。提示词必须为它的局限性做设计。
-
-### 自包含原则
-
-提示词不能假设Worker LLM知道任何背景信息。所有必要的上下文都要在提示词中显式提供：
-
-```markdown
-你是一个单据信息提取助手。你的任务是从以下发票文本中提取开票日期。
-
-提取规则：
-- 查找「开票日期」或「Date of Issue」字段
-- 日期格式统一输出为 YYYY-MM-DD
-- 如果找不到日期，输出 null
-- 只提取日期，不要做任何判断
-
-发票文本：
-{invoice_text}
-```
-
-### 结构化输出强制
-
-Worker LLM的输出必须是可解析的。在提示词中明确要求 JSON 格式输出：
-
-```markdown
-请严格按照以下 JSON 格式输出，不要输出任何其他内容：
-
-{
-  "invoice_date": "YYYY-MM-DD 或 null",
-  "extraction_confidence": "high / medium / low"
-}
 ```
 
-### 收窄上下文
-
-不要把整篇文档丢给Worker LLM。只传入它需要处理的那部分内容：
-
-- 如果只需要提取发票日期，只传发票头部区域的文本
-- 如果需要比对合同信息，只传合同中的相关条款段落
-- 上下文越窄，提取越准，成本越低
-
-### 使用单据语言
-
-提示词的指令语言应该与单据语言一致。核查中文单据时，提示词用中文写。这样可以避免Worker LLM在语言切换中引入错误。
-
-### 少量示例策略
-
-在提示词中提供 1-2 个精简的输入输出示例，但不要过多：
-
-- Worker LLM的上下文窗口有限，示例太多会挤占正文空间
-- 选择最典型的正例和一个常见的异常例
-- 示例要简短，只展示关键特征
-
-## 模型层级选择与逐级降级
+This is a reference, not a rigid contract. Adapt the structure to the specific rule. The important thing is that every workflow produces a result that can be compared against the skill-based ground truth.
 
-### 选择策略
+## Prompt Engineering for Worker LLMs
 
-从 TIER1 开始，逐步尝试更低层级。`.env` 中定义了四个层级：
+Worker LLMs have smaller context windows (typically 16K-32K tokens). Design prompts that:
 
-- `TIER1`：最强，适合复杂的语义理解和多步推理
-- `TIER2`：中等，适合需要一定推理的提取和判断
-- `TIER3`：轻量，适合结构化信息提取
-- `TIER4`：最便宜，适合简单的格式提取和分类
+1. **Are self-contained.** Include everything the model needs in the prompt. Do not assume the model has context from previous calls.
+2. **Specify the output format.** "Return a JSON object with fields: value, confidence, reasoning." Structured output reduces parsing errors.
+3. **Include the narrowed context.** Do not send the entire document. Use the tree-processing pipeline (full document → relevant chapter → relevant section) to narrow the context before calling the worker LLM.
+4. **Are written in the document's language.** Chinese documents get Chinese prompts. English documents get English prompts. Do not mix languages in a single prompt.
+5. **Provide examples sparingly.** One or two examples help. Ten examples waste context window and risk overfitting.
 
-### 降级流程
+## Model Tier Selection
 
-```
-1. 用 TIER1 运行全部测试样本，确立准确率天花板
-2. 用 TIER2 运行同一批测试样本，与 TIER1 结果对比
-3. 如果 TIER2 准确率接近 TIER1 → 继续尝试 TIER3
-4. 如果 TIER3 仍然接近 → 继续尝试 TIER4
-5. 选择满足 WORKFLOW_ACCURACY 阈值的最低层级
-6. 如果 TIER1 本身都不达标 → 回到技能层面检查提示词设计
-```
-
-注意：不同步骤可以使用不同层级。比如日期提取用 TIER4，语义判断用 TIER2。在 config.json 中按步骤记录最优层级。
-
-### 正式降级协议
-
-以下数值和流程是推荐起点，编程智能体和开发者用户应根据实际情况自由调整。重要的是模式本身（测试 → 对比 → 记录 → 退化时重评），而非具体数字。
+Start with the highest tier (TIER1) for each step. Measure accuracy. Then try lower tiers:
 
-**方向**：自上而下。先用 TIER1 建立准确率天花板，再逐级尝试更低层级，找到成本与准确率的最优平衡点。
+1. Run the workflow with TIER1 on all Samples/. Record accuracy per step.
+2. For each step, try TIER2. If accuracy stays above WORKFLOW_ACCURACY, keep TIER2.
+3. Continue downgrading per step until accuracy drops below threshold.
+4. Record the optimal tier per step in `config.json`.
 
-**最低测试量**：每个候选层级至少运行 min(10, total_samples) 篇文档。样本量太少则结论不可靠。
+Different steps within the same workflow can use different model tiers. Extraction might need TIER2 while judgment might work fine with TIER3.
 
-**准确率差值判定**：若低一级模型的准确率显著低于上一级（建议阈值：>5个百分点），则停留在较高层级。例如 TIER1 达到 96%、TIER2 只有 89%，则该步骤选定 TIER1。
+### Formal Downgrade Protocol
 
-**逐步骤独立评估**：工作流中每个 LLM 调用步骤独立评估模型层级。步骤 A 可能用 TIER3，步骤 B 可能需要 TIER1。最终结果按步骤分别记录在 config.json 的 `llm_steps` 配置中。
+The basic approach above works, but a more rigorous protocol prevents premature tier commitments:
 
-**退化触发重评**：生产环境质控发现准确率下降时（如 `quality-control` 技能检测到的退化信号），应对相关步骤重新执行降级评估。模型供应商更新、数据分布漂移都可能导致原有选择失效。
+**Direction**: Start top-down (TIER1 → TIER4) to establish the accuracy ceiling first. You need to know the best possible accuracy before trading it for cost savings.
 
-**模型-任务推荐表**：在项目级别维护 task_type → tier 的映射表，积累经验数据。例如「中文发票日期提取 → TIER4」「合同语义比对 → TIER1」。随着测试轮次增多，这张表会成为新规则蒸馏的起点参考。
+**Minimum test runs**: Run at least a meaningful number of documents (e.g., min(10, total_samples)) at each candidate tier before making a tier decision. Small samples are unreliable — a 3-document test could be misleading.
 
-**与文档解析的一致性**：此降级框架与 `document-parsing` 技能中解析器逐级升级的机制同构——都是在层级间做测试、对比、选择。两者可复用相同的评估脚本和判定逻辑。
+**Accuracy delta trigger**: If a lower tier's accuracy is significantly below the higher tier (e.g., >5 percentage points), stay at the higher tier for that step. If the delta is within tolerance, use the cheaper tier.
 
-## 对照真值测试
+**Per-step independence**: Each workflow step is assessed separately. Record the optimal tier per step in `config.json`. Do not assume the whole workflow must use one tier.
 
-技能的核查结果就是真值（Ground Truth）。工作流的测试方法是与技能结果逐字段对比。
+**Re-assessment trigger**: If production quality control shows a step's accuracy degrading (e.g., due to new document formats), re-run the tier assessment for that step.
 
-### 对比维度
+**Model-task recommendation list**: Maintain a per-project mapping of (task_type → recommended_tier) based on your testing experience. Over time, these lists can be collected across projects to build generalized tier recommendations.
 
-- **判定一致性**：工作流的 verdict 是否与技能的 verdict 一致
-- **字段提取准确性**：工作流提取的字段值是否与技能提取的一致
-- **置信度校准**：工作流报告高置信度的案例，是否确实准确率更高
+All numbers here (10 documents, 5 percentage points, etc.) are recommended starting points. The coding agent and developer user should calibrate these — or replace them entirely with a different assessment approach — based on their specific volume, accuracy requirements, and cost constraints. The pattern matters: **test at each tier → compare accuracy → commit when within tolerance → re-assess on degradation**.
 
-### 准确率计算
+This follows the same tier-transition framework as parser escalation in `document-parsing`: a quality/accuracy score drives the decision to stay, escalate, or skip.
 
-```
-工作流准确率 = 与技能判定一致的案例数 / 总案例数
-```
-
-分别计算总体准确率和分类准确率（通过、不通过、无法核查各自的准确率），避免类别不均衡导致的误判。
+## Testing Against Ground Truth
 
-## 版本管理
+The coding agent's skill-based results are the ground truth. For each document in Samples/:
 
-工作流的迭代以文件版本号标识：
+1. Run the workflow.
+2. Compare the workflow's result against the skill-based result.
+3. Log discrepancies: which step failed, what was expected vs actual.
+4. Compute accuracy: `(matching results) / (total documents)`.
+5. If accuracy < WORKFLOW_ACCURACY, diagnose and fix. Use `evolution-loop` methodology.
 
-- `workflow_v1.py` → 初始蒸馏版本
-- `workflow_v2.py` → 优化提示词后的版本
-- `workflow_v3.py` → 更换模型层级后的版本
+## Versioning
 
-不要覆盖旧版本文件。保留完整的版本历史，便于回退和对比。
+Each iteration of a workflow is a new version file: `workflow_v1.py`, `workflow_v2.py`, etc. Track which version is active in `config.json`. See `version-control` skill for the full methodology.
 
-## 成本追踪
+## Cost Tracking
 
-每次工作流运行都记录成本数据：
-
-```json
-{
-  "rule_id": "R001",
-  "workflow_version": "v2",
-  "document_id": "DOC-001",
-  "llm_calls": 2,
-  "total_tokens": 1850,
-  "estimated_cost_usd": 0.003,
-  "model_used": "TIER3",
-  "timestamp": "2025-04-01T10:30:00Z"
-}
-```
+Track the cost of each workflow run:
+- Number of LLM calls per document.
+- Total tokens consumed per document.
+- Model tier used per call.
 
-汇总后用于评估单据平均核查成本，指导模型层级优化方向。
+This data helps the developer user understand the production cost and informs further optimization.
 
-## SiliconFlow API 配置说明
+## Worker LLM API
 
-工作流中调用Worker LLM时，通过 `.env` 中配置的 `API_BASE_URL` 和 `API_KEY` 连接到 SiliconFlow 或其他兼容的 API 服务。
+Worker LLMs are accessed via SiliconFlow API. Connection details are in `.env`:
+- `SILICONFLOW_API_KEY` for authentication
+- `SILICONFLOW_BASE_URL` for the API endpoint
+- Model names in `TIER1` through `TIER4`
 
-调用时注意：
-- 使用标准的 OpenAI 兼容接口格式
-- 设置合理的超时和重试机制
-- 对 API 错误做好降级处理（如某模型不可用时切换到备选模型）
-- 记录每次调用的 token 用量和响应时间
+See `references/worker-llm-catalog.md` for current model capabilities and context window sizes.