kc-beta 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/kc-beta.js +16 -0
- package/package.json +32 -0
- package/src/agent/confidence-scorer.js +120 -0
- package/src/agent/context.js +124 -0
- package/src/agent/corner-case-registry.js +119 -0
- package/src/agent/engine.js +224 -0
- package/src/agent/events.js +27 -0
- package/src/agent/history.js +101 -0
- package/src/agent/llm-client.js +131 -0
- package/src/agent/pipelines/base.js +14 -0
- package/src/agent/pipelines/distillation.js +113 -0
- package/src/agent/pipelines/extraction.js +92 -0
- package/src/agent/pipelines/index.js +23 -0
- package/src/agent/pipelines/initializer.js +163 -0
- package/src/agent/pipelines/production-qc.js +99 -0
- package/src/agent/pipelines/skill-authoring.js +83 -0
- package/src/agent/pipelines/skill-testing.js +111 -0
- package/src/agent/tools/agent-tool.js +100 -0
- package/src/agent/tools/base.js +35 -0
- package/src/agent/tools/dashboard-render.js +146 -0
- package/src/agent/tools/document-parse.js +184 -0
- package/src/agent/tools/document-search.js +111 -0
- package/src/agent/tools/evolution-cycle.js +150 -0
- package/src/agent/tools/qc-sample.js +94 -0
- package/src/agent/tools/registry.js +55 -0
- package/src/agent/tools/rule-catalog.js +113 -0
- package/src/agent/tools/sandbox-exec.js +106 -0
- package/src/agent/tools/tier-downgrade.js +114 -0
- package/src/agent/tools/worker-llm-call.js +109 -0
- package/src/agent/tools/workflow-run.js +138 -0
- package/src/agent/tools/workspace-file.js +122 -0
- package/src/agent/version-manager.js +130 -0
- package/src/agent/workspace.js +82 -0
- package/src/cli/components.js +164 -0
- package/src/cli/index.js +329 -0
- package/src/cli/init.js +80 -0
- package/src/cli/onboard.js +182 -0
- package/src/cli/terminal.js +143 -0
- package/src/config.js +93 -0
- package/template/.env.template +31 -0
- package/template/CLAUDE.md +137 -0
- package/template/Input/.gitkeep +0 -0
- package/template/Output/.gitkeep +0 -0
- package/template/Rules/.gitkeep +0 -0
- package/template/Samples/.gitkeep +0 -0
- package/template/skills/en/meta/compliance-judgment/SKILL.md +114 -0
- package/template/skills/en/meta/compliance-judgment/references/output-format.md +151 -0
- package/template/skills/en/meta/confidence-system/SKILL.md +117 -0
- package/template/skills/en/meta/corner-case-management/SKILL.md +111 -0
- package/template/skills/en/meta/cross-document-verification/SKILL.md +131 -0
- package/template/skills/en/meta/cross-document-verification/references/contradiction-taxonomy.md +73 -0
- package/template/skills/en/meta/data-sensibility/SKILL.md +115 -0
- package/template/skills/en/meta/document-parsing/SKILL.md +108 -0
- package/template/skills/en/meta/document-parsing/references/parser-catalog.md +40 -0
- package/template/skills/en/meta/entity-extraction/SKILL.md +129 -0
- package/template/skills/en/meta/tree-processing/SKILL.md +103 -0
- package/template/skills/en/meta-meta/bootstrap-workspace/SKILL.md +70 -0
- package/template/skills/en/meta-meta/dashboard-reporting/SKILL.md +106 -0
- package/template/skills/en/meta-meta/dashboard-reporting/scripts/generate_dashboard.py +178 -0
- package/template/skills/en/meta-meta/evolution-loop/SKILL.md +210 -0
- package/template/skills/en/meta-meta/evolution-loop/references/convergence-guide.md +62 -0
- package/template/skills/en/meta-meta/quality-control/SKILL.md +138 -0
- package/template/skills/en/meta-meta/quality-control/references/qa-layers.md +92 -0
- package/template/skills/en/meta-meta/quality-control/references/sampling-strategies.md +76 -0
- package/template/skills/en/meta-meta/rule-extraction/SKILL.md +100 -0
- package/template/skills/en/meta-meta/rule-extraction/references/chunking-strategies.md +80 -0
- package/template/skills/en/meta-meta/rule-graph/SKILL.md +118 -0
- package/template/skills/en/meta-meta/skill-authoring/SKILL.md +108 -0
- package/template/skills/en/meta-meta/skill-authoring/references/skill-format-spec.md +78 -0
- package/template/skills/en/meta-meta/skill-to-workflow/SKILL.md +150 -0
- package/template/skills/en/meta-meta/skill-to-workflow/references/worker-llm-catalog.md +50 -0
- package/template/skills/en/meta-meta/task-decomposition/SKILL.md +129 -0
- package/template/skills/en/meta-meta/task-decomposition/references/decision-matrix.md +81 -0
- package/template/skills/en/meta-meta/version-control/SKILL.md +152 -0
- package/template/skills/en/meta-meta/version-control/references/trace-id-spec.md +79 -0
- package/template/skills/en/skill-creator/LICENSE.txt +202 -0
- package/template/skills/en/skill-creator/SKILL.md +479 -0
- package/template/skills/en/skill-creator/agents/analyzer.md +274 -0
- package/template/skills/en/skill-creator/agents/comparator.md +202 -0
- package/template/skills/en/skill-creator/agents/grader.md +223 -0
- package/template/skills/en/skill-creator/assets/eval_review.html +146 -0
- package/template/skills/en/skill-creator/eval-viewer/generate_review.py +471 -0
- package/template/skills/en/skill-creator/eval-viewer/viewer.html +1325 -0
- package/template/skills/en/skill-creator/references/schemas.md +430 -0
- package/template/skills/en/skill-creator/scripts/__init__.py +0 -0
- package/template/skills/en/skill-creator/scripts/aggregate_benchmark.py +401 -0
- package/template/skills/en/skill-creator/scripts/generate_report.py +326 -0
- package/template/skills/en/skill-creator/scripts/improve_description.py +248 -0
- package/template/skills/en/skill-creator/scripts/package_skill.py +136 -0
- package/template/skills/en/skill-creator/scripts/quick_validate.py +103 -0
- package/template/skills/en/skill-creator/scripts/run_eval.py +310 -0
- package/template/skills/en/skill-creator/scripts/run_loop.py +332 -0
- package/template/skills/en/skill-creator/scripts/utils.py +47 -0
- package/template/skills/zh/meta/compliance-judgment/SKILL.md +303 -0
- package/template/skills/zh/meta/compliance-judgment/references/output-format.md +151 -0
- package/template/skills/zh/meta/confidence-system/SKILL.md +228 -0
- package/template/skills/zh/meta/corner-case-management/SKILL.md +235 -0
- package/template/skills/zh/meta/cross-document-verification/SKILL.md +241 -0
- package/template/skills/zh/meta/cross-document-verification/references/contradiction-taxonomy.md +73 -0
- package/template/skills/zh/meta/data-sensibility/SKILL.md +235 -0
- package/template/skills/zh/meta/document-parsing/SKILL.md +168 -0
- package/template/skills/zh/meta/document-parsing/references/parser-catalog.md +40 -0
- package/template/skills/zh/meta/entity-extraction/SKILL.md +276 -0
- package/template/skills/zh/meta/tree-processing/SKILL.md +233 -0
- package/template/skills/zh/meta-meta/bootstrap-workspace/SKILL.md +147 -0
- package/template/skills/zh/meta-meta/dashboard-reporting/SKILL.md +281 -0
- package/template/skills/zh/meta-meta/dashboard-reporting/scripts/generate_dashboard.py +178 -0
- package/template/skills/zh/meta-meta/evolution-loop/SKILL.md +302 -0
- package/template/skills/zh/meta-meta/evolution-loop/references/convergence-guide.md +62 -0
- package/template/skills/zh/meta-meta/quality-control/SKILL.md +269 -0
- package/template/skills/zh/meta-meta/quality-control/references/qa-layers.md +92 -0
- package/template/skills/zh/meta-meta/quality-control/references/sampling-strategies.md +76 -0
- package/template/skills/zh/meta-meta/rule-extraction/SKILL.md +208 -0
- package/template/skills/zh/meta-meta/rule-extraction/references/chunking-strategies.md +80 -0
- package/template/skills/zh/meta-meta/rule-graph/SKILL.md +203 -0
- package/template/skills/zh/meta-meta/skill-authoring/SKILL.md +235 -0
- package/template/skills/zh/meta-meta/skill-authoring/references/skill-format-spec.md +78 -0
- package/template/skills/zh/meta-meta/skill-to-workflow/SKILL.md +275 -0
- package/template/skills/zh/meta-meta/skill-to-workflow/references/worker-llm-catalog.md +50 -0
- package/template/skills/zh/meta-meta/task-decomposition/SKILL.md +224 -0
- package/template/skills/zh/meta-meta/task-decomposition/references/decision-matrix.md +81 -0
- package/template/skills/zh/meta-meta/version-control/SKILL.md +284 -0
- package/template/skills/zh/meta-meta/version-control/references/trace-id-spec.md +79 -0
- package/template/skills/zh/skill-creator/LICENSE.txt +202 -0
- package/template/skills/zh/skill-creator/SKILL.md +479 -0
- package/template/skills/zh/skill-creator/agents/analyzer.md +274 -0
- package/template/skills/zh/skill-creator/agents/comparator.md +202 -0
- package/template/skills/zh/skill-creator/agents/grader.md +223 -0
- package/template/skills/zh/skill-creator/assets/eval_review.html +146 -0
- package/template/skills/zh/skill-creator/eval-viewer/generate_review.py +471 -0
- package/template/skills/zh/skill-creator/eval-viewer/viewer.html +1325 -0
- package/template/skills/zh/skill-creator/references/schemas.md +430 -0
- package/template/skills/zh/skill-creator/scripts/__init__.py +0 -0
- package/template/skills/zh/skill-creator/scripts/aggregate_benchmark.py +401 -0
- package/template/skills/zh/skill-creator/scripts/generate_report.py +326 -0
- package/template/skills/zh/skill-creator/scripts/improve_description.py +248 -0
- package/template/skills/zh/skill-creator/scripts/package_skill.py +136 -0
- package/template/skills/zh/skill-creator/scripts/quick_validate.py +103 -0
- package/template/skills/zh/skill-creator/scripts/run_eval.py +310 -0
- package/template/skills/zh/skill-creator/scripts/run_loop.py +332 -0
- package/template/skills/zh/skill-creator/scripts/utils.py +47 -0
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: skill-authoring
|
|
3
|
+
description: Write each verification rule into a Claude Code skill folder following the official skill format. Use when converting extracted rules into skill folders, when iterating on existing rule skills after testing, or when the developer user wants to capture domain knowledge as a skill. Each skill folder must be self-contained with business logic in SKILL.md, code in scripts/, regulation context in references/, and sample data in assets/. Also use the bundled skill-creator for the full eval/iterate workflow.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Skill Authoring
|
|
7
|
+
|
|
8
|
+
Each verification rule becomes a skill folder. The skill must be self-contained: anyone (or any agent) reading just this folder should have everything needed to verify compliance with that one rule.
|
|
9
|
+
|
|
10
|
+
## Skill Folder Structure
|
|
11
|
+
|
|
12
|
+
Follow the official Claude Code skill format strictly. See `references/skill-format-spec.md` for the complete specification.
|
|
13
|
+
|
|
14
|
+
```
|
|
15
|
+
rule-skills/
|
|
16
|
+
rule-001-capital-adequacy/
|
|
17
|
+
SKILL.md # The verification logic and methodology
|
|
18
|
+
scripts/
|
|
19
|
+
check.py # Deterministic checks (regex, calculations)
|
|
20
|
+
references/
|
|
21
|
+
regulation.md # Original regulation text, verbatim
|
|
22
|
+
interpretation.md # Expert notes on how to interpret edge cases
|
|
23
|
+
assets/
|
|
24
|
+
samples.json # Annotated sample extractions with expected results
|
|
25
|
+
corner_cases.json # Known edge cases with their resolutions
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Not every rule needs all of these. A simple threshold check might only need SKILL.md and a script. A complex semantic rule might need detailed references and many samples. Start minimal, add as needed during testing.
|
|
29
|
+
|
|
30
|
+
## Writing SKILL.md
|
|
31
|
+
|
|
32
|
+
### Frontmatter
|
|
33
|
+
|
|
34
|
+
```yaml
|
|
35
|
+
---
|
|
36
|
+
name: rule-001-capital-adequacy
|
|
37
|
+
description: Verify that the capital adequacy ratio reported in the document meets the regulatory minimum of 8%. Use when checking capital adequacy compliance in bank financial reports. Check the capital adequacy section or table for the reported ratio and compare against the threshold.
|
|
38
|
+
---
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
- **name**: Must match the directory name exactly. Use lowercase, hyphens, no spaces. Prefix with the rule ID from your catalog.
|
|
42
|
+
- **description**: Write it as if explaining to another coding agent when they should use this skill. Be specific about what the rule checks, where to look in the document, and what constitutes pass/fail. Be pushy — include trigger keywords.
|
|
43
|
+
|
|
44
|
+
### Body Content
|
|
45
|
+
|
|
46
|
+
The body should cover:
|
|
47
|
+
|
|
48
|
+
1. **What this rule checks** — one paragraph explaining the rule in plain language. Include the regulatory source and intent.
|
|
49
|
+
|
|
50
|
+
2. **Where to look** — which section, chapter, table, or part of the document contains the relevant information. Be specific. "The capital adequacy ratio is typically found in Chapter 2, Section 'Key Regulatory Metrics' or in the summary table on page 1."
|
|
51
|
+
|
|
52
|
+
3. **What to extract** — the specific entities needed. "Extract the reported capital adequacy ratio as a percentage." Define the expected format and any normalization needed.
|
|
53
|
+
|
|
54
|
+
4. **How to judge** — the logic for pass/fail. "The ratio must be >= 8.0%. If the ratio is missing, flag as MISSING rather than FAIL." For semantic judgments, describe the criteria in natural language.
|
|
55
|
+
|
|
56
|
+
5. **Edge cases** — known tricky situations. "Some reports express the ratio as a decimal (0.12) rather than a percentage (12%). Normalize before comparing."
|
|
57
|
+
|
|
58
|
+
6. **Comment format** — what to say when the rule fails. Keep it concise and actionable. "Capital adequacy ratio is X%, which is below the regulatory minimum of 8%."
|
|
59
|
+
|
|
60
|
+
### Length and Style
|
|
61
|
+
|
|
62
|
+
- Keep SKILL.md under 500 lines. Most rules should be 100-200 lines.
|
|
63
|
+
- Explain the WHY behind the rule, not just the mechanics. Understanding intent helps handle edge cases.
|
|
64
|
+
- Write in imperative form: "Extract the ratio" not "The ratio should be extracted."
|
|
65
|
+
- If detailed regulation text is long, put it in `references/regulation.md` and reference it from SKILL.md.
|
|
66
|
+
|
|
67
|
+
## Writing Scripts
|
|
68
|
+
|
|
69
|
+
Scripts in `scripts/` handle deterministic operations:
|
|
70
|
+
|
|
71
|
+
- **Regex patterns** for entity extraction (dates, amounts, ratios, identifiers).
|
|
72
|
+
- **Calculation logic** for threshold checks, ratio computations, cross-field validation.
|
|
73
|
+
- **Format normalization** (Chinese numerals → digits, date format standardization, unit conversion).
|
|
74
|
+
|
|
75
|
+
Scripts should be self-contained Python files that can be imported or executed. Include clear input/output documentation in the script's docstring.
|
|
76
|
+
|
|
77
|
+
Do not put LLM prompts in scripts. LLM interactions belong in the SKILL.md body or in the workflow (later phase).
|
|
78
|
+
|
|
79
|
+
## Writing References
|
|
80
|
+
|
|
81
|
+
`references/` holds content that the coding agent reads on demand:
|
|
82
|
+
|
|
83
|
+
- **regulation.md**: The original regulation text, verbatim. Include the source, date, and version. This is the ground truth that the rule is derived from.
|
|
84
|
+
- **interpretation.md**: Expert notes from the developer user or from the coding agent's own analysis. "When the regulation says 'adequate disclosure', in practice this means the section must be at least 2 paragraphs and cover risks A, B, and C."
|
|
85
|
+
|
|
86
|
+
Keep references factual and sourced. They are evidence, not instructions.
|
|
87
|
+
|
|
88
|
+
## Writing Assets
|
|
89
|
+
|
|
90
|
+
`assets/` holds data that supports testing and edge case handling:
|
|
91
|
+
|
|
92
|
+
- **samples.json**: Annotated examples. Each entry: the input (extracted text or entity), the expected result (pass/fail/missing), and the expected comment. Build this incrementally as you test.
|
|
93
|
+
- **corner_cases.json**: Edge cases that the standard logic does not handle. Each entry: description, detection pattern, resolution, and confidence threshold. See the `corner-case-management` skill for the methodology.
|
|
94
|
+
|
|
95
|
+
## Iteration
|
|
96
|
+
|
|
97
|
+
Skills evolve through testing. After each test iteration:
|
|
98
|
+
1. Update SKILL.md if the logic needs adjustment.
|
|
99
|
+
2. Add failing cases to `assets/samples.json`.
|
|
100
|
+
3. Add newly discovered edge cases to `assets/corner_cases.json`.
|
|
101
|
+
4. Update `references/interpretation.md` with new insights.
|
|
102
|
+
5. Log what changed and why.
|
|
103
|
+
|
|
104
|
+
Use the bundled `skill-creator` skill if you want to run the full eval/iterate workflow with quantitative benchmarks.
|
|
105
|
+
|
|
106
|
+
## Bilingual Skills
|
|
107
|
+
|
|
108
|
+
Write skills in the language matching the LANGUAGE setting in `.env`. If rules and documents are in Chinese, write the SKILL.md body in Chinese using proper financial/regulatory terminology. The frontmatter (name, description) stays in English for system compatibility.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# Claude Code Skill Format Specification
|
|
2
|
+
|
|
3
|
+
Distilled from the official Anthropic skill-creator. This is the authoritative reference for writing correctly formatted skill folders.
|
|
4
|
+
|
|
5
|
+
## Skill Folder Structure
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
skill-name/
|
|
9
|
+
├── SKILL.md (required) Metadata + instructions
|
|
10
|
+
├── scripts/ (optional) Executable code
|
|
11
|
+
├── references/ (optional) Detailed documentation, loaded on demand
|
|
12
|
+
└── assets/ (optional) Templates, data files, images
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The directory name must match the `name` field in SKILL.md frontmatter exactly.
|
|
16
|
+
|
|
17
|
+
## SKILL.md Format
|
|
18
|
+
|
|
19
|
+
### Frontmatter (YAML)
|
|
20
|
+
|
|
21
|
+
```yaml
|
|
22
|
+
---
|
|
23
|
+
name: skill-identifier
|
|
24
|
+
description: What this skill does and when to use it.
|
|
25
|
+
---
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
**Required fields:**
|
|
29
|
+
|
|
30
|
+
| Field | Constraints |
|
|
31
|
+
|-------|-------------|
|
|
32
|
+
| `name` | Max 64 chars. Lowercase letters, numbers, hyphens only. No leading/trailing/consecutive hyphens. Must match parent directory name. |
|
|
33
|
+
| `description` | Max 1024 chars. Non-empty. Describe what it does AND when to use it. |
|
|
34
|
+
|
|
35
|
+
**Optional fields:** `license`, `compatibility`, `metadata`
|
|
36
|
+
|
|
37
|
+
### Description Best Practices
|
|
38
|
+
|
|
39
|
+
The description is the primary triggering mechanism — Claude uses it to decide when to invoke the skill. Make descriptions "pushy" to combat under-triggering:
|
|
40
|
+
|
|
41
|
+
- Include both capability AND trigger contexts.
|
|
42
|
+
- Use specific keywords the user might mention.
|
|
43
|
+
- List concrete use cases.
|
|
44
|
+
- State what NOT to use it for.
|
|
45
|
+
- Aim for 100-200 words.
|
|
46
|
+
|
|
47
|
+
### Markdown Body
|
|
48
|
+
|
|
49
|
+
Following the frontmatter, write instructions in Markdown. Guidelines:
|
|
50
|
+
|
|
51
|
+
- **Under 500 lines.** If approaching this, move detail to references/.
|
|
52
|
+
- **Imperative form.** "Extract the value" not "The value should be extracted."
|
|
53
|
+
- **Explain the why** behind instructions, not just the what.
|
|
54
|
+
- **Include examples** when they clarify the expected behavior. One or two well-chosen examples beat ten mediocre ones.
|
|
55
|
+
|
|
56
|
+
## Progressive Disclosure
|
|
57
|
+
|
|
58
|
+
Skills use three-level loading:
|
|
59
|
+
|
|
60
|
+
1. **Metadata** (name + description): Always in context. ~100 tokens.
|
|
61
|
+
2. **SKILL.md body**: Loaded when skill triggers. <500 lines ideal.
|
|
62
|
+
3. **Bundled resources**: Loaded on demand. Unlimited size. Scripts can execute without loading.
|
|
63
|
+
|
|
64
|
+
Reference files clearly from SKILL.md with guidance on when to read them. For large reference files (>300 lines), include a table of contents.
|
|
65
|
+
|
|
66
|
+
## File Referencing
|
|
67
|
+
|
|
68
|
+
Use relative paths from the skill root:
|
|
69
|
+
```markdown
|
|
70
|
+
See [the reference guide](references/regulation.md) for the full regulation text.
|
|
71
|
+
Run the check script: `python scripts/check.py`
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
## Naming Conventions
|
|
75
|
+
|
|
76
|
+
- Directory names: lowercase with hyphens (`my-skill`, not `MySkill` or `my_skill`)
|
|
77
|
+
- Keep names short and descriptive
|
|
78
|
+
- For rule skills, prefix with the rule ID: `rule-001-capital-adequacy`
|
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: skill-to-workflow
|
|
3
|
+
description: Distill a proven verification skill into a Python workflow with worker LLM prompts. Use when a rule skill has been tested and reaches the SKILL_ACCURACY threshold defined in .env. Covers the decision of what to implement as code vs LLM calls, prompt engineering for small context windows, model tier selection and progressive downgrade, and testing workflows against the coding agent's own results as ground truth. Also use when optimizing existing workflows for cost or speed.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Skill to Workflow
|
|
7
|
+
|
|
8
|
+
The skill is the ground truth. The workflow is a cheaper, faster approximation. Your job is to make the approximation as good as the original while being as cheap as possible.
|
|
9
|
+
|
|
10
|
+
## When to Start
|
|
11
|
+
|
|
12
|
+
A skill is ready for workflow distillation when:
|
|
13
|
+
- It has been tested on all documents in Samples/.
|
|
14
|
+
- Its accuracy meets or exceeds the SKILL_ACCURACY threshold in `.env`.
|
|
15
|
+
- Edge cases are documented in the skill's `assets/corner_cases.json`.
|
|
16
|
+
- You understand the rule well enough to explain exactly how you verify it.
|
|
17
|
+
|
|
18
|
+
If any of these are not true, go back and iterate on the skill first.
|
|
19
|
+
|
|
20
|
+
## The Distillation Decision
|
|
21
|
+
|
|
22
|
+
For each step in your skill-based verification process, ask:
|
|
23
|
+
|
|
24
|
+
### Can this be done with regex or Python? (Cost: zero)
|
|
25
|
+
- Date extraction with known formats → regex
|
|
26
|
+
- Numeric comparison against threshold → Python arithmetic
|
|
27
|
+
- Chinese numeral conversion → Python lookup table
|
|
28
|
+
- Format validation (ID numbers, codes) → regex
|
|
29
|
+
- Table cell extraction from structured markdown → string manipulation
|
|
30
|
+
|
|
31
|
+
If yes, write it as code. These are free, fast, and deterministic.
|
|
32
|
+
|
|
33
|
+
### Does this require language understanding? (Cost: worker LLM call)
|
|
34
|
+
- Finding the relevant section in a document → LLM
|
|
35
|
+
- Extracting an entity described in natural language → LLM
|
|
36
|
+
- Judging semantic adequacy ("adequate risk disclosure") → LLM
|
|
37
|
+
- Resolving ambiguous references → LLM
|
|
38
|
+
|
|
39
|
+
If yes, design a worker LLM prompt. Use the smallest model tier that maintains accuracy.
|
|
40
|
+
|
|
41
|
+
### The hybrid approach (most common)
|
|
42
|
+
Most rules are a mix: regex extracts the number, Python compares it to the threshold, LLM handles the exceptional cases. Design the workflow as a pipeline where cheap steps run first and expensive steps run only when needed.
|
|
43
|
+
|
|
44
|
+
## Workflow Structure
|
|
45
|
+
|
|
46
|
+
A workflow is a Python file (or small set of files) in `workflows/`:
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
workflows/
|
|
50
|
+
rule_001_capital_adequacy/
|
|
51
|
+
workflow_v1.py # The main workflow script
|
|
52
|
+
prompts/
|
|
53
|
+
extract.txt # Worker LLM prompt for extraction
|
|
54
|
+
judge.txt # Worker LLM prompt for judgment (if needed)
|
|
55
|
+
config.json # Model assignments, thresholds
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The workflow file should have a clear entry point:
|
|
59
|
+
|
|
60
|
+
```python
|
|
61
|
+
def verify(document_text: str, config: dict) -> dict:
|
|
62
|
+
"""
|
|
63
|
+
Returns:
|
|
64
|
+
{
|
|
65
|
+
"rule_id": "R001",
|
|
66
|
+
"result": "pass" | "fail" | "missing" | "error",
|
|
67
|
+
"extracted_value": ...,
|
|
68
|
+
"confidence": 0.0-1.0,
|
|
69
|
+
"comment": "..." (only when fail),
|
|
70
|
+
"model_used": "...",
|
|
71
|
+
"llm_calls": int,
|
|
72
|
+
"llm_tokens": int
|
|
73
|
+
}
|
|
74
|
+
"""
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
This is a reference, not a rigid contract. Adapt the structure to the specific rule. The important thing is that every workflow produces a result that can be compared against the skill-based ground truth.
|
|
78
|
+
|
|
79
|
+
## Prompt Engineering for Worker LLMs
|
|
80
|
+
|
|
81
|
+
Worker LLMs have smaller context windows (typically 16K-32K tokens). Design prompts that:
|
|
82
|
+
|
|
83
|
+
1. **Are self-contained.** Include everything the model needs in the prompt. Do not assume the model has context from previous calls.
|
|
84
|
+
2. **Specify the output format.** "Return a JSON object with fields: value, confidence, reasoning." Structured output reduces parsing errors.
|
|
85
|
+
3. **Include the narrowed context.** Do not send the entire document. Use the tree-processing pipeline (full document → relevant chapter → relevant section) to narrow the context before calling the worker LLM.
|
|
86
|
+
4. **Are written in the document's language.** Chinese documents get Chinese prompts. English documents get English prompts. Do not mix languages in a single prompt.
|
|
87
|
+
5. **Provide examples sparingly.** One or two examples help. Ten examples waste context window and risk overfitting.
|
|
88
|
+
|
|
89
|
+
## Model Tier Selection
|
|
90
|
+
|
|
91
|
+
Start with the highest tier (TIER1) for each step. Measure accuracy. Then try lower tiers:
|
|
92
|
+
|
|
93
|
+
1. Run the workflow with TIER1 on all Samples/. Record accuracy per step.
|
|
94
|
+
2. For each step, try TIER2. If accuracy stays above WORKFLOW_ACCURACY, keep TIER2.
|
|
95
|
+
3. Continue downgrading per step until accuracy drops below threshold.
|
|
96
|
+
4. Record the optimal tier per step in `config.json`.
|
|
97
|
+
|
|
98
|
+
Different steps within the same workflow can use different model tiers. Extraction might need TIER2 while judgment might work fine with TIER3.
|
|
99
|
+
|
|
100
|
+
### Formal Downgrade Protocol
|
|
101
|
+
|
|
102
|
+
The basic approach above works, but a more rigorous protocol prevents premature tier commitments:
|
|
103
|
+
|
|
104
|
+
**Direction**: Start top-down (TIER1 → TIER4) to establish the accuracy ceiling first. You need to know the best possible accuracy before trading it for cost savings.
|
|
105
|
+
|
|
106
|
+
**Minimum test runs**: Run at least a meaningful number of documents (e.g., min(10, total_samples)) at each candidate tier before making a tier decision. Small samples are unreliable — a 3-document test could be misleading.
|
|
107
|
+
|
|
108
|
+
**Accuracy delta trigger**: If a lower tier's accuracy is significantly below the higher tier (e.g., >5 percentage points), stay at the higher tier for that step. If the delta is within tolerance, use the cheaper tier.
|
|
109
|
+
|
|
110
|
+
**Per-step independence**: Each workflow step is assessed separately. Record the optimal tier per step in `config.json`. Do not assume the whole workflow must use one tier.
|
|
111
|
+
|
|
112
|
+
**Re-assessment trigger**: If production quality control shows a step's accuracy degrading (e.g., due to new document formats), re-run the tier assessment for that step.
|
|
113
|
+
|
|
114
|
+
**Model-task recommendation list**: Maintain a per-project mapping of (task_type → recommended_tier) based on your testing experience. Over time, these lists can be collected across projects to build generalized tier recommendations.
|
|
115
|
+
|
|
116
|
+
All numbers here (10 documents, 5 percentage points, etc.) are recommended starting points. The coding agent and developer user should calibrate these — or replace them entirely with a different assessment approach — based on their specific volume, accuracy requirements, and cost constraints. The pattern matters: **test at each tier → compare accuracy → commit when within tolerance → re-assess on degradation**.
|
|
117
|
+
|
|
118
|
+
This follows the same tier-transition framework as parser escalation in `document-parsing`: a quality/accuracy score drives the decision to stay, escalate, or skip.
|
|
119
|
+
|
|
120
|
+
## Testing Against Ground Truth
|
|
121
|
+
|
|
122
|
+
The coding agent's skill-based results are the ground truth. For each document in Samples/:
|
|
123
|
+
|
|
124
|
+
1. Run the workflow.
|
|
125
|
+
2. Compare the workflow's result against the skill-based result.
|
|
126
|
+
3. Log discrepancies: which step failed, what was expected vs actual.
|
|
127
|
+
4. Compute accuracy: `(matching results) / (total documents)`.
|
|
128
|
+
5. If accuracy < WORKFLOW_ACCURACY, diagnose and fix. Use `evolution-loop` methodology.
|
|
129
|
+
|
|
130
|
+
## Versioning
|
|
131
|
+
|
|
132
|
+
Each iteration of a workflow is a new version file: `workflow_v1.py`, `workflow_v2.py`, etc. Track which version is active in `config.json`. See `version-control` skill for the full methodology.
|
|
133
|
+
|
|
134
|
+
## Cost Tracking
|
|
135
|
+
|
|
136
|
+
Track the cost of each workflow run:
|
|
137
|
+
- Number of LLM calls per document.
|
|
138
|
+
- Total tokens consumed per document.
|
|
139
|
+
- Model tier used per call.
|
|
140
|
+
|
|
141
|
+
This data helps the developer user understand the production cost and informs further optimization.
|
|
142
|
+
|
|
143
|
+
## Worker LLM API
|
|
144
|
+
|
|
145
|
+
Worker LLMs are accessed via SiliconFlow API. Connection details are in `.env`:
|
|
146
|
+
- `SILICONFLOW_API_KEY` for authentication
|
|
147
|
+
- `SILICONFLOW_BASE_URL` for the API endpoint
|
|
148
|
+
- Model names in `TIER1` through `TIER4`
|
|
149
|
+
|
|
150
|
+
See `references/worker-llm-catalog.md` for current model capabilities and context window sizes.
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# Worker LLM Catalog
|
|
2
|
+
|
|
3
|
+
Models available via SiliconFlow API for worker LLM tasks. Update this catalog as models change.
|
|
4
|
+
|
|
5
|
+
## Text Models
|
|
6
|
+
|
|
7
|
+
| Tier | Model | Context Window | Strengths | Notes |
|
|
8
|
+
|------|-------|---------------|-----------|-------|
|
|
9
|
+
| TIER1 | Pro/zai-org/GLM-5 | 128K | Strong reasoning, Chinese language | Top tier for complex judgment |
|
|
10
|
+
| TIER1 | Pro/moonshotai/Kimi-K2.5 | 128K | Long context, strong extraction | Good for full-document processing |
|
|
11
|
+
| TIER2 | Pro/deepseek-ai/DeepSeek-V3.2 | 64K | Balanced capability/cost | Good general purpose |
|
|
12
|
+
| TIER2 | Pro/MiniMaxAI/MiniMax-M2.5 | 64K | Strong Chinese, fast | Good for Chinese documents |
|
|
13
|
+
| TIER2 | Qwen/Qwen3.5-397B-A17B | 32K | Large MoE, strong reasoning | Cost-effective for complex tasks |
|
|
14
|
+
| TIER3 | Qwen/Qwen3.5-122B-A10B | 32K | Good accuracy, lower cost | Sweet spot for many tasks |
|
|
15
|
+
| TIER4 | Qwen/Qwen3.5-35B-A3B | 16K | Fast, cheap | Best for simple extraction |
|
|
16
|
+
|
|
17
|
+
## Vision/OCR Models
|
|
18
|
+
|
|
19
|
+
| Tier | Model | Strengths | Notes |
|
|
20
|
+
|------|-------|-----------|-------|
|
|
21
|
+
| OCR_TIER1 | zai-org/GLM-4.6V | Best OCR accuracy | Use for complex tables/charts |
|
|
22
|
+
| OCR_TIER2 | Qwen/Qwen3.5-397B-A17B | Good general vision | Multimodal version |
|
|
23
|
+
| OCR_TIER3 | PaddlePaddle/PaddleOCR-VL-1.5 | Fast, lightweight OCR | Best for standard text |
|
|
24
|
+
|
|
25
|
+
## Selection Guidelines
|
|
26
|
+
|
|
27
|
+
- Start with the highest tier that fits your context window needs.
|
|
28
|
+
- For extraction of simple entities (dates, amounts, names): TIER3-4 often sufficient.
|
|
29
|
+
- For semantic judgment (adequacy, compliance): TIER1-2 usually needed.
|
|
30
|
+
- For Chinese financial documents: prefer GLM and Qwen models over DeepSeek for domain terminology.
|
|
31
|
+
- Context window constraint: if the section to process exceeds the model's window, either narrow the context further (tree processing) or use a model with a larger window.
|
|
32
|
+
|
|
33
|
+
## API Configuration
|
|
34
|
+
|
|
35
|
+
```python
|
|
36
|
+
import openai
|
|
37
|
+
|
|
38
|
+
client = openai.OpenAI(
|
|
39
|
+
api_key=os.getenv("SILICONFLOW_API_KEY"),
|
|
40
|
+
base_url=os.getenv("SILICONFLOW_BASE_URL")
|
|
41
|
+
)
|
|
42
|
+
|
|
43
|
+
response = client.chat.completions.create(
|
|
44
|
+
model="Qwen/Qwen3.5-122B-A10B", # Use the model name from the table
|
|
45
|
+
messages=[{"role": "user", "content": prompt}],
|
|
46
|
+
temperature=0.1 # Low temperature for deterministic extraction
|
|
47
|
+
)
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
This catalog should be maintained by the coding agent. Add new models as they become available, remove deprecated models, and update capability assessments based on testing experience.
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: task-decomposition
|
|
3
|
+
description: Decompose each verification rule into independent sub-tasks and assign the optimal method (rule, code, LLM, manual) to each. Use when converting extracted rules into implementation plans, when a rule skill is too expensive or inaccurate and needs restructuring, or when designing a multi-step verification pipeline. Covers MECE decomposition, method selection via the four-dimension decision matrix, cost-benefit analysis, and source tagging. Also use when auditing an existing workflow for cost optimization opportunities.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Task Decomposition
|
|
7
|
+
|
|
8
|
+
Every verification rule is composite. Even the simplest-sounding rule — "check that the invoice date is within the contract period" — decomposes into a chain of distinct operations: locate the date field, extract its value, normalize the format, compare against the contract dates, and generate a comment if it fails.
|
|
9
|
+
|
|
10
|
+
The temptation is to throw the entire chain at an LLM. It works. It is also 100x more expensive than necessary and impossible to debug when it breaks.
|
|
11
|
+
|
|
12
|
+
The Lancet Method is scalpel-precision decomposition. Cut each rule into the smallest sub-tasks that are methodologically homogeneous — meaning each sub-task can be solved entirely by one method. Then assign the cheapest method that works for each sub-task. The name is deliberate: a lancet, not a cleaver. Precision matters because the cut points determine everything downstream — cost, debuggability, testability, and the eventual workflow architecture.
|
|
13
|
+
|
|
14
|
+
## MECE Decomposition
|
|
15
|
+
|
|
16
|
+
Decompose every rule into sub-tasks that are mutually exclusive and collectively exhaustive (MECE):
|
|
17
|
+
|
|
18
|
+
- **Mutually exclusive**: no two sub-tasks do the same work. If sub-task A extracts the invoice date, sub-task B does not also extract the invoice date.
|
|
19
|
+
- **Collectively exhaustive**: the sub-tasks together cover the entire rule. Nothing falls through the cracks. If you execute all sub-tasks in sequence, the rule is fully verified.
|
|
20
|
+
|
|
21
|
+
Each sub-task has exactly one input and one output. The output of one sub-task becomes the input of the next. This creates a pipeline with clean interfaces between stages.
|
|
22
|
+
|
|
23
|
+
Stop decomposing when a sub-task is **methodologically homogeneous** — it can be handled entirely by one method (regex, Python code, LLM call, or manual review). If a sub-task still requires two different methods, it is not yet atomic. Keep cutting.
|
|
24
|
+
|
|
25
|
+
A practical test: describe the sub-task in one sentence. If you need "and" or "then" in the sentence, it probably needs further decomposition. "Extract the date and compare it to the threshold" is two sub-tasks. "Extract the date" is one.
|
|
26
|
+
|
|
27
|
+
The canonical decomposition chain for most document verification rules is:
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
locate → extract → normalize → judge → comment
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Not every rule has all five stages. Some rules skip normalization. Some rules do not need a comment on pass. But this chain is a reliable starting framework.
|
|
34
|
+
|
|
35
|
+
For cross-document rules (e.g., "invoice amount matches contract amount"), the chain branches: two parallel locate-extract-normalize pipelines converge at a single judge step. Draw this out before implementing. The pipeline topology — linear, branching, or converging — determines how you structure the skill folder and later the workflow.
|
|
36
|
+
|
|
37
|
+
Three common topologies:
|
|
38
|
+
|
|
39
|
+
- **Linear**: Single document, single field. `locate → extract → normalize → judge → comment`. Most threshold checks follow this pattern.
|
|
40
|
+
- **Converging**: Two fields from different documents or different sections. Two parallel locate-extract chains merge at the judge step. Cross-field validations and cross-document matching follow this pattern.
|
|
41
|
+
- **Fan-out**: One rule applied to many items within a document (e.g., validating every line item in an invoice). The locate step produces N items, each of which flows through the remaining chain independently. Scale is the critical dimension here — if N is large, the method assignments must account for per-item cost.
|
|
42
|
+
|
|
43
|
+
## The Decision Matrix
|
|
44
|
+
|
|
45
|
+
After decomposition, assign a method to each sub-task. Do not guess. Use a structured evaluation based on four dimensions. See `references/decision-matrix.md` for the complete matrix with worked examples and a cost estimation template.
|
|
46
|
+
|
|
47
|
+
**Certainty** — How predictable is the input format? If the date is always in `YYYY-MM-DD` at a known position, certainty is high. If the date appears in free-form prose with varying formats, certainty is low.
|
|
48
|
+
|
|
49
|
+
**Scale** — How many items must be processed? One field per document is low scale. A thousand line items per invoice is high scale.
|
|
50
|
+
|
|
51
|
+
**Semantic depth** — How much language understanding is required? Comparing two numbers requires none. Judging whether a risk disclosure is "adequate" requires deep understanding.
|
|
52
|
+
|
|
53
|
+
**Cost sensitivity** — What is the budget per document? A bank processing 10,000 loan files per month has different economics than a one-time audit of 50 contracts.
|
|
54
|
+
|
|
55
|
+
These four dimensions map to a method hierarchy. Always prefer the cheapest method that achieves the required accuracy:
|
|
56
|
+
|
|
57
|
+
1. **Rule / Regex** — Zero cost, instant, deterministic. Use when certainty is high and semantic depth is zero.
|
|
58
|
+
2. **Code / Python** — Zero cost, instant, deterministic. Use for calculations, transformations, and structured comparisons.
|
|
59
|
+
3. **LLM** — Variable cost, latency, probabilistic. Use when semantic understanding is required and cheaper methods fail.
|
|
60
|
+
4. **Manual** — Highest cost, highest latency, highest accuracy. Reserve for edge cases that defeat all automated methods.
|
|
61
|
+
|
|
62
|
+
Do not skip levels. Try regex before code. Try code before LLM. Try LLM before manual. Each escalation must be justified by a failure at the lower level.
|
|
63
|
+
|
|
64
|
+
When scoring a sub-task on these dimensions, be honest about uncertainty. If you are unsure whether a regex can handle the input variability, score certainty conservatively and test the regex on samples before committing. A wrong method assignment wastes more time than a conservative initial assignment that gets optimized later.
|
|
65
|
+
|
|
66
|
+
Note that dimensions interact. High scale combined with high cost sensitivity pushes hard toward code-based solutions even when moderate semantic depth would normally suggest LLM. Conversely, low scale relaxes cost pressure, making LLM viable even for tasks that could theoretically be solved with complex regex. Let the combination of dimensions guide you, not any single dimension alone.
|
|
67
|
+
|
|
68
|
+
## Cost-Benefit Awareness
|
|
69
|
+
|
|
70
|
+
Method assignment is not an academic exercise. It directly determines the cost per document in production. Every LLM call that could have been a regex is money burned. Every regex that should have been an LLM call is accuracy lost.
|
|
71
|
+
|
|
72
|
+
Consider a real scenario: matching invoices against contracts in a large enterprise. There are 31,800 invoices and 15,940 contracts. The naive approach — send every possible pair to an LLM for comparison — means 507 million pairs. At any non-trivial cost per call, this is economically absurd.
|
|
73
|
+
|
|
74
|
+
The Lancet Method decomposes this into layers:
|
|
75
|
+
|
|
76
|
+
1. **Rule layer**: Match on exact supplier name and contract number. Cost: near zero. Eliminates 99.5% of pairs.
|
|
77
|
+
2. **Code layer**: Fuzzy match on amount ranges and date overlap. Cost: near zero. Reduces to 12,400 candidate pairs.
|
|
78
|
+
3. **LLM layer**: Semantic comparison of line-item descriptions against contract scope. Cost: moderate. Reduces to 7,652 confirmed matches.
|
|
79
|
+
4. **Manual layer**: Human review of ~200 low-confidence matches where the LLM was uncertain. Cost: labor hours. Resolves the final ambiguous cases.
|
|
80
|
+
|
|
81
|
+
The result: 200x lower cost than the naive approach. Same accuracy. Better debuggability because each layer's output is independently verifiable. And each layer can be tested, monitored, and optimized in isolation.
|
|
82
|
+
|
|
83
|
+
The principle: **filter cheap before reasoning expensive**. Always calculate the cost per document for each sub-task. If the LLM cost for one sub-task dominates the total, that sub-task is the optimization target.
|
|
84
|
+
|
|
85
|
+
Use the cost estimation template in `references/decision-matrix.md` to plan costs at decomposition time. Do not wait until production to discover that a workflow is too expensive. The developer user has a budget. Respect it by designing within it.
|
|
86
|
+
|
|
87
|
+
## Source Tagging
|
|
88
|
+
|
|
89
|
+
Every output from every sub-task must carry an `extraction_method` tag. This is not optional metadata — it is load-bearing infrastructure. Without it, the system degrades into an opaque pipeline that nobody can diagnose, cost-optimize, or trust.
|
|
90
|
+
|
|
91
|
+
Tags enable three capabilities that you cannot afford to lose:
|
|
92
|
+
|
|
93
|
+
1. **Debugging**: When a verification result is wrong, the tag tells you which sub-task produced the error and which method was responsible. Without tags, you are debugging a black box. With tags, you can immediately narrow the investigation to one sub-task and one method.
|
|
94
|
+
2. **Cost attribution**: Tags let you calculate the actual cost contribution of each method per rule and per document. This drives optimization decisions — you can identify which LLM calls are consuming the most budget and target them for replacement with cheaper methods.
|
|
95
|
+
3. **Confidence calibration**: Different methods have different reliability profiles. A regex extraction is either right or wrong — binary confidence. An LLM extraction has a confidence distribution that varies by model tier and prompt quality. Tags feed directly into the `confidence-system` method prior, enabling calibrated confidence scores that reflect the actual reliability of each extraction source.
|
|
96
|
+
|
|
97
|
+
Tag format: a simple string field on every intermediate output. Example values: `regex`, `python_calc`, `llm_tier2`, `manual_review`. Be consistent within a project. Define the tag vocabulary once at project setup and enforce it across all skills and workflows.
|
|
98
|
+
|
|
99
|
+
## Anti-Patterns
|
|
100
|
+
|
|
101
|
+
Five failure modes recur across projects. Learn to recognize them early.
|
|
102
|
+
|
|
103
|
+
**LLM-for-everything.** Sending an entire document to an LLM with "check if this complies with Rule X" works in demos. In production, it costs 100x more than a decomposed pipeline, provides no accuracy gain for deterministic checks, and is impossible to debug because you cannot tell which sub-check failed. The diagnostic signal: if a sub-task's input is fully predictable and requires zero language understanding, it does not belong in an LLM call.
|
|
104
|
+
|
|
105
|
+
**Rule over-engineering.** Building a 500-line regex to handle every possible date format when an LLM handles normalization better. If a rule becomes brittle and requires constant maintenance, the sub-task belongs at a higher method level. The diagnostic signal: if the regex needs patching after every new document batch, the sub-task has outgrown regex.
|
|
106
|
+
|
|
107
|
+
**Black-box pipeline.** Chaining sub-tasks without intermediate outputs. When the final result is wrong, you cannot tell where the error entered. Every sub-task must produce a logged, inspectable intermediate result. If debugging a rule requires re-running the entire pipeline end-to-end, the pipeline lacks checkpoints.
|
|
108
|
+
|
|
109
|
+
**Monolithic end-to-end.** Running every sub-task for every document, even when an early sub-task could short-circuit the pipeline. If the locate step finds that the relevant section does not exist, skip extract, normalize, judge, and comment. Go directly to "field missing." Short-circuit logic saves both cost and time.
|
|
110
|
+
|
|
111
|
+
**Premature optimization.** Spending days designing the optimal method assignment before testing anything. Get the decomposition right first. Assign all sub-tasks to LLM. Prove it works end-to-end on Samples/. Then optimize by pushing sub-tasks down to cheaper methods one at a time, verifying accuracy is maintained at each step. Correctness first, cost second. The decomposition itself is the hard part — method assignment can always be revised later.
|
|
112
|
+
|
|
113
|
+
## Integration
|
|
114
|
+
|
|
115
|
+
Task decomposition sits between rule extraction and skill authoring in the KC Reborn lifecycle. It is the bridge that translates abstract rules into concrete implementation plans.
|
|
116
|
+
|
|
117
|
+
**Input**: A rule catalog from `rule-extraction`. Each rule is an atomic, testable verification requirement. If a rule is not yet atomic, send it back to rule extraction for further decomposition before attempting task decomposition.
|
|
118
|
+
|
|
119
|
+
**Output**: A per-rule sub-task decomposition — a list of sub-tasks, each with a defined input, output, and assigned method. This decomposition feeds directly into `skill-authoring`, where each rule's sub-tasks become the implementation plan for the skill folder. The decomposition also serves as the testing contract: each sub-task's output is independently testable.
|
|
120
|
+
|
|
121
|
+
Method assignments also inform tier selection in `skill-to-workflow`. When a skill is distilled into a workflow, the method assignments from decomposition become the initial workflow architecture:
|
|
122
|
+
|
|
123
|
+
- Regex and code sub-tasks become deterministic code in `scripts/`.
|
|
124
|
+
- LLM sub-tasks become worker LLM prompts in `prompts/`, with model tier selected per the `skill-to-workflow` downgrade protocol.
|
|
125
|
+
- Manual sub-tasks become escalation paths in the `quality-control` layer, triggered by low confidence scores.
|
|
126
|
+
|
|
127
|
+
The decomposition is not static. As you test and iterate via `evolution-loop`, you will discover that some method assignments were wrong. A sub-task you thought was deterministic turns out to have edge cases that need LLM handling. A sub-task you assigned to LLM turns out to be solvable with a simple regex. Update the decomposition. Track changes with `version-control`.
|
|
128
|
+
|
|
129
|
+
A well-decomposed rule is a well-understood rule. If you struggle to decompose a rule into clean sub-tasks, that usually means you do not yet understand the rule well enough. Go back to the developer user. Ask how they verify this rule manually. Their manual process is often the best decomposition blueprint — it reveals the natural sub-task boundaries that no amount of abstract analysis will surface.
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Decision Matrix for Method Selection
|
|
2
|
+
|
|
3
|
+
This reference provides the detailed decision matrix for assigning methods to sub-tasks during task decomposition. Read `task-decomposition` SKILL.md first for the philosophy; this document is the operational reference.
|
|
4
|
+
|
|
5
|
+
## The Four Dimensions
|
|
6
|
+
|
|
7
|
+
| Dimension | Definition | 1 (Low) | 3 (Medium) | 5 (High) |
|
|
8
|
+
|---|---|---|---|---|
|
|
9
|
+
| **Certainty** | Predictability of input format and location | Free-form prose, no fixed structure | Semi-structured with known sections but variable formatting | Fixed template, exact field positions |
|
|
10
|
+
| **Scale** | Number of items to process per document | 1-5 items | 10-100 items | 1,000+ items |
|
|
11
|
+
| **Semantic Depth** | Language understanding required | None — pure pattern or numeric | Moderate — entity recognition, simple context | Deep — judgment, adequacy assessment, intent interpretation |
|
|
12
|
+
| **Cost Sensitivity** | Budget constraint per document | Unlimited (one-off audit) | Moderate (monthly batch of hundreds) | Tight (daily batch of thousands) |
|
|
13
|
+
|
|
14
|
+
## Method Assignment Rules
|
|
15
|
+
|
|
16
|
+
Use the highest-priority method whose requirements are met. Priority order: Rule/Regex > Code > LLM > Manual.
|
|
17
|
+
|
|
18
|
+
| Certainty | Scale | Semantic Depth | Cost Sensitivity | Assigned Method | Rationale |
|
|
19
|
+
|---|---|---|---|---|---|
|
|
20
|
+
| High (4-5) | Any | Low (1-2) | Any | **Rule / Regex** | Predictable input + no language understanding = deterministic pattern matching |
|
|
21
|
+
| High (4-5) | Any | Low (1-2) | Any | **Code / Python** | Calculations, comparisons, transformations on structured data |
|
|
22
|
+
| Medium (3) | High (4-5) | Low (1-2) | High (4-5) | **Code + Regex** | Volume demands speed; invest in parsing code to avoid per-item LLM cost |
|
|
23
|
+
| Medium (3) | Low (1-2) | Medium (3) | Low (1-2) | **LLM** | Moderate understanding needed, low volume makes LLM cost acceptable |
|
|
24
|
+
| Low (1-2) | Any | High (4-5) | Any | **LLM** | Deep semantic understanding has no cheaper alternative |
|
|
25
|
+
| Low (1-2) | High (4-5) | High (4-5) | High (4-5) | **LLM (low tier) + sampling** | Volume + semantics + budget = use cheapest LLM, sample-verify with higher tier |
|
|
26
|
+
| Any | Any | Any | — | **Manual** | Last resort when automated methods fail accuracy threshold |
|
|
27
|
+
|
|
28
|
+
The table covers common patterns, not every combination. When a sub-task falls between categories, test both candidate methods on a sample and measure accuracy and cost. Let data decide.
|
|
29
|
+
|
|
30
|
+
## Worked Example: Cross-Field Validation
|
|
31
|
+
|
|
32
|
+
**Rule**: "The loan amount must not exceed 70% of the appraised collateral value."
|
|
33
|
+
|
|
34
|
+
Decomposition into sub-tasks with method assignments:
|
|
35
|
+
|
|
36
|
+
| # | Sub-task | Input | Output | Method | Rationale |
|
|
37
|
+
|---|---|---|---|---|---|
|
|
38
|
+
| 1 | Locate loan amount field | Full document text | Page/section reference | LLM (Tier 3) | Field position varies across document types |
|
|
39
|
+
| 2 | Extract loan amount | Located section text | Numeric value (float) | Regex | Amount follows pattern: ¥/$/digits with commas |
|
|
40
|
+
| 3 | Locate collateral section | Full document text | Page/section reference | LLM (Tier 3) | Section name varies: "Collateral", "Security", "Pledged Assets" |
|
|
41
|
+
| 4 | Extract appraised value | Located section text | Numeric value (float) | Regex + Code | Regex extracts; code handles unit conversion (万/亿) |
|
|
42
|
+
| 5 | Calculate threshold | Loan amount, collateral value | 70% threshold value | Code | Pure arithmetic: `collateral * 0.70` |
|
|
43
|
+
| 6 | Compare | Loan amount, threshold | Pass/Fail | Code | Simple comparison: `loan_amount <= threshold` |
|
|
44
|
+
| 7 | Generate comment | All extracted values | Comment string | Code (template) | Template: "Loan amount {X} is {above/within} 70% of collateral value {Y} (threshold: {Z})" |
|
|
45
|
+
|
|
46
|
+
LLM calls: 2 (locate steps only). Everything else is regex or code. Total LLM cost per document: ~0.002 USD at Tier 3 pricing.
|
|
47
|
+
|
|
48
|
+
## Worked Example: Large-Scale Filtering
|
|
49
|
+
|
|
50
|
+
**Task**: Match 31,800 invoices against 15,940 contracts to find which invoices belong to which contracts.
|
|
51
|
+
|
|
52
|
+
Naive approach: 507M pairwise LLM comparisons. Estimated cost: $50,000+. Time: weeks.
|
|
53
|
+
|
|
54
|
+
Layered decomposition:
|
|
55
|
+
|
|
56
|
+
| Layer | Method | Input Size | Output Size | Reduction | Cost |
|
|
57
|
+
|---|---|---|---|---|---|
|
|
58
|
+
| 1. Exact match on supplier name + contract number | Rule/Regex | 507M pairs | 25,200 matches | 99.5% eliminated | ~$0 |
|
|
59
|
+
| 2. Fuzzy match on amount range (±5%) + date overlap | Code | Remaining unmatched pairs | 12,400 candidates | 97.6% of remainder eliminated | ~$0 |
|
|
60
|
+
| 3. Semantic comparison of line-item descriptions | LLM (Tier 3) | 12,400 candidates | 7,652 confirmed | Final precision filter | ~$25 |
|
|
61
|
+
| 4. Manual review of low-confidence matches | Manual | ~200 uncertain | ~200 resolved | Edge cases | ~$100 (labor) |
|
|
62
|
+
|
|
63
|
+
Total cost: ~$125. Time: hours. Same accuracy as the naive approach.
|
|
64
|
+
|
|
65
|
+
The key insight: each layer's method is chosen because it is the cheapest method that can reliably make the distinctions required at that stage.
|
|
66
|
+
|
|
67
|
+
## Cost Estimation Template
|
|
68
|
+
|
|
69
|
+
Use this template during decomposition planning to estimate per-document cost.
|
|
70
|
+
|
|
71
|
+
| Sub-task | Method | Est. Cost/Call | Calls/Document | Subtotal |
|
|
72
|
+
|---|---|---|---|---|
|
|
73
|
+
| Locate section | LLM Tier 3 | $0.001 | 2 | $0.002 |
|
|
74
|
+
| Extract fields | Regex | $0.000 | 5 | $0.000 |
|
|
75
|
+
| Normalize values | Python | $0.000 | 5 | $0.000 |
|
|
76
|
+
| Cross-field comparison | Python | $0.000 | 1 | $0.000 |
|
|
77
|
+
| Semantic judgment | LLM Tier 2 | $0.003 | 1 | $0.003 |
|
|
78
|
+
| Comment generation | Template | $0.000 | 1 | $0.000 |
|
|
79
|
+
| **Total per document** | | | | **$0.005** |
|
|
80
|
+
|
|
81
|
+
Multiply by expected document volume to get batch cost. Compare against the developer user's budget. If total exceeds budget, optimize the most expensive sub-tasks first — usually the LLM calls with the highest per-call cost or the highest call count.
|