kc-beta 0.1.0

package/template/skills/en/meta/data-sensibility/SKILL.md

@@ -0,0 +1,115 @@
+---
+name: data-sensibility
+description: Build intuition about document data before writing extraction logic. Use before designing any extraction schema or regex pattern, when onboarding a new document type, or when extraction accuracy is unexpectedly low and you suspect a data assumption is wrong. Covers systematic observation of raw documents, spot-checking extracted results, distribution analysis, and recognizing suspicious patterns. If you are about to write code that touches document data and you have not read at least five documents end-to-end, stop and use this skill first.
+---
+
+# Data Sensibility
+
+The most expensive errors in document verification come from assumptions about data that do not hold. You assume dates are in YYYY-MM-DD because the first three documents used it — then document four uses DD/MM/YYYY and every extraction is silently wrong. Data sensibility is the discipline of looking before coding.
+
+## Front-Load Observation
+
+Read 3-5 complete documents end-to-end before writing any extraction logic. Not skim — read. Open the raw parsed text, start at page one, and go to the end.
+
+While reading, note what surprises you:
+- A field you expected in a table is buried in a paragraph.
+- Amounts appear in both Chinese uppercase and digits, sometimes on the same page.
+- The document has two signature pages, not one.
+- Section headings use inconsistent numbering schemes.
+- A "standard" field like interest rate is expressed three different ways across five documents.
+
+Read with a text editor, not a PDF viewer. You need to see what the extraction code will see — parsed text with all its artifacts, not the pretty rendered version. If your pipeline starts with OCR, read the OCR output.
+
+Do this for each new document type. Do it again when document sources change. 30 minutes of reading saves 3 hours of debugging extraction logic that was built on wrong assumptions.
+
+## Systematic Observation Checklist
+
+After reading, answer these questions explicitly — write the answers down, not just think them:
+
+**What is consistent across all documents?**
+Header structure, field positions, terminology, date formats. These are your anchors. Design extraction around them.
+
+**What varies?**
+Table layouts, section ordering, field presence, formatting conventions. These are your risk points. Every variant needs a test case.
+
+**What is surprising?**
+Anything you did not expect. A field that is sometimes missing. A value expressed in different units across documents. A section that appears in some templates but not others.
+
+**Document subtypes?**
+Are there different templates, issuers, or time periods represented? A "loan contract" from Bank A may look nothing like one from Bank B. Identify subtypes early — they often need separate extraction paths.
+
+**Section lengths?**
+Measure them. A section that averages 200 tokens is fine for any model. A section that occasionally runs to 8,000 tokens will blow your context window budget. Plan accordingly.
+
+**Encoding issues?**
+Full-width vs half-width characters (１２.５% vs 12.5%). Unicode normalization problems. OCR artifacts. These cause silent extraction failures because the text looks correct to human eyes but does not match regex patterns.
+
+## Spot-Check Protocol
+
+After any extraction run, pick 10 random fields and verify manually against the source document. Not the easy ones — select across the confidence spectrum: 3 high-confidence, 4 medium, 3 low.
+
+For each field, check:
+- **Value correct?** Does the extracted value match what the document says?
+- **Source location correct?** Did the extraction find the value in the right place, or did it grab a similar value from somewhere else?
+- **Normalization correct?** If the raw text says "伍仟万元" and your output says 50000000, is that right?
+- **Type correct?** Did a percentage end up as a decimal? Did a date parse into the wrong century?
+
+If more than 1 out of 10 is wrong, stop. Do not continue processing the batch. Investigate the failures, identify the pattern, fix the extraction, and re-run. A 10% spot-check error rate implies a much higher overall error rate — you have not seen the errors that look correct but are not.
+
+## Distribution Visualization
+
+After extraction, visualize key field distributions. You do not need fancy tools — basic Python is enough.
+
+**Field lengths.** If a "contract number" field is usually 12-16 characters but some are 3 or 200, those outliers are almost certainly wrong extractions.
+
+**Value frequencies.** Run `Counter(values).most_common(20)` on categorical fields. Suspicious concentrations (one value appearing 90% of the time when you expect diversity) suggest extraction is grabbing a header or default value instead of the actual data.
+
+**Missing rates.** If a field that should be present in every document is missing in 30% of results, your extraction is failing silently. Missing data is the easiest signal to detect and the most commonly ignored.
+
+**Numeric ranges.** Plot or bucket numeric fields. Loan amounts should be positive and within a plausible range for the business. Interest rates should be between 0% and 30%, not 0.03 (forgot to multiply) or 3000 (grabbed the wrong column). Even a quick histogram of numeric field values will reveal bimodal distributions (two document subtypes?), impossible values (negative loan amounts?), or truncation artifacts.
+
+## Smelly Patterns
+
+Some extracted values look correct individually but form suspicious patterns in aggregate:
+
+- **Suspiciously round amounts.** Every loan is exactly 1,000,000.00 or 5,000,000.00. Possible in reality (banks do round), but worth verifying — you might be extracting a template placeholder.
+- **Date clusters.** All contracts signed on the 1st of the month. All documents dated the same day. May indicate you are extracting a print date instead of a signature date.
+- **Identical unique fields.** Multiple documents sharing the same contract number or borrower ID. Either a data quality issue upstream or an extraction error.
+- **Values at regulatory thresholds.** Capital adequacy at exactly 8.00%. Loan-to-value at exactly 70.00%. These may be real (institutions manage to thresholds), or they may be artifacts of extracting the threshold definition instead of the actual value.
+- **Perfect consistency.** Every single document passes every single rule. Real data has exceptions. If your system reports 100% compliance, your system is probably wrong.
+
+These patterns may be real. They may be artifacts. The point is to notice them and investigate.
+
+## Intermediate Output Materialization
+
+Save every processing stage to disk:
+1. Raw parsed text (from document parsing).
+2. Tree-chunked sections (from tree processing).
+3. Extracted entities (from entity extraction).
+4. Judgment results (from compliance judgment).
+
+Write intermediate results as JSON files in a `debug/` directory, named by document ID and stage:
+
+```
+debug/
+  DOC001_01_raw_text.json
+  DOC001_02_tree_sections.json
+  DOC001_03_extracted_entities.json
+  DOC001_04_judgments.json
+```
+
+Disk is cheap. Debugging without intermediates is guesswork.
+
+When something goes wrong — and it will — you can inspect each stage independently. Was the section split wrong? Was the extraction correct but the judgment wrong? Was the raw text garbled by OCR? Without intermediates, you are reduced to staring at the final output and guessing.
+
+Keep intermediates for at least the current iteration. Delete old iterations only when disk space becomes a real constraint.
+
+## Integration
+
+Feed your observations into downstream skills:
+- `entity-extraction`: Your observations about data formats, field variants, and encoding issues directly inform schema design and regex patterns.
+- `skill-authoring`: Edge cases and known variants you discover here become test cases and special handling in rule skills.
+- `confidence-system`: Your spot-check results and distribution analysis calibrate initial confidence expectations.
+- `evolution-loop`: Revisit data sensibility whenever the evolution loop reveals unexpected failures — the data may have changed, or your initial observations may have been incomplete.
+
+Data sensibility is not a one-time activity. Every new batch, every new document source, every format change is an occasion to look before coding.

package/template/skills/en/meta/document-parsing/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: document-parsing
+description: Parse source documents into machine-readable text with maximum fidelity. Use when processing any document in Samples/ or Input/ for the first time, when parsed text quality is poor, or when tables and charts need special handling. Covers multi-level parser selection from simple text extraction to OCR and vision models. Also use when a verification rule fails due to parsing issues (garbled text, missing tables, mangled layouts) and the parser needs to be upgraded for that document type.
+---
+
+# Document Parsing
+
+Parsing is the foundation. If the text is wrong, everything downstream is wrong. But parsing is also a cost center — do not use expensive vision models when simple text extraction works.
+
+## The Minimum Viable Parser Principle
+
+Start with the simplest parser. Escalate only when necessary. This is not about saving money — it is about producing the most reliable output. Simple parsers have fewer failure modes.
+
+### Level 1: Direct Text Extraction
+- Tool: pymupdf (PyMuPDF) or similar PDF text extraction.
+- When: Well-formed digital PDFs with embedded text. This covers most modern business documents.
+- Output: Raw text with basic structure preserved (paragraphs, basic formatting).
+- Limitations: Tables may come out as messy text. Charts and images are invisible. Scanned PDFs produce nothing.
+
+### Level 2: Layout-Aware Extraction
+- Tool: pdfplumber or similar layout-aware parser.
+- When: Level 1 produces messy table output, or when preserving spatial layout matters (forms, multi-column documents).
+- Output: Text with table detection and cell-level extraction.
+- Limitations: Still text-based. Cannot handle scanned content.
+
+### Level 3: OCR
+- Tool: Vision-capable models from OCR_MODEL_TIER in `.env` (PaddleOCR-VL, GLM-4.6V, etc.).
+- When: Scanned PDFs, image-based PDFs, or PDFs where Level 1-2 produce garbled/incomplete text.
+- Output: Recognized text from images.
+- Limitations: Slower, costs API calls, may introduce OCR errors.
+
+### Level 4: Vision Model Interpretation
+- Tool: High-capability vision models (OCR_MODEL_TIER1).
+- When: Complex tables that text extraction cannot parse correctly, charts that need data point extraction, mixed text-and-image layouts.
+- Output: Structured interpretation of visual content (table as markdown, chart data as JSON).
+- Limitations: Expensive, slow. Reserve for when the visual content genuinely needs interpretation.
+
+## Quality Detection
+
+How to know when to escalate:
+
+- **Low character count**: The document has pages but extracted text is very short. Likely a scanned PDF.
+- **Garbled text**: Unusual character sequences, encoding errors, or meaningless text patterns.
+- **Missing expected sections**: The table of contents mentions Chapter 5 but no Chapter 5 text was extracted.
+- **Table artifacts**: Columns of numbers without alignment, cell content mixed with headers, or table borders appearing as characters.
+- **Missing numbers in financial tables**: If a financial document's key metrics are not in the extracted text, the tables were probably not parsed.
+
+Write a quick quality check after parsing and before proceeding. If quality is insufficient, escalate to the next parser level.
+
+### Parse Quality Score
+
+Compute a quality score (0.0 to 1.0) from weighted heuristics to make escalation decisions systematic rather than ad-hoc. A recommended starting framework:
+
+- **Character density** (weight ~0.3): actual character count / expected characters for the document's page count. A 10-page PDF that yields only 200 characters likely failed.
+- **Garble ratio** (weight ~0.2): fraction of characters that are common CJK/Latin vs control characters, unusual sequences, or encoding artifacts.
+- **Section completeness** (weight ~0.3): if the document has a table of contents, what fraction of TOC entries have matching content in the extracted text?
+- **Table integrity** (weight ~0.2): for financial documents, are key numeric values that should appear in tables actually present in the extracted text?
+
+**Escalation thresholds** (recommended defaults — adjust freely):
+- Score >= 0.7: accept this parser level, proceed to downstream processing.
+- Score 0.4-0.7: escalate to the next parser level, re-parse, re-score.
+- Score < 0.4: skip directly to Level 3 (OCR) or Level 4 (vision) depending on document characteristics.
+
+**Lock-in**: once a parser level produces an acceptable score for a document type, record that level. Do not re-evaluate unless a downstream verification failure is traced back to a parsing issue.
+
+These weights, thresholds, and the scoring approach itself are starting points. The coding agent should design whatever quality assessment works for the specific document types at hand — a simple pass/fail heuristic may be sufficient for some scenarios; a more nuanced scoring function may be needed for others. The important pattern is: **measure quality → compare to threshold → decide whether to escalate**.
+
+This follows the same tier-transition pattern as model tier selection in `skill-to-workflow`: a quality/accuracy score drives the decision to stay, escalate, or skip tiers.
+
+## Table Handling
+
+Tables are critical in financial documents (balance sheets, ratio tables, compliance metrics). They deserve special attention:
+
+1. **Detection**: Identify table regions. Look for grid patterns, consistent column spacing, or explicit table markers.
+2. **Extraction**: Extract cell-by-cell content. Preserve the row-column relationship.
+3. **Reconstruction**: Convert to a structured format (markdown table, JSON array of rows, or CSV).
+4. **Validation**: Spot-check that key values in the reconstructed table match what is visible in the document.
+
+When the standard parser fails on tables, try the vision model approach: send the table image (cropped from the PDF page) to a vision model and ask it to produce a markdown table.
+
+## Chart Handling
+
+Charts (bar charts, line charts, pie charts) occasionally contain data needed for verification:
+
+- Extract the chart image from the document.
+- Send to a vision model with a prompt: "Extract the data points, labels, and values from this chart. Return as a JSON array."
+- Validate the extracted data against any nearby text or table that might contain the same numbers.
+
+This is expensive. Only do it when a verification rule specifically requires data from a chart and that data is not available in text elsewhere in the document.
+
+## Output Format
+
+Parsed documents should be saved as clean markdown:
+
+- Preserve the document's heading hierarchy (# Chapter, ## Section, ### Subsection).
+- Preserve lists, numbered or bulleted.
+- Convert tables to markdown table format.
+- Note page boundaries if relevant (some rules reference specific pages).
+- Strip noise: headers, footers, page numbers, watermarks (unless a rule specifically checks for them).
+
+Save parsed output alongside the original document for reuse across rules.
+
+## Caching
+
+Parsing is expensive (especially Level 3-4). Cache parsed output:
+- Store the parsed markdown alongside the original file.
+- Track which parser level produced it.
+- Re-parse only when: the original file changes, a rule requires higher-quality parsing than what is cached, or a verification failure is traced back to a parsing issue.

package/template/skills/en/meta/document-parsing/references/parser-catalog.md

@@ -0,0 +1,40 @@
+# Parser Catalog
+
+## Text-Based Parsers (No LLM Required)
+
+| Parser | Type | Strengths | Limitations | Install |
+|--------|------|-----------|-------------|---------|
+| PyMuPDF (fitz) | Text extraction | Fast, reliable, basic structure | No table awareness, no OCR | `pip install pymupdf` |
+| pdfplumber | Layout-aware | Good table detection, spatial layout | Text-only, no OCR | `pip install pdfplumber` |
+| python-docx | DOCX parser | Native DOCX support, preserves structure | DOCX only | `pip install python-docx` |
+| openpyxl | XLSX parser | Full spreadsheet support | XLSX only | `pip install openpyxl` |
+| MarkItDown | Multi-format | Handles PDF, DOCX, PPTX, XLSX → markdown | Basic parsing, may miss complex layouts | `pip install markitdown` |
+
+## OCR / Vision Models (Via SiliconFlow API)
+
+| Model | Tier | Strengths | Best For |
+|-------|------|-----------|----------|
+| zai-org/GLM-4.6V | OCR_TIER1 | Best accuracy, strong Chinese OCR | Complex tables, mixed layouts |
+| Qwen/Qwen3.5-397B-A17B | OCR_TIER2 | Good general vision, large model | Tables with context-dependent interpretation |
+| PaddlePaddle/PaddleOCR-VL-1.5 | OCR_TIER3 | Fast, lightweight | Standard text, simple tables |
+
+## Local Deployment Options
+
+For developer users who prefer local processing:
+
+| Tool | Type | Notes |
+|------|------|-------|
+| PaddleOCR | Local OCR | Open source, supports Chinese/English |
+| Surya | Local OCR | Modern OCR with table detection |
+| pdf2md-local | PDF → Markdown | Reference: github.com/Ruilin-mmwa/pdf2md-local |
+
+## Selection Decision Tree
+
+```
+Is the PDF text-based (not scanned)?
+├─ Yes → PyMuPDF or pdfplumber
+│   └─ Are tables parsed correctly?
+│       ├─ Yes → Done
+│       └─ No → Try pdfplumber → If still bad → Vision model on table regions
+└─ No (scanned) → OCR_TIER3 → If quality insufficient → OCR_TIER1
+```

package/template/skills/en/meta/entity-extraction/SKILL.md

@@ -0,0 +1,129 @@
+---
+name: entity-extraction
+description: Extract specific entities, values, and text segments from documents as required by verification rules. Use after tree processing has located the relevant section, when a rule needs a specific number, date, name, amount, clause, or any domain-specific entity extracted. Covers extraction method selection (regex vs LLM), schema design, postprocessing, and confidence annotation. Also use when designing the extraction step of a workflow for worker LLMs.
+---
+
+# Entity Extraction
+
+An entity is the thing you need to check. A number, a date, a name, a clause, a percentage, a statement. The rule says what to check; extraction is how you get the value to check it against.
+
+## Extraction Type Taxonomy
+
+Different extraction scenarios call for different approaches:
+
+### Single Entity from Single Section
+The simplest case. One rule needs one value from one place.
+- Example: "Extract the capital adequacy ratio from the Key Metrics table."
+- Approach: Locate the section, apply regex or LLM extraction.
+
+### Multiple Entities from Single Section
+One rule needs several related values from the same place.
+- Example: "Extract the borrower's name, loan amount, interest rate, and maturity date from the loan agreement summary."
+- Approach: Design a single extraction call that returns all values. More efficient than multiple calls.
+
+### Single Entity from Multiple Sections
+One value is scattered across multiple places, or needs cross-referencing.
+- Example: "Extract the total collateral value, which may be listed in the collateral section or in Appendix A."
+- Approach: Collect content from all relevant sections, then extract. Note which source the value came from.
+
+### Entity from Full Document
+The value could be anywhere, or the rule applies to the document as a whole.
+- Example: "Check whether the document contains a valid signature page."
+- Approach: For the coding agent, scan the full document. For worker LLM workflows, design a two-pass approach: first pass identifies the location, second pass extracts the value.
+
+## Method Selection
+
+### Regex / Python (Cost: zero, Speed: instant)
+
+Use when the entity has a predictable format:
+
+- **Dates**: `\d{4}[-/年]\d{1,2}[-/月]\d{1,2}[日]?` or specific patterns for the document type.
+- **Monetary amounts**: `[\d,]+\.?\d*\s*(元|万元|亿元|USD|RMB)` or similar.
+- **Percentages**: `\d+\.?\d*\s*%`
+- **Identifiers**: Loan numbers, registration codes, ID numbers — usually fixed formats.
+- **Specific phrases**: "I hereby agree" or "本人同意" — exact string matching.
+
+Build and test the regex on sample documents. A good regex is better than a good LLM prompt for structured values — it is faster, deterministic, and free.
+
+### LLM Extraction (Cost: API call, Speed: seconds)
+
+Use when the entity requires understanding:
+
+- **Named entities in context**: "the guarantor's main business" — requires understanding who the guarantor is and which text describes their business.
+- **Conditional values**: "the interest rate, including any adjustments" — requires understanding what constitutes an adjustment.
+- **Semantic matching**: "adequate risk disclosure" — requires judgment about what text constitutes risk disclosure.
+- **Table interpretation**: When a table's structure is not uniform and regex cannot reliably extract cells.
+
+Design the LLM prompt to:
+1. Include the narrowed context (from tree processing).
+2. Specify exactly what to extract.
+3. Define the output format (JSON with named fields).
+4. Provide one example if the extraction is non-obvious.
+
+### Hybrid Approach
+
+Often the best strategy:
+1. Use regex to extract candidates (fast, catches obvious matches).
+2. If regex finds a confident match, use it.
+3. If regex fails or is uncertain, fall back to LLM extraction.
+4. Use LLM to validate regex results when confidence matters.
+
+## Schema Design
+
+Define the expected output for each extraction. Keep it simple and JIT:
+
+```json
+{
+  "entity_name": "capital_adequacy_ratio",
+  "value": 12.5,
+  "unit": "%",
+  "raw_text": "资本充足率为12.5%",
+  "source_location": "Chapter 2, Table 1, Row 3",
+  "confidence": 0.95,
+  "extraction_method": "regex"
+}
+```
+
+The schema should capture:
+- **value**: The extracted value, normalized.
+- **unit**: If applicable (%, 元, days, etc.).
+- **raw_text**: The original text fragment where the value was found. This is evidence for the judgment step.
+- **source_location**: Where in the document the value was found.
+- **confidence**: How sure you are (see `confidence-system`).
+- **extraction_method**: What extracted it (regex, LLM-TIER2, etc.).
+
+Do not over-engineer the schema. Add fields as needed during testing.
+
+## Postprocessing
+
+Raw extracted values often need normalization:
+
+- **Chinese numerals → digits**: 一百二十万 → 1200000
+- **Date standardization**: 2024年3月15日 → 2024-03-15
+- **Unit conversion**: 万元 → multiply by 10000 if comparing to a threshold in 元.
+- **Whitespace and noise removal**: Strip extra spaces, line breaks, formatting artifacts.
+- **Percentage normalization**: 0.125 → 12.5% or vice versa, depending on what the rule expects.
+
+Build postprocessing as Python functions in the rule skill's `scripts/` directory. They are deterministic and reusable.
+
+## Confidence Annotation
+
+Every extraction should carry a confidence estimate:
+
+- **Regex match, validated format**: 0.90-0.95
+- **LLM extraction, high certainty**: 0.80-0.85
+- **LLM extraction, some ambiguity**: 0.60-0.75
+- **Fallback or inferred value**: 0.40-0.60
+- **No value found**: 0.0 (flag as MISSING)
+
+These are starting points. Calibrate based on actual accuracy (see `confidence-system`).
+
+## Fitting Worker LLM Context
+
+When designing extraction for worker LLM workflows:
+
+1. Calculate the prompt size: system prompt + instructions + examples + output format = N tokens.
+2. Available context for document content = model's context window - N.
+3. If the section exceeds available context, narrow further via tree processing.
+4. Always leave room for the model's response.
+5. Test with the actual model to verify the context fits — token counts from the coding agent may differ from the worker LLM's tokenizer.

package/template/skills/en/meta/tree-processing/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: tree-processing
+description: Build hierarchical document trees and navigate to specific chapters or sections required by verification rules. Use when a rule targets a specific part of a document (e.g., "Chapter 3 must contain..."), when documents are too long for a single LLM context window, or when you need to find where a specific entity lives within a large document. Implements the "onion peeler" approach for chunking. Also use for documents over 100 pages where full-document processing is impractical.
+---
+
+# Tree Processing
+
+Most verification rules do not need the entire document. They need a specific section, a specific table, a specific disclosure. The tree is your map for navigating large documents efficiently.
+
+## Why Trees
+
+Two reasons:
+
+1. **Rules have scope.** "The risk disclosure in Chapter 5 must contain..." — you need to find Chapter 5, not read 1000 pages.
+2. **Worker LLMs have limits.** A 16K-32K context window cannot hold a 1000-page document. You must narrow to the relevant section.
+
+The tree structure solves both: it tells you WHERE things are, and lets you extract JUST what you need.
+
+## Building the Tree
+
+### Step 1: Discover the Structure
+
+Before building a tree parser, explore several sample documents to find structural patterns. Look for:
+
+- **Header conventions**: Do chapters start with "Chapter X"? "第X章"? "Part X"? A Roman numeral?
+- **Numbering systems**: "1.1.2", "Article 3", "(a)(i)", hierarchical numbering?
+- **Visual markers**: Bold text, larger font, horizontal rules, page breaks before chapters?
+- **Table of contents**: Most formal documents have one. It is the document's own tree.
+
+Spend time here. The patterns you find determine whether the tree builder is a simple regex or a complex parser.
+
+### Step 2: Choose the Parser
+
+**If patterns are consistent** (they usually are in regulated documents):
+- Write a regex-based splitter. For example:
+  - `^第[一二三四五六七八九十百千]+章` for Chinese chapter headers
+  - `^Chapter \d+` for English
+  - `^\d+\.\d+(\.\d+)*\s` for numbered sections
+- This is fast, deterministic, and reliable. Prefer this when it works.
+
+**If patterns are inconsistent or absent**:
+- Use the LLM-guided wedge-driving approach (see `rule-extraction/references/chunking-strategies.md` for the full algorithm: rolling context window, K-token quoting, Levenshtein fuzzy matching).
+- This is slower and costs LLM calls, but handles unstructured documents. The rolling window means even very large unstructured leaf nodes can be chunked incrementally.
+
+**If the document has a table of contents**:
+- Parse the TOC first. It gives you the tree structure and page numbers for free.
+- Then use the TOC-derived structure to split the document body.
+
+### Step 3: Build the Tree
+
+The tree is a simple nested structure:
+
+```
+Document
+├── Part I: General Provisions
+│   ├── Chapter 1: Definitions (pages 1-15)
+│   └── Chapter 2: Scope (pages 16-22)
+├── Part II: Capital Requirements
+│   ├── Chapter 3: Minimum Capital (pages 23-45)
+│   │   ├── Section 3.1: Tier 1 Capital
+│   │   └── Section 3.2: Tier 2 Capital
+│   └── Chapter 4: Risk Weighting (pages 46-78)
+└── Part III: Disclosure
+    └── Chapter 5: Risk Disclosure (pages 79-120)
+```
+
+Each node stores: the header text, the level, the start/end positions in the document, and the content size (in tokens or characters).
+
+### Step 4: Use the Tree
+
+Given a rule that says "check the risk disclosure section":
+
+1. **Search the tree** for the relevant node. Match the rule's scope description against node headers.
+   - Exact match: "Chapter 5" → find node with "Chapter 5" header.
+   - Semantic match: "risk disclosure section" → find node whose header or content relates to risk disclosure. May need fuzzy matching or LLM classification.
+2. **Extract the content** of that node (and optionally its children).
+3. **Check the size.** If the content fits in the worker LLM's context window, use it directly. If not, descend to child nodes and find the specific subsection needed.
+
+## The Full Context → Chapter → Entity Pipeline
+
+This is the standard narrowing funnel for extracting entities for verification:
+
+1. **Full context**: Use the tree to understand the document structure. Know where everything is.
+2. **Chapter**: Navigate to the specific section that the rule targets. Extract its content.
+3. **Entity**: Within the chapter content, extract the specific entity (number, text, clause) using the techniques from `entity-extraction`.
+
+For worker LLMs with 16K-32K context:
+- The chapter content + the extraction prompt must fit in the context window.
+- If a chapter is too large, descend further in the tree.
+- Always include the parent header chain for context: "Part II > Chapter 3 > Section 3.1" so the LLM knows where this content sits in the document.
+
+## Caching and Reuse
+
+Build the tree once per document, reuse across all rules:
+- Save the tree structure as JSON alongside the parsed document.
+- Multiple rules may need different sections of the same document. The tree lets each rule navigate directly to its section without re-parsing.
+
+## Edge Cases
+
+- **Flat documents**: Some documents have no structural hierarchy. Treat the entire document as one node. Use LLM-guided chunking if it exceeds the context window.
+- **Deeply nested structures**: Some legal documents have 6+ nesting levels. Build all levels but typically only navigate 2-3 levels deep for any given rule.
+- **Cross-section references**: A section might reference "as defined in Section 1.2." When extracting, you may need content from multiple tree nodes. Collect them into a single context for the LLM.
+- **Appendices and annexes**: Often contain critical tables and data. Include them as top-level nodes in the tree.

package/template/skills/en/meta-meta/bootstrap-workspace/SKILL.md

@@ -0,0 +1,70 @@
+---
+name: bootstrap-workspace
+description: Initialize and configure a document verification workspace. Use when a developer user first opens this workspace, when .env needs configuration, or when the business scenario needs to be understood. Guides the coding agent through reading regulation documents, understanding the developer user's business context, configuring model tiers and thresholds, and establishing the working relationship. Covers initial conversation with developer user to scope the verification task, set expectations, and agree on checkpoints.
+---
+
+# Bootstrap Workspace
+
+You are replacing a team of business analysts, prompt engineers, and QA engineers. Before writing a single line of code, understand what that team would ask first.
+
+## First Actions
+
+1. Read everything in `Rules/`. Understand the domain, the regulation structure, the language, and the level of specificity.
+2. Scan `Samples/` to understand the document types, formats (PDF, DOCX, scans), typical length, and structural patterns.
+3. Review `.env` to understand the current configuration. Check that API keys are set and models are accessible.
+
+## Discuss with Developer User
+
+Before proceeding, have a focused conversation with the developer user to align on:
+
+### Scope
+- What type of documents are being verified? (contracts, filings, reports, etc.)
+- What regulations or rules apply? Are they all in Rules/, or are some implicit domain knowledge?
+- How many distinct rules are there, roughly? Dozens? Hundreds?
+- Are there rules already sorted into a spreadsheet (xlsx/csv) with clear scope definitions?
+
+### Granularity
+- How fine-grained should each rule be? A single clause? A paragraph? A section?
+- Some rules are naturally atomic ("the interest rate must not exceed X%"). Others are compound ("the disclosure section must contain items A, B, C, and each must meet criteria D, E, F"). Discuss how to decompose compound rules.
+- If the developer user has pre-sorted rules, follow their structure. Do not re-decompose unless they ask.
+
+### Expectations
+- What accuracy is acceptable? The default in `.env` is 0.9 for both skills and workflows. Is this right for this business scenario?
+- Are some rules more critical than others? Should critical rules have higher thresholds?
+- What is the expected production volume? How many documents per batch?
+- How fast do results need to be? Same-day? Real-time?
+
+### Checkpoints
+- Explain the lifecycle to the developer user: you will first verify documents yourself (skill phase), then build automated workflows (workflow phase), then monitor in production (QC phase).
+- Agree on when the developer user wants to review progress. After the first rule? After all rules? After workflow distillation?
+- Establish how the developer user wants to see results (HTML dashboard, direct conversation, output files).
+
+## .env Configuration Guidance
+
+Walk through `.env` parameters with the developer user:
+
+- **TIER1-4**: Model tiers from most capable to most efficient. TIER1 is used when accuracy matters most. TIER4 is used when the task is well-understood and speed/cost matters. The coding agent decides per-task, but the developer user should confirm the available models match their API access.
+- **OCR_MODEL_TIER1-3**: For document parsing when text extraction fails. Only relevant if documents include scanned pages, complex tables, or charts.
+- **SKILL_ACCURACY**: The accuracy threshold before a skill is considered "proven" and ready for workflow distillation. Default 0.9. Higher means more iteration before moving on, but more reliable workflows.
+- **WORKFLOW_ACCURACY**: The accuracy threshold before a workflow is deployed to production. Default 0.9. This is measured against the coding agent's own skill-based results as ground truth.
+- **MONITOR_FREQUENCY**: How aggressively to sample and review production results. `high` = slower decay (review more for longer), `mid` = balanced, `low` = faster decay (trust workflows sooner).
+- **MAX_ITERATIONS**: Safety valve. After this many evolution cycles on a single rule, escalate to the developer user instead of continuing to iterate.
+
+## Setting Up the Workspace
+
+After the conversation:
+
+1. Ensure all workspace folders exist (Rules/, Samples/, Input/, Output/).
+2. Create a `logs/` directory for iteration logs.
+3. Create a `workflows/` directory for distilled Python workflows.
+4. Create a `rule-skills/` directory where individual rule skill folders will live.
+5. Initialize version tracking (a `versions.json` manifest).
+6. Log the bootstrap conversation summary for future reference.
+
+## When to Re-Bootstrap
+
+Return to this skill when:
+- The developer user adds new regulation documents to Rules/.
+- The business scenario changes significantly.
+- Model availability changes (new models added, old models deprecated).
+- Thresholds need adjustment based on production experience.