@1aboveio/skills 0.0.0-stage → 1.0.0

package/skills/core/reg-parser/SKILL.md

@@ -0,0 +1,212 @@
+---
+name: reg-parser
+version: 1.1.0
+description: "Parse regulatory/scheme documents into structured outputs: convert a regulation (AML, VARS, PCI-DSS, KYC, GDPR) into an audit-ready compliance checklist, or trace contract clauses to a mandate source. Suggest proactively when a regulation PDF accompanies a contract review or compliance task. NOT general summarization (doc-to-md), or writing the gap analysis or policy from the checklist (write-doc)."
+release-version: 0.36.0
+---
+**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
+# reg-parser
+
+Parse a regulatory or scheme document into one of two structured outputs:
+
+- **Mode A — Clause Annotation Extract**: flat obligation list for contract review P10 origin tracing
+- **Mode B — Compliance Checklist**: audit-ready checklist with IDs, severity, verification steps
+
+## Mode Inference
+
+Infer mode from context — only ask if genuinely ambiguous:
+
+| Context signals | Mode |
+|-----------------|------|
+| "contract", "clause", "review", "sign", "channel agreement", "is this clause required by…" | **A** |
+| "audit", "checklist", "compliance", "are we compliant", "VARS checklist" | **B** |
+| Ambiguous | Ask: "Do you want a clause annotation extract (for contract review) or a compliance checklist (for auditing)?" |
+
+---
+
+## Step 1: Convert document
+
+Use the `doc-to-md` skill:
+
+```bash
+python3 {doc-to-md baseDir}/scripts/convert.py /path/to/regulation.pdf ./reg-md/
+```
+
+---
+
+## Step 2: Extract obligations
+
+Read the converted Markdown. Extract every **shall**, **must**, and mandatory requirement.
+Ignore preamble, definitions, and non-binding guidance unless user requests them.
+
+Group obligations by the document's section structure. For each obligation, capture:
+- Section reference (`section <number>` or heading). Write clause references as `section 2(1)(a)`, never `§2(1)(a)`, even when the source uses §; a schedule, part or appendix is named as such (`Schedule 2 section 2(1)`, `Part 3 section 9`). Write ranges as `sections 4.8.1–4.8.20`, and expand `s.` / `Sch.` to `section` / `Schedule`.
+- The obligation in plain language
+- Who it applies to (regulator → licensed entity → downstream entity)
+
+---
+
+## Step 3A: Output — Clause Annotation Extract (Mode A)
+
+Save to `{baseDir}/references/mandates/<slug>.md` (known sources) or `{baseDir}/references/mandates/adhoc-<slug>-<YYYYMMDD>.md` (ad-hoc).
+
+```bash
+bash {baseDir}/scripts/preprocess-mandate.sh <converted-md> \
+  --source "<Source Name>" \
+  --out {baseDir}/references/mandates/<slug>.md
+```
+
+The script refuses to overwrite a finished extract (one with no `Status: RAW` line). To refresh a built-in mandate, write a dated `adhoc-` file and merge by hand, or pass `--force` only when you mean to replace the curated file.
+
+The script writes a template with a `Status: RAW` marker, an embedded raw-text dump, and HTML-comment scaffolding. After extraction, the final file must have the raw-text dump and all scaffolding comments removed, the `Status: RAW` line removed, and the four canonical sections filled in (use `_None_` if a section truly has no entries).
+
+Output format:
+
+```markdown
+# <Source Name> — Contract Obligation Extract
+
+> **Source:** <full citation>
+> **Jurisdiction:** <scope>
+> **Version / Effective Date:** <…>
+
+## Mandatory Downstream Agreement Clauses
+**section [ref] [title]**: [obligation in one sentence]
+
+## Prohibited Conduct
+**section [ref]**: [prohibition]
+
+## Thresholds & Triggers
+**section [ref] [metric]**: [value + consequence]
+
+## Audit & Oversight Rights
+**section [ref]**: [right]
+
+## Notes
+[Jurisdiction, effective date, version, caveats]
+```
+
+After generating, validate:
+
+```bash
+python3 {baseDir}/scripts/validate_mandate.py <mandate.md> [--source <converted-md>]
+```
+
+The optional `--source` flag enables citation-existence warnings (each `section <ref>` in the extract is fuzzy-matched against the converted source). Fix errors and re-run until clean (exit 0); warnings are advisory.
+
+**Downstream use:** contract-review Step 0b loads this file for P10 clause origin tracing.
+Annotations: `[Mandated: <source> section <ref>]` / `[Permitted: <source> section <ref>]` / `[Counterparty-Added]`
+
+---
+
+## Step 3B: Output — Compliance Checklist (Mode B)
+
+Read `{baseDir}/references/checklist-schema.md` for full schema before generating.
+
+Save to `{baseDir}/references/checklists/<slug>-checklist.md`.
+
+### Pick an `id_scheme` first
+
+| Scheme            | Use for                                                                                   | Item ID example   | Severity values             |
+|-------------------|-------------------------------------------------------------------------------------------|-------------------|-----------------------------|
+| `cat` (default)   | Obligation lists with no canonical source IDs (AMLD articles, GDPR articles, FATF Recs)   | `CAT-01-001`      | critical / major / minor    |
+| `source-native`   | Control frameworks with native ID hierarchies (Visa VARS, PCI-DSS, COBIT)                 | `AACQ.C1.1`       | mandatory / recommended     |
+
+Use `source-native` when the source already groups its requirements under canonical IDs you'd lose by re-numbering. Otherwise default to `cat`. Source-native checklists may add optional `Risk Domain`, `Risk Sub-Domain`, `Risk Statement`, and `Parent Control` fields per item to preserve source taxonomy.
+
+Under source-native, prefix every Verification step with **(N)**, **(a)**, **(i)** matching the source's numbered Mandatory/Recommended Controls list — see checklist-schema.md "Verification step prefixes" for the convention, including three-level nesting. Preserve the source's nesting depth verbatim; flattening lettered/roman sub-bullets into a single comma-separated bullet destroys auditor traceability and is the most common failure mode of this skill. When in doubt, preserve. The canonical example is `references/checklists/visa-vars-checklist.md` — see entries like `ATPA.C1.1`, `ATPA.C1.2`, `ATPA.C8.1` for the expected nesting pattern.
+
+### Source-native specifics
+
+**Severity rule.** Many control frameworks (e.g. VARS) split each control into a Mandatory Controls list and a Recommended Controls list. Set the item's `Severity`:
+- `mandatory` if the source lists *any* Mandatory Controls under the control — even if Recommended Controls also exist
+- `recommended` only when the source lists exclusively Recommended Controls
+
+**Cross-archetype / cross-layer controls.** Some frameworks layer archetypes (e.g. VARS: ATPA-Acquirers inherit AACQ controls implicitly without restating them). Do not duplicate inherited items into the dependent archetype's category. Keep each control under the archetype where the source defines it; cross-references in the source's prose stand, and the audit consumer is expected to read parent archetypes too. If a control is genuinely re-stated by the source under a second archetype with different obligations, that's a new item under the second archetype's ID.
+
+**Source-document defects.** Real regulation PDFs contain typos and TOC artefacts (e.g. `ATM.C3` for `AATM.C3`, "4.11" for "4.1.1"). Use the canonical form found in the body of the source; do not propagate the defect into the checklist. Note material corrections in the YAML `scope` field if they could affect interpretation.
+
+Each obligation from Step 2 becomes one checklist item:
+
+```markdown
+### CAT-XX-NNN: [Requirement title]                  # cat scheme
+### AACQ.C1.1: [Requirement title]                   # source-native scheme
+
+- **Requirement:** [plain-language obligation]
+- **Source:** section [ref], [regulation]
+- **Severity:** critical | major | minor             # cat scheme
+- **Severity:** mandatory | recommended              # source-native scheme
+- **Verification:**
+  - [ ] [Concrete observable step]
+  - [ ] [Another step]
+- **Evidence:**
+  - [Document or record that proves compliance]
+```
+
+After generating, validate:
+
+```bash
+python3 {baseDir}/scripts/validate_checklist.py <checklist.md> [--source <converted-md>]
+```
+
+Fix errors and re-run until clean (exit 0). Every field needs content, and Verification needs at least one `- [ ]` step.
+
+The optional `--source` flag enables three semantic warnings (advisory, never block exit):
+- **Citation existence** — each item's `Source:` field must contain a clause anchor (e.g. `13(1)(a)`) found in the source markdown as a whole reference. The regulation's own number (`Cap. 615`, `2015/849`) does not count, and a Source with no clause anchor is flagged.
+- **Weak modal language** — Requirement contains `should`/`may`/`recommended` with no overriding `must`/`shall` anchor.
+- **Verification independence** — Verification steps must not be a near-paraphrase of the Requirement (Jaccard > 0.5).
+
+**Built-in reference checklists** (in `references/checklists/`):
+- `kyc-aml-checklist.md` — KYC/AML (EU AMLD4/5), `cat` scheme
+- `visa-vars-checklist.md` — Visa VARS Acquirer/TPA/HIR/ATM/Visa-Direct controls (45 items, 38 mandatory + 7 recommended), `source-native` scheme
+- `hk-amlo-checklist.md` — Hong Kong AMLO (Cap. 615), with JSON
+- `hk-mso-guideline-checklist.md` — CCE Guideline for Money Service Operators, with JSON
+- `hk-fi-corporate-onboarding-cdd-checklist.md` — HK financial-institution corporate onboarding CDD, with JSON
+
+**Built-in mandates** (in `references/mandates/`): `visa-vars.md`, `hk-amlo-cap615.md`.
+
+### JSON dump for downstream consumers
+
+Markdown is the authored source of truth — humans read and review it, GitHub renders it, and authors edit it directly. Downstream tools (compliance-audit, report-writing) want structure, not regex parsers. Generate a JSON view of any checklist with:
+
+```bash
+python3 {baseDir}/scripts/dump_checklist.py <checklist.md> --out <out.json>
+```
+
+The JSON preserves the verification tree verbatim (each node has `prefix`, `text`, `children`), so consumers can iterate items, walk nested controls, and look up severity/evidence without re-parsing markdown. Treat the JSON as a build artifact — regenerate on every checklist change rather than hand-editing it.
+
+**Downstream use:** compliance-audit skill loads the checklist (markdown for review, JSON for programmatic consumption) to drive the audit workflow.
+
+---
+
+## Known Source Slugs
+
+| Source | Slug |
+|--------|------|
+| EU AML Directive 2015/849 (AMLD4/5) | `kyc-aml` |
+| Visa VARS | `visa-vars` |
+| HK AMLO (Cap. 615) | `hk-amlo` (mandate file `hk-amlo-cap615.md`) |
+| HK MSO Guideline (CCE) | `hk-mso-guideline` |
+| HK FI corporate onboarding CDD | `hk-fi-corporate-onboarding-cdd` |
+| Mastercard SDP/MDRP | `mastercard-sdp` |
+| PCI-DSS | `pci-dss` |
+| UnionPay Agreement Rules | `unionpay-rules` |
+| MAS Payment Services Act | `mas-psa` |
+| FATF Recommendations | `fatf` |
+
+Ad-hoc sources use `adhoc-<slug>-<YYYYMMDD>` — no permanent slug needed.
+
+---
+
+## Testing
+
+End-to-end fixtures live in `references/tests/` (PDF input + ground-truth target). Because the test source overlaps with a built-in canonical checklist, an agent rerun must hide the canonical first, or it will copy it. Use the harness:
+
+```bash
+bash {baseDir}/scripts/test_quarantine.sh setup <slug>      # hide canonical
+# ... run agent against references/tests/vars-source.pdf, write to a workspace path ...
+python3 {baseDir}/scripts/validate_checklist.py <out.md> --source <converted-source.md>
+python3 {baseDir}/scripts/compare_checklist.py <out.md> {baseDir}/references/tests/vars-checklist.md   # counts item headings only
+bash {baseDir}/scripts/test_quarantine.sh teardown <slug>   # restore canonical
+```
+
+See `references/tests/README.md` for the full protocol.

package/skills/core/reg-parser/references/checklist-schema.md

@@ -0,0 +1,223 @@
+# Checklist Schema Reference
+
+Full format specification for compliance checklists. Read this when creating or validating a checklist.
+
+## File Structure
+
+A checklist is a single `.md` file with two sections: a YAML metadata block and requirement categories.
+
+## ID schemes
+
+Two schemes are supported. Pick one per checklist via the `id_scheme` metadata field:
+
+| Scheme          | When to use                                                                                         |
+|-----------------|-----------------------------------------------------------------------------------------------------|
+| `cat` (default) | Obligation-list regulations where the source has no canonical control IDs (AMLD articles, GDPR articles, FATF Recommendations). The skill assigns synthetic `CAT-XX-NNN` IDs. |
+| `source-native` | Control-framework documents where the source already defines its own hierarchy (Visa VARS `AACQ.C1.1`, PCI-DSS `Req. 3.2.1`, COBIT `APO13.01`). Native IDs are preserved verbatim. |
+
+The two schemes differ only in: category/item heading regex, severity enum, and summary count fields. All other rules (required item fields, item-field syntax, citation/modal/Jaccard semantic checks) are identical.
+
+## Metadata Block
+
+````markdown
+```yaml
+name: "Short Checklist Title"
+regulation: "Full regulation name and number"
+version: "1.0"
+scope: "Brief description of what this checklist covers"
+date: "YYYY-MM-DD"
+id_scheme: "cat"                       # optional — "cat" (default) or "source-native"
+extends: "path/to/base-checklist.md"   # optional
+summary:
+  categories: 5
+  total_items: 16
+  critical: 6
+  major: 7
+  minor: 3
+```
+````
+
+For `id_scheme: source-native` the `summary` block uses `mandatory` and `recommended` instead of `critical`/`major`/`minor`:
+
+```yaml
+id_scheme: "source-native"
+summary:
+  categories: 5
+  total_items: 45
+  mandatory: 38
+  recommended: 7
+```
+
+| Field        | Required | Description                                         |
+|--------------|----------|-----------------------------------------------------|
+| `name`       | yes      | Human-readable title                                |
+| `regulation` | yes      | Source regulation (e.g. "EU AML Directive 2015/849")|
+| `version`    | yes      | Checklist version (semver recommended)              |
+| `scope`      | yes      | What the checklist covers                           |
+| `date`       | yes      | Date of last revision (YYYY-MM-DD)                  |
+| `id_scheme`  | no       | `cat` (default) or `source-native`                  |
+| `extends`    | no       | Path to a base checklist, relative to this file     |
+| `summary`    | yes      | Scheme-dependent counts (see above)                 |
+
+## Categories
+
+H2 heading with an ID prefix:
+
+```markdown
+## CAT-01: Customer Due Diligence            # cat scheme
+## AACQ: Acquirer Controls                   # source-native scheme
+```
+
+| Scheme          | Category ID format                            |
+|-----------------|-----------------------------------------------|
+| `cat`           | `CAT-` plus two digits (`CAT-01`, `CAT-02`)   |
+| `source-native` | Uppercase token of letters/digits (`AACQ`, `ATPA`, `PCIDSS`) |
+
+Categories appear in logical audit order.
+
+## Checklist Items
+
+H3 under their category. Item IDs must be unique within the file; the prefix before the first dot/dash must match an existing category ID.
+
+```markdown
+### CAT-01-001: Verify customer identity     # cat scheme
+### AACQ.C1.1: Alignment Between Business Plans and Risk Policies   # source-native
+```
+
+| Scheme          | Item ID format                                                                |
+|-----------------|-------------------------------------------------------------------------------|
+| `cat`           | `<CAT-ID>-<SEQ>` — SEQ is zero-padded three digits (`CAT-01-001`)             |
+| `source-native` | `<CAT-ID>.<segment>(\.<segment>)+` — preserves source punctuation (`AACQ.C1.1`, `AACQ.C8.2`) |
+
+Example item:
+
+```markdown
+### CAT-01-001: Verify customer identity
+
+- **Requirement:** Collect and verify government-issued photo ID for all
+  natural persons before establishing a business relationship.
+- **Source:** Art. 13(1)(a), Directive 2015/849
+- **Severity:** critical
+- **Verification:**
+  - [ ] Photo ID collected (passport, national ID, or driver licence)
+  - [ ] ID verified against an independent, reliable source
+  - [ ] Verification date and method recorded
+- **Evidence:**
+  - Copy of verified ID document
+  - Verification log entry with timestamp and officer name
+```
+
+### Item fields
+
+| Field             | Required | Description                                          |
+|-------------------|----------|------------------------------------------------------|
+| **Requirement**   | yes      | Plain-language statement of what must be done        |
+| **Source**        | yes      | Article, section, or clause in the source regulation |
+| **Severity**      | yes      | Scheme-dependent (see below)                         |
+| **Verification**  | yes      | Checkbox list of concrete steps to confirm compliance|
+| **Evidence**      | yes      | List of documents or artefacts that prove compliance |
+| **Risk Domain**   | no       | Source-native taxonomy (e.g. "4.1.2 Operational Risk") |
+| **Risk Sub-Domain** | no     | Source-native sub-grouping (e.g. "Written Agreements") |
+| **Risk Statement**| no       | One-line statement of the risk being mitigated       |
+| **Parent Control**| no       | Source-native parent control ID (e.g. `AACQ.C2`)     |
+
+The four optional fields preserve source taxonomy when `id_scheme: source-native` is used. They are silently ignored under the `cat` scheme.
+
+### Verification step prefixes (source-native only)
+
+Under `id_scheme: source-native`, each Verification checkbox **must** carry a bold prefix identifying the source-numbered control item it audits. Use parenthesized numbering that mirrors the source document's outline:
+
+```markdown
+- **Verification:**
+  - [ ] **(1)** Written agreement contains the following provisions:
+    - [ ] **(a)** Standards: Visa minimum standards incorporated
+    - [ ] **(b)** Visa's rights and authority: audits, accessibility, termination clause
+    - [ ] **(c)** Compliance: TPA does not submit illegal transactions
+      - [ ] **(i)** TPA complies with Visa Rules and applicable laws
+      - [ ] **(ii)** TPA adheres to Acquirer policies and AISP
+  - [ ] **(2)** (Recommended) Agreement names Acquirer as primary party
+    - [ ] **(a)** Defines duties and responsibilities of TPA and Acquirer
+    - [ ] **(b)** Allows transfer/assignment to another Acquirer
+```
+
+Within a parent bullet's nested list, the sub-prefix is just `(a)` / `(i)` — the parent number is implied. The fully qualified path (`AACQ.C1.1.(1.c.ii)`) is what an auditor reconstructs when citing back to source; the on-page prefix stays short.
+
+**Preserve the source's nesting depth verbatim.** Every lettered or roman sub-bullet in the source becomes its own checkbox at the matching depth. The whole point of source-native is auditor traceability back to specific source bullets, and flattening destroys it. Three-level nesting — `(N)` → `(a)` → `(i)` — is fully supported and expected for control frameworks like VARS where mandatory provisions sit two or three levels deep.
+
+**Mandatory and Recommended controls use independent numbering.** When a source control has both a "Mandatory Controls" and a "Recommended Controls" section (e.g. VARS AACQ.C2.1 with 3 mandatory + 1 recommended), each section's numbering restarts from 1 in the source. Mirror that exactly: mandatory controls render as `(1)`, `(2)`, `(3)`; the first recommended control restarts at `(1)` and is marked inline with the `(Recommended)` prefix in the bullet text. Do not continue mandatory numbering into the recommended section. Auditors cite back to source as e.g. "AACQ.C2.1 Mandatory (1.a)" or "AACQ.C2.1 Recommended (1.c)" — same `(N.x)` form within each scope.
+
+The only acceptable folding case is when the source uses sub-bullets as inline prose enumeration within a single sentence (e.g. "must include: (a) name, (b) address, (c) phone"). If the source presents each sub-bullet as a structurally separate provision — its own paragraph or its own labeled clause — preserve it as a nested checkbox. **When in doubt, preserve.** A future auditor can fold a faithful checklist; they cannot recover structure the parser threw away.
+
+This prefix scheme is not used under `id_scheme: cat`, where items aren't anchored to a numbered source list.
+
+### Severity definitions
+
+`id_scheme: cat` (default):
+
+| Level      | Meaning                                                       |
+|------------|---------------------------------------------------------------|
+| `critical` | Non-compliance triggers regulatory sanctions or licence risk  |
+| `major`    | Non-compliance creates significant operational or legal risk  |
+| `minor`    | Best-practice gap; low immediate risk but should be addressed |
+
+`id_scheme: source-native`:
+
+| Level         | Meaning                                                         |
+|---------------|-----------------------------------------------------------------|
+| `mandatory`   | Source classifies the control as required (e.g. VARS Mandatory) |
+| `recommended` | Source classifies the control as best-practice/recommended      |
+
+## Extending a Base Checklist
+
+Set `extends: "path/to/base-checklist.md"` in the metadata, relative to the extending checklist's own directory. The extending checklist inherits all base categories and items. `dump_checklist.py` does not merge inherited items into the JSON; consumers that need them load the base checklist too.
+
+**Allowed operations:**
+- **Add new categories** — use IDs that don't conflict with the base (e.g. start at CAT-06 if base uses CAT-01 to CAT-05)
+- **Add items to existing categories** — use the base category ID with new sequence numbers
+- **Override an item** — redefine the same item ID; the extending version fully replaces the base version for that ID
+- **Mark N/A** — override the item and prepend `[N/A]` to the Requirement with a justification
+
+**Not allowed:** removing base items.
+
+### Extension example
+
+```yaml
+name: "ACME Corp KYC/AML Checklist"
+regulation: "EU AML Directive 2015/849 — ACME Supplementary"
+version: "1.0"
+scope: "ACME-specific additions to base KYC/AML checklist"
+date: "2026-03-30"
+extends: "kyc-aml-checklist.md"   # same directory as this checklist
+```
+
+```markdown
+## CAT-01: Customer Due Diligence
+
+### CAT-01-020: Verify beneficial owner via company registry
+
+- **Requirement:** Cross-check beneficial ownership declarations against
+  the national company registry before onboarding.
+- **Source:** Internal Policy section 4.2
+- **Severity:** major
+- **Verification:**
+  - [ ] Company registry search performed
+  - [ ] Results compared with customer declaration
+  - [ ] Discrepancies escalated to compliance officer
+- **Evidence:**
+  - Company registry extract
+  - Comparison report
+
+## CAT-06: ACME Internal Controls
+
+### CAT-06-001: Quarterly compliance self-assessment
+
+- **Requirement:** Each business unit completes a quarterly self-assessment.
+- **Source:** Internal Policy section 8.1
+- **Severity:** major
+- **Verification:**
+  - [ ] Self-assessment form distributed and completed
+  - [ ] Results reviewed by compliance officer
+- **Evidence:**
+  - Completed self-assessment forms
+  - Review summary report
+```