@1aboveio/skills 0.20.2 → 0.20.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -6
- package/package.json +1 -1
- package/runtime/skills/distribution/generated/recipes.json +220 -25
- package/runtime/skills/distribution/scripts/bundles.mjs +167 -48
- package/runtime/skills/engineering/engineering-runtime/scripts/workflow-coherence.mjs +4 -0
- package/runtime/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs +1 -1
- package/skills/compliance/compliance-audit/LICENSE +3 -0
- package/skills/compliance/compliance-audit/SKILL.md +466 -0
- package/skills/compliance/compliance-audit/references/findings-schema.md +290 -0
- package/skills/compliance/compliance-audit/references/vars-checklist.md +1068 -0
- package/skills/compliance/compliance-audit/references/vars-findings-schema.md +395 -0
- package/skills/compliance/compliance-audit/scripts/reconcile_tracker.py +454 -0
- package/skills/compliance/compliance-audit/scripts/test_validate_tracker.py +90 -0
- package/skills/compliance/compliance-audit/scripts/validate_tracker.py +752 -0
- package/skills/compliance/doc-authenticity/LICENSE +3 -0
- package/skills/compliance/doc-authenticity/SKILL.md +158 -0
- package/skills/compliance/doc-authenticity/references/agreement.md +66 -0
- package/skills/compliance/doc-authenticity/references/bank-statement.md +150 -0
- package/skills/compliance/doc-authenticity/references/forensic-signals.md +157 -0
- package/skills/compliance/doc-authenticity/references/image-signals.md +114 -0
- package/skills/compliance/doc-authenticity/references/invoice.md +66 -0
- package/skills/compliance/doc-authenticity/references/issuer-profiles.md +86 -0
- package/skills/compliance/doc-authenticity/references/verdict-and-report.md +95 -0
- package/skills/compliance/doc-authenticity/references/verdict.schema.json +81 -0
- package/skills/compliance/doc-authenticity/scripts/image_forensics.py +294 -0
- package/skills/compliance/doc-authenticity/scripts/pdf_forensics.py +401 -0
- package/skills/compliance/doc-authenticity/scripts/validate_codes.py +224 -0
- package/skills/compliance/doc-authenticity/scripts/validate_verdict.py +308 -0
- package/skills/compliance/report-writing/LICENSE +3 -0
- package/skills/compliance/report-writing/SKILL.md +206 -0
- package/skills/compliance/report-writing/references/audit-report-template.md +121 -0
- package/skills/compliance/report-writing/references/external-assessment-template.md +167 -0
- package/skills/compliance/report-writing/references/management-report-template.md +145 -0
- package/skills/compliance/report-writing/references/vars-report-template.md +224 -0
- package/skills/document-utils/doc-to-md/LICENSE +3 -0
- package/skills/document-utils/doc-to-md/SKILL.md +77 -0
- package/skills/document-utils/doc-to-md/scripts/convert.py +157 -0
- package/skills/document-utils/reg-parser/LICENSE +3 -0
- package/skills/document-utils/reg-parser/SKILL.md +201 -0
- package/skills/document-utils/reg-parser/references/checklist-schema.md +223 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-amlo-checklist.json +1306 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-amlo-checklist.md +469 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-fi-corporate-onboarding-cdd-checklist.json +634 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-fi-corporate-onboarding-cdd-checklist.md +233 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-mso-guideline-checklist.json +2194 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-mso-guideline-checklist.md +771 -0
- package/skills/document-utils/reg-parser/references/checklists/kyc-aml-checklist.md +258 -0
- package/skills/document-utils/reg-parser/references/checklists/visa-vars-checklist.md +1058 -0
- package/skills/document-utils/reg-parser/references/mandates/hk-amlo-cap615.md +389 -0
- package/skills/document-utils/reg-parser/references/mandates/visa-vars.md +205 -0
- package/skills/document-utils/reg-parser/scripts/compare_checklist.py +228 -0
- package/skills/document-utils/reg-parser/scripts/dump_checklist.py +348 -0
- package/skills/document-utils/reg-parser/scripts/preprocess-mandate.sh +171 -0
- package/skills/document-utils/reg-parser/scripts/test_quarantine.sh +102 -0
- package/skills/document-utils/reg-parser/scripts/validate_checklist.py +532 -0
- package/skills/document-utils/reg-parser/scripts/validate_mandate.py +242 -0
- package/skills/engineering/engineering-runtime/coherence/workflow.json +16 -16
- package/skills/engineering/engineering-runtime/scripts/workflow-coherence.mjs +4 -0
- package/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs +1 -1
- package/skills/payment/similar-domain/LICENSE +3 -0
- package/skills/payment/similar-domain/SKILL.md +178 -0
- package/skills/payment/similar-domain/references/entities.md +58 -0
- package/skills/payment/similar-domain/references/fingerprints.md +82 -0
- package/skills/payment/similar-domain/references/pivots.md +66 -0
- package/skills/payment/similar-domain/references/report-template.md +76 -0
- package/skills/payment/similar-domain/references/scoring.md +61 -0
- package/skills/payment/similar-domain/references/terminology.md +37 -0
- package/skills/payment/similar-domain/references/workflow.md +79 -0
- package/skills/payment/tld-detection/SKILL.md +76 -21
- package/skills/payment/tld-detection/references/clean.md +1 -1
- package/skills/payment/tld-detection/references/clustering.md +21 -6
- package/skills/payment/tld-detection/references/decision.md +7 -2
- package/skills/payment/tld-detection/references/fx-conversion.md +229 -0
- package/skills/payment/tld-detection/references/report-template.md +63 -27
- package/skills/payment/tld-detection/references/streams.md +90 -0
- package/skills/payment/tld-detection/references/terminology.md +15 -7
- package/skills/payment/tld-detection/references/workflow.md +1 -1
- package/skills/payment/tld-detection/scripts/detect_streams.py +542 -0
- package/skills/engineering/ensure-coverage/evals/evals.json +0 -125
- package/skills/engineering/rush-issues/evals/evals.json +0 -65
- package/skills/engineering/rush-release/evals/evals.json +0 -58
- package/skills/fullstack/shadcn/evals/evals.json +0 -90
- package/skills/payment/fraud-analysis/evals/evals.json +0 -40
- package/skills/payment/tld-detection/evals/evals.json +0 -107
- package/skills/payment/tld-detection/tests/test_detect_ladders.py +0 -132
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""
|
|
3
|
+
doc-to-md: convert documents (docx, pptx, pdf, images, html, xlsx) to markdown.
|
|
4
|
+
|
|
5
|
+
PDF strategy:
|
|
6
|
+
- Primary: pdftotext (poppler-utils) — cleaner layout, better whitespace
|
|
7
|
+
- Fallback: markitdown — used when pdftotext is unavailable or produces empty output
|
|
8
|
+
|
|
9
|
+
Non-PDF formats: markitdown handles all (docx, pptx, xlsx, html, images).
|
|
10
|
+
|
|
11
|
+
Usage:
|
|
12
|
+
python3 convert.py <input_file_or_dir> [output_dir]
|
|
13
|
+
python3 convert.py policy.docx
|
|
14
|
+
python3 convert.py ./evidence-docs/ ./evidence-md/
|
|
15
|
+
python3 convert.py report.pdf ./out/
|
|
16
|
+
|
|
17
|
+
Output:
|
|
18
|
+
<output_dir>/<original_stem>.md (default output_dir = same dir as input)
|
|
19
|
+
|
|
20
|
+
Exits 0 on success (all converted), 1 if any file failed.
|
|
21
|
+
"""
|
|
22
|
+
|
|
23
|
+
import sys
|
|
24
|
+
import os
|
|
25
|
+
import pathlib
|
|
26
|
+
import argparse
|
|
27
|
+
import shutil
|
|
28
|
+
import subprocess
|
|
29
|
+
|
|
30
|
+
# Support non-system markitdown install
|
|
31
|
+
_MARKITDOWN_PATH = os.path.expanduser("~/.local/lib/python3-markitdown")
|
|
32
|
+
if _MARKITDOWN_PATH not in sys.path:
|
|
33
|
+
sys.path.insert(0, _MARKITDOWN_PATH)
|
|
34
|
+
|
|
35
|
+
try:
|
|
36
|
+
from markitdown import MarkItDown
|
|
37
|
+
_MARKITDOWN_AVAILABLE = True
|
|
38
|
+
except ImportError:
|
|
39
|
+
_MARKITDOWN_AVAILABLE = False
|
|
40
|
+
|
|
41
|
+
SUPPORTED = {".docx", ".pptx", ".pdf", ".html", ".htm", ".xlsx", ".xls",
|
|
42
|
+
".jpg", ".jpeg", ".png", ".gif", ".webp", ".bmp", ".tiff", ".tif"}
|
|
43
|
+
|
|
44
|
+
_PDFTOTEXT = shutil.which("pdftotext")
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
def convert_pdf(src: pathlib.Path, out_path: pathlib.Path) -> tuple[bool, str]:
|
|
48
|
+
"""Convert PDF using pdftotext (primary) or markitdown (fallback)."""
|
|
49
|
+
# Primary: pdftotext
|
|
50
|
+
if _PDFTOTEXT:
|
|
51
|
+
try:
|
|
52
|
+
result = subprocess.run(
|
|
53
|
+
[_PDFTOTEXT, "-layout", str(src), str(out_path)],
|
|
54
|
+
capture_output=True, text=True
|
|
55
|
+
)
|
|
56
|
+
if result.returncode == 0 and out_path.exists():
|
|
57
|
+
text = out_path.read_text(encoding="utf-8", errors="replace")
|
|
58
|
+
if text.strip():
|
|
59
|
+
lines = text.count("\n")
|
|
60
|
+
return True, f"pdftotext ({lines} lines)"
|
|
61
|
+
# Empty output — fall through to markitdown
|
|
62
|
+
except Exception:
|
|
63
|
+
pass # fall through
|
|
64
|
+
|
|
65
|
+
# Fallback: markitdown
|
|
66
|
+
if not _MARKITDOWN_AVAILABLE:
|
|
67
|
+
return False, "pdftotext produced empty output and markitdown not installed"
|
|
68
|
+
|
|
69
|
+
try:
|
|
70
|
+
md = MarkItDown()
|
|
71
|
+
result = md.convert(str(src))
|
|
72
|
+
text = result.text_content
|
|
73
|
+
if not text or not text.strip():
|
|
74
|
+
placeholder = f"<!-- doc-to-md: no text extracted from {src.name} — may be scanned/image-only -->\n"
|
|
75
|
+
out_path.write_text(placeholder, encoding="utf-8")
|
|
76
|
+
return True, "markitdown (empty — scanned PDF?)"
|
|
77
|
+
out_path.write_text(text, encoding="utf-8")
|
|
78
|
+
lines = text.count("\n")
|
|
79
|
+
return True, f"markitdown fallback ({lines} lines)"
|
|
80
|
+
except Exception as e:
|
|
81
|
+
return False, str(e)
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
def convert_file(src: pathlib.Path, out_dir: pathlib.Path) -> bool:
|
|
85
|
+
"""Convert a single file to markdown. Returns True on success."""
|
|
86
|
+
if src.suffix.lower() not in SUPPORTED:
|
|
87
|
+
print(f" SKIP {src.name} (unsupported type: {src.suffix})")
|
|
88
|
+
return True # not a failure
|
|
89
|
+
|
|
90
|
+
out_path = out_dir / (src.stem + ".md")
|
|
91
|
+
|
|
92
|
+
# PDF: dedicated handler
|
|
93
|
+
if src.suffix.lower() == ".pdf":
|
|
94
|
+
ok, detail = convert_pdf(src, out_path)
|
|
95
|
+
if ok:
|
|
96
|
+
print(f" OK {src.name} → {out_path.name} [{detail}]")
|
|
97
|
+
else:
|
|
98
|
+
print(f" FAIL {src.name} → {detail}")
|
|
99
|
+
return ok
|
|
100
|
+
|
|
101
|
+
# All other formats: markitdown
|
|
102
|
+
if not _MARKITDOWN_AVAILABLE:
|
|
103
|
+
print(f" FAIL {src.name} → markitdown not installed (run: uv pip install --system --target ~/.local/lib/python3-markitdown 'markitdown[all]')")
|
|
104
|
+
return False
|
|
105
|
+
|
|
106
|
+
try:
|
|
107
|
+
md = MarkItDown()
|
|
108
|
+
result = md.convert(str(src))
|
|
109
|
+
text = result.text_content
|
|
110
|
+
if not text or not text.strip():
|
|
111
|
+
print(f" WARN {src.name} → empty output")
|
|
112
|
+
text = f"<!-- doc-to-md: no text extracted from {src.name} -->\n"
|
|
113
|
+
|
|
114
|
+
out_path.write_text(text, encoding="utf-8")
|
|
115
|
+
lines = text.count("\n")
|
|
116
|
+
print(f" OK {src.name} → {out_path.name} ({lines} lines)")
|
|
117
|
+
return True
|
|
118
|
+
except Exception as e:
|
|
119
|
+
print(f" FAIL {src.name} → {e}")
|
|
120
|
+
return False
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
def main():
|
|
124
|
+
parser = argparse.ArgumentParser(description="Convert documents to markdown")
|
|
125
|
+
parser.add_argument("input", help="Input file or directory")
|
|
126
|
+
parser.add_argument("output_dir", nargs="?", default=None,
|
|
127
|
+
help="Output directory (default: same as input)")
|
|
128
|
+
args = parser.parse_args()
|
|
129
|
+
|
|
130
|
+
inp = pathlib.Path(args.input).expanduser().resolve()
|
|
131
|
+
if not inp.exists():
|
|
132
|
+
print(f"ERROR: input not found: {inp}", file=sys.stderr)
|
|
133
|
+
sys.exit(1)
|
|
134
|
+
|
|
135
|
+
if inp.is_dir():
|
|
136
|
+
files = sorted(f for f in inp.iterdir() if f.is_file())
|
|
137
|
+
out_dir = pathlib.Path(args.output_dir).expanduser().resolve() if args.output_dir else inp
|
|
138
|
+
else:
|
|
139
|
+
files = [inp]
|
|
140
|
+
out_dir = pathlib.Path(args.output_dir).expanduser().resolve() if args.output_dir else inp.parent
|
|
141
|
+
|
|
142
|
+
out_dir.mkdir(parents=True, exist_ok=True)
|
|
143
|
+
|
|
144
|
+
pdf_engine = f"pdftotext ({_PDFTOTEXT})" if _PDFTOTEXT else "markitdown (pdftotext not found)"
|
|
145
|
+
print(f"doc-to-md: converting {len(files)} file(s) → {out_dir}/")
|
|
146
|
+
print(f" PDF engine: {pdf_engine}")
|
|
147
|
+
print()
|
|
148
|
+
|
|
149
|
+
results = [convert_file(f, out_dir) for f in files]
|
|
150
|
+
ok = sum(results)
|
|
151
|
+
fail = len(results) - ok
|
|
152
|
+
print(f"\n {ok} converted, {fail} failed")
|
|
153
|
+
sys.exit(0 if fail == 0 else 1)
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
if __name__ == "__main__":
|
|
157
|
+
main()
|
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: reg-parser
|
|
3
|
+
version: 1.0.0
|
|
4
|
+
description: "Parse regulatory/scheme documents into structured outputs: convert a regulation (AML, VARS, PCI-DSS, KYC, GDPR) into an audit-ready compliance checklist, or trace contract clauses to a mandate source. Suggest proactively when a regulation PDF accompanies a contract review or compliance task. NOT general summarization (doc-to-md)."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# reg-parser
|
|
8
|
+
|
|
9
|
+
Parse a regulatory or scheme document into one of two structured outputs:
|
|
10
|
+
|
|
11
|
+
- **Mode A — Clause Annotation Extract**: flat obligation list for contract review P10 origin tracing
|
|
12
|
+
- **Mode B — Compliance Checklist**: audit-ready checklist with IDs, severity, verification steps
|
|
13
|
+
|
|
14
|
+
## Mode Inference
|
|
15
|
+
|
|
16
|
+
Infer mode from context — only ask if genuinely ambiguous:
|
|
17
|
+
|
|
18
|
+
| Context signals | Mode |
|
|
19
|
+
|-----------------|------|
|
|
20
|
+
| "contract", "clause", "review", "sign", "channel agreement", "is this clause required by…" | **A** |
|
|
21
|
+
| "audit", "checklist", "compliance", "gap analysis", "are we compliant", "VARS checklist" | **B** |
|
|
22
|
+
| Ambiguous | Ask: "Do you want a clause annotation extract (for contract review) or a compliance checklist (for auditing)?" |
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Step 1: Convert document
|
|
27
|
+
|
|
28
|
+
Use the `doc-to-md` skill:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
python3 {doc-to-md baseDir}/scripts/convert.py /path/to/regulation.pdf ./reg-md/
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Step 2: Extract obligations
|
|
37
|
+
|
|
38
|
+
Read the converted Markdown. Extract every **shall**, **must**, and mandatory requirement.
|
|
39
|
+
Ignore preamble, definitions, and non-binding guidance unless user requests them.
|
|
40
|
+
|
|
41
|
+
Group obligations by the document's section structure. For each obligation, capture:
|
|
42
|
+
- Section reference (§ number or heading)
|
|
43
|
+
- The obligation in plain language
|
|
44
|
+
- Who it applies to (regulator → licensed entity → downstream entity)
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## Step 3A: Output — Clause Annotation Extract (Mode A)
|
|
49
|
+
|
|
50
|
+
Save to `{baseDir}/references/mandates/<slug>.md` (known sources) or `references/mandates/adhoc-<slug>-<YYYYMMDD>.md` (ad-hoc).
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
bash {baseDir}/scripts/preprocess-mandate.sh <converted-md> \
|
|
54
|
+
--source "<Source Name>" \
|
|
55
|
+
--out {baseDir}/references/mandates/<slug>.md
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The script writes a template with a `Status: RAW` marker, an embedded raw-text dump, and HTML-comment scaffolding. After extraction, the final file must have the raw-text dump and all scaffolding comments removed, the `Status: RAW` line removed, and the four canonical sections filled in (use `_None_` if a section truly has no entries).
|
|
59
|
+
|
|
60
|
+
Output format:
|
|
61
|
+
|
|
62
|
+
```markdown
|
|
63
|
+
# <Source Name> — Contract Obligation Extract
|
|
64
|
+
|
|
65
|
+
> **Source:** <full citation>
|
|
66
|
+
> **Jurisdiction:** <scope>
|
|
67
|
+
> **Version / Effective Date:** <…>
|
|
68
|
+
|
|
69
|
+
## Mandatory Downstream Agreement Clauses
|
|
70
|
+
**§[ref] [title]**: [obligation in one sentence]
|
|
71
|
+
|
|
72
|
+
## Prohibited Conduct
|
|
73
|
+
**§[ref]**: [prohibition]
|
|
74
|
+
|
|
75
|
+
## Thresholds & Triggers
|
|
76
|
+
**§[ref] [metric]**: [value + consequence]
|
|
77
|
+
|
|
78
|
+
## Audit & Oversight Rights
|
|
79
|
+
**§[ref]**: [right]
|
|
80
|
+
|
|
81
|
+
## Notes
|
|
82
|
+
[Jurisdiction, effective date, version, caveats]
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
After generating, validate:
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
python3 {baseDir}/scripts/validate_mandate.py <mandate.md> [--source <converted-md>]
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
The optional `--source` flag enables citation-existence warnings (each `§ref` in the extract is fuzzy-matched against the converted source). Fix errors and re-run until clean (exit 0); warnings are advisory.
|
|
92
|
+
|
|
93
|
+
**Downstream use:** contract-review Step 0b loads this file for P10 clause origin tracing.
|
|
94
|
+
Annotations: `[Mandated: <source> §ref]` / `[Permitted: <source> §ref]` / `[Counterparty-Added]`
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## Step 3B: Output — Compliance Checklist (Mode B)
|
|
99
|
+
|
|
100
|
+
Read `{baseDir}/references/checklist-schema.md` for full schema before generating.
|
|
101
|
+
|
|
102
|
+
Save to `{baseDir}/references/checklists/<slug>-checklist.md`.
|
|
103
|
+
|
|
104
|
+
### Pick an `id_scheme` first
|
|
105
|
+
|
|
106
|
+
| Scheme | Use for | Item ID example | Severity values |
|
|
107
|
+
|-------------------|-------------------------------------------------------------------------------------------|-------------------|-----------------------------|
|
|
108
|
+
| `cat` (default) | Obligation lists with no canonical source IDs (AMLD articles, GDPR articles, FATF Recs) | `CAT-01-001` | critical / major / minor |
|
|
109
|
+
| `source-native` | Control frameworks with native ID hierarchies (Visa VARS, PCI-DSS, COBIT) | `AACQ.C1.1` | mandatory / recommended |
|
|
110
|
+
|
|
111
|
+
Use `source-native` when the source already groups its requirements under canonical IDs you'd lose by re-numbering. Otherwise default to `cat`. Source-native checklists may add optional `Risk Domain`, `Risk Sub-Domain`, `Risk Statement`, and `Parent Control` fields per item to preserve source taxonomy.
|
|
112
|
+
|
|
113
|
+
Under source-native, prefix every Verification step with **(N)**, **(a)**, **(i)** matching the source's numbered Mandatory/Recommended Controls list — see checklist-schema.md "Verification step prefixes" for the convention, including three-level nesting. Preserve the source's nesting depth verbatim; flattening lettered/roman sub-bullets into a single comma-separated bullet destroys auditor traceability and is the most common failure mode of this skill. When in doubt, preserve. The canonical example is `references/checklists/visa-vars-checklist.md` — see entries like `ATPA.C1.1`, `ATPA.C1.2`, `ATPA.C8.1` for the expected nesting pattern.
|
|
114
|
+
|
|
115
|
+
### Source-native specifics
|
|
116
|
+
|
|
117
|
+
**Severity rule.** Many control frameworks (e.g. VARS) split each control into a Mandatory Controls list and a Recommended Controls list. Set the item's `Severity`:
|
|
118
|
+
- `mandatory` if the source lists *any* Mandatory Controls under the control — even if Recommended Controls also exist
|
|
119
|
+
- `recommended` only when the source lists exclusively Recommended Controls
|
|
120
|
+
|
|
121
|
+
**Cross-archetype / cross-layer controls.** Some frameworks layer archetypes (e.g. VARS: ATPA-Acquirers inherit AACQ controls implicitly without restating them). Do not duplicate inherited items into the dependent archetype's category. Keep each control under the archetype where the source defines it; cross-references in the source's prose stand, and the audit consumer is expected to read parent archetypes too. If a control is genuinely re-stated by the source under a second archetype with different obligations, that's a new item under the second archetype's ID.
|
|
122
|
+
|
|
123
|
+
**Source-document defects.** Real regulation PDFs contain typos and TOC artefacts (e.g. `ATM.C3` for `AATM.C3`, "4.11" for "4.1.1"). Use the canonical form found in the body of the source; do not propagate the defect into the checklist. Note material corrections in the YAML `scope` field if they could affect interpretation.
|
|
124
|
+
|
|
125
|
+
Each obligation from Step 2 becomes one checklist item:
|
|
126
|
+
|
|
127
|
+
```markdown
|
|
128
|
+
### CAT-XX-NNN: [Requirement title] # cat scheme
|
|
129
|
+
### AACQ.C1.1: [Requirement title] # source-native scheme
|
|
130
|
+
|
|
131
|
+
- **Requirement**: [plain-language obligation]
|
|
132
|
+
- **Source**: §[ref]
|
|
133
|
+
- **Severity**: critical | major | minor # cat scheme
|
|
134
|
+
- **Severity**: mandatory | recommended # source-native scheme
|
|
135
|
+
- **Verification**:
|
|
136
|
+
- [ ] [Concrete observable step]
|
|
137
|
+
- [ ] [Another step]
|
|
138
|
+
- **Evidence**: [Documents or records that prove compliance]
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
After generating, validate:
|
|
142
|
+
|
|
143
|
+
```bash
|
|
144
|
+
python3 {baseDir}/scripts/validate_checklist.py <checklist.md> [--source <converted-md>]
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Fix errors and re-run until clean (exit 0).
|
|
148
|
+
|
|
149
|
+
The optional `--source` flag enables three semantic warnings (advisory, never block exit):
|
|
150
|
+
- **Citation existence** — each item's `Source:` field must contain at least one digit-bearing anchor (e.g. `13(1)(a)`, `2015/849`) that fuzzy-matches the source markdown.
|
|
151
|
+
- **Weak modal language** — Requirement contains `should`/`may`/`recommended` with no overriding `must`/`shall` anchor.
|
|
152
|
+
- **Verification independence** — Verification steps must not be a near-paraphrase of the Requirement (Jaccard > 0.5).
|
|
153
|
+
|
|
154
|
+
**Built-in reference checklists** (in `references/checklists/`):
|
|
155
|
+
- `kyc-aml-checklist.md` — KYC/AML (EU AMLD4/5), `cat` scheme
|
|
156
|
+
- `visa-vars-checklist.md` — Visa VARS Acquirer/TPA/HIR/ATM/Visa-Direct controls (45 items, 38 mandatory + 7 recommended), `source-native` scheme
|
|
157
|
+
|
|
158
|
+
### JSON dump for downstream consumers
|
|
159
|
+
|
|
160
|
+
Markdown is the authored source of truth — humans read and review it, GitHub renders it, and authors edit it directly. Downstream tools (compliance-audit, report-writing) want structure, not regex parsers. Generate a JSON view of any checklist with:
|
|
161
|
+
|
|
162
|
+
```bash
|
|
163
|
+
python3 {baseDir}/scripts/dump_checklist.py <checklist.md> --out <out.json>
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
The JSON preserves the verification tree verbatim (each node has `prefix`, `text`, `children`), so consumers can iterate items, walk nested controls, and look up severity/evidence without re-parsing markdown. Treat the JSON as a build artifact — regenerate on every checklist change rather than hand-editing it.
|
|
167
|
+
|
|
168
|
+
**Downstream use:** compliance-audit skill loads the checklist (markdown for review, JSON for programmatic consumption) to drive the audit workflow.
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## Known Source Slugs
|
|
173
|
+
|
|
174
|
+
| Source | Slug |
|
|
175
|
+
|--------|------|
|
|
176
|
+
| EU AML Directive 2015/849 (AMLD4/5) | `kyc-aml` |
|
|
177
|
+
| Visa VARS | `visa-vars` |
|
|
178
|
+
| HKMA AML Guide | `hk-aml-guide` |
|
|
179
|
+
| Mastercard SDP/MDRP | `mastercard-sdp` |
|
|
180
|
+
| PCI-DSS | `pci-dss` |
|
|
181
|
+
| UnionPay Agreement Rules | `unionpay-rules` |
|
|
182
|
+
| MAS Payment Services Act | `mas-psa` |
|
|
183
|
+
| FATF Recommendations | `fatf` |
|
|
184
|
+
|
|
185
|
+
Ad-hoc sources use `adhoc-<slug>-<YYYYMMDD>` — no permanent slug needed.
|
|
186
|
+
|
|
187
|
+
---
|
|
188
|
+
|
|
189
|
+
## Testing
|
|
190
|
+
|
|
191
|
+
End-to-end fixtures live in `references/tests/` (PDF input + ground-truth target). Because the test source overlaps with a built-in canonical checklist, an agent rerun must hide the canonical first, or it will copy it. Use the harness:
|
|
192
|
+
|
|
193
|
+
```bash
|
|
194
|
+
bash {baseDir}/scripts/test_quarantine.sh setup <slug> # hide canonical
|
|
195
|
+
# ... run agent against references/tests/<source>.pdf, write to a workspace path ...
|
|
196
|
+
python3 {baseDir}/scripts/validate_checklist.py <out.md> --source <converted-source.md>
|
|
197
|
+
python3 {baseDir}/scripts/compare_checklist.py <out.md> {baseDir}/references/tests/<source>-checklist.md
|
|
198
|
+
bash {baseDir}/scripts/test_quarantine.sh teardown <slug> # restore canonical
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
See `references/tests/README.md` for the full protocol.
|
|
@@ -0,0 +1,223 @@
|
|
|
1
|
+
# Checklist Schema Reference
|
|
2
|
+
|
|
3
|
+
Full format specification for compliance checklists. Read this when creating or validating a checklist.
|
|
4
|
+
|
|
5
|
+
## File Structure
|
|
6
|
+
|
|
7
|
+
A checklist is a single `.md` file with two sections: a YAML metadata block and requirement categories.
|
|
8
|
+
|
|
9
|
+
## ID schemes
|
|
10
|
+
|
|
11
|
+
Two schemes are supported. Pick one per checklist via the `id_scheme` metadata field:
|
|
12
|
+
|
|
13
|
+
| Scheme | When to use |
|
|
14
|
+
|-----------------|-----------------------------------------------------------------------------------------------------|
|
|
15
|
+
| `cat` (default) | Obligation-list regulations where the source has no canonical control IDs (AMLD articles, GDPR articles, FATF Recommendations). The skill assigns synthetic `CAT-XX-NNN` IDs. |
|
|
16
|
+
| `source-native` | Control-framework documents where the source already defines its own hierarchy (Visa VARS `AACQ.C1.1`, PCI-DSS `Req. 3.2.1`, COBIT `APO13.01`). Native IDs are preserved verbatim. |
|
|
17
|
+
|
|
18
|
+
The two schemes differ only in: category/item heading regex, severity enum, and summary count fields. All other rules (required item fields, item-field syntax, citation/modal/Jaccard semantic checks) are identical.
|
|
19
|
+
|
|
20
|
+
## Metadata Block
|
|
21
|
+
|
|
22
|
+
````markdown
|
|
23
|
+
```yaml
|
|
24
|
+
name: "Short Checklist Title"
|
|
25
|
+
regulation: "Full regulation name and number"
|
|
26
|
+
version: "1.0"
|
|
27
|
+
scope: "Brief description of what this checklist covers"
|
|
28
|
+
date: "YYYY-MM-DD"
|
|
29
|
+
id_scheme: "cat" # optional — "cat" (default) or "source-native"
|
|
30
|
+
extends: "path/to/base-checklist.md" # optional
|
|
31
|
+
summary:
|
|
32
|
+
categories: 5
|
|
33
|
+
total_items: 16
|
|
34
|
+
critical: 6
|
|
35
|
+
major: 7
|
|
36
|
+
minor: 3
|
|
37
|
+
```
|
|
38
|
+
````
|
|
39
|
+
|
|
40
|
+
For `id_scheme: source-native` the `summary` block uses `mandatory` and `recommended` instead of `critical`/`major`/`minor`:
|
|
41
|
+
|
|
42
|
+
```yaml
|
|
43
|
+
id_scheme: "source-native"
|
|
44
|
+
summary:
|
|
45
|
+
categories: 5
|
|
46
|
+
total_items: 45
|
|
47
|
+
mandatory: 38
|
|
48
|
+
recommended: 7
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
| Field | Required | Description |
|
|
52
|
+
|--------------|----------|-----------------------------------------------------|
|
|
53
|
+
| `name` | yes | Human-readable title |
|
|
54
|
+
| `regulation` | yes | Source regulation (e.g. "EU AML Directive 2015/849")|
|
|
55
|
+
| `version` | yes | Checklist version (semver recommended) |
|
|
56
|
+
| `scope` | yes | What the checklist covers |
|
|
57
|
+
| `date` | yes | Date of last revision (YYYY-MM-DD) |
|
|
58
|
+
| `id_scheme` | no | `cat` (default) or `source-native` |
|
|
59
|
+
| `extends` | no | Relative path to a base checklist to inherit from |
|
|
60
|
+
| `summary` | yes | Scheme-dependent counts (see above) |
|
|
61
|
+
|
|
62
|
+
## Categories
|
|
63
|
+
|
|
64
|
+
H2 heading with an ID prefix:
|
|
65
|
+
|
|
66
|
+
```markdown
|
|
67
|
+
## CAT-01: Customer Due Diligence # cat scheme
|
|
68
|
+
## AACQ: Acquirer Controls # source-native scheme
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
| Scheme | Category ID format |
|
|
72
|
+
|-----------------|-----------------------------------------------|
|
|
73
|
+
| `cat` | `CAT-` plus two digits (`CAT-01`, `CAT-02`) |
|
|
74
|
+
| `source-native` | Uppercase token of letters/digits (`AACQ`, `ATPA`, `PCIDSS`) |
|
|
75
|
+
|
|
76
|
+
Categories appear in logical audit order.
|
|
77
|
+
|
|
78
|
+
## Checklist Items
|
|
79
|
+
|
|
80
|
+
H3 under their category. Item IDs must be unique within the file; the prefix before the first dot/dash must match an existing category ID.
|
|
81
|
+
|
|
82
|
+
```markdown
|
|
83
|
+
### CAT-01-001: Verify customer identity # cat scheme
|
|
84
|
+
### AACQ.C1.1: Alignment Between Business Plans and Risk Policies # source-native
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
| Scheme | Item ID format |
|
|
88
|
+
|-----------------|-------------------------------------------------------------------------------|
|
|
89
|
+
| `cat` | `<CAT-ID>-<SEQ>` — SEQ is zero-padded three digits (`CAT-01-001`) |
|
|
90
|
+
| `source-native` | `<CAT-ID>.<segment>(\.<segment>)+` — preserves source punctuation (`AACQ.C1.1`, `AACQ.C8.2`) |
|
|
91
|
+
|
|
92
|
+
Example item:
|
|
93
|
+
|
|
94
|
+
```markdown
|
|
95
|
+
### CAT-01-001: Verify customer identity
|
|
96
|
+
|
|
97
|
+
- **Requirement:** Collect and verify government-issued photo ID for all
|
|
98
|
+
natural persons before establishing a business relationship.
|
|
99
|
+
- **Source:** Art. 13(1)(a), Directive 2015/849
|
|
100
|
+
- **Severity:** critical
|
|
101
|
+
- **Verification:**
|
|
102
|
+
- [ ] Photo ID collected (passport, national ID, or driver licence)
|
|
103
|
+
- [ ] ID verified against an independent, reliable source
|
|
104
|
+
- [ ] Verification date and method recorded
|
|
105
|
+
- **Evidence:**
|
|
106
|
+
- Copy of verified ID document
|
|
107
|
+
- Verification log entry with timestamp and officer name
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
### Item fields
|
|
111
|
+
|
|
112
|
+
| Field | Required | Description |
|
|
113
|
+
|-------------------|----------|------------------------------------------------------|
|
|
114
|
+
| **Requirement** | yes | Plain-language statement of what must be done |
|
|
115
|
+
| **Source** | yes | Article, section, or clause in the source regulation |
|
|
116
|
+
| **Severity** | yes | Scheme-dependent (see below) |
|
|
117
|
+
| **Verification** | yes | Checkbox list of concrete steps to confirm compliance|
|
|
118
|
+
| **Evidence** | yes | List of documents or artefacts that prove compliance |
|
|
119
|
+
| **Risk Domain** | no | Source-native taxonomy (e.g. "4.1.2 Operational Risk") |
|
|
120
|
+
| **Risk Sub-Domain** | no | Source-native sub-grouping (e.g. "Written Agreements") |
|
|
121
|
+
| **Risk Statement**| no | One-line statement of the risk being mitigated |
|
|
122
|
+
| **Parent Control**| no | Source-native parent control ID (e.g. `AACQ.C2`) |
|
|
123
|
+
|
|
124
|
+
The four optional fields preserve source taxonomy when `id_scheme: source-native` is used. They are silently ignored under the `cat` scheme.
|
|
125
|
+
|
|
126
|
+
### Verification step prefixes (source-native only)
|
|
127
|
+
|
|
128
|
+
Under `id_scheme: source-native`, each Verification checkbox **must** carry a bold prefix identifying the source-numbered control item it audits. Use parenthesized numbering that mirrors the source document's outline:
|
|
129
|
+
|
|
130
|
+
```markdown
|
|
131
|
+
- **Verification:**
|
|
132
|
+
- [ ] **(1)** Written agreement contains the following provisions:
|
|
133
|
+
- [ ] **(a)** Standards: Visa minimum standards incorporated
|
|
134
|
+
- [ ] **(b)** Visa's rights and authority: audits, accessibility, termination clause
|
|
135
|
+
- [ ] **(c)** Compliance: TPA does not submit illegal transactions
|
|
136
|
+
- [ ] **(i)** TPA complies with Visa Rules and applicable laws
|
|
137
|
+
- [ ] **(ii)** TPA adheres to Acquirer policies and AISP
|
|
138
|
+
- [ ] **(2)** (Recommended) Agreement names Acquirer as primary party
|
|
139
|
+
- [ ] **(a)** Defines duties and responsibilities of TPA and Acquirer
|
|
140
|
+
- [ ] **(b)** Allows transfer/assignment to another Acquirer
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
Within a parent bullet's nested list, the sub-prefix is just `(a)` / `(i)` — the parent number is implied. The fully qualified path (`AACQ.C1.1.(1.c.ii)`) is what an auditor reconstructs when citing back to source; the on-page prefix stays short.
|
|
144
|
+
|
|
145
|
+
**Preserve the source's nesting depth verbatim.** Every lettered or roman sub-bullet in the source becomes its own checkbox at the matching depth. The whole point of source-native is auditor traceability back to specific source bullets, and flattening destroys it. Three-level nesting — `(N)` → `(a)` → `(i)` — is fully supported and expected for control frameworks like VARS where mandatory provisions sit two or three levels deep.
|
|
146
|
+
|
|
147
|
+
**Mandatory and Recommended controls use independent numbering.** When a source control has both a "Mandatory Controls" and a "Recommended Controls" section (e.g. VARS AACQ.C2.1 with 3 mandatory + 1 recommended), each section's numbering restarts from 1 in the source. Mirror that exactly: mandatory controls render as `(1)`, `(2)`, `(3)`; the first recommended control restarts at `(1)` and is marked inline with the `(Recommended)` prefix in the bullet text. Do not continue mandatory numbering into the recommended section. Auditors cite back to source as e.g. "AACQ.C2.1 Mandatory (1.a)" or "AACQ.C2.1 Recommended (1.c)" — same `(N.x)` form within each scope.
|
|
148
|
+
|
|
149
|
+
The only acceptable folding case is when the source uses sub-bullets as inline prose enumeration within a single sentence (e.g. "must include: (a) name, (b) address, (c) phone"). If the source presents each sub-bullet as a structurally separate provision — its own paragraph or its own labeled clause — preserve it as a nested checkbox. **When in doubt, preserve.** A future auditor can fold a faithful checklist; they cannot recover structure the parser threw away.
|
|
150
|
+
|
|
151
|
+
This prefix scheme is not used under `id_scheme: cat`, where items aren't anchored to a numbered source list.
|
|
152
|
+
|
|
153
|
+
### Severity definitions
|
|
154
|
+
|
|
155
|
+
`id_scheme: cat` (default):
|
|
156
|
+
|
|
157
|
+
| Level | Meaning |
|
|
158
|
+
|------------|---------------------------------------------------------------|
|
|
159
|
+
| `critical` | Non-compliance triggers regulatory sanctions or licence risk |
|
|
160
|
+
| `major` | Non-compliance creates significant operational or legal risk |
|
|
161
|
+
| `minor` | Best-practice gap; low immediate risk but should be addressed |
|
|
162
|
+
|
|
163
|
+
`id_scheme: source-native`:
|
|
164
|
+
|
|
165
|
+
| Level | Meaning |
|
|
166
|
+
|---------------|-----------------------------------------------------------------|
|
|
167
|
+
| `mandatory` | Source classifies the control as required (e.g. VARS Mandatory) |
|
|
168
|
+
| `recommended` | Source classifies the control as best-practice/recommended |
|
|
169
|
+
|
|
170
|
+
## Extending a Base Checklist
|
|
171
|
+
|
|
172
|
+
Set `extends: "path/to/base-checklist.md"` in the metadata. The extending checklist inherits all base categories and items.
|
|
173
|
+
|
|
174
|
+
**Allowed operations:**
|
|
175
|
+
- **Add new categories** — use IDs that don't conflict with the base (e.g. start at CAT-06 if base uses CAT-01 to CAT-05)
|
|
176
|
+
- **Add items to existing categories** — use the base category ID with new sequence numbers
|
|
177
|
+
- **Override an item** — redefine the same item ID; the extending version fully replaces the base version for that ID
|
|
178
|
+
- **Mark N/A** — override the item and prepend `[N/A]` to the Requirement with a justification
|
|
179
|
+
|
|
180
|
+
**Not allowed:** removing base items.
|
|
181
|
+
|
|
182
|
+
### Extension example
|
|
183
|
+
|
|
184
|
+
```yaml
|
|
185
|
+
name: "ACME Corp KYC/AML Checklist"
|
|
186
|
+
regulation: "EU AML Directive 2015/849 — ACME Supplementary"
|
|
187
|
+
version: "1.0"
|
|
188
|
+
scope: "ACME-specific additions to base KYC/AML checklist"
|
|
189
|
+
date: "2026-03-30"
|
|
190
|
+
extends: "references/kyc-aml-checklist.md"
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
```markdown
|
|
194
|
+
## CAT-01: Customer Due Diligence
|
|
195
|
+
|
|
196
|
+
### CAT-01-020: Verify beneficial owner via company registry
|
|
197
|
+
|
|
198
|
+
- **Requirement:** Cross-check beneficial ownership declarations against
|
|
199
|
+
the national company registry before onboarding.
|
|
200
|
+
- **Source:** Internal Policy §4.2
|
|
201
|
+
- **Severity:** major
|
|
202
|
+
- **Verification:**
|
|
203
|
+
- [ ] Company registry search performed
|
|
204
|
+
- [ ] Results compared with customer declaration
|
|
205
|
+
- [ ] Discrepancies escalated to compliance officer
|
|
206
|
+
- **Evidence:**
|
|
207
|
+
- Company registry extract
|
|
208
|
+
- Comparison report
|
|
209
|
+
|
|
210
|
+
## CAT-06: ACME Internal Controls
|
|
211
|
+
|
|
212
|
+
### CAT-06-001: Quarterly compliance self-assessment
|
|
213
|
+
|
|
214
|
+
- **Requirement:** Each business unit completes a quarterly self-assessment.
|
|
215
|
+
- **Source:** Internal Policy §8.1
|
|
216
|
+
- **Severity:** major
|
|
217
|
+
- **Verification:**
|
|
218
|
+
- [ ] Self-assessment form distributed and completed
|
|
219
|
+
- [ ] Results reviewed by compliance officer
|
|
220
|
+
- **Evidence:**
|
|
221
|
+
- Completed self-assessment forms
|
|
222
|
+
- Review summary report
|
|
223
|
+
```
|