@1aboveio/skills 0.20.2 → 0.20.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -6
- package/package.json +1 -1
- package/runtime/skills/distribution/generated/recipes.json +220 -25
- package/runtime/skills/distribution/scripts/bundles.mjs +167 -48
- package/runtime/skills/engineering/engineering-runtime/scripts/workflow-coherence.mjs +4 -0
- package/runtime/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs +1 -1
- package/skills/compliance/compliance-audit/LICENSE +3 -0
- package/skills/compliance/compliance-audit/SKILL.md +466 -0
- package/skills/compliance/compliance-audit/references/findings-schema.md +290 -0
- package/skills/compliance/compliance-audit/references/vars-checklist.md +1068 -0
- package/skills/compliance/compliance-audit/references/vars-findings-schema.md +395 -0
- package/skills/compliance/compliance-audit/scripts/reconcile_tracker.py +454 -0
- package/skills/compliance/compliance-audit/scripts/test_validate_tracker.py +90 -0
- package/skills/compliance/compliance-audit/scripts/validate_tracker.py +752 -0
- package/skills/compliance/doc-authenticity/LICENSE +3 -0
- package/skills/compliance/doc-authenticity/SKILL.md +158 -0
- package/skills/compliance/doc-authenticity/references/agreement.md +66 -0
- package/skills/compliance/doc-authenticity/references/bank-statement.md +150 -0
- package/skills/compliance/doc-authenticity/references/forensic-signals.md +157 -0
- package/skills/compliance/doc-authenticity/references/image-signals.md +114 -0
- package/skills/compliance/doc-authenticity/references/invoice.md +66 -0
- package/skills/compliance/doc-authenticity/references/issuer-profiles.md +86 -0
- package/skills/compliance/doc-authenticity/references/verdict-and-report.md +95 -0
- package/skills/compliance/doc-authenticity/references/verdict.schema.json +81 -0
- package/skills/compliance/doc-authenticity/scripts/image_forensics.py +294 -0
- package/skills/compliance/doc-authenticity/scripts/pdf_forensics.py +401 -0
- package/skills/compliance/doc-authenticity/scripts/validate_codes.py +224 -0
- package/skills/compliance/doc-authenticity/scripts/validate_verdict.py +308 -0
- package/skills/compliance/report-writing/LICENSE +3 -0
- package/skills/compliance/report-writing/SKILL.md +206 -0
- package/skills/compliance/report-writing/references/audit-report-template.md +121 -0
- package/skills/compliance/report-writing/references/external-assessment-template.md +167 -0
- package/skills/compliance/report-writing/references/management-report-template.md +145 -0
- package/skills/compliance/report-writing/references/vars-report-template.md +224 -0
- package/skills/document-utils/doc-to-md/LICENSE +3 -0
- package/skills/document-utils/doc-to-md/SKILL.md +77 -0
- package/skills/document-utils/doc-to-md/scripts/convert.py +157 -0
- package/skills/document-utils/reg-parser/LICENSE +3 -0
- package/skills/document-utils/reg-parser/SKILL.md +201 -0
- package/skills/document-utils/reg-parser/references/checklist-schema.md +223 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-amlo-checklist.json +1306 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-amlo-checklist.md +469 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-fi-corporate-onboarding-cdd-checklist.json +634 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-fi-corporate-onboarding-cdd-checklist.md +233 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-mso-guideline-checklist.json +2194 -0
- package/skills/document-utils/reg-parser/references/checklists/hk-mso-guideline-checklist.md +771 -0
- package/skills/document-utils/reg-parser/references/checklists/kyc-aml-checklist.md +258 -0
- package/skills/document-utils/reg-parser/references/checklists/visa-vars-checklist.md +1058 -0
- package/skills/document-utils/reg-parser/references/mandates/hk-amlo-cap615.md +389 -0
- package/skills/document-utils/reg-parser/references/mandates/visa-vars.md +205 -0
- package/skills/document-utils/reg-parser/scripts/compare_checklist.py +228 -0
- package/skills/document-utils/reg-parser/scripts/dump_checklist.py +348 -0
- package/skills/document-utils/reg-parser/scripts/preprocess-mandate.sh +171 -0
- package/skills/document-utils/reg-parser/scripts/test_quarantine.sh +102 -0
- package/skills/document-utils/reg-parser/scripts/validate_checklist.py +532 -0
- package/skills/document-utils/reg-parser/scripts/validate_mandate.py +242 -0
- package/skills/engineering/engineering-runtime/coherence/workflow.json +16 -16
- package/skills/engineering/engineering-runtime/scripts/workflow-coherence.mjs +4 -0
- package/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs +1 -1
- package/skills/payment/similar-domain/LICENSE +3 -0
- package/skills/payment/similar-domain/SKILL.md +178 -0
- package/skills/payment/similar-domain/references/entities.md +58 -0
- package/skills/payment/similar-domain/references/fingerprints.md +82 -0
- package/skills/payment/similar-domain/references/pivots.md +66 -0
- package/skills/payment/similar-domain/references/report-template.md +76 -0
- package/skills/payment/similar-domain/references/scoring.md +61 -0
- package/skills/payment/similar-domain/references/terminology.md +37 -0
- package/skills/payment/similar-domain/references/workflow.md +79 -0
- package/skills/payment/tld-detection/SKILL.md +76 -21
- package/skills/payment/tld-detection/references/clean.md +1 -1
- package/skills/payment/tld-detection/references/clustering.md +21 -6
- package/skills/payment/tld-detection/references/decision.md +7 -2
- package/skills/payment/tld-detection/references/fx-conversion.md +229 -0
- package/skills/payment/tld-detection/references/report-template.md +63 -27
- package/skills/payment/tld-detection/references/streams.md +90 -0
- package/skills/payment/tld-detection/references/terminology.md +15 -7
- package/skills/payment/tld-detection/references/workflow.md +1 -1
- package/skills/payment/tld-detection/scripts/detect_streams.py +542 -0
- package/skills/engineering/ensure-coverage/evals/evals.json +0 -125
- package/skills/engineering/rush-issues/evals/evals.json +0 -65
- package/skills/engineering/rush-release/evals/evals.json +0 -58
- package/skills/fullstack/shadcn/evals/evals.json +0 -90
- package/skills/payment/fraud-analysis/evals/evals.json +0 -40
- package/skills/payment/tld-detection/evals/evals.json +0 -107
- package/skills/payment/tld-detection/tests/test_detect_ladders.py +0 -132
|
@@ -0,0 +1,242 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Validate a mandate (Mode A — Clause Annotation Extract) Markdown file.
|
|
3
|
+
|
|
4
|
+
Catches the failure modes specific to mandate extracts produced by the
|
|
5
|
+
`reg-parser` skill:
|
|
6
|
+
|
|
7
|
+
- Agent left the raw template scaffolding behind (Status: RAW, HTML
|
|
8
|
+
comment placeholders, the embedded raw-text fenced block).
|
|
9
|
+
- One of the four canonical sections is missing or silently empty.
|
|
10
|
+
- Header metadata (Source attribution, jurisdiction/version/date) is
|
|
11
|
+
missing.
|
|
12
|
+
- Notes section is empty or stub.
|
|
13
|
+
|
|
14
|
+
Optional --source <converted-md> enables citation-existence warnings:
|
|
15
|
+
each `§<ref>` cited in the extract is fuzzy-matched against the source
|
|
16
|
+
markdown. Missing citations emit warnings, not errors (refs are often
|
|
17
|
+
restyled during extraction).
|
|
18
|
+
|
|
19
|
+
Usage:
|
|
20
|
+
python3 validate_mandate.py <mandate.md> [--source <converted-md>]
|
|
21
|
+
|
|
22
|
+
Exit code 0 = valid. Non-zero = errors found.
|
|
23
|
+
"""
|
|
24
|
+
|
|
25
|
+
import argparse
|
|
26
|
+
import re
|
|
27
|
+
import sys
|
|
28
|
+
import os
|
|
29
|
+
|
|
30
|
+
REQUIRED_SECTIONS = [
|
|
31
|
+
"Mandatory Downstream Agreement Clauses",
|
|
32
|
+
"Prohibited Conduct",
|
|
33
|
+
"Thresholds & Triggers",
|
|
34
|
+
"Audit & Oversight Rights",
|
|
35
|
+
"Notes",
|
|
36
|
+
]
|
|
37
|
+
|
|
38
|
+
# Markers an agent may use to indicate a section legitimately has no entries.
|
|
39
|
+
EMPTY_MARKERS = ("_none_", "(none)", "n/a", "not applicable", "none.")
|
|
40
|
+
|
|
41
|
+
# Header metadata keywords — at least one must appear in the header block
|
|
42
|
+
# (between H1 and the first H2). Matches the conventions in the existing
|
|
43
|
+
# extracts (visa-vars.md, hk-amlo-cap615.md).
|
|
44
|
+
HEADER_KEYWORDS = ("source", "jurisdiction", "version", "effective", "updated", "date")
|
|
45
|
+
|
|
46
|
+
# Patterns that indicate the agent left raw template artifacts behind.
|
|
47
|
+
RAW_STATUS_RE = re.compile(r"^\s*>\s*\*?\*?Status:\*?\*?\s*RAW", re.IGNORECASE)
|
|
48
|
+
RAW_TEXT_HEADING_RE = re.compile(r"^##\s+Raw Text\b", re.IGNORECASE)
|
|
49
|
+
SCAFFOLDING_COMMENT_RES = [
|
|
50
|
+
re.compile(r"<!--\s*Agent:\s*replace", re.IGNORECASE),
|
|
51
|
+
re.compile(r"<!--\s*Format:\s*\*\*§\[ref\]"),
|
|
52
|
+
re.compile(r"<!--\s*Clauses the counterparty"),
|
|
53
|
+
re.compile(r"<!--\s*Explicitly prohibited"),
|
|
54
|
+
re.compile(r"<!--\s*Numeric thresholds"),
|
|
55
|
+
re.compile(r"<!--\s*Rights granted"),
|
|
56
|
+
re.compile(r"<!--\s*Jurisdiction scope, effective date"),
|
|
57
|
+
]
|
|
58
|
+
|
|
59
|
+
# A §-style citation. Captures common forms: §AACQ.C2.1, §76E,
|
|
60
|
+
# §Sched.2 §2(1)(a), § 14(3)(b). The capturing group keeps just the
|
|
61
|
+
# alphanumeric/punctuation token; the leading § is consumed.
|
|
62
|
+
CITATION_RE = re.compile(r"§\s*([A-Za-z0-9][A-Za-z0-9.\-]*(?:\([^)]+\))*)")
|
|
63
|
+
|
|
64
|
+
H1_RE = re.compile(r"^#\s+(.+?)\s*$")
|
|
65
|
+
H2_RE = re.compile(r"^##\s+(.+?)\s*$")
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
def split_sections(lines: list[str]) -> tuple[list[str], dict[str, list[str]]]:
|
|
69
|
+
"""Return (header_lines, {section_title: body_lines})."""
|
|
70
|
+
header: list[str] = []
|
|
71
|
+
sections: dict[str, list[str]] = {}
|
|
72
|
+
current: str | None = None
|
|
73
|
+
seen_h1 = False
|
|
74
|
+
|
|
75
|
+
for line in lines:
|
|
76
|
+
if not seen_h1 and H1_RE.match(line):
|
|
77
|
+
seen_h1 = True
|
|
78
|
+
continue
|
|
79
|
+
h2 = H2_RE.match(line)
|
|
80
|
+
if h2:
|
|
81
|
+
current = h2.group(1).strip()
|
|
82
|
+
sections.setdefault(current, [])
|
|
83
|
+
continue
|
|
84
|
+
if current is None:
|
|
85
|
+
header.append(line)
|
|
86
|
+
else:
|
|
87
|
+
sections[current].append(line)
|
|
88
|
+
|
|
89
|
+
return header, sections
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
def section_is_empty(body: list[str]) -> bool:
|
|
93
|
+
"""A section is empty if it has no §-bullet entries AND no explicit
|
|
94
|
+
'none' marker."""
|
|
95
|
+
text = "\n".join(body).strip()
|
|
96
|
+
if not text:
|
|
97
|
+
return True
|
|
98
|
+
lower = text.lower()
|
|
99
|
+
if any(marker in lower for marker in EMPTY_MARKERS):
|
|
100
|
+
return False
|
|
101
|
+
if "**§" in text or re.search(r"^\s*§", text, re.MULTILINE):
|
|
102
|
+
return False
|
|
103
|
+
if "§" in text and re.search(r"§\s*[A-Za-z0-9]", text):
|
|
104
|
+
return False
|
|
105
|
+
return True
|
|
106
|
+
|
|
107
|
+
|
|
108
|
+
def extract_citations(text: str) -> set[str]:
|
|
109
|
+
"""Pull §-style citation tokens out of arbitrary text."""
|
|
110
|
+
return {m.group(1) for m in CITATION_RE.finditer(text)}
|
|
111
|
+
|
|
112
|
+
|
|
113
|
+
def fuzzy_present(token: str, haystack_lower: str) -> bool:
|
|
114
|
+
"""Best-effort: token appears in source if either the literal token
|
|
115
|
+
or a heavily-normalized form (lowercased, punctuation stripped) does.
|
|
116
|
+
"""
|
|
117
|
+
if token.lower() in haystack_lower:
|
|
118
|
+
return True
|
|
119
|
+
norm = re.sub(r"[^a-z0-9]", "", token.lower())
|
|
120
|
+
if not norm:
|
|
121
|
+
return False
|
|
122
|
+
norm_haystack = re.sub(r"[^a-z0-9]", "", haystack_lower)
|
|
123
|
+
return norm in norm_haystack
|
|
124
|
+
|
|
125
|
+
|
|
126
|
+
def validate(filepath: str, source_path: str | None) -> tuple[list[str], list[str]]:
|
|
127
|
+
"""Return (errors, warnings)."""
|
|
128
|
+
errors: list[str] = []
|
|
129
|
+
warnings: list[str] = []
|
|
130
|
+
|
|
131
|
+
if not os.path.isfile(filepath):
|
|
132
|
+
return [f"File not found: {filepath}"], []
|
|
133
|
+
|
|
134
|
+
with open(filepath, "r", encoding="utf-8") as f:
|
|
135
|
+
raw = f.read()
|
|
136
|
+
lines = raw.splitlines()
|
|
137
|
+
|
|
138
|
+
# --- Raw template artifacts ---
|
|
139
|
+
for line_num, line in enumerate(lines, 1):
|
|
140
|
+
if RAW_STATUS_RE.match(line):
|
|
141
|
+
errors.append(
|
|
142
|
+
f"Line {line_num}: 'Status: RAW' marker still present — "
|
|
143
|
+
f"agent extraction not completed"
|
|
144
|
+
)
|
|
145
|
+
if RAW_TEXT_HEADING_RE.match(line):
|
|
146
|
+
errors.append(
|
|
147
|
+
f"Line {line_num}: '## Raw Text' section still present — "
|
|
148
|
+
f"remove the embedded raw-text dump after extraction"
|
|
149
|
+
)
|
|
150
|
+
for pat in SCAFFOLDING_COMMENT_RES:
|
|
151
|
+
if pat.search(line):
|
|
152
|
+
errors.append(
|
|
153
|
+
f"Line {line_num}: leftover template scaffolding comment"
|
|
154
|
+
)
|
|
155
|
+
break
|
|
156
|
+
|
|
157
|
+
# --- Structure ---
|
|
158
|
+
header_lines, sections = split_sections(lines)
|
|
159
|
+
|
|
160
|
+
if not any(H1_RE.match(line) for line in lines):
|
|
161
|
+
errors.append("Missing H1 title (expected '# <Source> — Contract Obligation Extract')")
|
|
162
|
+
|
|
163
|
+
# Header metadata: at least one of Source/Jurisdiction/Version/etc.
|
|
164
|
+
header_text = "\n".join(header_lines).lower()
|
|
165
|
+
if not any(kw in header_text for kw in HEADER_KEYWORDS):
|
|
166
|
+
errors.append(
|
|
167
|
+
"Header metadata block missing — expected blockquote with "
|
|
168
|
+
"Source / Jurisdiction / Version / Effective Date"
|
|
169
|
+
)
|
|
170
|
+
|
|
171
|
+
# The Source attribution specifically is critical.
|
|
172
|
+
if "source" not in header_text:
|
|
173
|
+
errors.append("Header missing 'Source:' attribution")
|
|
174
|
+
|
|
175
|
+
# Required sections present
|
|
176
|
+
for required in REQUIRED_SECTIONS:
|
|
177
|
+
if required not in sections:
|
|
178
|
+
errors.append(f"Missing required section: ## {required}")
|
|
179
|
+
|
|
180
|
+
# Required content sections non-empty
|
|
181
|
+
for required in REQUIRED_SECTIONS[:-1]: # everything except Notes
|
|
182
|
+
body = sections.get(required)
|
|
183
|
+
if body is not None and section_is_empty(body):
|
|
184
|
+
errors.append(
|
|
185
|
+
f"Section '## {required}' is empty — if there are no "
|
|
186
|
+
f"items, mark explicitly as '_None_'"
|
|
187
|
+
)
|
|
188
|
+
|
|
189
|
+
# Notes substance check
|
|
190
|
+
notes_body = sections.get("Notes")
|
|
191
|
+
if notes_body is not None:
|
|
192
|
+
notes_text = "\n".join(notes_body).strip()
|
|
193
|
+
if len(notes_text) < 50:
|
|
194
|
+
errors.append(
|
|
195
|
+
"Section '## Notes' is too short — include jurisdiction "
|
|
196
|
+
"scope, effective date, version, and caveats"
|
|
197
|
+
)
|
|
198
|
+
|
|
199
|
+
# --- Optional: citation existence check ---
|
|
200
|
+
if source_path:
|
|
201
|
+
if not os.path.isfile(source_path):
|
|
202
|
+
warnings.append(f"--source file not found: {source_path}")
|
|
203
|
+
else:
|
|
204
|
+
with open(source_path, "r", encoding="utf-8") as sf:
|
|
205
|
+
source_lower = sf.read().lower()
|
|
206
|
+
cites = extract_citations(raw)
|
|
207
|
+
missing = sorted(c for c in cites if not fuzzy_present(c, source_lower))
|
|
208
|
+
if missing:
|
|
209
|
+
preview = ", ".join(f"§{c}" for c in missing[:8])
|
|
210
|
+
more = "" if len(missing) <= 8 else f" (+{len(missing) - 8} more)"
|
|
211
|
+
warnings.append(
|
|
212
|
+
f"{len(missing)} citation(s) not found in source: {preview}{more}"
|
|
213
|
+
)
|
|
214
|
+
|
|
215
|
+
return errors, warnings
|
|
216
|
+
|
|
217
|
+
|
|
218
|
+
def main() -> None:
|
|
219
|
+
p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
|
220
|
+
p.add_argument("filepath", help="Path to mandate Markdown file")
|
|
221
|
+
p.add_argument(
|
|
222
|
+
"--source",
|
|
223
|
+
help="Optional path to the converted source Markdown for citation checks",
|
|
224
|
+
)
|
|
225
|
+
args = p.parse_args()
|
|
226
|
+
|
|
227
|
+
errors, warnings = validate(args.filepath, args.source)
|
|
228
|
+
|
|
229
|
+
for w in warnings:
|
|
230
|
+
print(f" ⚠️ {w}", file=sys.stderr)
|
|
231
|
+
|
|
232
|
+
if errors:
|
|
233
|
+
print(f"❌ Validation failed for {args.filepath}:", file=sys.stderr)
|
|
234
|
+
for err in errors:
|
|
235
|
+
print(f" • {err}", file=sys.stderr)
|
|
236
|
+
sys.exit(1)
|
|
237
|
+
print(f"✅ {args.filepath} is valid.")
|
|
238
|
+
sys.exit(0)
|
|
239
|
+
|
|
240
|
+
|
|
241
|
+
if __name__ == "__main__":
|
|
242
|
+
main()
|
|
@@ -342,7 +342,7 @@
|
|
|
342
342
|
"id": "first-party",
|
|
343
343
|
"type": "first-party",
|
|
344
344
|
"package": "@1aboveio/skills",
|
|
345
|
-
"version": "0.20.
|
|
345
|
+
"version": "0.20.4"
|
|
346
346
|
},
|
|
347
347
|
"contentDigest": "eff6c7b5931bce5b2265a619bccddd371a89df6ce7bc2edc74f11d00060a71dd",
|
|
348
348
|
"digestExcludes": []
|
|
@@ -355,7 +355,7 @@
|
|
|
355
355
|
"id": "first-party",
|
|
356
356
|
"type": "first-party",
|
|
357
357
|
"package": "@1aboveio/skills",
|
|
358
|
-
"version": "0.20.
|
|
358
|
+
"version": "0.20.4"
|
|
359
359
|
},
|
|
360
360
|
"contentDigest": "d97abeff7bdc5ae2e4be2f631d09dba46d4753ce1e67334793364329af59f066",
|
|
361
361
|
"digestExcludes": []
|
|
@@ -368,9 +368,9 @@
|
|
|
368
368
|
"id": "first-party",
|
|
369
369
|
"type": "first-party",
|
|
370
370
|
"package": "@1aboveio/skills",
|
|
371
|
-
"version": "0.20.
|
|
371
|
+
"version": "0.20.4"
|
|
372
372
|
},
|
|
373
|
-
"contentDigest": "
|
|
373
|
+
"contentDigest": "fe2ec3bae2bfec2dbef19db30ab0272391d7c1d9b6893341a76a9f339adefc57",
|
|
374
374
|
"digestExcludes": []
|
|
375
375
|
},
|
|
376
376
|
{
|
|
@@ -381,7 +381,7 @@
|
|
|
381
381
|
"id": "first-party",
|
|
382
382
|
"type": "first-party",
|
|
383
383
|
"package": "@1aboveio/skills",
|
|
384
|
-
"version": "0.20.
|
|
384
|
+
"version": "0.20.4"
|
|
385
385
|
},
|
|
386
386
|
"contentDigest": "58b0556228a9271cf9e33727a05fc08a4fdfd29512ec0936b169f0a55d60e6ef",
|
|
387
387
|
"digestExcludes": []
|
|
@@ -394,9 +394,9 @@
|
|
|
394
394
|
"id": "first-party",
|
|
395
395
|
"type": "first-party",
|
|
396
396
|
"package": "@1aboveio/skills",
|
|
397
|
-
"version": "0.20.
|
|
397
|
+
"version": "0.20.4"
|
|
398
398
|
},
|
|
399
|
-
"contentDigest": "
|
|
399
|
+
"contentDigest": "d28dd4d3c4394df0cc7e6b48d0e5daa7d2277701ea459e39fba99acf1d8ff670",
|
|
400
400
|
"digestExcludes": []
|
|
401
401
|
},
|
|
402
402
|
{
|
|
@@ -407,9 +407,9 @@
|
|
|
407
407
|
"id": "first-party",
|
|
408
408
|
"type": "first-party",
|
|
409
409
|
"package": "@1aboveio/skills",
|
|
410
|
-
"version": "0.20.
|
|
410
|
+
"version": "0.20.4"
|
|
411
411
|
},
|
|
412
|
-
"contentDigest": "
|
|
412
|
+
"contentDigest": "56fb5843788c9073b0d5a65ffa72bd7d2fa80dc6a1ff06fb0e19fab513c51502",
|
|
413
413
|
"digestExcludes": []
|
|
414
414
|
},
|
|
415
415
|
{
|
|
@@ -420,7 +420,7 @@
|
|
|
420
420
|
"id": "first-party",
|
|
421
421
|
"type": "first-party",
|
|
422
422
|
"package": "@1aboveio/skills",
|
|
423
|
-
"version": "0.20.
|
|
423
|
+
"version": "0.20.4"
|
|
424
424
|
},
|
|
425
425
|
"contentDigest": "f048fd00c69f2dc666fc7a3096cfeee6dfba73933bfb69602f3f0e26159798cc",
|
|
426
426
|
"digestExcludes": []
|
|
@@ -433,7 +433,7 @@
|
|
|
433
433
|
"id": "first-party",
|
|
434
434
|
"type": "first-party",
|
|
435
435
|
"package": "@1aboveio/skills",
|
|
436
|
-
"version": "0.20.
|
|
436
|
+
"version": "0.20.4"
|
|
437
437
|
},
|
|
438
438
|
"contentDigest": "d533684db4483cfc90be2aa7161980b6548057ef18a4b54f0cdbcd53c79dd46d",
|
|
439
439
|
"digestExcludes": []
|
|
@@ -446,15 +446,15 @@
|
|
|
446
446
|
"id": "first-party",
|
|
447
447
|
"type": "first-party",
|
|
448
448
|
"package": "@1aboveio/skills",
|
|
449
|
-
"version": "0.20.
|
|
449
|
+
"version": "0.20.4"
|
|
450
450
|
},
|
|
451
|
-
"contentDigest": "
|
|
451
|
+
"contentDigest": "86a91b567eabe6143a4ecac3cbec7271da252978202f292e287bb3a8da95a797",
|
|
452
452
|
"digestExcludes": [
|
|
453
453
|
"coherence/workflow.json"
|
|
454
454
|
]
|
|
455
455
|
}
|
|
456
456
|
],
|
|
457
|
-
"releaseIdentity": "
|
|
457
|
+
"releaseIdentity": "ffa735d74d402c79fbd8cbca332b415422884ee88f914412d4aa88c2d1353cb9",
|
|
458
458
|
"lifecycleAuthority": "native-skills-cli",
|
|
459
459
|
"repairRecipe": {
|
|
460
460
|
"id": "engineering-workflow-dependency-first",
|
|
@@ -512,7 +512,7 @@
|
|
|
512
512
|
"sourceId": "first-party",
|
|
513
513
|
"sourceType": "first-party",
|
|
514
514
|
"package": "@1aboveio/skills",
|
|
515
|
-
"version": "0.20.
|
|
515
|
+
"version": "0.20.4",
|
|
516
516
|
"installPath": null,
|
|
517
517
|
"members": [
|
|
518
518
|
"harness-runtime",
|
|
@@ -528,7 +528,7 @@
|
|
|
528
528
|
"commands": [
|
|
529
529
|
{
|
|
530
530
|
"transport": "npm",
|
|
531
|
-
"command": "npx @1aboveio/skills@0.20.
|
|
531
|
+
"command": "npx @1aboveio/skills@0.20.4 install --group engineering-workflow --yes"
|
|
532
532
|
}
|
|
533
533
|
],
|
|
534
534
|
"onFailure": {
|
|
@@ -154,6 +154,10 @@ const TEST_FIXTURE = /\.fixtures?\.[^.]+$/;
|
|
|
154
154
|
export function publishedContentExcluded(name) {
|
|
155
155
|
return name.startsWith('.')
|
|
156
156
|
|| name === 'node_modules'
|
|
157
|
+
|| name === '__pycache__'
|
|
158
|
+
|| name === 'evals'
|
|
159
|
+
|| name === 'samples'
|
|
160
|
+
|| name === 'tests'
|
|
157
161
|
|| name.endsWith('-workspace')
|
|
158
162
|
|| name.endsWith('.test.mjs')
|
|
159
163
|
|| TEST_FIXTURE.test(name);
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: similar-domain
|
|
3
|
+
description: >
|
|
4
|
+
Find the domains, mirrors, and operating entities related to one or more seed
|
|
5
|
+
domains: collect technical fingerprints (analytics IDs, block-page templates,
|
|
6
|
+
DNS, mail, certificates, registration), reverse-pivot on each fingerprint,
|
|
7
|
+
score every link, and expand until the cluster stops growing. Use whenever
|
|
8
|
+
the user gives a domain or URL and asks who is behind it, whether several
|
|
9
|
+
sites are related or the same operator, for sister sites, mirror domains,
|
|
10
|
+
the cluster, or a merchant-website background check — even if they never say
|
|
11
|
+
similar-domain. Stops at websites, mirrors and operators: anything inside
|
|
12
|
+
transaction data (streams, preset menus, amount ladders) is tld-detection,
|
|
13
|
+
and transaction-level fraud rules are fraud-analysis.
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
# Similar Domain
|
|
17
|
+
|
|
18
|
+
Start from a seed domain and find every other domain that the same people run.
|
|
19
|
+
A cluster is defined by **shared infrastructure**, not by shared branding or
|
|
20
|
+
shared industry. Corporate records (operator, licence, affiliate program)
|
|
21
|
+
confirm and name the cluster; they do not create it on their own.
|
|
22
|
+
|
|
23
|
+
The method is a loop: read fingerprints off a domain, ask "who else has this?"
|
|
24
|
+
for each fingerprint, score what comes back, and feed every new domain back
|
|
25
|
+
into the loop.
|
|
26
|
+
|
|
27
|
+
**Produce:** a scored cluster with an evidence matrix, the entity chain behind
|
|
28
|
+
it, and a ≤3-sentence verdict, under `reports/similar-domain/`.
|
|
29
|
+
**Do not produce:** an ownership claim with no corporate evidence, a cluster
|
|
30
|
+
built from one weak signal, or a list of every site on the same CDN.
|
|
31
|
+
|
|
32
|
+
Paths under `references/` are relative to this skill.
|
|
33
|
+
|
|
34
|
+
## Workflow
|
|
35
|
+
|
|
36
|
+
Follow these steps in order. Detail:
|
|
37
|
+
[references/workflow.md](references/workflow.md).
|
|
38
|
+
|
|
39
|
+
| Step | Do | Gate |
|
|
40
|
+
|---|---|---|
|
|
41
|
+
| 0 Pre-flight | Seeds, question, jurisdiction, access limits | Stop if there is no resolvable domain; note geo-blocks and challenge pages |
|
|
42
|
+
| 1 Fingerprint | Record every non-brand identifier on each domain | Every fingerprint has a source and a timestamp; local resolver artefacts are flagged |
|
|
43
|
+
| 2 Pivot | Reverse-search each fingerprint for other domains | Each pivot is logged with source class and hit count; high-noise pivots are filtered by a second fingerprint |
|
|
44
|
+
| 3 Expand names | Enumerate mirror and numbering variants of every brand found | Candidates are only kept if they share a fingerprint from step 1 |
|
|
45
|
+
| 4 Entity | Operator, licence, address, affiliate program, terms page | Each entity fact cites a page; conflicting operator names are kept, not resolved by guess |
|
|
46
|
+
| 5 Score | Weight every link; label each domain | Two independent strong signals for Confirmed; one for Probable; nothing weaker than Possible enters the cluster table |
|
|
47
|
+
| 6 Loop | Send every new Confirmed or Probable domain back to step 1 | Stop when a full pass adds no new domain |
|
|
48
|
+
| 7 Report | Verdict, cluster table, evidence matrix, entity chain, outliers, pivot log | ≤3-sentence verdict; no ownership wording without an entity citation |
|
|
49
|
+
|
|
50
|
+
### 0 — Pre-flight
|
|
51
|
+
|
|
52
|
+
Record the seed domains, what the user actually wants to know (same
|
|
53
|
+
operator? mirrors? merchant background?), and any jurisdiction or time limits.
|
|
54
|
+
Note what you cannot see: geo-blocks, bot challenges, private WHOIS. A blocked
|
|
55
|
+
site still yields a block page, DNS, certificates, and history, so blocking is
|
|
56
|
+
a finding, not a dead end. Grain and limits:
|
|
57
|
+
[workflow.md](references/workflow.md).
|
|
58
|
+
|
|
59
|
+
### 1 — Fingerprint
|
|
60
|
+
|
|
61
|
+
[references/fingerprints.md](references/fingerprints.md). Collect
|
|
62
|
+
everything about the domain that is not the brand name: registration facts,
|
|
63
|
+
DNS and mail records, certificate history, analytics and pixel IDs, the
|
|
64
|
+
block-page or error-page template, support-address pattern, hard-coded
|
|
65
|
+
partner hosts, and naming scheme. Record each with where it came from.
|
|
66
|
+
Fingerprint the seed fully before pivoting; a half-collected profile makes
|
|
67
|
+
later pivots look independent when they are not.
|
|
68
|
+
|
|
69
|
+
### 2 — Pivot
|
|
70
|
+
|
|
71
|
+
[references/pivots.md](references/pivots.md). For each fingerprint, query the
|
|
72
|
+
source class that can answer "who else has this": page-source indexes for
|
|
73
|
+
analytics IDs and templates, passive DNS for nameservers and mail records,
|
|
74
|
+
certificate transparency for subdomains and naming schemes, reverse WHOIS for
|
|
75
|
+
registration batches. Ordinary web search does not index page source; do not
|
|
76
|
+
treat an empty web-search result as a negative. Filter noisy pivots
|
|
77
|
+
(shared CDN nameservers, common mail includes) by requiring a second
|
|
78
|
+
fingerprint before a hit becomes a candidate.
|
|
79
|
+
|
|
80
|
+
### 3 — Expand names
|
|
81
|
+
|
|
82
|
+
Brands that rotate mirrors follow a scheme: numbered suffixes, hyphenated
|
|
83
|
+
digits, double-dash prefixes, country TLDs. Enumerate the scheme for every
|
|
84
|
+
brand found and keep only the variants that share a step-1 fingerprint.
|
|
85
|
+
A name match alone is a candidate, never a member.
|
|
86
|
+
|
|
87
|
+
### 4 — Entity
|
|
88
|
+
|
|
89
|
+
[references/entities.md](references/entities.md). Once a brand name exists,
|
|
90
|
+
collect the legal entity, licence number, address, and affiliate program from
|
|
91
|
+
the site's own terms page and from review aggregators, then reverse each of
|
|
92
|
+
those: licence registers list every domain under one licence, affiliate
|
|
93
|
+
programs list every brand they pay for, entity names appear in other sites'
|
|
94
|
+
terms pages. Keep conflicting operator names side by side with dates; operators
|
|
95
|
+
migrate and the conflict is often the history.
|
|
96
|
+
|
|
97
|
+
### 5 — Score
|
|
98
|
+
|
|
99
|
+
[references/scoring.md](references/scoring.md). Weight each link by how hard
|
|
100
|
+
it is to share by accident. One analytics account or one custom template is
|
|
101
|
+
very strong; one registrar or one CDN nameserver pair is medium. Two signals
|
|
102
|
+
from the same vendor account count once. Label each domain Confirmed,
|
|
103
|
+
Probable, Possible, or Unrelated, and keep the reasoning visible in the
|
|
104
|
+
evidence matrix.
|
|
105
|
+
|
|
106
|
+
### 6 — Loop
|
|
107
|
+
|
|
108
|
+
Every Confirmed or Probable domain becomes a new seed and goes through steps 1
|
|
109
|
+
to 5. Possible domains are parked with the fingerprint that would promote them.
|
|
110
|
+
Stop when a complete pass finds nothing new, or when the user's question is
|
|
111
|
+
answered and further expansion would only add mirrors.
|
|
112
|
+
|
|
113
|
+
### 7 — Report
|
|
114
|
+
|
|
115
|
+
[references/report-template.md](references/report-template.md). Terms:
|
|
116
|
+
[references/terminology.md](references/terminology.md).
|
|
117
|
+
|
|
118
|
+
```text
|
|
119
|
+
reports/similar-domain/<seed>_cluster.md
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
Lead with the verdict, then the cluster table sorted Confirmed → Probable →
|
|
123
|
+
Possible, the evidence matrix (domain × fingerprint), the entity chain with
|
|
124
|
+
dates, outliers that were checked and rejected, coverage caveats, and the
|
|
125
|
+
pivot log. Grep banned strings after writing.
|
|
126
|
+
|
|
127
|
+
## Progress
|
|
128
|
+
|
|
129
|
+
Track a task plan. One item is `in_progress` at a time.
|
|
130
|
+
|
|
131
|
+
1. Pass pre-flight
|
|
132
|
+
2. Fingerprint every seed
|
|
133
|
+
3. Run and log pivots
|
|
134
|
+
4. Expand naming schemes
|
|
135
|
+
5. Collect entity chain
|
|
136
|
+
6. Score and label
|
|
137
|
+
7. Loop until no new domain
|
|
138
|
+
8. Write report and proofread
|
|
139
|
+
|
|
140
|
+
## Boundary
|
|
141
|
+
|
|
142
|
+
| Need | Skill |
|
|
143
|
+
|---|---|
|
|
144
|
+
| Related domains, mirrors, sister sites, who runs a website, merchant-site background | `similar-domain` |
|
|
145
|
+
| Anything inside a transaction file: upstream streams behind a descriptor, wallet currency, preset menus, frozen-rate batches, retail constants, currency-amount ladders | `tld-detection` (`scripts/detect_streams.py`, then the ladder run) |
|
|
146
|
+
| Supervised fraud labels, holdout precision/recall, rule packages | `fraud-analysis` |
|
|
147
|
+
| Volume, journey, auth rate, decline mix | `payment-analysis` |
|
|
148
|
+
|
|
149
|
+
A domain cluster from this skill can feed a merchant-risk decision; it is not
|
|
150
|
+
itself a fraud finding. When the user also has authorization data for the
|
|
151
|
+
merchant, hand that part to `tld-detection`: this skill stops at the websites,
|
|
152
|
+
mirrors and operators.
|
|
153
|
+
|
|
154
|
+
## Reference index
|
|
155
|
+
|
|
156
|
+
| Need | Read |
|
|
157
|
+
|---|---|
|
|
158
|
+
| Inputs, limits, order that must not move | [workflow.md](references/workflow.md) |
|
|
159
|
+
| What to collect per domain and why it identifies an operator | [fingerprints.md](references/fingerprints.md) |
|
|
160
|
+
| Which source class answers each "who else" question, and the traps | [pivots.md](references/pivots.md) |
|
|
161
|
+
| Operator, licence, affiliate, terms-page pivots | [entities.md](references/entities.md) |
|
|
162
|
+
| Link weights, independence, labels | [scoring.md](references/scoring.md) |
|
|
163
|
+
| Report chapters | [report-template.md](references/report-template.md) |
|
|
164
|
+
| Language and banned strings | [terminology.md](references/terminology.md) |
|
|
165
|
+
|
|
166
|
+
## Sanity check
|
|
167
|
+
|
|
168
|
+
- [ ] Seeds, question, and access limits are stated; blocked sites are still fingerprinted
|
|
169
|
+
- [ ] Every fingerprint has a source and a timestamp; local resolver artefacts are excluded
|
|
170
|
+
- [ ] Every pivot is logged with source class, query, and hit count, including empty ones
|
|
171
|
+
- [ ] Noisy pivots (CDN nameservers, common mail includes) never add a domain on their own
|
|
172
|
+
- [ ] Name-scheme candidates are kept only when they share a fingerprint
|
|
173
|
+
- [ ] Entity facts cite the page they came from; conflicting operator names are dated, not merged
|
|
174
|
+
- [ ] Two signals from one vendor account are counted once
|
|
175
|
+
- [ ] Every cluster member has a label and the matrix shows why
|
|
176
|
+
- [ ] Loop ran until a full pass added nothing, or the stop reason is stated
|
|
177
|
+
- [ ] Verdict is ≤3 sentences; no "owned by" without an entity citation
|
|
178
|
+
- [ ] Outliers checked and rejected are listed with the reason
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# Entities
|
|
2
|
+
|
|
3
|
+
Step 4 names and dates the cluster. It runs after at least one brand name
|
|
4
|
+
is known and after the technical links exist; it explains the cluster, it
|
|
5
|
+
does not replace the evidence.
|
|
6
|
+
|
|
7
|
+
## What to collect per brand
|
|
8
|
+
|
|
9
|
+
| Fact | Where it usually is |
|
|
10
|
+
|---|---|
|
|
11
|
+
| Legal operator name and company number | Site footer, terms and conditions, "about" page, review aggregators |
|
|
12
|
+
| Licence authority and number | Footer seal, validator link, regulator register |
|
|
13
|
+
| Registered address | Terms page, regulator register |
|
|
14
|
+
| Launch year and relaunches | Review aggregators, archives, certificate history |
|
|
15
|
+
| Affiliate program | Footer "Affiliates" link, affiliate directories |
|
|
16
|
+
| Platform vendor | Footer, script hosts, review sites |
|
|
17
|
+
| Previous operators | Older reviews, forum threads, archive snapshots |
|
|
18
|
+
| Complaint patterns | Review aggregators, complaint forums |
|
|
19
|
+
|
|
20
|
+
Record each fact with its page and the date the page was written or
|
|
21
|
+
captured. Review aggregators disagree with each other because operators
|
|
22
|
+
migrate; keep every version with its date rather than picking one.
|
|
23
|
+
|
|
24
|
+
## Entity pivots
|
|
25
|
+
|
|
26
|
+
| Have | Reverse into | Yields |
|
|
27
|
+
|---|---|---|
|
|
28
|
+
| Operator name | Aggregator "owner" pages, licence register, web search of terms-page text | Brand list under that operator |
|
|
29
|
+
| Licence number | Regulator register | Every licensed domain, including ones with no technical link yet |
|
|
30
|
+
| Affiliate program | Affiliate directories and forums | Brands marketed together |
|
|
31
|
+
| Address | Web search | Other entities at the address; treat corporate-service addresses as noise |
|
|
32
|
+
| Platform vendor | Vendor's client list, review sites | Vendor-level neighbours; filter, do not add |
|
|
33
|
+
| Previous operator | Same as operator name | The migration path (for example Curaçao entity → transitional entity → Anjouan entity) |
|
|
34
|
+
|
|
35
|
+
## Migration patterns worth expecting
|
|
36
|
+
|
|
37
|
+
Offshore operators reorganise often. Common shapes:
|
|
38
|
+
|
|
39
|
+
- **Entity swap, same platform** — the brand keeps its template and
|
|
40
|
+
tokens; only the footer entity and licence change. Technical evidence
|
|
41
|
+
stays continuous; corporate evidence shows a break.
|
|
42
|
+
- **Brand acquisition** — an old, once-reputable domain is bought and
|
|
43
|
+
relaunched on a new operator's platform. Registration date is old;
|
|
44
|
+
everything else matches the new cluster. Weight the technical evidence.
|
|
45
|
+
- **Mirror farms** — numbered or hyphenated domains registered in batches,
|
|
46
|
+
each redirecting or serving the parent brand. They inherit every
|
|
47
|
+
fingerprint and are usually Confirmed on the first pass.
|
|
48
|
+
- **White-label tenants** — several unrelated operators on one vendor
|
|
49
|
+
platform share vendor-level fingerprints but not operator-level ones.
|
|
50
|
+
This is the main source of false clusters.
|
|
51
|
+
|
|
52
|
+
## Adding a domain from entity evidence alone
|
|
53
|
+
|
|
54
|
+
Allowed only when the source is authoritative for domain membership: a
|
|
55
|
+
regulator's licence register, or the operator's own terms page listing its
|
|
56
|
+
domains. Label such domains Probable until a technical fingerprint
|
|
57
|
+
confirms them. Aggregator sister-site lists are hypotheses to test, not
|
|
58
|
+
grounds for membership.
|