@ia-qa/self-healing 1.3.5 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +53 -2
- package/TUTORIAL.md +31 -0
- package/dist/cli/discover.js +70 -12
- package/dist/cli/discover.js.map +1 -1
- package/dist/cli/index.js +1 -0
- package/dist/cli/index.js.map +1 -1
- package/dist/crawl.d.ts +2 -0
- package/dist/crawl.js +9 -2
- package/dist/crawl.js.map +1 -1
- package/dist/discovery/safety.d.ts +35 -5
- package/dist/discovery/safety.js +37 -15
- package/dist/discovery/safety.js.map +1 -1
- package/dist/discovery/sitemap.d.ts +26 -5
- package/dist/discovery/sitemap.js +29 -8
- package/dist/discovery/sitemap.js.map +1 -1
- package/dist/index.d.ts +2 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -29,7 +29,7 @@ This one package ships **three commands**. Use just the first, or chain all thre
|
|
|
29
29
|
|
|
30
30
|
```
|
|
31
31
|
① ia-qa-heal deterministic core (humans & CI) — no AI, no key, nothing leaves your machine
|
|
32
|
-
init → map → baseline → (app changes) → map → diff → fix
|
|
32
|
+
init → (discover) → map → baseline → (app changes) → map → diff → fix
|
|
33
33
|
└ diff gives a PASS / FIX / BLOCK verdict you can gate CI on; fix applies only the safe rewrites
|
|
34
34
|
|
|
35
35
|
② ia-qa-heal-ai optional AI add-on (BYOK) — runs AFTER diff, only on what ① gave up on
|
|
@@ -125,6 +125,57 @@ ia-qa-heal map --config apps/xcp # a different portal, its own contract
|
|
|
125
125
|
> - **Behind a login and no credentials to wire?** Prefer the `IAQA_CAPTURE=1` + `ia-qa-heal run` path so the human's own suite handles auth — you never touch their passwords.
|
|
126
126
|
> - A view that shares a URL with others is reachable **by name only** (`steps`), not by URL — say so rather than mapping the wrong view.
|
|
127
127
|
|
|
128
|
+
## Discover pages to map — `ia-qa-heal discover`
|
|
129
|
+
|
|
130
|
+
`map` only ever sees the pages in `config.pages` — and so does drift detection. The question `discover` answers is the one nothing else does: **which pages of your app are you *not* covering?** It reads your whole surface (the sitemap, or a safe crawl), compares it to `config.pages`, and leads with the gap:
|
|
131
|
+
|
|
132
|
+
```
|
|
133
|
+
🗺 Coverage — 2 of 143 discovered pages in config.pages · 141 not yet covered:
|
|
134
|
+
all-tools /all-tools (sitemap)
|
|
135
|
+
resources /resources (sitemap)
|
|
136
|
+
…
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
It only ever **proposes**: it prints the uncovered pages and writes nothing unless you pass `--apply` (which appends them to `config.pages`). You confirm — exactly like the AI add-on's `suggest`.
|
|
140
|
+
|
|
141
|
+
> **Why not just capture?** A live suite (`ia-qa-heal run` with `IAQA_CAPTURE=1`) records the pages your tests *visit* — but that is precisely its blind spot: it cannot tell you about the pages your tests *don't* visit. The app overview has the same limit (it only sees links out of pages already mapped). `discover` is the **only** source that sees the whole declared surface, so it is what tells you what you are missing. Use capture to map what you touch; use `discover` to find what you don't. Within it: **`--sitemap`** reads the public surface instantly (no browser); **`--crawl`** reaches behind the login.
|
|
142
|
+
|
|
143
|
+
Two sources, one output (`{ name, url }` pairs appended to `config.pages`):
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
npx ia-qa-heal discover # default: read sitemap.xml (pure HTTP, no browser)
|
|
147
|
+
npx ia-qa-heal discover --sitemap https://…/custom.xml # an explicit sitemap
|
|
148
|
+
npx ia-qa-heal discover --crawl # follow links from inside the logged-in app
|
|
149
|
+
npx ia-qa-heal discover --crawl --apply # …and write the new pages to config.json
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
- **`--sitemap` (the default, zero-risk).** A single GET on `<baseUrl>/sitemap.xml` — falling back to the `Sitemap:` lines in `/robots.txt`, and following a sitemap *index* one level into its children. No browser, no login, nothing touched. Its blind spot is the mirror of the crawler's: a sitemap is the **public, SEO** surface and stops exactly where the authenticated app begins.
|
|
153
|
+
|
|
154
|
+
It leads with what you'll map — `Found 140 pages.` — never a bare "0 URLs". **`www.` and the apex are treated as one site**, so an apex `baseUrl` (`https://ia-qa.com`) against a `www.` sitemap (or the reverse, or a `www.`→apex 301) just works; the kept URLs are stored as **relative paths**, so `map` loads them under whatever host `baseUrl` names. The only time it mentions a skip is the one you can act on: when URLs were dropped because they sit on a **different host**, it names that host and how to reconcile it — so the old silent "0 URLs" over an apex/`www.` mismatch can't happen. Pass **`--strict-host`** to require an exact host match instead.
|
|
155
|
+
|
|
156
|
+
- **`--crawl` (higher reach, still ultra-safe).** Starts *inside* the app — reusing the very `auth` login `map` uses — and follows links outward, so it finds the authenticated pages a sitemap never lists. It is built to be side-effect-free:
|
|
157
|
+
- **same-origin GET navigations only**, and it **never visits a URL that reads like an action** — `/logout` (which would end the session for every page after it), `/delete`, `/pay`, `/order`, `/unsubscribe`… A GET that mutates is bad REST but real, so the guard is a denylist on the URL itself.
|
|
158
|
+
- to find links hidden behind a menu, dropdown, or dialog, it **opens** them — and does nothing else. Only pure **disclosure toggles** are clicked (an `aria-haspopup`, a collapsed `aria-expanded="false"`, a `<summary>`); never a link, never a submit, never a control whose name reads like an action, never one already open. If a click unexpectedly navigates, that was not a disclosure — it records where it went and steps back. After reading, it presses `Escape` to close. It never clicks the revealed items; it only harvests their `href`s for the crawl. Reveal goes **one level deep** — open it and that's all. Turn it off entirely with **`--no-reveal`** (crawl links only).
|
|
159
|
+
|
|
160
|
+
```
|
|
161
|
+
--depth <n> crawl link-depth from the start pages (default 2)
|
|
162
|
+
--max <n> cap candidates per source
|
|
163
|
+
--strict-host require an exact host match (default: www./apex are one site)
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
Both sources can run together (`discover --sitemap --crawl`) — the sitemap hits seed the crawl frontier. Pages already in `config.pages` count as covered (not re-listed); a newly discovered page whose name would collide with an existing one gets a numeric suffix (a page name is a contract's filename — a collision would overwrite it). It is idempotent: once you have added the uncovered pages, a re-run reports full coverage.
|
|
167
|
+
|
|
168
|
+
```
|
|
169
|
+
🗺 Coverage — 6 of 8 discovered pages in config.pages · 2 not yet covered:
|
|
170
|
+
|
|
171
|
+
name url (source)
|
|
172
|
+
dashboard /dashboard (crawl)
|
|
173
|
+
settings /settings (crawl)
|
|
174
|
+
Run again with --apply to add the uncovered pages to config.pages.
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Then `ia-qa-heal map` captures their contracts as usual. Discovery never maps, heals, or edits a test — it only ever edits `config.json`, and only with `--apply`.
|
|
178
|
+
|
|
128
179
|
## The one-verb loop: `ia-qa-heal run`
|
|
129
180
|
|
|
130
181
|
`run` chains the whole loop and stops where a human belongs:
|
|
@@ -207,7 +258,7 @@ An icon button or image link with no `aria-label`/text has **no role + name iden
|
|
|
207
258
|
| | For | Entry point |
|
|
208
259
|
|---|---|---|
|
|
209
260
|
| **MCP server** (stdio) | AI agents — "map my app at \<url\>" | `ia-qa-heal-mcp` → tools `map_app`, `diff_mappings`, `fix_tests`, `suggest_heal` (optional AI) |
|
|
210
|
-
| **CLI** | humans & CI | `ia-qa-heal` → `init`, `map`, `baseline`, `diff`, `fix`, `ingest`, `run` |
|
|
261
|
+
| **CLI** | humans & CI | `ia-qa-heal` → `init`, `discover`, `map`, `baseline`, `diff`, `fix`, `ingest`, `run` |
|
|
211
262
|
| **AI add-on** (optional, BYOK) | semantic renames the deterministic engine gives up on | `ia-qa-heal-ai` → `suggest` — see [Optional AI add-on](#optional-ai-add-on--semantic-suggestions-ia-qa-heal-ai-byok) |
|
|
212
263
|
|
|
213
264
|
### Locale
|
package/TUTORIAL.md
CHANGED
|
@@ -215,6 +215,36 @@ The full annotated `config.json` — every field explained, plus a note for AI a
|
|
|
215
215
|
|
|
216
216
|
> 🔐 **Your password is never written to that file** — only the *name* of the environment variable that holds it.
|
|
217
217
|
|
|
218
|
+
### Step 5.2b — (optional) Find the pages you're *not* testing: `discover`
|
|
219
|
+
|
|
220
|
+
You list a handful of pages to map — but how many pages does your app actually have? `discover` answers that: it reads your whole site (the sitemap, or a safe crawl) and shows you **what you're not covering yet**. It **only suggests** — it changes nothing until you add `--apply`.
|
|
221
|
+
|
|
222
|
+
```bash
|
|
223
|
+
npx ia-qa-heal discover # reads your site's sitemap.xml — just an HTTP read, no browser
|
|
224
|
+
npx ia-qa-heal discover --crawl # or explores your app by following links, from inside the login
|
|
225
|
+
npx ia-qa-heal discover --crawl --apply # happy with the list? add the missing pages to config.json
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
- **The sitemap way** (default) is instant and touches nothing — it treats `www.yoursite.com` and `yoursite.com` as the same site, so you don't have to match them exactly. Its limit: a sitemap lists your *public* pages and stops at the login.
|
|
229
|
+
- **The crawl way** (`--crawl`) goes further: it logs in like `map` does and follows links to find the pages behind the login. It's built to be **safe** — it only ever *reads* pages (never clicks "Delete", "Log out", "Pay"…), and to peek inside a menu or dropdown it just **opens** it and moves on. Nothing is submitted, nothing is changed.
|
|
230
|
+
|
|
231
|
+
You'll see your coverage gap:
|
|
232
|
+
|
|
233
|
+
```
|
|
234
|
+
🗺 Coverage — 2 of 27 discovered pages in config.pages · 25 not yet covered:
|
|
235
|
+
|
|
236
|
+
name url
|
|
237
|
+
pricing /pricing
|
|
238
|
+
dashboard /dashboard
|
|
239
|
+
settings /settings
|
|
240
|
+
…
|
|
241
|
+
Run again with --apply to add the uncovered pages to config.pages.
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
That "25 not yet covered" is the point: pages that exist but nothing is testing. Look them over, drop the ones you don't care about (a privacy page, 100 near-identical blog posts), and `--apply` the rest — then `map` covers them too.
|
|
245
|
+
|
|
246
|
+
> **Why not just let my test suite find them?** A live suite (`ia-qa-heal run`, Step 6.7) only ever sees the pages your tests *visit* — which is exactly why it can't tell you about the ones they *miss*. `discover` sees the whole site, so it's the one thing that shows you the blind spots. Use your suite to cover what you touch; use `discover` to find what you don't.
|
|
247
|
+
|
|
218
248
|
### Step 5.3 — Map
|
|
219
249
|
|
|
220
250
|
```bash
|
|
@@ -539,6 +569,7 @@ Yes, any URL your machine can reach.
|
|
|
539
569
|
|
|
540
570
|
```bash
|
|
541
571
|
npx ia-qa-heal init # one-time setup
|
|
572
|
+
npx ia-qa-heal discover [--crawl] [--apply] # optional: propose pages for config.json (sitemap or safe crawl)
|
|
542
573
|
npx ia-qa-heal map # scan → *.json + *.md + _overview.md + _navigation.md/.svg
|
|
543
574
|
npx ia-qa-heal baseline # promote mapping/ → baseline/ (commit it)
|
|
544
575
|
# …change the app, then `map` again…
|
package/dist/cli/discover.js
CHANGED
|
@@ -30,11 +30,22 @@ async function runDiscover(args) {
|
|
|
30
30
|
const candidates = [];
|
|
31
31
|
if (doSitemap) {
|
|
32
32
|
console.log('🔎 Reading sitemap…');
|
|
33
|
-
const found = await (0, sitemap_1.discoverFromSitemap)(config.baseUrl, {
|
|
33
|
+
const { candidates: found, stats } = await (0, sitemap_1.discoverFromSitemap)(config.baseUrl, {
|
|
34
34
|
sitemapUrl: opts.sitemapUrl,
|
|
35
35
|
maxUrls: opts.max,
|
|
36
|
+
strictHost: opts.strictHost,
|
|
36
37
|
});
|
|
37
|
-
|
|
38
|
+
// Say nothing on the happy path — the coverage summary below carries the count.
|
|
39
|
+
// Only speak up for the two things a user must act on: a host mismatch (the
|
|
40
|
+
// reported bug — never a silent "0"), or an empty result and why.
|
|
41
|
+
if (stats.skippedHost > 0) {
|
|
42
|
+
warnHostSkip(stats.skippedHost, stats.otherHosts, config.baseUrl, opts.strictHost);
|
|
43
|
+
}
|
|
44
|
+
else if (found.length === 0) {
|
|
45
|
+
console.log(stats.parsed === 0
|
|
46
|
+
? ' No sitemap found (tried /sitemap.xml and robots.txt).'
|
|
47
|
+
: ` The sitemap listed ${stats.parsed} URL${stats.parsed === 1 ? '' : 's'}, but none map to a page here (downloads, actions or duplicates).`);
|
|
48
|
+
}
|
|
38
49
|
candidates.push(...found);
|
|
39
50
|
}
|
|
40
51
|
if (doCrawl) {
|
|
@@ -43,6 +54,7 @@ async function runDiscover(args) {
|
|
|
43
54
|
maxDepth: opts.depth,
|
|
44
55
|
maxPages: opts.max,
|
|
45
56
|
reveal: opts.reveal,
|
|
57
|
+
strictHost: opts.strictHost,
|
|
46
58
|
// Sitemap hits seed the crawl frontier when both run together.
|
|
47
59
|
seeds: candidates.filter((c) => c.source === 'sitemap').map((c) => c.url),
|
|
48
60
|
onVisit: (url, n) => console.log(` visited ${url} (${n} link${n === 1 ? '' : 's'})`),
|
|
@@ -51,33 +63,76 @@ async function runDiscover(args) {
|
|
|
51
63
|
candidates.push(...found);
|
|
52
64
|
}
|
|
53
65
|
const added = (0, safety_1.mergePages)(config.pages, candidates, config.baseUrl);
|
|
54
|
-
|
|
66
|
+
// The whole surface discovery could see, deduped across both sources by path.
|
|
67
|
+
const discovered = new Set(candidates.map((c) => (0, safety_1.normalizedPath)(c.url, config.baseUrl))).size;
|
|
68
|
+
reportCoverage(discovered, added);
|
|
55
69
|
if (added.length === 0)
|
|
56
70
|
return;
|
|
57
71
|
if (opts.apply) {
|
|
58
72
|
const pages = [...config.pages, ...added.map((c) => ({ name: c.name, url: c.url }))];
|
|
59
73
|
const file = (0, config_1.saveConfig)({ ...config, pages });
|
|
60
|
-
console.log(`\n✅ Added ${added.length} page${added.length === 1 ? '' : 's'} to ${file}`);
|
|
74
|
+
console.log(`\n✅ Added ${added.length} page${added.length === 1 ? '' : 's'} to config.pages → ${file}`);
|
|
61
75
|
console.log(' Next: `ia-qa-heal map` to capture their contracts.');
|
|
62
76
|
}
|
|
63
77
|
else {
|
|
64
|
-
console.log('\nRun again with --apply to add
|
|
78
|
+
console.log('\nRun again with --apply to add the uncovered pages to config.pages.');
|
|
65
79
|
}
|
|
66
80
|
}
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
81
|
+
/**
|
|
82
|
+
* The point of the whole verb: not "here are some URLs", but "here is how much of
|
|
83
|
+
* your app you are NOT covering". `map` only ever sees `config.pages`; a live suite
|
|
84
|
+
* (capture) only sees what it visits; the app overview only sees links out of pages
|
|
85
|
+
* already mapped. Every one of those has the same blind spot — pages you don't
|
|
86
|
+
* already touch. Discovery is the only source that sees the whole declared surface,
|
|
87
|
+
* so the summary leads with the gap, and the list is the actionable part: the pages
|
|
88
|
+
* to add before `map` can cover them.
|
|
89
|
+
*/
|
|
90
|
+
function reportCoverage(discovered, uncovered) {
|
|
91
|
+
if (discovered === 0)
|
|
92
|
+
return; // nothing found — the source block already said why
|
|
93
|
+
const mapped = discovered - uncovered.length;
|
|
94
|
+
if (uncovered.length === 0) {
|
|
95
|
+
console.log(`\n🗺 Coverage — all ${discovered} discovered page${discovered === 1 ? '' : 's'} already in config.pages ✅`);
|
|
71
96
|
return;
|
|
72
97
|
}
|
|
73
|
-
|
|
98
|
+
console.log(`\n🗺 Coverage — ${mapped} of ${discovered} discovered pages in config.pages · ` +
|
|
99
|
+
`${uncovered.length} not yet covered:`);
|
|
100
|
+
const nameW = Math.max(4, ...uncovered.map((c) => c.name.length));
|
|
74
101
|
console.log(`\n ${'name'.padEnd(nameW)} url (source)`);
|
|
75
|
-
for (const c of
|
|
102
|
+
for (const c of uncovered) {
|
|
76
103
|
console.log(` ${c.name.padEnd(nameW)} ${c.url} (${c.source})`);
|
|
77
104
|
}
|
|
78
105
|
}
|
|
106
|
+
/**
|
|
107
|
+
* Name the host mismatch and the way out. This is the whole point of the fix: the
|
|
108
|
+
* old code dropped every off-host URL and printed "0 URLs", so an apex `baseUrl`
|
|
109
|
+
* against a `www.` sitemap looked like an empty sitemap. Here we say which host the
|
|
110
|
+
* URLs were on, which host `baseUrl` is, and — depending on whether normalization is
|
|
111
|
+
* already on — how to reconcile them.
|
|
112
|
+
*/
|
|
113
|
+
function warnHostSkip(count, otherHosts, baseUrl, strictHost) {
|
|
114
|
+
let baseHost = baseUrl;
|
|
115
|
+
try {
|
|
116
|
+
baseHost = new URL(baseUrl).host;
|
|
117
|
+
}
|
|
118
|
+
catch {
|
|
119
|
+
/* keep the raw string */
|
|
120
|
+
}
|
|
121
|
+
const on = otherHosts.length > 0 ? `on ${otherHosts.map((h) => `"${h}"`).join(', ')}` : 'on a different host';
|
|
122
|
+
console.warn(` ⚠ ${count} sitemap URL${count === 1 ? '' : 's'} ${on}, not baseUrl's host "${baseHost}" — skipped.`);
|
|
123
|
+
if (strictHost) {
|
|
124
|
+
// The classic apex/www case, now behind the opt-out the user chose.
|
|
125
|
+
console.warn(` If it is the same site, drop --strict-host (www./apex are then one site), ` +
|
|
126
|
+
`or set baseUrl to that host.`);
|
|
127
|
+
}
|
|
128
|
+
else {
|
|
129
|
+
// Normalization is already on, so this is a genuinely different host.
|
|
130
|
+
console.warn(` That is a different site, not a www./apex variant. If you meant to include it, ` +
|
|
131
|
+
`set baseUrl to match it (discovery stores relative paths, so pick the host map should load).`);
|
|
132
|
+
}
|
|
133
|
+
}
|
|
79
134
|
function parseArgs(args) {
|
|
80
|
-
const opts = { sitemap: false, crawl: false, reveal: true, apply: false };
|
|
135
|
+
const opts = { sitemap: false, crawl: false, reveal: true, apply: false, strictHost: false };
|
|
81
136
|
for (let i = 0; i < args.length; i++) {
|
|
82
137
|
const a = args[i];
|
|
83
138
|
switch (a) {
|
|
@@ -96,6 +151,9 @@ function parseArgs(args) {
|
|
|
96
151
|
case '--no-reveal':
|
|
97
152
|
opts.reveal = false;
|
|
98
153
|
break;
|
|
154
|
+
case '--strict-host':
|
|
155
|
+
opts.strictHost = true;
|
|
156
|
+
break;
|
|
99
157
|
case '--apply':
|
|
100
158
|
opts.apply = true;
|
|
101
159
|
break;
|
package/dist/cli/discover.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"discover.js","sourceRoot":"","sources":["../../src/cli/discover.ts"],"names":[],"mappings":";;AAqBA,
|
|
1
|
+
{"version":3,"file":"discover.js","sourceRoot":"","sources":["../../src/cli/discover.ts"],"names":[],"mappings":";;AAqBA,kCA8DC;AAnFD,sCAA+D;AAC/D,gDAA4E;AAC5E,kDAA2D;AAC3D,oCAAoC;AAEpC;;;;;;;;;;;;;;;GAeG;AACI,KAAK,UAAU,WAAW,CAAC,IAAc;IAC9C,MAAM,IAAI,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;IAC7B,MAAM,MAAM,GAAG,IAAA,mBAAU,GAAE,CAAC;IAE5B,oEAAoE;IACpE,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,IAAI,CAAC,CAAC,IAAI,CAAC,OAAO,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IACjE,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC;IAE3B,MAAM,UAAU,GAAgB,EAAE,CAAC;IAEnC,IAAI,SAAS,EAAE,CAAC;QACd,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,CAAC;QACnC,MAAM,EAAE,UAAU,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,MAAM,IAAA,6BAAmB,EAAC,MAAM,CAAC,OAAO,EAAE;YAC7E,UAAU,EAAE,IAAI,CAAC,UAAU;YAC3B,OAAO,EAAE,IAAI,CAAC,GAAG;YACjB,UAAU,EAAE,IAAI,CAAC,UAAU;SAC5B,CAAC,CAAC;QACH,gFAAgF;QAChF,4EAA4E;QAC5E,kEAAkE;QAClE,IAAI,KAAK,CAAC,WAAW,GAAG,CAAC,EAAE,CAAC;YAC1B,YAAY,CAAC,KAAK,CAAC,WAAW,EAAE,KAAK,CAAC,UAAU,EAAE,MAAM,CAAC,OAAO,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;QACrF,CAAC;aAAM,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YAC9B,OAAO,CAAC,GAAG,CACT,KAAK,CAAC,MAAM,KAAK,CAAC;gBAChB,CAAC,CAAC,0DAA0D;gBAC5D,CAAC,CAAC,yBAAyB,KAAK,CAAC,MAAM,OAAO,KAAK,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,mEAAmE,CACjJ,CAAC;QACJ,CAAC;QACD,UAAU,CAAC,IAAI,CAAC,GAAG,KAAK,CAAC,CAAC;IAC5B,CAAC;IAED,IAAI,OAAO,EAAE,CAAC;QACZ,OAAO,CAAC,GAAG,CAAC,qBAAqB,MAAM,CAAC,OAAO,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,yCAAyC,CAAC,CAAC,CAAC,eAAe,GAAG,CAAC,CAAC;QAChI,MAAM,KAAK,GAAG,MAAM,IAAA,gBAAQ,EAAC,MAAM,EAAE;YACnC,QAAQ,EAAE,IAAI,CAAC,KAAK;YACpB,QAAQ,EAAE,IAAI,CAAC,GAAG;YAClB,MAAM,EAAE,IAAI,CAAC,MAAM;YACnB,UAAU,EAAE,IAAI,CAAC,UAAU;YAC3B,+DAA+D;YAC/D,KAAK,EAAE,UAAU,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC;YACzE,OAAO,EAAE,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC,OAAO,CAAC,GAAG,CAAC,cAAc,GAAG,KAAK,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC;SACvF,CAAC,CAAC;QACH,OAAO,CAAC,GAAG,CAAC,MAAM,KAAK,CAAC,MAAM,QAAQ,KAAK,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,yBAAyB,CAAC,CAAC;QAC9F,UAAU,CAAC,IAAI,CAAC,GAAG,KAAK,CAAC,CAAC;IAC5B,CAAC;IAED,MAAM,KAAK,GAAG,IAAA,mBAAU,EAAC,MAAM,CAAC,KAAK,EAAE,UAAU,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC;IACnE,8EAA8E;IAC9E,MAAM,UAAU,GAAG,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAA,uBAAc,EAAC,CAAC,CAAC,GAAG,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IAE9F,cAAc,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;IAClC,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO;IAE/B,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;QACf,MAAM,KAAK,GAAiB,CAAC,GAAG,MAAM,CAAC,KAAK,EAAE,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,GAAG,EAAE,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,CAAC;QACnG,MAAM,IAAI,GAAG,IAAA,mBAAU,EAAC,EAAE,GAAG,MAAM,EAAE,KAAK,EAAE,CAAC,CAAC;QAC9C,OAAO,CAAC,GAAG,CAAC,aAAa,KAAK,CAAC,MAAM,QAAQ,KAAK,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,sBAAsB,IAAI,EAAE,CAAC,CAAC;QACxG,OAAO,CAAC,GAAG,CAAC,uDAAuD,CAAC,CAAC;IACvE,CAAC;SAAM,CAAC;QACN,OAAO,CAAC,GAAG,CAAC,sEAAsE,CAAC,CAAC;IACtF,CAAC;AACH,CAAC;AAED;;;;;;;;GAQG;AACH,SAAS,cAAc,CAAC,UAAkB,EAAE,SAAsB;IAChE,IAAI,UAAU,KAAK,CAAC;QAAE,OAAO,CAAC,oDAAoD;IAClF,MAAM,MAAM,GAAG,UAAU,GAAG,SAAS,CAAC,MAAM,CAAC;IAE7C,IAAI,SAAS,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC3B,OAAO,CAAC,GAAG,CACT,wBAAwB,UAAU,mBAAmB,UAAU,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,4BAA4B,CAC7G,CAAC;QACF,OAAO;IACT,CAAC;IAED,OAAO,CAAC,GAAG,CACT,oBAAoB,MAAM,OAAO,UAAU,sCAAsC;QAC/E,GAAG,SAAS,CAAC,MAAM,mBAAmB,CACzC,CAAC;IACF,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,SAAS,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC;IAClE,OAAO,CAAC,GAAG,CAAC,QAAQ,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,iBAAiB,CAAC,CAAC;IAC3D,KAAK,MAAM,CAAC,IAAI,SAAS,EAAE,CAAC;QAC1B,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC;IACrE,CAAC;AACH,CAAC;AAED;;;;;;GAMG;AACH,SAAS,YAAY,CAAC,KAAa,EAAE,UAAoB,EAAE,OAAe,EAAE,UAAoB;IAC9F,IAAI,QAAQ,GAAG,OAAO,CAAC;IACvB,IAAI,CAAC;QACH,QAAQ,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC;IACnC,CAAC;IAAC,MAAM,CAAC;QACP,yBAAyB;IAC3B,CAAC;IACD,MAAM,EAAE,GAAG,UAAU,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,MAAM,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,qBAAqB,CAAC;IAC9G,OAAO,CAAC,IAAI,CACV,QAAQ,KAAK,eAAe,KAAK,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,IAAI,EAAE,yBAAyB,QAAQ,cAAc,CACxG,CAAC;IACF,IAAI,UAAU,EAAE,CAAC;QACf,oEAAoE;QACpE,OAAO,CAAC,IAAI,CACV,iFAAiF;YAC/E,8BAA8B,CACjC,CAAC;IACJ,CAAC;SAAM,CAAC;QACN,sEAAsE;QACtE,OAAO,CAAC,IAAI,CACV,sFAAsF;YACpF,8FAA8F,CACjG,CAAC;IACJ,CAAC;AACH,CAAC;AAaD,SAAS,SAAS,CAAC,IAAc;IAC/B,MAAM,IAAI,GAAiB,EAAE,OAAO,EAAE,KAAK,EAAE,KAAK,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,KAAK,EAAE,UAAU,EAAE,KAAK,EAAE,CAAC;IAC3G,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,QAAQ,CAAC,EAAE,CAAC;YACV,KAAK,WAAW,CAAC,CAAC,CAAC;gBACjB,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC;gBACpB,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;gBACzB,IAAI,IAAI,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;oBAClC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC;oBACvB,CAAC,EAAE,CAAC;gBACN,CAAC;gBACD,MAAM;YACR,CAAC;YACD,KAAK,SAAS;gBACZ,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC;gBAClB,MAAM;YACR,KAAK,aAAa;gBAChB,IAAI,CAAC,MAAM,GAAG,KAAK,CAAC;gBACpB,MAAM;YACR,KAAK,eAAe;gBAClB,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC;gBACvB,MAAM;YACR,KAAK,SAAS;gBACZ,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC;gBAClB,MAAM;YACR,KAAK,OAAO;gBACV,IAAI,CAAC,GAAG,GAAG,UAAU,CAAC,CAAC,EAAE,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;gBACpC,MAAM;YACR,KAAK,SAAS;gBACZ,IAAI,CAAC,KAAK,GAAG,UAAU,CAAC,CAAC,EAAE,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;gBACtC,MAAM;YACR;gBACE,MAAM,IAAI,KAAK,CAAC,mBAAmB,CAAC,8CAA8C,CAAC,CAAC;QACxF,CAAC;IACH,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC;AAED,SAAS,UAAU,CAAC,IAAY,EAAE,KAAyB;IACzD,MAAM,CAAC,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IACxB,IAAI,CAAC,KAAK,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QAC7C,MAAM,IAAI,KAAK,CAAC,GAAG,IAAI,mCAAmC,KAAK,IAAI,EAAE,IAAI,CAAC,CAAC;IAC7E,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC"}
|
package/dist/cli/index.js
CHANGED
|
@@ -35,6 +35,7 @@ Usage:
|
|
|
35
35
|
--no-reveal crawl links only; do not open any menu/dropdown
|
|
36
36
|
--depth <n> crawl link-depth from the start pages (default 2)
|
|
37
37
|
--max <n> cap candidates per source
|
|
38
|
+
--strict-host treat www. and apex as different hosts (default: one site)
|
|
38
39
|
--apply append the new pages to config.json (else print only)
|
|
39
40
|
ia-qa-heal map [page] Map configured pages → .ia-qa/mapping/<page>.json + .md
|
|
40
41
|
Also rebuilds _overview.md (the app index) and
|
package/dist/cli/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/cli/index.ts"],"names":[],"mappings":";;;AACA,sCAAuC;AACvC,iCAAiC;AACjC,+BAA+B;AAC/B,yCAAyC;AACzC,iCAAiC;AACjC,+BAA+B;AAC/B,qCAAqC;AACrC,+BAA+B;AAC/B,yCAAyC;AAEzC,qEAAqE;AACrE,MAAM,OAAO,GAAG,OAAO,CAAC,oBAAoB,CAAC,CAAC,OAAiB,CAAC;AAEhE,MAAM,IAAI,GAAG
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/cli/index.ts"],"names":[],"mappings":";;;AACA,sCAAuC;AACvC,iCAAiC;AACjC,+BAA+B;AAC/B,yCAAyC;AACzC,iCAAiC;AACjC,+BAA+B;AAC/B,qCAAqC;AACrC,+BAA+B;AAC/B,yCAAyC;AAEzC,qEAAqE;AACrE,MAAM,OAAO,GAAG,OAAO,CAAC,oBAAoB,CAAC,CAAC,OAAiB,CAAC;AAEhE,MAAM,IAAI,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAwEZ,CAAC;AAEF;;;;;;GAMG;AACH,SAAS,gBAAgB,CAAC,IAAc;IACtC,MAAM,IAAI,GAAa,EAAE,CAAC;IAC1B,IAAI,GAAuB,CAAC;IAC5B,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QAClB,IAAI,CAAC,KAAK,UAAU,EAAE,CAAC;YACrB,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YACzB,IAAI,IAAI,KAAK,SAAS,IAAI,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;gBAC/C,MAAM,IAAI,KAAK,CAAC,8DAA8D,CAAC,CAAC;YAClF,CAAC;YACD,GAAG,GAAG,IAAI,CAAC;YACX,CAAC,EAAE,CAAC;YACJ,SAAS;QACX,CAAC;QACD,IAAI,CAAC,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;YAC9B,GAAG,GAAG,CAAC,CAAC,KAAK,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC;YAClC,SAAS;QACX,CAAC;QACD,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACf,CAAC;IACD,OAAO,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC;AACvB,CAAC;AAED,KAAK,UAAU,IAAI;IACjB,MAAM,EAAE,GAAG,EAAE,SAAS,EAAE,IAAI,EAAE,IAAI,EAAE,GAAG,gBAAgB,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IAC/E,IAAI,SAAS,KAAK,SAAS,EAAE,CAAC;QAC5B,IAAI,CAAC,SAAS;YAAE,MAAM,IAAI,KAAK,CAAC,sCAAsC,CAAC,CAAC;QACxE,IAAA,mBAAU,EAAC,SAAS,CAAC,CAAC;IACxB,CAAC;IACD,MAAM,CAAC,OAAO,EAAE,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC;IAChC,MAAM,GAAG,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;IACpB,QAAQ,OAAO,EAAE,CAAC;QAChB,KAAK,MAAM;YACT,MAAM,IAAA,cAAO,GAAE,CAAC;YAChB,MAAM;QACR,KAAK,UAAU;YACb,MAAM,IAAA,sBAAW,EAAC,IAAI,CAAC,CAAC;YACxB,MAAM;QACR,KAAK,KAAK;YACR,MAAM,IAAA,YAAM,EAAC,GAAG,CAAC,CAAC;YAClB,MAAM;QACR,KAAK,UAAU;YACb,MAAM,IAAA,sBAAW,EAAC,IAAI,CAAC,CAAC;YACxB,MAAM;QACR,KAAK,MAAM;YACT,MAAM,IAAA,cAAO,EAAC,IAAI,CAAC,CAAC;YACpB,MAAM;QACR,KAAK,KAAK;YACR,MAAM,IAAA,YAAM,EAAC,IAAI,CAAC,CAAC;YACnB,MAAM;QACR,KAAK,QAAQ;YACX,MAAM,IAAA,kBAAS,EAAC,IAAI,CAAC,CAAC;YACtB,MAAM;QACR,KAAK,KAAK;YACR,MAAM,IAAA,YAAM,EAAC,IAAI,CAAC,CAAC;YACnB,MAAM;QACR,KAAK,SAAS,CAAC;QACf,KAAK,WAAW,CAAC;QACjB,KAAK,IAAI;YACP,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;YACrB,MAAM;QACR,KAAK,MAAM,CAAC;QACZ,KAAK,QAAQ,CAAC;QACd,KAAK,IAAI,CAAC;QACV,KAAK,SAAS;YACZ,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;YAClB,MAAM;QACR;YACE,OAAO,CAAC,KAAK,CAAC,oBAAoB,OAAO,SAAS,IAAI,EAAE,CAAC,CAAC;YAC1D,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACpB,CAAC;AACH,CAAC;AAED,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,GAAY,EAAE,EAAE;IAC5B,OAAO,CAAC,KAAK,CAAC,OAAO,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;IACzE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC,CAAC,CAAC"}
|
package/dist/crawl.d.ts
CHANGED
|
@@ -31,6 +31,8 @@ export interface CrawlOptions {
|
|
|
31
31
|
reveal?: boolean;
|
|
32
32
|
/** Extra start URLs beyond baseUrl + configured pages (e.g. sitemap hits). */
|
|
33
33
|
seeds?: string[];
|
|
34
|
+
/** Treat www./apex as distinct hosts. Off by default (they are one site). */
|
|
35
|
+
strictHost?: boolean;
|
|
34
36
|
/** Called once per visited URL, for progress output. */
|
|
35
37
|
onVisit?: (url: string, found: number) => void;
|
|
36
38
|
}
|
package/dist/crawl.js
CHANGED
|
@@ -17,18 +17,19 @@ async function crawlApp(config, options = {}) {
|
|
|
17
17
|
const maxDepth = options.maxDepth ?? 2;
|
|
18
18
|
const maxPages = options.maxPages ?? 60;
|
|
19
19
|
const reveal = options.reveal !== false;
|
|
20
|
+
const hostOpts = { strictHost: options.strictHost };
|
|
20
21
|
const base = new URL(config.baseUrl);
|
|
21
22
|
const start = new URL(config.baseUrl).toString();
|
|
22
23
|
const found = new Map();
|
|
23
24
|
const visited = new Set();
|
|
24
25
|
const queue = [{ url: start, depth: 0 }];
|
|
25
26
|
const enqueue = (href, depth) => {
|
|
26
|
-
if (!(0, safety_1.isSafeCandidateUrl)(href, config.baseUrl))
|
|
27
|
+
if (!(0, safety_1.isSafeCandidateUrl)(href, config.baseUrl, hostOpts))
|
|
27
28
|
return;
|
|
28
29
|
const abs = new URL(href, config.baseUrl).toString();
|
|
29
30
|
const key = (0, safety_1.normalizedPath)(abs, config.baseUrl);
|
|
30
31
|
if (!found.has(key)) {
|
|
31
|
-
const candidate = (0, safety_1.toCandidate)(abs, config.baseUrl, 'crawl');
|
|
32
|
+
const candidate = (0, safety_1.toCandidate)(abs, config.baseUrl, 'crawl', hostOpts);
|
|
32
33
|
if (candidate)
|
|
33
34
|
found.set(key, candidate);
|
|
34
35
|
}
|
|
@@ -55,6 +56,12 @@ async function crawlApp(config, options = {}) {
|
|
|
55
56
|
if (visited.has(key))
|
|
56
57
|
continue;
|
|
57
58
|
visited.add(key);
|
|
59
|
+
// A page we actually loaded is a real, reachable page — a candidate in its own
|
|
60
|
+
// right, even when nothing links to it. Without this the start page (and any
|
|
61
|
+
// JS-only entry) is dropped unless some other page happens to link back to it.
|
|
62
|
+
const self = (0, safety_1.toCandidate)(url, config.baseUrl, 'crawl', hostOpts);
|
|
63
|
+
if (self && !found.has(key))
|
|
64
|
+
found.set(key, self);
|
|
58
65
|
let hrefs;
|
|
59
66
|
try {
|
|
60
67
|
hrefs = await visit(page, url, reveal && depth < maxDepth);
|
package/dist/crawl.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"crawl.js","sourceRoot":"","sources":["../src/crawl.ts"],"names":[],"mappings":";;
|
|
1
|
+
{"version":3,"file":"crawl.js","sourceRoot":"","sources":["../src/crawl.ts"],"names":[],"mappings":";;AAsDA,4BAqEC;AA1HD,yCAA2C;AAC3C,mCAAkC;AAClC,+BAAmD;AACnD,iDAAuD;AAEvD,+CAAgG;AAwChG,uEAAuE;AACvE,MAAM,gBAAgB,GAAG,GAAG,CAAC;AAE7B;;;;GAIG;AACI,KAAK,UAAU,QAAQ,CAAC,MAAkB,EAAE,UAAwB,EAAE;IAC3E,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,CAAC,CAAC;IACvC,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,EAAE,CAAC;IACxC,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,KAAK,KAAK,CAAC;IACxC,MAAM,QAAQ,GAAG,EAAE,UAAU,EAAE,OAAO,CAAC,UAAU,EAAE,CAAC;IAEpD,MAAM,IAAI,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;IACrC,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC;IAEjD,MAAM,KAAK,GAAG,IAAI,GAAG,EAAqB,CAAC;IAC3C,MAAM,OAAO,GAAG,IAAI,GAAG,EAAU,CAAC;IAClC,MAAM,KAAK,GAA0C,CAAC,EAAE,GAAG,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC,EAAE,CAAC,CAAC;IAEhF,MAAM,OAAO,GAAG,CAAC,IAAY,EAAE,KAAa,EAAE,EAAE;QAC9C,IAAI,CAAC,IAAA,2BAAkB,EAAC,IAAI,EAAE,MAAM,CAAC,OAAO,EAAE,QAAQ,CAAC;YAAE,OAAO;QAChE,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,IAAI,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC;QACrD,MAAM,GAAG,GAAG,IAAA,uBAAc,EAAC,GAAG,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC;QAChD,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;YACpB,MAAM,SAAS,GAAG,IAAA,oBAAW,EAAC,GAAG,EAAE,MAAM,CAAC,OAAO,EAAE,OAAO,EAAE,QAAQ,CAAC,CAAC;YACtE,IAAI,SAAS;gBAAE,KAAK,CAAC,GAAG,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;QAC3C,CAAC;QACD,IAAI,KAAK,IAAI,QAAQ,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;YAC3C,KAAK,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,GAAG,IAAI,CAAC,MAAM,GAAG,GAAG,EAAE,EAAE,KAAK,EAAE,CAAC,CAAC;QACrD,CAAC;IACH,CAAC,CAAC;IAEF,KAAK,MAAM,IAAI,IAAI,OAAO,CAAC,KAAK,IAAI,EAAE;QAAE,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IAEzD,MAAM,EAAE,OAAO,EAAE,KAAK,EAAE,GAAG,MAAM,IAAA,wBAAa,GAAE,CAAC;IACjD,OAAO,CAAC,GAAG,CAAC,YAAY,KAAK,EAAE,CAAC,CAAC;IACjC,IAAI,CAAC;QACH,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM;YAC3B,CAAC,CAAC,MAAM,OAAO,CAAC,UAAU,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,CAAC;YACrD,CAAC,CAAC,MAAM,OAAO,CAAC,UAAU,EAAE,CAAC;QAC/B,MAAM,IAAI,GAAG,MAAM,OAAO,CAAC,OAAO,EAAE,CAAC;QAErC,IAAI,MAAM,CAAC,IAAI,EAAE,CAAC;YAChB,OAAO,CAAC,GAAG,CAAC,gBAAgB,CAAC,CAAC;YAC9B,MAAM,IAAA,WAAK,EAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QAC5B,CAAC;QAED,OAAO,KAAK,CAAC,MAAM,GAAG,CAAC,IAAI,OAAO,CAAC,IAAI,GAAG,QAAQ,EAAE,CAAC;YACnD,MAAM,EAAE,GAAG,EAAE,KAAK,EAAE,GAAG,KAAK,CAAC,KAAK,EAAG,CAAC;YACtC,MAAM,GAAG,GAAG,IAAA,uBAAc,EAAC,GAAG,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC;YAChD,IAAI,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC;gBAAE,SAAS;YAC/B,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;YAEjB,+EAA+E;YAC/E,6EAA6E;YAC7E,+EAA+E;YAC/E,MAAM,IAAI,GAAG,IAAA,oBAAW,EAAC,GAAG,EAAE,MAAM,CAAC,OAAO,EAAE,OAAO,EAAE,QAAQ,CAAC,CAAC;YACjE,IAAI,IAAI,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC;gBAAE,KAAK,CAAC,GAAG,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;YAElD,IAAI,KAAe,CAAC;YACpB,IAAI,CAAC;gBACH,KAAK,GAAG,MAAM,KAAK,CAAC,IAAI,EAAE,GAAG,EAAE,MAAM,IAAI,KAAK,GAAG,QAAQ,CAAC,CAAC;YAC7D,CAAC;YAAC,MAAM,CAAC;gBACP,2EAA2E;gBAC3E,qEAAqE;gBACrE,SAAS;YACX,CAAC;YACD,OAAO,CAAC,OAAO,EAAE,CAAC,GAAG,EAAE,KAAK,CAAC,MAAM,CAAC,CAAC;YACrC,KAAK,MAAM,IAAI,IAAI,KAAK;gBAAE,OAAO,CAAC,IAAI,EAAE,KAAK,GAAG,CAAC,CAAC,CAAC;QACrD,CAAC;IACH,CAAC;YAAS,CAAC;QACT,MAAM,OAAO,CAAC,KAAK,EAAE,CAAC;IACxB,CAAC;IAED,OAAO,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC;AACpC,CAAC;AAED;;;;;;GAMG;AACH,KAAK,UAAU,KAAK,CAAC,IAAU,EAAE,GAAW,EAAE,MAAe;IAC3D,MAAM,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,aAAa,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC,CAAC;IACnE,MAAM,KAAK,GAAG,IAAI,GAAG,EAAU,CAAC;IAChC,KAAK,MAAM,EAAE,IAAI,MAAM,IAAA,gCAA0B,EAAC,IAAI,CAAC,EAAE,CAAC;QACxD,IAAI,EAAE,CAAC,IAAI;YAAE,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,CAAC,CAAC;IAClC,CAAC;IACD,IAAI,CAAC,MAAM;QAAE,OAAO,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAEtC,gFAAgF;IAChF,6DAA6D;IAC7D,MAAM,OAAO,GAAa,MAAM,IAAI,CAAC,QAAQ,CAAC,4BAAiB,CAAC,CAAC;IACjE,KAAK,MAAM,QAAQ,IAAI,OAAO,EAAE,CAAC;QAC/B,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QAC7B,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,KAAK,EAAE,CAAC,KAAK,CAAC,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC;QAChE,CAAC;QAAC,MAAM,CAAC;YACP,SAAS,CAAC,4DAA4D;QACxE,CAAC;QAED,+EAA+E;QAC/E,8EAA8E;QAC9E,+BAA+B;QAC/B,IAAI,IAAI,CAAC,GAAG,EAAE,KAAK,SAAS,EAAE,CAAC;YAC7B,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;YACtB,MAAM,IAAI,CAAC,MAAM,CAAC,EAAE,SAAS,EAAE,kBAAkB,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;YACrF,SAAS;QACX,CAAC;QAED,MAAM,IAAI,CAAC,cAAc,CAAC,gBAAgB,CAAC,CAAC;QAC5C,KAAK,MAAM,EAAE,IAAI,MAAM,IAAA,gCAA0B,EAAC,IAAI,CAAC,EAAE,CAAC;YACxD,IAAI,EAAE,CAAC,IAAI;gBAAE,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,CAAC,CAAC;QAClC,CAAC;QACD,iFAAiF;QACjF,gFAAgF;QAChF,8CAA8C;QAC9C,MAAM,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;IACtD,CAAC;IAED,OAAO,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;AAC3B,CAAC"}
|
|
@@ -26,14 +26,44 @@ import type { PageTarget } from '../config';
|
|
|
26
26
|
* "ultra-safe" tool that trade is not close.
|
|
27
27
|
*/
|
|
28
28
|
export declare const UNSAFE_URL_RE: RegExp;
|
|
29
|
+
export interface HostOptions {
|
|
30
|
+
/**
|
|
31
|
+
* Require the host to match `baseUrl`'s exactly. Off by default: `www.` is
|
|
32
|
+
* stripped from both sides first, so an apex `baseUrl` and a `www.` sitemap (or a
|
|
33
|
+
* `www.`→apex 301) are treated as one site. Turn this on to keep them distinct.
|
|
34
|
+
*/
|
|
35
|
+
strictHost?: boolean;
|
|
36
|
+
}
|
|
37
|
+
/**
|
|
38
|
+
* The "same site" key of a URL: protocol + host, with a leading `www.` folded away
|
|
39
|
+
* unless `strictHost`. This is the whole apex/`www.` fix — comparing raw `origin`
|
|
40
|
+
* drops every `www.` sitemap URL when `baseUrl` is the apex (or vice versa), which
|
|
41
|
+
* is the single most common way a real `baseUrl` and a real sitemap differ. Protocol
|
|
42
|
+
* and port stay significant: `http` vs `https` is a real difference, not a `www.`.
|
|
43
|
+
*/
|
|
44
|
+
export declare function siteKey(u: URL, strictHost?: boolean): string;
|
|
45
|
+
/** Why a URL was not kept — `host` is the one worth surfacing to the user. */
|
|
46
|
+
export type SkipReason = 'scheme' | 'host' | 'download' | 'action' | 'unparseable';
|
|
47
|
+
export interface UrlVerdict {
|
|
48
|
+
ok: boolean;
|
|
49
|
+
reason?: SkipReason;
|
|
50
|
+
/** The URL's host when it parsed — lets the caller name the mismatched host. */
|
|
51
|
+
host?: string;
|
|
52
|
+
}
|
|
53
|
+
/**
|
|
54
|
+
* Classify a URL against `baseUrl`, returning *why* it was rejected — so a caller
|
|
55
|
+
* can tell "on a different host" (worth a warning) from "a PDF" (silently skipped).
|
|
56
|
+
* The order matters: an off-host download reports `host`, the more actionable cause.
|
|
57
|
+
*/
|
|
58
|
+
export declare function classifyUrl(href: string, baseUrl: string, opts?: HostOptions): UrlVerdict;
|
|
29
59
|
/**
|
|
30
60
|
* May the crawler touch this href, resolved against `baseUrl`?
|
|
31
61
|
*
|
|
32
|
-
* Returns false for anything off-
|
|
33
|
-
* `UNSAFE_URL_RE`. Unparseable input is unsafe by
|
|
34
|
-
* be conservative about what gets a navigation.
|
|
62
|
+
* Returns false for anything off-site (see `siteKey` for the apex/`www.` rule),
|
|
63
|
+
* non-http, a download, or matching `UNSAFE_URL_RE`. Unparseable input is unsafe by
|
|
64
|
+
* default — the whole point is to be conservative about what gets a navigation.
|
|
35
65
|
*/
|
|
36
|
-
export declare function isSafeCandidateUrl(href: string, baseUrl: string): boolean;
|
|
66
|
+
export declare function isSafeCandidateUrl(href: string, baseUrl: string, opts?: HostOptions): boolean;
|
|
37
67
|
/**
|
|
38
68
|
* Compare two URLs by pathname alone — query and hash do not make another page,
|
|
39
69
|
* and a trailing slash is not a distinction. This is the dedup key for candidates,
|
|
@@ -53,7 +83,7 @@ export interface Candidate {
|
|
|
53
83
|
* The stored `url` is the same shape the rest of the config uses: a path for
|
|
54
84
|
* same-origin (`/dashboard`), so a contract stays comparable across environments.
|
|
55
85
|
*/
|
|
56
|
-
export declare function toCandidate(href: string, baseUrl: string, source: Candidate['source']): Candidate | null;
|
|
86
|
+
export declare function toCandidate(href: string, baseUrl: string, source: Candidate['source'], opts?: HostOptions): Candidate | null;
|
|
57
87
|
/**
|
|
58
88
|
* Fold candidates into the pages already in config, dropping duplicates and
|
|
59
89
|
* anything already configured, and making every new page name unique.
|
package/dist/discovery/safety.js
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
3
|
exports.UNSAFE_URL_RE = void 0;
|
|
4
|
+
exports.siteKey = siteKey;
|
|
5
|
+
exports.classifyUrl = classifyUrl;
|
|
4
6
|
exports.isSafeCandidateUrl = isSafeCandidateUrl;
|
|
5
7
|
exports.normalizedPath = normalizedPath;
|
|
6
8
|
exports.toCandidate = toCandidate;
|
|
@@ -36,13 +38,22 @@ exports.UNSAFE_URL_RE = /(?:^|[/_.?=&-])(?:log[\s_-]?out|sign[\s_-]?out|logout|s
|
|
|
36
38
|
/** File extensions that are downloads, not pages — never worth mapping. */
|
|
37
39
|
const DOWNLOAD_EXT_RE = /\.(?:pdf|zip|gz|tar|rar|7z|csv|xlsx?|docx?|pptx?|png|jpe?g|gif|svg|webp|ico|mp[34]|mov|avi|woff2?|ttf|eot|dmg|exe|apk|pkg)(?:$|[?#])/i;
|
|
38
40
|
/**
|
|
39
|
-
*
|
|
40
|
-
*
|
|
41
|
-
*
|
|
42
|
-
*
|
|
43
|
-
*
|
|
41
|
+
* The "same site" key of a URL: protocol + host, with a leading `www.` folded away
|
|
42
|
+
* unless `strictHost`. This is the whole apex/`www.` fix — comparing raw `origin`
|
|
43
|
+
* drops every `www.` sitemap URL when `baseUrl` is the apex (or vice versa), which
|
|
44
|
+
* is the single most common way a real `baseUrl` and a real sitemap differ. Protocol
|
|
45
|
+
* and port stay significant: `http` vs `https` is a real difference, not a `www.`.
|
|
46
|
+
*/
|
|
47
|
+
function siteKey(u, strictHost = false) {
|
|
48
|
+
const host = strictHost ? u.host : u.host.replace(/^www\./i, '');
|
|
49
|
+
return `${u.protocol}//${host}`;
|
|
50
|
+
}
|
|
51
|
+
/**
|
|
52
|
+
* Classify a URL against `baseUrl`, returning *why* it was rejected — so a caller
|
|
53
|
+
* can tell "on a different host" (worth a warning) from "a PDF" (silently skipped).
|
|
54
|
+
* The order matters: an off-host download reports `host`, the more actionable cause.
|
|
44
55
|
*/
|
|
45
|
-
function
|
|
56
|
+
function classifyUrl(href, baseUrl, opts = {}) {
|
|
46
57
|
let url;
|
|
47
58
|
let base;
|
|
48
59
|
try {
|
|
@@ -50,17 +61,28 @@ function isSafeCandidateUrl(href, baseUrl) {
|
|
|
50
61
|
url = new URL(href, baseUrl);
|
|
51
62
|
}
|
|
52
63
|
catch {
|
|
53
|
-
return false;
|
|
64
|
+
return { ok: false, reason: 'unparseable' };
|
|
54
65
|
}
|
|
55
66
|
if (url.protocol !== 'http:' && url.protocol !== 'https:')
|
|
56
|
-
return false;
|
|
57
|
-
if (url.
|
|
58
|
-
return false;
|
|
67
|
+
return { ok: false, reason: 'scheme' };
|
|
68
|
+
if (siteKey(url, opts.strictHost) !== siteKey(base, opts.strictHost)) {
|
|
69
|
+
return { ok: false, reason: 'host', host: url.host };
|
|
70
|
+
}
|
|
59
71
|
if (DOWNLOAD_EXT_RE.test(url.pathname + url.search))
|
|
60
|
-
return false;
|
|
72
|
+
return { ok: false, reason: 'download', host: url.host };
|
|
61
73
|
if (exports.UNSAFE_URL_RE.test(url.pathname + url.search))
|
|
62
|
-
return false;
|
|
63
|
-
return true;
|
|
74
|
+
return { ok: false, reason: 'action', host: url.host };
|
|
75
|
+
return { ok: true, host: url.host };
|
|
76
|
+
}
|
|
77
|
+
/**
|
|
78
|
+
* May the crawler touch this href, resolved against `baseUrl`?
|
|
79
|
+
*
|
|
80
|
+
* Returns false for anything off-site (see `siteKey` for the apex/`www.` rule),
|
|
81
|
+
* non-http, a download, or matching `UNSAFE_URL_RE`. Unparseable input is unsafe by
|
|
82
|
+
* default — the whole point is to be conservative about what gets a navigation.
|
|
83
|
+
*/
|
|
84
|
+
function isSafeCandidateUrl(href, baseUrl, opts = {}) {
|
|
85
|
+
return classifyUrl(href, baseUrl, opts).ok;
|
|
64
86
|
}
|
|
65
87
|
/**
|
|
66
88
|
* Compare two URLs by pathname alone — query and hash do not make another page,
|
|
@@ -85,8 +107,8 @@ function normalizedPath(href, baseUrl) {
|
|
|
85
107
|
* The stored `url` is the same shape the rest of the config uses: a path for
|
|
86
108
|
* same-origin (`/dashboard`), so a contract stays comparable across environments.
|
|
87
109
|
*/
|
|
88
|
-
function toCandidate(href, baseUrl, source) {
|
|
89
|
-
if (!isSafeCandidateUrl(href, baseUrl))
|
|
110
|
+
function toCandidate(href, baseUrl, source, opts = {}) {
|
|
111
|
+
if (!isSafeCandidateUrl(href, baseUrl, opts))
|
|
90
112
|
return null;
|
|
91
113
|
const path = normalizedPath(href, baseUrl);
|
|
92
114
|
return { name: (0, pageName_1.pageNameFromUrl)(new URL(href, baseUrl).toString()), url: path, source };
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"safety.js","sourceRoot":"","sources":["../../src/discovery/safety.ts"],"names":[],"mappings":";;;
|
|
1
|
+
{"version":3,"file":"safety.js","sourceRoot":"","sources":["../../src/discovery/safety.ts"],"names":[],"mappings":";;;AAqDA,0BAGC;AAiBD,kCAgBC;AASD,gDAEC;AAQD,wCASC;AAeD,kCASC;AAcD,gCA4BC;AAvLD,0CAA8C;AAG9C;;;;;;;;;;;;;;;;GAgBG;AAEH;;;;;;;;GAQG;AACU,QAAA,aAAa,GACxB,+NAA+N,CAAC;AAElO,2EAA2E;AAC3E,MAAM,eAAe,GACnB,uIAAuI,CAAC;AAW1I;;;;;;GAMG;AACH,SAAgB,OAAO,CAAC,CAAM,EAAE,UAAU,GAAG,KAAK;IAChD,MAAM,IAAI,GAAG,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACjE,OAAO,GAAG,CAAC,CAAC,QAAQ,KAAK,IAAI,EAAE,CAAC;AAClC,CAAC;AAYD;;;;GAIG;AACH,SAAgB,WAAW,CAAC,IAAY,EAAE,OAAe,EAAE,OAAoB,EAAE;IAC/E,IAAI,GAAQ,CAAC;IACb,IAAI,IAAS,CAAC;IACd,IAAI,CAAC;QACH,IAAI,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC;QACxB,GAAG,GAAG,IAAI,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;IAC/B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,aAAa,EAAE,CAAC;IAC9C,CAAC;IACD,IAAI,GAAG,CAAC,QAAQ,KAAK,OAAO,IAAI,GAAG,CAAC,QAAQ,KAAK,QAAQ;QAAE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,CAAC;IAClG,IAAI,OAAO,CAAC,GAAG,EAAE,IAAI,CAAC,UAAU,CAAC,KAAK,OAAO,CAAC,IAAI,EAAE,IAAI,CAAC,UAAU,CAAC,EAAE,CAAC;QACrE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,IAAI,EAAE,GAAG,CAAC,IAAI,EAAE,CAAC;IACvD,CAAC;IACD,IAAI,eAAe,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ,GAAG,GAAG,CAAC,MAAM,CAAC;QAAE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,UAAU,EAAE,IAAI,EAAE,GAAG,CAAC,IAAI,EAAE,CAAC;IAC9G,IAAI,qBAAa,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ,GAAG,GAAG,CAAC,MAAM,CAAC;QAAE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,IAAI,EAAE,GAAG,CAAC,IAAI,EAAE,CAAC;IAC1G,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,GAAG,CAAC,IAAI,EAAE,CAAC;AACtC,CAAC;AAED;;;;;;GAMG;AACH,SAAgB,kBAAkB,CAAC,IAAY,EAAE,OAAe,EAAE,OAAoB,EAAE;IACtF,OAAO,WAAW,CAAC,IAAI,EAAE,OAAO,EAAE,IAAI,CAAC,CAAC,EAAE,CAAC;AAC7C,CAAC;AAED;;;;;GAKG;AACH,SAAgB,cAAc,CAAC,IAAY,EAAE,OAAe;IAC1D,IAAI,GAAQ,CAAC;IACb,IAAI,CAAC;QACH,GAAG,GAAG,IAAI,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;IAC/B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,IAAI,CAAC;IACd,CAAC;IACD,MAAM,CAAC,GAAG,GAAG,CAAC,QAAQ,CAAC;IACvB,OAAO,CAAC,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;AAC9D,CAAC;AASD;;;;;GAKG;AACH,SAAgB,WAAW,CACzB,IAAY,EACZ,OAAe,EACf,MAA2B,EAC3B,OAAoB,EAAE;IAEtB,IAAI,CAAC,kBAAkB,CAAC,IAAI,EAAE,OAAO,EAAE,IAAI,CAAC;QAAE,OAAO,IAAI,CAAC;IAC1D,MAAM,IAAI,GAAG,cAAc,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;IAC3C,OAAO,EAAE,IAAI,EAAE,IAAA,0BAAe,EAAC,IAAI,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC,EAAE,GAAG,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC;AACzF,CAAC;AAED;;;;;;;;;;;GAWG;AACH,SAAgB,UAAU,CACxB,QAAsB,EACtB,UAAuB,EACvB,OAAe;IAEf,MAAM,UAAU,GAAG,IAAI,GAAG,CACxB,QAAQ;SACL,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,KAAK,EAAE,MAAM,CAAC;SAC/B,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,cAAc,CAAC,CAAC,CAAC,GAAG,EAAE,OAAO,CAAC,CAAC,CAC9C,CAAC;IACF,MAAM,UAAU,GAAG,IAAI,GAAG,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;IACxD,MAAM,KAAK,GAAgB,EAAE,CAAC;IAE9B,KAAK,MAAM,CAAC,IAAI,UAAU,EAAE,CAAC;QAC3B,MAAM,GAAG,GAAG,cAAc,CAAC,CAAC,CAAC,GAAG,EAAE,OAAO,CAAC,CAAC;QAC3C,IAAI,UAAU,CAAC,GAAG,CAAC,GAAG,CAAC;YAAE,SAAS;QAClC,UAAU,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QAEpB,IAAI,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC;QAClB,IAAI,UAAU,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;YACzB,IAAI,CAAC,GAAG,CAAC,CAAC;YACV,OAAO,UAAU,CAAC,GAAG,CAAC,GAAG,IAAI,IAAI,CAAC,EAAE,CAAC;gBAAE,CAAC,EAAE,CAAC;YAC3C,IAAI,GAAG,GAAG,IAAI,IAAI,CAAC,EAAE,CAAC;QACxB,CAAC;QACD,UAAU,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACrB,KAAK,CAAC,IAAI,CAAC,EAAE,GAAG,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC;IAC7B,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC"}
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { Candidate } from './safety';
|
|
1
|
+
import { Candidate, HostOptions } from './safety';
|
|
2
2
|
/**
|
|
3
3
|
* Sitemap discovery — the zero-risk source.
|
|
4
4
|
*
|
|
@@ -18,7 +18,7 @@ export type FetchLike = (url: string) => Promise<{
|
|
|
18
18
|
status: number;
|
|
19
19
|
text: () => Promise<string>;
|
|
20
20
|
}>;
|
|
21
|
-
export interface SitemapOptions {
|
|
21
|
+
export interface SitemapOptions extends HostOptions {
|
|
22
22
|
/** Explicit sitemap URL. Defaults to `<baseUrl>/sitemap.xml`, then robots.txt. */
|
|
23
23
|
sitemapUrl?: string;
|
|
24
24
|
/** Injected for tests; defaults to the global `fetch` (Node 18+). */
|
|
@@ -28,6 +28,24 @@ export interface SitemapOptions {
|
|
|
28
28
|
/** Bound on nested sitemap files followed from an index. */
|
|
29
29
|
maxSitemaps?: number;
|
|
30
30
|
}
|
|
31
|
+
/**
|
|
32
|
+
* An honest account of what the sitemap held versus what survived — the fix for the
|
|
33
|
+
* silent "0 URLs" that a host mismatch used to produce. `parsed` is every page
|
|
34
|
+
* `<loc>` seen; the three skip buckets say why the rest did not make it, and
|
|
35
|
+
* `otherHosts` names the hosts behind a `host` skip so the CLI can point at the
|
|
36
|
+
* apex/`www.` mismatch by name instead of leaving the user to guess.
|
|
37
|
+
*/
|
|
38
|
+
export interface SitemapStats {
|
|
39
|
+
parsed: number;
|
|
40
|
+
retained: number;
|
|
41
|
+
skippedHost: number;
|
|
42
|
+
skippedOther: number;
|
|
43
|
+
otherHosts: string[];
|
|
44
|
+
}
|
|
45
|
+
export interface SitemapResult {
|
|
46
|
+
candidates: Candidate[];
|
|
47
|
+
stats: SitemapStats;
|
|
48
|
+
}
|
|
31
49
|
/** Pull every `<loc>` out of a sitemap or sitemap-index document. Pure. */
|
|
32
50
|
export declare function extractLocs(xml: string): string[];
|
|
33
51
|
/** A sitemap index points at other sitemaps rather than at pages. */
|
|
@@ -39,10 +57,13 @@ export declare function sitemapsFromRobots(robots: string): string[];
|
|
|
39
57
|
*
|
|
40
58
|
* Resolution: the explicit `sitemapUrl` if given, else `/sitemap.xml`; if that
|
|
41
59
|
* 404s, the `Sitemap:` lines in `/robots.txt`. A sitemap index is followed one
|
|
42
|
-
* level into its child sitemaps (bounded by `maxSitemaps`).
|
|
43
|
-
*
|
|
60
|
+
* level into its child sitemaps (bounded by `maxSitemaps`). Every page URL is
|
|
61
|
+
* classified (see `classifyUrl`), so off-site, unsafe, and download URLs never
|
|
62
|
+
* survive — but the count of what was dropped, and why, comes back in `stats` so
|
|
63
|
+
* the caller can tell a host mismatch from "nothing there" instead of printing a
|
|
64
|
+
* silent "0 URLs".
|
|
44
65
|
*
|
|
45
66
|
* Never throws on a missing or malformed sitemap: absence is the common case
|
|
46
67
|
* (SPAs, staging), and it is a "found nothing", not an error.
|
|
47
68
|
*/
|
|
48
|
-
export declare function discoverFromSitemap(baseUrl: string, options?: SitemapOptions): Promise<
|
|
69
|
+
export declare function discoverFromSitemap(baseUrl: string, options?: SitemapOptions): Promise<SitemapResult>;
|
|
@@ -5,6 +5,7 @@ exports.isSitemapIndex = isSitemapIndex;
|
|
|
5
5
|
exports.sitemapsFromRobots = sitemapsFromRobots;
|
|
6
6
|
exports.discoverFromSitemap = discoverFromSitemap;
|
|
7
7
|
const safety_1 = require("./safety");
|
|
8
|
+
const pageName_1 = require("../pageName");
|
|
8
9
|
const LOC_RE = /<loc>\s*([^<\s][^<]*?)\s*<\/loc>/gi;
|
|
9
10
|
/** Pull every `<loc>` out of a sitemap or sitemap-index document. Pure. */
|
|
10
11
|
function extractLocs(xml) {
|
|
@@ -45,8 +46,11 @@ function sitemapsFromRobots(robots) {
|
|
|
45
46
|
*
|
|
46
47
|
* Resolution: the explicit `sitemapUrl` if given, else `/sitemap.xml`; if that
|
|
47
48
|
* 404s, the `Sitemap:` lines in `/robots.txt`. A sitemap index is followed one
|
|
48
|
-
* level into its child sitemaps (bounded by `maxSitemaps`).
|
|
49
|
-
*
|
|
49
|
+
* level into its child sitemaps (bounded by `maxSitemaps`). Every page URL is
|
|
50
|
+
* classified (see `classifyUrl`), so off-site, unsafe, and download URLs never
|
|
51
|
+
* survive — but the count of what was dropped, and why, comes back in `stats` so
|
|
52
|
+
* the caller can tell a host mismatch from "nothing there" instead of printing a
|
|
53
|
+
* silent "0 URLs".
|
|
50
54
|
*
|
|
51
55
|
* Never throws on a missing or malformed sitemap: absence is the common case
|
|
52
56
|
* (SPAs, staging), and it is a "found nothing", not an error.
|
|
@@ -83,18 +87,35 @@ async function discoverFromSitemap(baseUrl, options = {}) {
|
|
|
83
87
|
}
|
|
84
88
|
const candidates = [];
|
|
85
89
|
const seenPaths = new Set();
|
|
90
|
+
const otherHosts = new Set();
|
|
91
|
+
const stats = { parsed: 0, retained: 0, skippedHost: 0, skippedOther: 0, otherHosts: [] };
|
|
86
92
|
for (const raw of pageUrls) {
|
|
87
|
-
|
|
88
|
-
|
|
93
|
+
stats.parsed++;
|
|
94
|
+
const verdict = (0, safety_1.classifyUrl)(raw, baseUrl, { strictHost: options.strictHost });
|
|
95
|
+
if (!verdict.ok) {
|
|
96
|
+
if (verdict.reason === 'host') {
|
|
97
|
+
stats.skippedHost++;
|
|
98
|
+
if (verdict.host)
|
|
99
|
+
otherHosts.add(verdict.host);
|
|
100
|
+
}
|
|
101
|
+
else {
|
|
102
|
+
stats.skippedOther++;
|
|
103
|
+
}
|
|
89
104
|
continue;
|
|
90
|
-
|
|
105
|
+
}
|
|
106
|
+
const path = (0, safety_1.normalizedPath)(raw, baseUrl);
|
|
107
|
+
if (seenPaths.has(path)) {
|
|
108
|
+
stats.skippedOther++;
|
|
91
109
|
continue;
|
|
92
|
-
|
|
93
|
-
|
|
110
|
+
}
|
|
111
|
+
seenPaths.add(path);
|
|
112
|
+
candidates.push({ name: (0, pageName_1.pageNameFromUrl)(new URL(raw, baseUrl).toString()), url: path, source: 'sitemap' });
|
|
113
|
+
stats.retained++;
|
|
94
114
|
if (candidates.length >= maxUrls)
|
|
95
115
|
break;
|
|
96
116
|
}
|
|
97
|
-
|
|
117
|
+
stats.otherHosts = Array.from(otherHosts).slice(0, 5);
|
|
118
|
+
return { candidates, stats };
|
|
98
119
|
}
|
|
99
120
|
async function resolveSitemapUrls(baseUrl, fetchImpl) {
|
|
100
121
|
const primary = new URL('/sitemap.xml', baseUrl).toString();
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"sitemap.js","sourceRoot":"","sources":["../../src/discovery/sitemap.ts"],"names":[],"mappings":";;
|
|
1
|
+
{"version":3,"file":"sitemap.js","sourceRoot":"","sources":["../../src/discovery/sitemap.ts"],"names":[],"mappings":";;AAsDA,kCASC;AAGD,wCAEC;AAYD,gDAOC;AAgBD,kDAgEC;AAvKD,qCAA+E;AAC/E,0CAA8C;AAkD9C,MAAM,MAAM,GAAG,oCAAoC,CAAC;AAEpD,2EAA2E;AAC3E,SAAgB,WAAW,CAAC,GAAW;IACrC,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,IAAI,CAAyB,CAAC;IAC9B,MAAM,CAAC,SAAS,GAAG,CAAC,CAAC;IACrB,OAAO,CAAC,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,KAAK,IAAI,EAAE,CAAC;QACvC,MAAM,GAAG,GAAG,cAAc,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;QACxC,IAAI,GAAG;YAAE,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IACzB,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,qEAAqE;AACrE,SAAgB,cAAc,CAAC,GAAW;IACxC,OAAO,qBAAqB,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AACzC,CAAC;AAED,SAAS,cAAc,CAAC,CAAS;IAC/B,OAAO,CAAC;SACL,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC;SACtB,OAAO,CAAC,OAAO,EAAE,GAAG,CAAC;SACrB,OAAO,CAAC,OAAO,EAAE,GAAG,CAAC;SACrB,OAAO,CAAC,SAAS,EAAE,GAAG,CAAC;SACvB,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC;AAC5B,CAAC;AAED,iFAAiF;AACjF,SAAgB,kBAAkB,CAAC,MAAc;IAC/C,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,KAAK,CAAC,OAAO,CAAC,EAAE,CAAC;QACzC,MAAM,CAAC,GAAG,uBAAuB,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QAC7C,IAAI,CAAC;YAAE,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;IAC/B,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED;;;;;;;;;;;;;GAaG;AACI,KAAK,UAAU,mBAAmB,CACvC,OAAe,EACf,UAA0B,EAAE;IAE5B,MAAM,SAAS,GAAG,OAAO,CAAC,SAAS,IAAK,UAAU,CAAC,KAA8B,CAAC;IAClF,IAAI,CAAC,SAAS;QAAE,MAAM,IAAI,KAAK,CAAC,iEAAiE,CAAC,CAAC;IACnG,MAAM,OAAO,GAAG,OAAO,CAAC,OAAO,IAAI,GAAG,CAAC;IACvC,MAAM,WAAW,GAAG,OAAO,CAAC,WAAW,IAAI,EAAE,CAAC;IAE9C,MAAM,KAAK,GAAG,OAAO,CAAC,UAAU;QAC9B,CAAC,CAAC,CAAC,OAAO,CAAC,UAAU,CAAC;QACtB,CAAC,CAAC,MAAM,kBAAkB,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IAEjD,MAAM,YAAY,GAAG,IAAI,GAAG,EAAU,CAAC;IACvC,MAAM,QAAQ,GAAa,EAAE,CAAC;IAC9B,MAAM,KAAK,GAAG,CAAC,GAAG,KAAK,CAAC,CAAC;IAEzB,OAAO,KAAK,CAAC,MAAM,GAAG,CAAC,IAAI,YAAY,CAAC,IAAI,GAAG,WAAW,IAAI,QAAQ,CAAC,MAAM,GAAG,OAAO,EAAE,CAAC;QACxF,MAAM,GAAG,GAAG,KAAK,CAAC,KAAK,EAAG,CAAC;QAC3B,IAAI,YAAY,CAAC,GAAG,CAAC,GAAG,CAAC;YAAE,SAAS;QACpC,YAAY,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QAEtB,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,SAAS,EAAE,GAAG,CAAC,CAAC;QAC3C,IAAI,CAAC,GAAG;YAAE,SAAS;QAEnB,IAAI,cAAc,CAAC,GAAG,CAAC,EAAE,CAAC;YACxB,KAAK,MAAM,KAAK,IAAI,WAAW,CAAC,GAAG,CAAC,EAAE,CAAC;gBACrC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,KAAK,CAAC;oBAAE,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YAClD,CAAC;QACH,CAAC;aAAM,CAAC;YACN,QAAQ,CAAC,IAAI,CAAC,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC,CAAC;QACrC,CAAC;IACH,CAAC;IAED,MAAM,UAAU,GAAgB,EAAE,CAAC;IACnC,MAAM,SAAS,GAAG,IAAI,GAAG,EAAU,CAAC;IACpC,MAAM,UAAU,GAAG,IAAI,GAAG,EAAU,CAAC;IACrC,MAAM,KAAK,GAAiB,EAAE,MAAM,EAAE,CAAC,EAAE,QAAQ,EAAE,CAAC,EAAE,WAAW,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,UAAU,EAAE,EAAE,EAAE,CAAC;IAExG,KAAK,MAAM,GAAG,IAAI,QAAQ,EAAE,CAAC;QAC3B,KAAK,CAAC,MAAM,EAAE,CAAC;QACf,MAAM,OAAO,GAAG,IAAA,oBAAW,EAAC,GAAG,EAAE,OAAO,EAAE,EAAE,UAAU,EAAE,OAAO,CAAC,UAAU,EAAE,CAAC,CAAC;QAC9E,IAAI,CAAC,OAAO,CAAC,EAAE,EAAE,CAAC;YAChB,IAAI,OAAO,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;gBAC9B,KAAK,CAAC,WAAW,EAAE,CAAC;gBACpB,IAAI,OAAO,CAAC,IAAI;oBAAE,UAAU,CAAC,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;YACjD,CAAC;iBAAM,CAAC;gBACN,KAAK,CAAC,YAAY,EAAE,CAAC;YACvB,CAAC;YACD,SAAS;QACX,CAAC;QACD,MAAM,IAAI,GAAG,IAAA,uBAAc,EAAC,GAAG,EAAE,OAAO,CAAC,CAAC;QAC1C,IAAI,SAAS,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;YACxB,KAAK,CAAC,YAAY,EAAE,CAAC;YACrB,SAAS;QACX,CAAC;QACD,SAAS,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACpB,UAAU,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,IAAA,0BAAe,EAAC,IAAI,GAAG,CAAC,GAAG,EAAE,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC,EAAE,GAAG,EAAE,IAAI,EAAE,MAAM,EAAE,SAAS,EAAE,CAAC,CAAC;QAC3G,KAAK,CAAC,QAAQ,EAAE,CAAC;QACjB,IAAI,UAAU,CAAC,MAAM,IAAI,OAAO;YAAE,MAAM;IAC1C,CAAC;IAED,KAAK,CAAC,UAAU,GAAG,KAAK,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACtD,OAAO,EAAE,UAAU,EAAE,KAAK,EAAE,CAAC;AAC/B,CAAC;AAED,KAAK,UAAU,kBAAkB,CAAC,OAAe,EAAE,SAAoB;IACrE,MAAM,OAAO,GAAG,IAAI,GAAG,CAAC,cAAc,EAAE,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC;IAC5D,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,SAAS,EAAE,OAAO,CAAC,CAAC;IAChD,IAAI,IAAI;QAAE,OAAO,CAAC,OAAO,CAAC,CAAC;IAE3B,MAAM,MAAM,GAAG,MAAM,QAAQ,CAAC,SAAS,EAAE,IAAI,GAAG,CAAC,aAAa,EAAE,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC;IACrF,IAAI,MAAM,EAAE,CAAC;QACX,MAAM,UAAU,GAAG,kBAAkB,CAAC,MAAM,CAAC,CAAC;QAC9C,IAAI,UAAU,CAAC,MAAM,GAAG,CAAC;YAAE,OAAO,UAAU,CAAC;IAC/C,CAAC;IACD,OAAO,CAAC,OAAO,CAAC,CAAC;AACnB,CAAC;AAED,KAAK,UAAU,QAAQ,CAAC,SAAoB,EAAE,GAAW;IACvD,IAAI,CAAC;QACH,MAAM,GAAG,GAAG,MAAM,SAAS,CAAC,GAAG,CAAC,CAAC;QACjC,IAAI,CAAC,GAAG,CAAC,EAAE;YAAE,OAAO,IAAI,CAAC;QACzB,OAAO,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC;IAC1B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,IAAI,CAAC;IACd,CAAC;AACH,CAAC"}
|
package/dist/index.d.ts
CHANGED
|
@@ -10,9 +10,9 @@ export { renderPageContract, saveMarkdown, markdownPath } from './markdown';
|
|
|
10
10
|
export { mapUrl, pageNameFromUrl } from './mapUrl';
|
|
11
11
|
export { resolvePageName } from './pageName';
|
|
12
12
|
export { isSafeCandidateUrl, normalizedPath, toCandidate, mergePages, UNSAFE_URL_RE, } from './discovery/safety';
|
|
13
|
-
export type { Candidate } from './discovery/safety';
|
|
13
|
+
export type { Candidate, HostOptions } from './discovery/safety';
|
|
14
14
|
export { discoverFromSitemap, extractLocs, isSitemapIndex, sitemapsFromRobots, } from './discovery/sitemap';
|
|
15
|
-
export type { SitemapOptions, FetchLike } from './discovery/sitemap';
|
|
15
|
+
export type { SitemapOptions, SitemapResult, SitemapStats, FetchLike } from './discovery/sitemap';
|
|
16
16
|
export { crawlApp } from './crawl';
|
|
17
17
|
export type { CrawlOptions } from './crawl';
|
|
18
18
|
export type { PageNameResolution } from './pageName';
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ia-qa/self-healing",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.4.0",
|
|
4
4
|
"description": "Local-first self-healing for UI tests: a local MCP server + CLI that map your app's pages to a role/name/selector contract, diff selector drift (PASS/FIX/BLOCK), and apply deterministic fixes to Cypress/Playwright/Selenium tests. Runs on your machine — nothing leaves it.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"self-healing",
|