@goflag/cli 0.2.4 → 0.2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +76 -14
- package/dist/{chunk-TVJVVSLV.js → chunk-UOKYUKHQ.js} +176 -20
- package/dist/chunk-UOKYUKHQ.js.map +1 -0
- package/dist/cli.js +500 -8
- package/dist/cli.js.map +1 -1
- package/dist/index.d.ts +30 -4
- package/dist/index.js +1 -1
- package/package.json +4 -2
- package/rules.json +517 -0
- package/dist/chunk-TVJVVSLV.js.map +0 -1
package/README.md
CHANGED
|
@@ -16,9 +16,14 @@ on the site. A human cannot check those on 400 pages. That is the whole job.
|
|
|
16
16
|
## What it checks
|
|
17
17
|
|
|
18
18
|
- **Broken links** — scrapes every crawled page, dedupes targets globally (a footer link on 500 pages is probed once), and checks each with `HEAD`→`GET` fallback, redirect-chain/loop detection, soft-404 and anti-bot (403/429) triage. Broken links are mapped back to the pages that reference them.
|
|
19
|
-
- **Missing translation pages** — builds a `route × locale` matrix
|
|
19
|
+
- **Missing translation pages** — builds a `route × locale` matrix and flags every route that exists in one locale but is missing in another, plus hreflang reciprocity gaps (`A → B` with no `B → A`), missing `x-default`, and locale tags naming a language, script or region that does not exist. Rows are keyed by pathname, unless your sitemap declares its translation clusters with `xhtml:link` — then those declarations decide which URLs are one page, whatever their slugs, and the run reports `diagnostics.declaredClusters`.
|
|
20
20
|
- **robots.txt policy** — flags a site that forbids crawling while its pages ask to be indexed. The two declarations cannot both hold, and robots.txt wins, so the meta tag is never even read.
|
|
21
21
|
- **SEO metadata** — lints each page's `<head>` for the handful of mistakes that actually hurt in search and social and are invisible in a browser: missing/oversized `<title>` and description, missing/relative canonical, missing `og:title`/`og:description`/`og:image`, missing viewport, and contradictory `robots`/`googlebot`/`X-Robots-Tag` directives.
|
|
22
|
+
- **Unreachable pages** — a page the crawl reached that answered non-2xx, or that never answered at all (timeout, reset, DNS failure — recorded as `status: 0`), is reported in `unreachablePages`. goflag asks a second time, from the back of the queue, before calling a page unreachable. Each one is an `error`-severity finding: it turns the verdict red, counts toward `--max-debt`, and fails the build even under `--fail-on error`. A page that silently dropped out of the audited set is what poisons a baseline, so it is reported rather than warned about.
|
|
23
|
+
|
|
24
|
+
**Which pages get audited.** When goflag finds a sitemap it does not audit every URL in it. It groups URLs that share a path shape — and therefore a template — into families, audits every page that stands alone, and samples three pages per family. A cap answers "how many" and never "which": on a site of thousands of pages built from thirty templates, the first 200 pages a crawl reaches are four templates out of thirty.
|
|
25
|
+
|
|
26
|
+
The trade is stated in every report rather than hidden. Template rules (`canonical.*`, `hreflang.*`, `og.image.missing`) are conclusive on a sampled family; copy rules (`title.length`, `description.length`), broken links and the translation matrix are only conclusive on the pages that were drawn. The terminal prints a `COVERAGE` line, and `diagnostics.coverage` carries `mode`, `considered`, `selected` and every sampled family with its ratio. Pass `--coverage all` to audit what the sitemap lists in order, up to `--max-pages`, which is the pre-0.2.3 behaviour.
|
|
22
27
|
|
|
23
28
|
Pages that declare a `<link rel="canonical">` pointing at another crawled page are excluded from the checks — the site has said they are duplicates, so linting them would multiply every finding by the number of filtered variants. The count is reported as `diagnostics.duplicatePages`.
|
|
24
29
|
|
|
@@ -125,16 +130,57 @@ npx @goflag/cli https://example.com --report report.json # write JSON to a
|
|
|
125
130
|
|
|
126
131
|
The terminal view is just a render of that JSON.
|
|
127
132
|
|
|
133
|
+
### The catalogue is data too
|
|
134
|
+
|
|
135
|
+
```sh
|
|
136
|
+
npx @goflag/cli rules > rules.json
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
`rules` answers a question about goflag rather than about a site: no URL, no
|
|
140
|
+
crawl, no network. Every entry carries its scope, severity and summary; the
|
|
141
|
+
eleven page rules and four prose rules also carry a rigor, the documents they
|
|
142
|
+
cite and — where a remedy is a line of code — a fix snippet. The three site
|
|
143
|
+
rules carry none of those three: `SiteRule` has no rigor field yet, and the
|
|
144
|
+
catalogue emits `rigor: null` with an empty `sources` rather than inventing an
|
|
145
|
+
authority nobody assigned. Write your consumer against that. The same document
|
|
146
|
+
ships inside the package as `rules.json`, if you would rather read it than run
|
|
147
|
+
anything.
|
|
148
|
+
|
|
149
|
+
It deliberately omits the message a finding prints — that is built at audit time
|
|
150
|
+
from what the page actually says, so a static copy would be a sample rather than
|
|
151
|
+
a fact, and a sample presented as the truth is how this project's own
|
|
152
|
+
documentation site came to quote a message the engine had stopped printing.
|
|
153
|
+
|
|
128
154
|
### Options
|
|
129
155
|
|
|
130
156
|
```
|
|
131
157
|
--json Print the JSON report to stdout (nothing else).
|
|
158
|
+
--summary, -s Roll findings up (dedup by link/rule/code). Pairs with
|
|
159
|
+
--json for a compact payload; --report always writes
|
|
160
|
+
the full report regardless.
|
|
132
161
|
--report <file> Write the JSON report to <file>.
|
|
133
|
-
--depth <n>
|
|
134
|
-
|
|
162
|
+
--depth <n> How far to follow links out of each page (0 = follow
|
|
163
|
+
none). Sitemap URLs are seeded regardless, so --depth 0
|
|
164
|
+
alone is not "entry page only" — add --no-sitemap.
|
|
165
|
+
Default: 2.
|
|
166
|
+
--max-pages <n> Page budget for the crawl. Default: 200. A hard cap
|
|
167
|
+
under --coverage all; under structural coverage the
|
|
168
|
+
selection wins and the budget is max(<n>, selected + 5),
|
|
169
|
+
so a lower value cannot cut the run short.
|
|
170
|
+
--coverage <mode> How pages are chosen: "structural" (default when a
|
|
171
|
+
sitemap is found) keeps every standalone page and
|
|
172
|
+
samples three pages per family of template-generated
|
|
173
|
+
pages; "all" audits what the sitemap lists, in order,
|
|
174
|
+
up to --max-pages.
|
|
135
175
|
--include <glob> Only crawl paths matching <glob> (repeatable).
|
|
136
176
|
--exclude <glob> Skip paths matching <glob> (repeatable).
|
|
137
177
|
--locales <list> Comma-separated locales the site serves ("fr,en,pt-br").
|
|
178
|
+
Unioned with the prefixes your sitemap shows, never
|
|
179
|
+
substituted for them, so a locale the sitemap serves
|
|
180
|
+
stays on the axis whether or not you list it. Also what
|
|
181
|
+
folds /en/… and /fr/… into one route family for
|
|
182
|
+
structural coverage — pass it on any locale-prefixed
|
|
183
|
+
site.
|
|
138
184
|
--ignore-holes <glob> Route deliberately not translated everywhere
|
|
139
185
|
(repeatable).
|
|
140
186
|
--no-sitemap Do not discover the sitemap; crawl from <url> only.
|
|
@@ -160,7 +206,11 @@ The terminal view is just a render of that JSON.
|
|
|
160
206
|
--no-external Do not probe off-origin (external) links.
|
|
161
207
|
--static Static HTML only; never launch headless Chromium.
|
|
162
208
|
--allow-insecure-tls Accept self-signed / invalid TLS (localhost, tunnels).
|
|
163
|
-
--timeout <ms> Per-request timeout in ms
|
|
209
|
+
--timeout <ms> Per-request timeout in ms, for page fetches and link
|
|
210
|
+
probes alike. Unset, the defaults differ: 8000 for link
|
|
211
|
+
probes, 15000 for page fetches.
|
|
212
|
+
--verbose, -V Log every page as it is analyzed.
|
|
213
|
+
--quiet, -q Suppress the live progress output.
|
|
164
214
|
--no-color Disable coloured output.
|
|
165
215
|
-h, --help Show help.
|
|
166
216
|
-v, --version Show the version.
|
|
@@ -351,9 +401,16 @@ goflag https://example.com --locales fr,en,pt-br
|
|
|
351
401
|
|
|
352
402
|
When a site publishes no usable sitemap and you pass no `--locales`, goflag
|
|
353
403
|
does **not** guess the locale axis — it reports the prefixes it saw with the
|
|
354
|
-
evidence for each and turns the
|
|
355
|
-
|
|
356
|
-
|
|
404
|
+
evidence for each and turns the axis-dependent checks off (holes,
|
|
405
|
+
`hreflang.missing`, `hreflang.sitemap-mismatch`; the reciprocity checks read the
|
|
406
|
+
markup and keep running). Guessing from path shape alone once turned `/cv` (a CV
|
|
407
|
+
page served in French) into a locale, because `cv` is a registered ISO 639-1
|
|
408
|
+
code, and produced 31 findings that were never real.
|
|
409
|
+
|
|
410
|
+
Pass `--locales` even when the sitemap is fine. It is the axis goflag folds when
|
|
411
|
+
it groups URLs into route families, so without it a site with many top-level
|
|
412
|
+
sections under `/en` can have those sections sampled as one family instead of
|
|
413
|
+
audited.
|
|
357
414
|
|
|
358
415
|
Some pages are meant to exist in one locale and not another — a
|
|
359
416
|
jurisdiction-specific legal notice, a post written for one market. A site has
|
|
@@ -398,7 +455,7 @@ Three rules, whichever you use:
|
|
|
398
455
|
stages: [build, deploy, audit]
|
|
399
456
|
|
|
400
457
|
variables:
|
|
401
|
-
GOFLAG_VERSION: "0.
|
|
458
|
+
GOFLAG_VERSION: "0.2.7"
|
|
402
459
|
|
|
403
460
|
# Before the merge: build the branch, let goflag boot it, audit that.
|
|
404
461
|
seo:mr:
|
|
@@ -412,6 +469,7 @@ seo:mr:
|
|
|
412
469
|
- npx --yes "@goflag/cli@$GOFLAG_VERSION" http://localhost:3000
|
|
413
470
|
--start "pnpm start" --static --no-external
|
|
414
471
|
--baseline .goflag/baseline.json --regressions-only --max-debt 41
|
|
472
|
+
--report goflag-report.json
|
|
415
473
|
artifacts:
|
|
416
474
|
when: always
|
|
417
475
|
paths: [goflag-report.json]
|
|
@@ -449,9 +507,10 @@ jobs:
|
|
|
449
507
|
- run: corepack enable && pnpm install --frozen-lockfile
|
|
450
508
|
- run: pnpm build
|
|
451
509
|
- run: |
|
|
452
|
-
npx --yes @goflag/cli@0.
|
|
510
|
+
npx --yes @goflag/cli@0.2.7 http://localhost:3000 \
|
|
453
511
|
--start "pnpm start" --static --no-external \
|
|
454
|
-
--baseline .goflag/baseline.json --regressions-only --max-debt 41
|
|
512
|
+
--baseline .goflag/baseline.json --regressions-only --max-debt 41 \
|
|
513
|
+
--report goflag-report.json
|
|
455
514
|
- uses: actions/upload-artifact@v4
|
|
456
515
|
if: always()
|
|
457
516
|
with:
|
|
@@ -484,7 +543,8 @@ you what it accepted, and that number belongs in the commit message.
|
|
|
484
543
|
import { runAudit } from "@goflag/cli";
|
|
485
544
|
|
|
486
545
|
const report = await runAudit("https://example.com", { depth: 2 });
|
|
487
|
-
console.log(report.summary);
|
|
546
|
+
console.log(report.summary);
|
|
547
|
+
// { brokenLinks, missingTranslations, seoIssues, siteIssues, unreachablePages, verdict }
|
|
488
548
|
```
|
|
489
549
|
|
|
490
550
|
`runAudit` returns the same `GoflagReport` object the CLI emits.
|
|
@@ -543,9 +603,11 @@ pnpm typecheck # tsc --noEmit
|
|
|
543
603
|
pnpm format # prettier
|
|
544
604
|
```
|
|
545
605
|
|
|
546
|
-
`
|
|
547
|
-
|
|
548
|
-
|
|
606
|
+
`format` and `format:check` run once over the whole repository. `lint` runs
|
|
607
|
+
eslint once at the root and then fans out to any package that defines its own
|
|
608
|
+
`lint` script — `apps/website` does, with the Next config — and the rest fan out
|
|
609
|
+
across workspace packages with `pnpm -r`. To work on one package, filter:
|
|
610
|
+
`pnpm --filter @goflag/cli test`.
|
|
549
611
|
|
|
550
612
|
The engine is framework-agnostic and lives in [`packages/cli/src/lib/core/`](packages/cli/src/lib/core) (crawl, fetch/extract, link audit, i18n) and [`packages/cli/src/lib/rules/`](packages/cli/src/lib/rules) (SEO checks). The CLI shell — orchestration, the `GoflagReport` schema, and rendering — lives in [`packages/cli/src/report/`](packages/cli/src/report) and [`packages/cli/src/cli.ts`](packages/cli/src/cli.ts).
|
|
551
613
|
|
|
@@ -970,7 +970,8 @@ async function inspect(url, options = {}) {
|
|
|
970
970
|
extractor: {
|
|
971
971
|
mode: "static",
|
|
972
972
|
escalated: false,
|
|
973
|
-
escalationReason:
|
|
973
|
+
escalationReason: heuristic.reason,
|
|
974
|
+
escalationBlocked: err.message
|
|
974
975
|
},
|
|
975
976
|
html: { static: staticBody },
|
|
976
977
|
probes: probes2
|
|
@@ -1115,6 +1116,7 @@ function push(out, seen, href, source) {
|
|
|
1115
1116
|
}
|
|
1116
1117
|
|
|
1117
1118
|
// src/lib/core/crawl.ts
|
|
1119
|
+
var MAX_ATTEMPTS = 2;
|
|
1118
1120
|
async function crawl(options) {
|
|
1119
1121
|
const depthLimit = options.depth ?? 1;
|
|
1120
1122
|
const concurrency = Math.max(1, options.concurrency ?? 4);
|
|
@@ -1128,6 +1130,7 @@ async function crawl(options) {
|
|
|
1128
1130
|
pages: [],
|
|
1129
1131
|
visited: [],
|
|
1130
1132
|
errors: [{ url: options.entryUrl, message: "entry URL is not a valid http(s) URL" }],
|
|
1133
|
+
recovered: [],
|
|
1131
1134
|
truncated: false
|
|
1132
1135
|
};
|
|
1133
1136
|
}
|
|
@@ -1143,7 +1146,9 @@ async function crawl(options) {
|
|
|
1143
1146
|
}
|
|
1144
1147
|
const pages = [];
|
|
1145
1148
|
const errors = [];
|
|
1149
|
+
const recovered = [];
|
|
1146
1150
|
let truncated = false;
|
|
1151
|
+
const seenFinal = /* @__PURE__ */ new Set();
|
|
1147
1152
|
while (queue.length > 0) {
|
|
1148
1153
|
if (options.inspectOptions?.signal?.aborted) break;
|
|
1149
1154
|
const wave = queue.splice(0, concurrency);
|
|
@@ -1162,9 +1167,18 @@ async function crawl(options) {
|
|
|
1162
1167
|
const results = await Promise.all(inspections);
|
|
1163
1168
|
for (const result of results) {
|
|
1164
1169
|
if (result.kind === "err") {
|
|
1170
|
+
const attempt = (result.item.attempt ?? 0) + 1;
|
|
1171
|
+
if (attempt < MAX_ATTEMPTS) {
|
|
1172
|
+
queue.push({ ...result.item, attempt });
|
|
1173
|
+
continue;
|
|
1174
|
+
}
|
|
1165
1175
|
errors.push({ url: result.item.url, message: result.error });
|
|
1166
1176
|
continue;
|
|
1167
1177
|
}
|
|
1178
|
+
if (result.item.attempt) recovered.push(result.item.url);
|
|
1179
|
+
const finalUrl = canonicaliseUrl(result.page.fetch.finalUrl) ?? result.page.fetch.finalUrl;
|
|
1180
|
+
if (seenFinal.has(finalUrl)) continue;
|
|
1181
|
+
seenFinal.add(finalUrl);
|
|
1168
1182
|
pages.push(result.page);
|
|
1169
1183
|
if (options.onPage) {
|
|
1170
1184
|
options.onPage(result.page, { visited: pages.length, queued: queue.length });
|
|
@@ -1194,6 +1208,7 @@ async function crawl(options) {
|
|
|
1194
1208
|
pages,
|
|
1195
1209
|
visited: [...visitedSet],
|
|
1196
1210
|
errors,
|
|
1211
|
+
recovered,
|
|
1197
1212
|
truncated
|
|
1198
1213
|
};
|
|
1199
1214
|
}
|
|
@@ -1780,17 +1795,60 @@ function sortIssues(issues) {
|
|
|
1780
1795
|
});
|
|
1781
1796
|
}
|
|
1782
1797
|
|
|
1783
|
-
// src/lib/core/
|
|
1784
|
-
var
|
|
1785
|
-
|
|
1786
|
-
|
|
1787
|
-
|
|
1798
|
+
// src/lib/core/bcp47.ts
|
|
1799
|
+
var languageNames = new Intl.DisplayNames(["en"], { type: "language", fallback: "none" });
|
|
1800
|
+
var regionNames = new Intl.DisplayNames(["en"], { type: "region", fallback: "none" });
|
|
1801
|
+
var scriptNames = new Intl.DisplayNames(["en"], { type: "script", fallback: "none" });
|
|
1802
|
+
var NOT_AN_AUDIENCE_REGION = /* @__PURE__ */ new Set(["ZZ", "XA", "XB"]);
|
|
1803
|
+
var NOT_AN_AUDIENCE_LANGUAGE = /* @__PURE__ */ new Set(["mul", "zxx", "und", "mis"]);
|
|
1804
|
+
var NOT_A_SCRIPT = /* @__PURE__ */ new Set(["Zzzz"]);
|
|
1805
|
+
var ICU_KNOWS_LANGUAGES = languageNames.of("fr") !== void 0 && languageNames.of("qq") === void 0 && regionNames.of("BR") !== void 0;
|
|
1806
|
+
var ICU_UNAVAILABLE_WARNING = "This Node build cannot tell a real language tag from an invented one (small-icu). hreflang tags were checked for shape only, so `locale.invalid` may have missed tags that do not exist.";
|
|
1807
|
+
var BCP47_SHAPE = /^[a-z]{2,3}(-[a-z]{2}|-\d{3})?$/i;
|
|
1788
1808
|
function looksLikeLocaleSegment(segment) {
|
|
1789
|
-
return
|
|
1809
|
+
return BCP47_SHAPE.test(segment);
|
|
1790
1810
|
}
|
|
1791
1811
|
function isValidLocale(tag) {
|
|
1792
1812
|
if (tag === "x-default") return true;
|
|
1793
|
-
return
|
|
1813
|
+
if (!ICU_KNOWS_LANGUAGES) return BCP47_SHAPE.test(tag);
|
|
1814
|
+
let canonical;
|
|
1815
|
+
try {
|
|
1816
|
+
canonical = Intl.getCanonicalLocales(tag)[0] ?? "";
|
|
1817
|
+
} catch {
|
|
1818
|
+
return false;
|
|
1819
|
+
}
|
|
1820
|
+
if (!canonical) return false;
|
|
1821
|
+
let parts;
|
|
1822
|
+
try {
|
|
1823
|
+
parts = new Intl.Locale(canonical);
|
|
1824
|
+
} catch {
|
|
1825
|
+
return false;
|
|
1826
|
+
}
|
|
1827
|
+
const language = (parts.language ?? "").toLowerCase();
|
|
1828
|
+
if (!language) return false;
|
|
1829
|
+
if (NOT_AN_AUDIENCE_LANGUAGE.has(language)) return false;
|
|
1830
|
+
if (languageNames.of(language) === void 0) return false;
|
|
1831
|
+
if (parts.script) {
|
|
1832
|
+
if (NOT_A_SCRIPT.has(parts.script)) return false;
|
|
1833
|
+
if (scriptNames.of(parts.script) === void 0) return false;
|
|
1834
|
+
}
|
|
1835
|
+
if (parts.region) {
|
|
1836
|
+
if (NOT_AN_AUDIENCE_REGION.has(parts.region)) return false;
|
|
1837
|
+
if (regionNames.of(parts.region) === void 0) return false;
|
|
1838
|
+
}
|
|
1839
|
+
return true;
|
|
1840
|
+
}
|
|
1841
|
+
function isKnownLanguageTag(tag) {
|
|
1842
|
+
const primary = (tag.toLowerCase().split("-")[0] ?? "").trim();
|
|
1843
|
+
if (!primary) return false;
|
|
1844
|
+
if (NOT_AN_AUDIENCE_LANGUAGE.has(primary)) return false;
|
|
1845
|
+
if (!ICU_KNOWS_LANGUAGES) return BCP47_SHAPE.test(primary);
|
|
1846
|
+
return languageNames.of(primary) !== void 0;
|
|
1847
|
+
}
|
|
1848
|
+
|
|
1849
|
+
// src/lib/core/i18n.ts
|
|
1850
|
+
function localeIdentity(tag) {
|
|
1851
|
+
return tag.trim().toLowerCase();
|
|
1794
1852
|
}
|
|
1795
1853
|
function buildI18nMatrix(pages, options = {}) {
|
|
1796
1854
|
const inspectedByUrl = /* @__PURE__ */ new Map();
|
|
@@ -1802,6 +1860,9 @@ function buildI18nMatrix(pages, options = {}) {
|
|
|
1802
1860
|
const tag = locale.trim().toLowerCase();
|
|
1803
1861
|
if (tag) locales.add(tag);
|
|
1804
1862
|
}
|
|
1863
|
+
function rowOf(url, pathnameRoute) {
|
|
1864
|
+
return options.clusterRouteOf?.(url) ?? pathnameRoute;
|
|
1865
|
+
}
|
|
1805
1866
|
function record(route, locale, url) {
|
|
1806
1867
|
routes.add(route);
|
|
1807
1868
|
locales.add(locale);
|
|
@@ -1817,7 +1878,7 @@ function buildI18nMatrix(pages, options = {}) {
|
|
|
1817
1878
|
for (const page of pages) {
|
|
1818
1879
|
const url = new URL(page.fetch.finalUrl);
|
|
1819
1880
|
const { route: selfRoute, locale: selfLocale } = splitRoute(url.pathname);
|
|
1820
|
-
record(selfRoute, selfLocale, page.fetch.finalUrl);
|
|
1881
|
+
record(rowOf(page.fetch.finalUrl, selfRoute), selfLocale, page.fetch.finalUrl);
|
|
1821
1882
|
for (const alt of page.links.alternates) {
|
|
1822
1883
|
let altUrl;
|
|
1823
1884
|
try {
|
|
@@ -1827,7 +1888,7 @@ function buildI18nMatrix(pages, options = {}) {
|
|
|
1827
1888
|
}
|
|
1828
1889
|
const { route } = splitRoute(altUrl.pathname);
|
|
1829
1890
|
const locale = alt.isXDefault ? "x-default" : localeIdentity(alt.hreflang);
|
|
1830
|
-
record(route || selfRoute, locale, altUrl.toString());
|
|
1891
|
+
record(rowOf(altUrl.toString(), route || selfRoute), locale, altUrl.toString());
|
|
1831
1892
|
}
|
|
1832
1893
|
}
|
|
1833
1894
|
for (const declared of options.declaredUrls ?? []) {
|
|
@@ -1838,7 +1899,7 @@ function buildI18nMatrix(pages, options = {}) {
|
|
|
1838
1899
|
continue;
|
|
1839
1900
|
}
|
|
1840
1901
|
const { route, locale } = splitRoute(url.pathname);
|
|
1841
|
-
record(route, locale, url.toString());
|
|
1902
|
+
record(rowOf(url.toString(), route), locale, url.toString());
|
|
1842
1903
|
}
|
|
1843
1904
|
const sortedLocales = sortLocales([...locales]);
|
|
1844
1905
|
const sortedRoutes = [...routes].sort((a, b) => a.localeCompare(b));
|
|
@@ -2155,13 +2216,6 @@ function sortSiteIssues(drafts) {
|
|
|
2155
2216
|
}
|
|
2156
2217
|
|
|
2157
2218
|
// src/lib/core/locales.ts
|
|
2158
|
-
var ISO_639_1 = new Set(
|
|
2159
|
-
"aa ab ae af ak am an ar as av ay az ba be bg bh bi bm bn bo br bs ca ce ch co cr cs cu cv cy da de dv dz ee el en eo es et eu fa ff fi fj fo fr fy ga gd gl gn gu gv ha he hi ho hr ht hu hy hz ia id ie ig ii ik io is it iu ja jv ka kg ki kj kk kl km kn ko kr ks ku kv kw ky la lb lg li ln lo lt lu lv mg mh mi mk ml mn mr ms mt my na nb nd ne ng nl nn no nr nv ny oc oj om or os pa pi pl ps pt qu rm rn ro ru rw sa sc sd se sg si sk sl sm sn so sq sr ss st su sv sw ta te tg th ti tk tl tn to tr ts tt tw ty ug uk ur uz ve vi vo wa wo xh yi yo za zh zu".split(" ")
|
|
2160
|
-
);
|
|
2161
|
-
function isKnownLanguageTag(tag) {
|
|
2162
|
-
const primary = tag.toLowerCase().split("-")[0] ?? "";
|
|
2163
|
-
return ISO_639_1.has(primary);
|
|
2164
|
-
}
|
|
2165
2219
|
function localePrefixOf(url) {
|
|
2166
2220
|
let pathname;
|
|
2167
2221
|
try {
|
|
@@ -2242,6 +2296,26 @@ import { gunzipSync } from "zlib";
|
|
|
2242
2296
|
|
|
2243
2297
|
// src/lib/core/sitemap/parse.ts
|
|
2244
2298
|
import * as cheerio2 from "cheerio";
|
|
2299
|
+
function alternatesOf(node) {
|
|
2300
|
+
const out = [];
|
|
2301
|
+
const seen = /* @__PURE__ */ new Set();
|
|
2302
|
+
node.children().each((_, el) => {
|
|
2303
|
+
const tag = "tagName" in el ? el.tagName ?? "" : "";
|
|
2304
|
+
if (tag.split(":").pop()?.toLowerCase() !== "link") return;
|
|
2305
|
+
const attrs = "attribs" in el ? el.attribs : void 0;
|
|
2306
|
+
if (!attrs) return;
|
|
2307
|
+
const rel = (attrs.rel ?? "alternate").trim().toLowerCase();
|
|
2308
|
+
if (rel !== "alternate") return;
|
|
2309
|
+
const hreflang = (attrs.hreflang ?? "").trim();
|
|
2310
|
+
const href = (attrs.href ?? "").trim();
|
|
2311
|
+
if (!hreflang || !href) return;
|
|
2312
|
+
const key = `${hreflang.toLowerCase()}\0${href}`;
|
|
2313
|
+
if (seen.has(key)) return;
|
|
2314
|
+
seen.add(key);
|
|
2315
|
+
out.push({ hreflang, href });
|
|
2316
|
+
});
|
|
2317
|
+
return out;
|
|
2318
|
+
}
|
|
2245
2319
|
function parseSitemap(body) {
|
|
2246
2320
|
const trimmed = body.trim();
|
|
2247
2321
|
if (trimmed.length === 0) {
|
|
@@ -2280,6 +2354,8 @@ function parseSitemap(body) {
|
|
|
2280
2354
|
if (changefreq) entry.changefreq = changefreq;
|
|
2281
2355
|
const priority = node.find("priority").first().text().trim();
|
|
2282
2356
|
if (priority) entry.priority = priority;
|
|
2357
|
+
const alternates = alternatesOf(node);
|
|
2358
|
+
if (alternates.length > 0) entry.alternates = alternates;
|
|
2283
2359
|
urls.push(entry);
|
|
2284
2360
|
});
|
|
2285
2361
|
return { kind: "urlset", wellFormed: true, urls };
|
|
@@ -2499,6 +2575,54 @@ function relaxTlsIfRequested2(enabled) {
|
|
|
2499
2575
|
};
|
|
2500
2576
|
}
|
|
2501
2577
|
|
|
2578
|
+
// src/lib/core/clusters.ts
|
|
2579
|
+
function normalise(raw) {
|
|
2580
|
+
let url;
|
|
2581
|
+
try {
|
|
2582
|
+
url = new URL(raw);
|
|
2583
|
+
} catch {
|
|
2584
|
+
return null;
|
|
2585
|
+
}
|
|
2586
|
+
url.hash = "";
|
|
2587
|
+
const path = url.pathname.length > 1 ? url.pathname.replace(/\/+$/, "") : url.pathname;
|
|
2588
|
+
return `${url.origin}${path}${url.search}`;
|
|
2589
|
+
}
|
|
2590
|
+
var X_DEFAULT = "x-default";
|
|
2591
|
+
function buildClusterIndex(entries) {
|
|
2592
|
+
const routeByUrl = /* @__PURE__ */ new Map();
|
|
2593
|
+
const conflicts = [];
|
|
2594
|
+
const anchors = /* @__PURE__ */ new Set();
|
|
2595
|
+
for (const entry of entries) {
|
|
2596
|
+
const alternates = entry.alternates;
|
|
2597
|
+
if (!alternates || alternates.length === 0) continue;
|
|
2598
|
+
const xDefault = alternates.find((a) => a.hreflang.trim().toLowerCase() === X_DEFAULT);
|
|
2599
|
+
if (!xDefault) continue;
|
|
2600
|
+
const anchorUrl = normalise(xDefault.href);
|
|
2601
|
+
if (!anchorUrl) continue;
|
|
2602
|
+
const { route: anchorRoute } = splitRoute(new URL(anchorUrl).pathname);
|
|
2603
|
+
anchors.add(anchorRoute);
|
|
2604
|
+
const members = [entry.loc, ...alternates.map((a) => a.href)];
|
|
2605
|
+
for (const member of members) {
|
|
2606
|
+
const url = normalise(member);
|
|
2607
|
+
if (!url) continue;
|
|
2608
|
+
const existing = routeByUrl.get(url);
|
|
2609
|
+
if (existing === void 0) {
|
|
2610
|
+
routeByUrl.set(url, anchorRoute);
|
|
2611
|
+
continue;
|
|
2612
|
+
}
|
|
2613
|
+
if (existing !== anchorRoute && !conflicts.includes(url)) conflicts.push(url);
|
|
2614
|
+
}
|
|
2615
|
+
}
|
|
2616
|
+
return {
|
|
2617
|
+
routeOf: (url) => {
|
|
2618
|
+
const key = normalise(url);
|
|
2619
|
+
return key === null ? void 0 : routeByUrl.get(key);
|
|
2620
|
+
},
|
|
2621
|
+
size: anchors.size,
|
|
2622
|
+
conflicts
|
|
2623
|
+
};
|
|
2624
|
+
}
|
|
2625
|
+
|
|
2502
2626
|
// src/lib/core/coverage.ts
|
|
2503
2627
|
var DEFAULT_PER_FAMILY = 3;
|
|
2504
2628
|
var DEFAULT_THRESHOLD = 8;
|
|
@@ -3431,6 +3555,11 @@ function localeOfUrl(url) {
|
|
|
3431
3555
|
}
|
|
3432
3556
|
return null;
|
|
3433
3557
|
}
|
|
3558
|
+
function blockedEscalationWarning(pages) {
|
|
3559
|
+
const blocked = pages.filter((p) => p.extractor.escalationBlocked);
|
|
3560
|
+
if (blocked.length === 0) return null;
|
|
3561
|
+
return `${blocked.length} page(s) looked client-rendered and were judged on their static HTML, because the browser could not be started: ${blocked[0].extractor.escalationBlocked} Their metadata findings may be phantoms \u2014 install Playwright and Chromium, or pass --static to say the static HTML is what you meant to judge.`;
|
|
3562
|
+
}
|
|
3434
3563
|
function deriveTranslationHoles(matrix, ignoreRoutes = []) {
|
|
3435
3564
|
const realLocales = matrix.locales.filter((l) => l !== "x-default");
|
|
3436
3565
|
if (realLocales.length < 2) return [];
|
|
@@ -3497,6 +3626,7 @@ async function runAudit(inputUrl, options = {}) {
|
|
|
3497
3626
|
crawlFallback: false
|
|
3498
3627
|
}).catch(() => void 0);
|
|
3499
3628
|
const allSitemapUrls = (discovery?.urls ?? []).map((u) => u.loc);
|
|
3629
|
+
const clusters = buildClusterIndex(discovery?.urls ?? []);
|
|
3500
3630
|
const coverageMode = options.coverage ?? (allSitemapUrls.length > 0 ? "structural" : "all");
|
|
3501
3631
|
const selection = coverageMode === "structural" && allSitemapUrls.length > 0 ? selectByStructure(allSitemapUrls, { locales: options.locales }) : void 0;
|
|
3502
3632
|
const sitemapUrls = selection?.urls ?? allSitemapUrls;
|
|
@@ -3529,6 +3659,13 @@ async function runAudit(inputUrl, options = {}) {
|
|
|
3529
3659
|
});
|
|
3530
3660
|
const pages = crawlResult.pages;
|
|
3531
3661
|
const warnings = crawlResult.errors.map((e) => `crawl: ${e.url} \u2014 ${e.message}`);
|
|
3662
|
+
if (crawlResult.recovered.length > 0) {
|
|
3663
|
+
warnings.push(
|
|
3664
|
+
`${crawlResult.recovered.length} page(s) failed once and answered on retry: ${crawlResult.recovered.slice(0, 3).join(", ")}${crawlResult.recovered.length > 3 ? `, +${crawlResult.recovered.length - 3} more` : ""}.`
|
|
3665
|
+
);
|
|
3666
|
+
}
|
|
3667
|
+
const escalationWarning = blockedEscalationWarning(pages);
|
|
3668
|
+
if (escalationWarning) warnings.push(escalationWarning);
|
|
3532
3669
|
const okPages = pages.filter((p) => isOkStatus(p.fetch.status));
|
|
3533
3670
|
const documents = okPages.filter(isHtmlPage);
|
|
3534
3671
|
const { kept: htmlPages, dropped: duplicatePages } = dropCanonicalDuplicates(documents);
|
|
@@ -3591,10 +3728,17 @@ async function runAudit(inputUrl, options = {}) {
|
|
|
3591
3728
|
suggestion ? `No sitemap and no --locales: i18n checks are off. Locale-looking prefixes found \u2014 re-run with \`--locales ${suggestion}\` to enable them.` : `No sitemap and no --locales: i18n checks are off. Prefixes seen (${localeAxis.candidates.map((c) => c.tag).join(", ")}) do not look like locales; pass --locales if any of them is one.`
|
|
3592
3729
|
);
|
|
3593
3730
|
}
|
|
3731
|
+
if (!ICU_KNOWS_LANGUAGES) warnings.push(ICU_UNAVAILABLE_WARNING);
|
|
3594
3732
|
const matrix = buildI18nMatrix(htmlPages, {
|
|
3595
3733
|
declaredUrls: sitemapUrls,
|
|
3596
|
-
locales: localeAxis.locales
|
|
3734
|
+
locales: localeAxis.locales,
|
|
3735
|
+
clusterRouteOf: clusters.routeOf
|
|
3597
3736
|
});
|
|
3737
|
+
if (clusters.conflicts.length > 0) {
|
|
3738
|
+
warnings.push(
|
|
3739
|
+
`${clusters.conflicts.length} URL(s) are declared in two different hreflang clusters by the sitemap; the first declaration was kept: ${clusters.conflicts.slice(0, 3).join(", ")}${clusters.conflicts.length > 3 ? `, +${clusters.conflicts.length - 3} more` : ""}.`
|
|
3740
|
+
);
|
|
3741
|
+
}
|
|
3598
3742
|
const holes = localeAxis.multilingual ? deriveTranslationHoles(matrix, options.ignoreHoles) : [];
|
|
3599
3743
|
const ignoredHoles = localeAxis.multilingual && (options.ignoreHoles?.length ?? 0) > 0 ? deriveTranslationHoles(matrix).length - holes.length : 0;
|
|
3600
3744
|
const reciprocity = reciprocityIssues(htmlPages).map((issue) => ({
|
|
@@ -3739,6 +3883,15 @@ async function runAudit(inputUrl, options = {}) {
|
|
|
3739
3883
|
},
|
|
3740
3884
|
...ignoredHoles > 0 ? { ignoredHoles } : {},
|
|
3741
3885
|
...duplicatePages > 0 ? { duplicatePages } : {},
|
|
3886
|
+
// Only when the site declared clusters. A merge that changes which rows
|
|
3887
|
+
// exist must be visible: the alternative is a hole count that moved for
|
|
3888
|
+
// a reason nothing in the report explains.
|
|
3889
|
+
...clusters.size > 0 ? {
|
|
3890
|
+
declaredClusters: {
|
|
3891
|
+
count: clusters.size,
|
|
3892
|
+
...clusters.conflicts.length > 0 ? { conflicts: clusters.conflicts.length } : {}
|
|
3893
|
+
}
|
|
3894
|
+
} : {},
|
|
3742
3895
|
...discovery ? {
|
|
3743
3896
|
sitemap: {
|
|
3744
3897
|
found: discovery.diagnostics.found,
|
|
@@ -4395,9 +4548,12 @@ var Logger = class {
|
|
|
4395
4548
|
};
|
|
4396
4549
|
|
|
4397
4550
|
export {
|
|
4551
|
+
RULES,
|
|
4552
|
+
SITE_RULES,
|
|
4398
4553
|
DEFAULT_PROFILE,
|
|
4399
4554
|
PROFILES,
|
|
4400
4555
|
PROFILE_NAMES,
|
|
4556
|
+
PROSE_RULES,
|
|
4401
4557
|
routeKey,
|
|
4402
4558
|
targetKey,
|
|
4403
4559
|
fingerprint,
|
|
@@ -4410,4 +4566,4 @@ export {
|
|
|
4410
4566
|
summarize,
|
|
4411
4567
|
Logger
|
|
4412
4568
|
};
|
|
4413
|
-
//# sourceMappingURL=chunk-
|
|
4569
|
+
//# sourceMappingURL=chunk-UOKYUKHQ.js.map
|