@goflag/cli 0.2.5 → 0.2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +76 -14
- package/dist/{chunk-5FO2Q3II.js → chunk-UOKYUKHQ.js} +172 -20
- package/dist/chunk-UOKYUKHQ.js.map +1 -0
- package/dist/cli.js +500 -8
- package/dist/cli.js.map +1 -1
- package/dist/index.d.ts +30 -4
- package/dist/index.js +1 -1
- package/package.json +4 -2
- package/rules.json +517 -0
- package/dist/chunk-5FO2Q3II.js.map +0 -1
package/README.md
CHANGED
|
@@ -16,9 +16,14 @@ on the site. A human cannot check those on 400 pages. That is the whole job.
|
|
|
16
16
|
## What it checks
|
|
17
17
|
|
|
18
18
|
- **Broken links** — scrapes every crawled page, dedupes targets globally (a footer link on 500 pages is probed once), and checks each with `HEAD`→`GET` fallback, redirect-chain/loop detection, soft-404 and anti-bot (403/429) triage. Broken links are mapped back to the pages that reference them.
|
|
19
|
-
- **Missing translation pages** — builds a `route × locale` matrix
|
|
19
|
+
- **Missing translation pages** — builds a `route × locale` matrix and flags every route that exists in one locale but is missing in another, plus hreflang reciprocity gaps (`A → B` with no `B → A`), missing `x-default`, and locale tags naming a language, script or region that does not exist. Rows are keyed by pathname, unless your sitemap declares its translation clusters with `xhtml:link` — then those declarations decide which URLs are one page, whatever their slugs, and the run reports `diagnostics.declaredClusters`.
|
|
20
20
|
- **robots.txt policy** — flags a site that forbids crawling while its pages ask to be indexed. The two declarations cannot both hold, and robots.txt wins, so the meta tag is never even read.
|
|
21
21
|
- **SEO metadata** — lints each page's `<head>` for the handful of mistakes that actually hurt in search and social and are invisible in a browser: missing/oversized `<title>` and description, missing/relative canonical, missing `og:title`/`og:description`/`og:image`, missing viewport, and contradictory `robots`/`googlebot`/`X-Robots-Tag` directives.
|
|
22
|
+
- **Unreachable pages** — a page the crawl reached that answered non-2xx, or that never answered at all (timeout, reset, DNS failure — recorded as `status: 0`), is reported in `unreachablePages`. goflag asks a second time, from the back of the queue, before calling a page unreachable. Each one is an `error`-severity finding: it turns the verdict red, counts toward `--max-debt`, and fails the build even under `--fail-on error`. A page that silently dropped out of the audited set is what poisons a baseline, so it is reported rather than warned about.
|
|
23
|
+
|
|
24
|
+
**Which pages get audited.** When goflag finds a sitemap it does not audit every URL in it. It groups URLs that share a path shape — and therefore a template — into families, audits every page that stands alone, and samples three pages per family. A cap answers "how many" and never "which": on a site of thousands of pages built from thirty templates, the first 200 pages a crawl reaches are four templates out of thirty.
|
|
25
|
+
|
|
26
|
+
The trade is stated in every report rather than hidden. Template rules (`canonical.*`, `hreflang.*`, `og.image.missing`) are conclusive on a sampled family; copy rules (`title.length`, `description.length`), broken links and the translation matrix are only conclusive on the pages that were drawn. The terminal prints a `COVERAGE` line, and `diagnostics.coverage` carries `mode`, `considered`, `selected` and every sampled family with its ratio. Pass `--coverage all` to audit what the sitemap lists in order, up to `--max-pages`, which is the pre-0.2.3 behaviour.
|
|
22
27
|
|
|
23
28
|
Pages that declare a `<link rel="canonical">` pointing at another crawled page are excluded from the checks — the site has said they are duplicates, so linting them would multiply every finding by the number of filtered variants. The count is reported as `diagnostics.duplicatePages`.
|
|
24
29
|
|
|
@@ -125,16 +130,57 @@ npx @goflag/cli https://example.com --report report.json # write JSON to a
|
|
|
125
130
|
|
|
126
131
|
The terminal view is just a render of that JSON.
|
|
127
132
|
|
|
133
|
+
### The catalogue is data too
|
|
134
|
+
|
|
135
|
+
```sh
|
|
136
|
+
npx @goflag/cli rules > rules.json
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
`rules` answers a question about goflag rather than about a site: no URL, no
|
|
140
|
+
crawl, no network. Every entry carries its scope, severity and summary; the
|
|
141
|
+
eleven page rules and four prose rules also carry a rigor, the documents they
|
|
142
|
+
cite and — where a remedy is a line of code — a fix snippet. The three site
|
|
143
|
+
rules carry none of those three: `SiteRule` has no rigor field yet, and the
|
|
144
|
+
catalogue emits `rigor: null` with an empty `sources` rather than inventing an
|
|
145
|
+
authority nobody assigned. Write your consumer against that. The same document
|
|
146
|
+
ships inside the package as `rules.json`, if you would rather read it than run
|
|
147
|
+
anything.
|
|
148
|
+
|
|
149
|
+
It deliberately omits the message a finding prints — that is built at audit time
|
|
150
|
+
from what the page actually says, so a static copy would be a sample rather than
|
|
151
|
+
a fact, and a sample presented as the truth is how this project's own
|
|
152
|
+
documentation site came to quote a message the engine had stopped printing.
|
|
153
|
+
|
|
128
154
|
### Options
|
|
129
155
|
|
|
130
156
|
```
|
|
131
157
|
--json Print the JSON report to stdout (nothing else).
|
|
158
|
+
--summary, -s Roll findings up (dedup by link/rule/code). Pairs with
|
|
159
|
+
--json for a compact payload; --report always writes
|
|
160
|
+
the full report regardless.
|
|
132
161
|
--report <file> Write the JSON report to <file>.
|
|
133
|
-
--depth <n>
|
|
134
|
-
|
|
162
|
+
--depth <n> How far to follow links out of each page (0 = follow
|
|
163
|
+
none). Sitemap URLs are seeded regardless, so --depth 0
|
|
164
|
+
alone is not "entry page only" — add --no-sitemap.
|
|
165
|
+
Default: 2.
|
|
166
|
+
--max-pages <n> Page budget for the crawl. Default: 200. A hard cap
|
|
167
|
+
under --coverage all; under structural coverage the
|
|
168
|
+
selection wins and the budget is max(<n>, selected + 5),
|
|
169
|
+
so a lower value cannot cut the run short.
|
|
170
|
+
--coverage <mode> How pages are chosen: "structural" (default when a
|
|
171
|
+
sitemap is found) keeps every standalone page and
|
|
172
|
+
samples three pages per family of template-generated
|
|
173
|
+
pages; "all" audits what the sitemap lists, in order,
|
|
174
|
+
up to --max-pages.
|
|
135
175
|
--include <glob> Only crawl paths matching <glob> (repeatable).
|
|
136
176
|
--exclude <glob> Skip paths matching <glob> (repeatable).
|
|
137
177
|
--locales <list> Comma-separated locales the site serves ("fr,en,pt-br").
|
|
178
|
+
Unioned with the prefixes your sitemap shows, never
|
|
179
|
+
substituted for them, so a locale the sitemap serves
|
|
180
|
+
stays on the axis whether or not you list it. Also what
|
|
181
|
+
folds /en/… and /fr/… into one route family for
|
|
182
|
+
structural coverage — pass it on any locale-prefixed
|
|
183
|
+
site.
|
|
138
184
|
--ignore-holes <glob> Route deliberately not translated everywhere
|
|
139
185
|
(repeatable).
|
|
140
186
|
--no-sitemap Do not discover the sitemap; crawl from <url> only.
|
|
@@ -160,7 +206,11 @@ The terminal view is just a render of that JSON.
|
|
|
160
206
|
--no-external Do not probe off-origin (external) links.
|
|
161
207
|
--static Static HTML only; never launch headless Chromium.
|
|
162
208
|
--allow-insecure-tls Accept self-signed / invalid TLS (localhost, tunnels).
|
|
163
|
-
--timeout <ms> Per-request timeout in ms
|
|
209
|
+
--timeout <ms> Per-request timeout in ms, for page fetches and link
|
|
210
|
+
probes alike. Unset, the defaults differ: 8000 for link
|
|
211
|
+
probes, 15000 for page fetches.
|
|
212
|
+
--verbose, -V Log every page as it is analyzed.
|
|
213
|
+
--quiet, -q Suppress the live progress output.
|
|
164
214
|
--no-color Disable coloured output.
|
|
165
215
|
-h, --help Show help.
|
|
166
216
|
-v, --version Show the version.
|
|
@@ -351,9 +401,16 @@ goflag https://example.com --locales fr,en,pt-br
|
|
|
351
401
|
|
|
352
402
|
When a site publishes no usable sitemap and you pass no `--locales`, goflag
|
|
353
403
|
does **not** guess the locale axis — it reports the prefixes it saw with the
|
|
354
|
-
evidence for each and turns the
|
|
355
|
-
|
|
356
|
-
|
|
404
|
+
evidence for each and turns the axis-dependent checks off (holes,
|
|
405
|
+
`hreflang.missing`, `hreflang.sitemap-mismatch`; the reciprocity checks read the
|
|
406
|
+
markup and keep running). Guessing from path shape alone once turned `/cv` (a CV
|
|
407
|
+
page served in French) into a locale, because `cv` is a registered ISO 639-1
|
|
408
|
+
code, and produced 31 findings that were never real.
|
|
409
|
+
|
|
410
|
+
Pass `--locales` even when the sitemap is fine. It is the axis goflag folds when
|
|
411
|
+
it groups URLs into route families, so without it a site with many top-level
|
|
412
|
+
sections under `/en` can have those sections sampled as one family instead of
|
|
413
|
+
audited.
|
|
357
414
|
|
|
358
415
|
Some pages are meant to exist in one locale and not another — a
|
|
359
416
|
jurisdiction-specific legal notice, a post written for one market. A site has
|
|
@@ -398,7 +455,7 @@ Three rules, whichever you use:
|
|
|
398
455
|
stages: [build, deploy, audit]
|
|
399
456
|
|
|
400
457
|
variables:
|
|
401
|
-
GOFLAG_VERSION: "0.
|
|
458
|
+
GOFLAG_VERSION: "0.2.7"
|
|
402
459
|
|
|
403
460
|
# Before the merge: build the branch, let goflag boot it, audit that.
|
|
404
461
|
seo:mr:
|
|
@@ -412,6 +469,7 @@ seo:mr:
|
|
|
412
469
|
- npx --yes "@goflag/cli@$GOFLAG_VERSION" http://localhost:3000
|
|
413
470
|
--start "pnpm start" --static --no-external
|
|
414
471
|
--baseline .goflag/baseline.json --regressions-only --max-debt 41
|
|
472
|
+
--report goflag-report.json
|
|
415
473
|
artifacts:
|
|
416
474
|
when: always
|
|
417
475
|
paths: [goflag-report.json]
|
|
@@ -449,9 +507,10 @@ jobs:
|
|
|
449
507
|
- run: corepack enable && pnpm install --frozen-lockfile
|
|
450
508
|
- run: pnpm build
|
|
451
509
|
- run: |
|
|
452
|
-
npx --yes @goflag/cli@0.
|
|
510
|
+
npx --yes @goflag/cli@0.2.7 http://localhost:3000 \
|
|
453
511
|
--start "pnpm start" --static --no-external \
|
|
454
|
-
--baseline .goflag/baseline.json --regressions-only --max-debt 41
|
|
512
|
+
--baseline .goflag/baseline.json --regressions-only --max-debt 41 \
|
|
513
|
+
--report goflag-report.json
|
|
455
514
|
- uses: actions/upload-artifact@v4
|
|
456
515
|
if: always()
|
|
457
516
|
with:
|
|
@@ -484,7 +543,8 @@ you what it accepted, and that number belongs in the commit message.
|
|
|
484
543
|
import { runAudit } from "@goflag/cli";
|
|
485
544
|
|
|
486
545
|
const report = await runAudit("https://example.com", { depth: 2 });
|
|
487
|
-
console.log(report.summary);
|
|
546
|
+
console.log(report.summary);
|
|
547
|
+
// { brokenLinks, missingTranslations, seoIssues, siteIssues, unreachablePages, verdict }
|
|
488
548
|
```
|
|
489
549
|
|
|
490
550
|
`runAudit` returns the same `GoflagReport` object the CLI emits.
|
|
@@ -543,9 +603,11 @@ pnpm typecheck # tsc --noEmit
|
|
|
543
603
|
pnpm format # prettier
|
|
544
604
|
```
|
|
545
605
|
|
|
546
|
-
`
|
|
547
|
-
|
|
548
|
-
|
|
606
|
+
`format` and `format:check` run once over the whole repository. `lint` runs
|
|
607
|
+
eslint once at the root and then fans out to any package that defines its own
|
|
608
|
+
`lint` script — `apps/website` does, with the Next config — and the rest fan out
|
|
609
|
+
across workspace packages with `pnpm -r`. To work on one package, filter:
|
|
610
|
+
`pnpm --filter @goflag/cli test`.
|
|
549
611
|
|
|
550
612
|
The engine is framework-agnostic and lives in [`packages/cli/src/lib/core/`](packages/cli/src/lib/core) (crawl, fetch/extract, link audit, i18n) and [`packages/cli/src/lib/rules/`](packages/cli/src/lib/rules) (SEO checks). The CLI shell — orchestration, the `GoflagReport` schema, and rendering — lives in [`packages/cli/src/report/`](packages/cli/src/report) and [`packages/cli/src/cli.ts`](packages/cli/src/cli.ts).
|
|
551
613
|
|
|
@@ -970,7 +970,8 @@ async function inspect(url, options = {}) {
|
|
|
970
970
|
extractor: {
|
|
971
971
|
mode: "static",
|
|
972
972
|
escalated: false,
|
|
973
|
-
escalationReason:
|
|
973
|
+
escalationReason: heuristic.reason,
|
|
974
|
+
escalationBlocked: err.message
|
|
974
975
|
},
|
|
975
976
|
html: { static: staticBody },
|
|
976
977
|
probes: probes2
|
|
@@ -1115,6 +1116,7 @@ function push(out, seen, href, source) {
|
|
|
1115
1116
|
}
|
|
1116
1117
|
|
|
1117
1118
|
// src/lib/core/crawl.ts
|
|
1119
|
+
var MAX_ATTEMPTS = 2;
|
|
1118
1120
|
async function crawl(options) {
|
|
1119
1121
|
const depthLimit = options.depth ?? 1;
|
|
1120
1122
|
const concurrency = Math.max(1, options.concurrency ?? 4);
|
|
@@ -1128,6 +1130,7 @@ async function crawl(options) {
|
|
|
1128
1130
|
pages: [],
|
|
1129
1131
|
visited: [],
|
|
1130
1132
|
errors: [{ url: options.entryUrl, message: "entry URL is not a valid http(s) URL" }],
|
|
1133
|
+
recovered: [],
|
|
1131
1134
|
truncated: false
|
|
1132
1135
|
};
|
|
1133
1136
|
}
|
|
@@ -1143,6 +1146,7 @@ async function crawl(options) {
|
|
|
1143
1146
|
}
|
|
1144
1147
|
const pages = [];
|
|
1145
1148
|
const errors = [];
|
|
1149
|
+
const recovered = [];
|
|
1146
1150
|
let truncated = false;
|
|
1147
1151
|
const seenFinal = /* @__PURE__ */ new Set();
|
|
1148
1152
|
while (queue.length > 0) {
|
|
@@ -1163,9 +1167,15 @@ async function crawl(options) {
|
|
|
1163
1167
|
const results = await Promise.all(inspections);
|
|
1164
1168
|
for (const result of results) {
|
|
1165
1169
|
if (result.kind === "err") {
|
|
1170
|
+
const attempt = (result.item.attempt ?? 0) + 1;
|
|
1171
|
+
if (attempt < MAX_ATTEMPTS) {
|
|
1172
|
+
queue.push({ ...result.item, attempt });
|
|
1173
|
+
continue;
|
|
1174
|
+
}
|
|
1166
1175
|
errors.push({ url: result.item.url, message: result.error });
|
|
1167
1176
|
continue;
|
|
1168
1177
|
}
|
|
1178
|
+
if (result.item.attempt) recovered.push(result.item.url);
|
|
1169
1179
|
const finalUrl = canonicaliseUrl(result.page.fetch.finalUrl) ?? result.page.fetch.finalUrl;
|
|
1170
1180
|
if (seenFinal.has(finalUrl)) continue;
|
|
1171
1181
|
seenFinal.add(finalUrl);
|
|
@@ -1198,6 +1208,7 @@ async function crawl(options) {
|
|
|
1198
1208
|
pages,
|
|
1199
1209
|
visited: [...visitedSet],
|
|
1200
1210
|
errors,
|
|
1211
|
+
recovered,
|
|
1201
1212
|
truncated
|
|
1202
1213
|
};
|
|
1203
1214
|
}
|
|
@@ -1784,17 +1795,60 @@ function sortIssues(issues) {
|
|
|
1784
1795
|
});
|
|
1785
1796
|
}
|
|
1786
1797
|
|
|
1787
|
-
// src/lib/core/
|
|
1788
|
-
var
|
|
1789
|
-
|
|
1790
|
-
|
|
1791
|
-
|
|
1798
|
+
// src/lib/core/bcp47.ts
|
|
1799
|
+
var languageNames = new Intl.DisplayNames(["en"], { type: "language", fallback: "none" });
|
|
1800
|
+
var regionNames = new Intl.DisplayNames(["en"], { type: "region", fallback: "none" });
|
|
1801
|
+
var scriptNames = new Intl.DisplayNames(["en"], { type: "script", fallback: "none" });
|
|
1802
|
+
var NOT_AN_AUDIENCE_REGION = /* @__PURE__ */ new Set(["ZZ", "XA", "XB"]);
|
|
1803
|
+
var NOT_AN_AUDIENCE_LANGUAGE = /* @__PURE__ */ new Set(["mul", "zxx", "und", "mis"]);
|
|
1804
|
+
var NOT_A_SCRIPT = /* @__PURE__ */ new Set(["Zzzz"]);
|
|
1805
|
+
var ICU_KNOWS_LANGUAGES = languageNames.of("fr") !== void 0 && languageNames.of("qq") === void 0 && regionNames.of("BR") !== void 0;
|
|
1806
|
+
var ICU_UNAVAILABLE_WARNING = "This Node build cannot tell a real language tag from an invented one (small-icu). hreflang tags were checked for shape only, so `locale.invalid` may have missed tags that do not exist.";
|
|
1807
|
+
var BCP47_SHAPE = /^[a-z]{2,3}(-[a-z]{2}|-\d{3})?$/i;
|
|
1792
1808
|
function looksLikeLocaleSegment(segment) {
|
|
1793
|
-
return
|
|
1809
|
+
return BCP47_SHAPE.test(segment);
|
|
1794
1810
|
}
|
|
1795
1811
|
function isValidLocale(tag) {
|
|
1796
1812
|
if (tag === "x-default") return true;
|
|
1797
|
-
return
|
|
1813
|
+
if (!ICU_KNOWS_LANGUAGES) return BCP47_SHAPE.test(tag);
|
|
1814
|
+
let canonical;
|
|
1815
|
+
try {
|
|
1816
|
+
canonical = Intl.getCanonicalLocales(tag)[0] ?? "";
|
|
1817
|
+
} catch {
|
|
1818
|
+
return false;
|
|
1819
|
+
}
|
|
1820
|
+
if (!canonical) return false;
|
|
1821
|
+
let parts;
|
|
1822
|
+
try {
|
|
1823
|
+
parts = new Intl.Locale(canonical);
|
|
1824
|
+
} catch {
|
|
1825
|
+
return false;
|
|
1826
|
+
}
|
|
1827
|
+
const language = (parts.language ?? "").toLowerCase();
|
|
1828
|
+
if (!language) return false;
|
|
1829
|
+
if (NOT_AN_AUDIENCE_LANGUAGE.has(language)) return false;
|
|
1830
|
+
if (languageNames.of(language) === void 0) return false;
|
|
1831
|
+
if (parts.script) {
|
|
1832
|
+
if (NOT_A_SCRIPT.has(parts.script)) return false;
|
|
1833
|
+
if (scriptNames.of(parts.script) === void 0) return false;
|
|
1834
|
+
}
|
|
1835
|
+
if (parts.region) {
|
|
1836
|
+
if (NOT_AN_AUDIENCE_REGION.has(parts.region)) return false;
|
|
1837
|
+
if (regionNames.of(parts.region) === void 0) return false;
|
|
1838
|
+
}
|
|
1839
|
+
return true;
|
|
1840
|
+
}
|
|
1841
|
+
function isKnownLanguageTag(tag) {
|
|
1842
|
+
const primary = (tag.toLowerCase().split("-")[0] ?? "").trim();
|
|
1843
|
+
if (!primary) return false;
|
|
1844
|
+
if (NOT_AN_AUDIENCE_LANGUAGE.has(primary)) return false;
|
|
1845
|
+
if (!ICU_KNOWS_LANGUAGES) return BCP47_SHAPE.test(primary);
|
|
1846
|
+
return languageNames.of(primary) !== void 0;
|
|
1847
|
+
}
|
|
1848
|
+
|
|
1849
|
+
// src/lib/core/i18n.ts
|
|
1850
|
+
function localeIdentity(tag) {
|
|
1851
|
+
return tag.trim().toLowerCase();
|
|
1798
1852
|
}
|
|
1799
1853
|
function buildI18nMatrix(pages, options = {}) {
|
|
1800
1854
|
const inspectedByUrl = /* @__PURE__ */ new Map();
|
|
@@ -1806,6 +1860,9 @@ function buildI18nMatrix(pages, options = {}) {
|
|
|
1806
1860
|
const tag = locale.trim().toLowerCase();
|
|
1807
1861
|
if (tag) locales.add(tag);
|
|
1808
1862
|
}
|
|
1863
|
+
function rowOf(url, pathnameRoute) {
|
|
1864
|
+
return options.clusterRouteOf?.(url) ?? pathnameRoute;
|
|
1865
|
+
}
|
|
1809
1866
|
function record(route, locale, url) {
|
|
1810
1867
|
routes.add(route);
|
|
1811
1868
|
locales.add(locale);
|
|
@@ -1821,7 +1878,7 @@ function buildI18nMatrix(pages, options = {}) {
|
|
|
1821
1878
|
for (const page of pages) {
|
|
1822
1879
|
const url = new URL(page.fetch.finalUrl);
|
|
1823
1880
|
const { route: selfRoute, locale: selfLocale } = splitRoute(url.pathname);
|
|
1824
|
-
record(selfRoute, selfLocale, page.fetch.finalUrl);
|
|
1881
|
+
record(rowOf(page.fetch.finalUrl, selfRoute), selfLocale, page.fetch.finalUrl);
|
|
1825
1882
|
for (const alt of page.links.alternates) {
|
|
1826
1883
|
let altUrl;
|
|
1827
1884
|
try {
|
|
@@ -1831,7 +1888,7 @@ function buildI18nMatrix(pages, options = {}) {
|
|
|
1831
1888
|
}
|
|
1832
1889
|
const { route } = splitRoute(altUrl.pathname);
|
|
1833
1890
|
const locale = alt.isXDefault ? "x-default" : localeIdentity(alt.hreflang);
|
|
1834
|
-
record(route || selfRoute, locale, altUrl.toString());
|
|
1891
|
+
record(rowOf(altUrl.toString(), route || selfRoute), locale, altUrl.toString());
|
|
1835
1892
|
}
|
|
1836
1893
|
}
|
|
1837
1894
|
for (const declared of options.declaredUrls ?? []) {
|
|
@@ -1842,7 +1899,7 @@ function buildI18nMatrix(pages, options = {}) {
|
|
|
1842
1899
|
continue;
|
|
1843
1900
|
}
|
|
1844
1901
|
const { route, locale } = splitRoute(url.pathname);
|
|
1845
|
-
record(route, locale, url.toString());
|
|
1902
|
+
record(rowOf(url.toString(), route), locale, url.toString());
|
|
1846
1903
|
}
|
|
1847
1904
|
const sortedLocales = sortLocales([...locales]);
|
|
1848
1905
|
const sortedRoutes = [...routes].sort((a, b) => a.localeCompare(b));
|
|
@@ -2159,13 +2216,6 @@ function sortSiteIssues(drafts) {
|
|
|
2159
2216
|
}
|
|
2160
2217
|
|
|
2161
2218
|
// src/lib/core/locales.ts
|
|
2162
|
-
var ISO_639_1 = new Set(
|
|
2163
|
-
"aa ab ae af ak am an ar as av ay az ba be bg bh bi bm bn bo br bs ca ce ch co cr cs cu cv cy da de dv dz ee el en eo es et eu fa ff fi fj fo fr fy ga gd gl gn gu gv ha he hi ho hr ht hu hy hz ia id ie ig ii ik io is it iu ja jv ka kg ki kj kk kl km kn ko kr ks ku kv kw ky la lb lg li ln lo lt lu lv mg mh mi mk ml mn mr ms mt my na nb nd ne ng nl nn no nr nv ny oc oj om or os pa pi pl ps pt qu rm rn ro ru rw sa sc sd se sg si sk sl sm sn so sq sr ss st su sv sw ta te tg th ti tk tl tn to tr ts tt tw ty ug uk ur uz ve vi vo wa wo xh yi yo za zh zu".split(" ")
|
|
2164
|
-
);
|
|
2165
|
-
function isKnownLanguageTag(tag) {
|
|
2166
|
-
const primary = tag.toLowerCase().split("-")[0] ?? "";
|
|
2167
|
-
return ISO_639_1.has(primary);
|
|
2168
|
-
}
|
|
2169
2219
|
function localePrefixOf(url) {
|
|
2170
2220
|
let pathname;
|
|
2171
2221
|
try {
|
|
@@ -2246,6 +2296,26 @@ import { gunzipSync } from "zlib";
|
|
|
2246
2296
|
|
|
2247
2297
|
// src/lib/core/sitemap/parse.ts
|
|
2248
2298
|
import * as cheerio2 from "cheerio";
|
|
2299
|
+
function alternatesOf(node) {
|
|
2300
|
+
const out = [];
|
|
2301
|
+
const seen = /* @__PURE__ */ new Set();
|
|
2302
|
+
node.children().each((_, el) => {
|
|
2303
|
+
const tag = "tagName" in el ? el.tagName ?? "" : "";
|
|
2304
|
+
if (tag.split(":").pop()?.toLowerCase() !== "link") return;
|
|
2305
|
+
const attrs = "attribs" in el ? el.attribs : void 0;
|
|
2306
|
+
if (!attrs) return;
|
|
2307
|
+
const rel = (attrs.rel ?? "alternate").trim().toLowerCase();
|
|
2308
|
+
if (rel !== "alternate") return;
|
|
2309
|
+
const hreflang = (attrs.hreflang ?? "").trim();
|
|
2310
|
+
const href = (attrs.href ?? "").trim();
|
|
2311
|
+
if (!hreflang || !href) return;
|
|
2312
|
+
const key = `${hreflang.toLowerCase()}\0${href}`;
|
|
2313
|
+
if (seen.has(key)) return;
|
|
2314
|
+
seen.add(key);
|
|
2315
|
+
out.push({ hreflang, href });
|
|
2316
|
+
});
|
|
2317
|
+
return out;
|
|
2318
|
+
}
|
|
2249
2319
|
function parseSitemap(body) {
|
|
2250
2320
|
const trimmed = body.trim();
|
|
2251
2321
|
if (trimmed.length === 0) {
|
|
@@ -2284,6 +2354,8 @@ function parseSitemap(body) {
|
|
|
2284
2354
|
if (changefreq) entry.changefreq = changefreq;
|
|
2285
2355
|
const priority = node.find("priority").first().text().trim();
|
|
2286
2356
|
if (priority) entry.priority = priority;
|
|
2357
|
+
const alternates = alternatesOf(node);
|
|
2358
|
+
if (alternates.length > 0) entry.alternates = alternates;
|
|
2287
2359
|
urls.push(entry);
|
|
2288
2360
|
});
|
|
2289
2361
|
return { kind: "urlset", wellFormed: true, urls };
|
|
@@ -2503,6 +2575,54 @@ function relaxTlsIfRequested2(enabled) {
|
|
|
2503
2575
|
};
|
|
2504
2576
|
}
|
|
2505
2577
|
|
|
2578
|
+
// src/lib/core/clusters.ts
|
|
2579
|
+
function normalise(raw) {
|
|
2580
|
+
let url;
|
|
2581
|
+
try {
|
|
2582
|
+
url = new URL(raw);
|
|
2583
|
+
} catch {
|
|
2584
|
+
return null;
|
|
2585
|
+
}
|
|
2586
|
+
url.hash = "";
|
|
2587
|
+
const path = url.pathname.length > 1 ? url.pathname.replace(/\/+$/, "") : url.pathname;
|
|
2588
|
+
return `${url.origin}${path}${url.search}`;
|
|
2589
|
+
}
|
|
2590
|
+
var X_DEFAULT = "x-default";
|
|
2591
|
+
function buildClusterIndex(entries) {
|
|
2592
|
+
const routeByUrl = /* @__PURE__ */ new Map();
|
|
2593
|
+
const conflicts = [];
|
|
2594
|
+
const anchors = /* @__PURE__ */ new Set();
|
|
2595
|
+
for (const entry of entries) {
|
|
2596
|
+
const alternates = entry.alternates;
|
|
2597
|
+
if (!alternates || alternates.length === 0) continue;
|
|
2598
|
+
const xDefault = alternates.find((a) => a.hreflang.trim().toLowerCase() === X_DEFAULT);
|
|
2599
|
+
if (!xDefault) continue;
|
|
2600
|
+
const anchorUrl = normalise(xDefault.href);
|
|
2601
|
+
if (!anchorUrl) continue;
|
|
2602
|
+
const { route: anchorRoute } = splitRoute(new URL(anchorUrl).pathname);
|
|
2603
|
+
anchors.add(anchorRoute);
|
|
2604
|
+
const members = [entry.loc, ...alternates.map((a) => a.href)];
|
|
2605
|
+
for (const member of members) {
|
|
2606
|
+
const url = normalise(member);
|
|
2607
|
+
if (!url) continue;
|
|
2608
|
+
const existing = routeByUrl.get(url);
|
|
2609
|
+
if (existing === void 0) {
|
|
2610
|
+
routeByUrl.set(url, anchorRoute);
|
|
2611
|
+
continue;
|
|
2612
|
+
}
|
|
2613
|
+
if (existing !== anchorRoute && !conflicts.includes(url)) conflicts.push(url);
|
|
2614
|
+
}
|
|
2615
|
+
}
|
|
2616
|
+
return {
|
|
2617
|
+
routeOf: (url) => {
|
|
2618
|
+
const key = normalise(url);
|
|
2619
|
+
return key === null ? void 0 : routeByUrl.get(key);
|
|
2620
|
+
},
|
|
2621
|
+
size: anchors.size,
|
|
2622
|
+
conflicts
|
|
2623
|
+
};
|
|
2624
|
+
}
|
|
2625
|
+
|
|
2506
2626
|
// src/lib/core/coverage.ts
|
|
2507
2627
|
var DEFAULT_PER_FAMILY = 3;
|
|
2508
2628
|
var DEFAULT_THRESHOLD = 8;
|
|
@@ -3435,6 +3555,11 @@ function localeOfUrl(url) {
|
|
|
3435
3555
|
}
|
|
3436
3556
|
return null;
|
|
3437
3557
|
}
|
|
3558
|
+
function blockedEscalationWarning(pages) {
|
|
3559
|
+
const blocked = pages.filter((p) => p.extractor.escalationBlocked);
|
|
3560
|
+
if (blocked.length === 0) return null;
|
|
3561
|
+
return `${blocked.length} page(s) looked client-rendered and were judged on their static HTML, because the browser could not be started: ${blocked[0].extractor.escalationBlocked} Their metadata findings may be phantoms \u2014 install Playwright and Chromium, or pass --static to say the static HTML is what you meant to judge.`;
|
|
3562
|
+
}
|
|
3438
3563
|
function deriveTranslationHoles(matrix, ignoreRoutes = []) {
|
|
3439
3564
|
const realLocales = matrix.locales.filter((l) => l !== "x-default");
|
|
3440
3565
|
if (realLocales.length < 2) return [];
|
|
@@ -3501,6 +3626,7 @@ async function runAudit(inputUrl, options = {}) {
|
|
|
3501
3626
|
crawlFallback: false
|
|
3502
3627
|
}).catch(() => void 0);
|
|
3503
3628
|
const allSitemapUrls = (discovery?.urls ?? []).map((u) => u.loc);
|
|
3629
|
+
const clusters = buildClusterIndex(discovery?.urls ?? []);
|
|
3504
3630
|
const coverageMode = options.coverage ?? (allSitemapUrls.length > 0 ? "structural" : "all");
|
|
3505
3631
|
const selection = coverageMode === "structural" && allSitemapUrls.length > 0 ? selectByStructure(allSitemapUrls, { locales: options.locales }) : void 0;
|
|
3506
3632
|
const sitemapUrls = selection?.urls ?? allSitemapUrls;
|
|
@@ -3533,6 +3659,13 @@ async function runAudit(inputUrl, options = {}) {
|
|
|
3533
3659
|
});
|
|
3534
3660
|
const pages = crawlResult.pages;
|
|
3535
3661
|
const warnings = crawlResult.errors.map((e) => `crawl: ${e.url} \u2014 ${e.message}`);
|
|
3662
|
+
if (crawlResult.recovered.length > 0) {
|
|
3663
|
+
warnings.push(
|
|
3664
|
+
`${crawlResult.recovered.length} page(s) failed once and answered on retry: ${crawlResult.recovered.slice(0, 3).join(", ")}${crawlResult.recovered.length > 3 ? `, +${crawlResult.recovered.length - 3} more` : ""}.`
|
|
3665
|
+
);
|
|
3666
|
+
}
|
|
3667
|
+
const escalationWarning = blockedEscalationWarning(pages);
|
|
3668
|
+
if (escalationWarning) warnings.push(escalationWarning);
|
|
3536
3669
|
const okPages = pages.filter((p) => isOkStatus(p.fetch.status));
|
|
3537
3670
|
const documents = okPages.filter(isHtmlPage);
|
|
3538
3671
|
const { kept: htmlPages, dropped: duplicatePages } = dropCanonicalDuplicates(documents);
|
|
@@ -3595,10 +3728,17 @@ async function runAudit(inputUrl, options = {}) {
|
|
|
3595
3728
|
suggestion ? `No sitemap and no --locales: i18n checks are off. Locale-looking prefixes found \u2014 re-run with \`--locales ${suggestion}\` to enable them.` : `No sitemap and no --locales: i18n checks are off. Prefixes seen (${localeAxis.candidates.map((c) => c.tag).join(", ")}) do not look like locales; pass --locales if any of them is one.`
|
|
3596
3729
|
);
|
|
3597
3730
|
}
|
|
3731
|
+
if (!ICU_KNOWS_LANGUAGES) warnings.push(ICU_UNAVAILABLE_WARNING);
|
|
3598
3732
|
const matrix = buildI18nMatrix(htmlPages, {
|
|
3599
3733
|
declaredUrls: sitemapUrls,
|
|
3600
|
-
locales: localeAxis.locales
|
|
3734
|
+
locales: localeAxis.locales,
|
|
3735
|
+
clusterRouteOf: clusters.routeOf
|
|
3601
3736
|
});
|
|
3737
|
+
if (clusters.conflicts.length > 0) {
|
|
3738
|
+
warnings.push(
|
|
3739
|
+
`${clusters.conflicts.length} URL(s) are declared in two different hreflang clusters by the sitemap; the first declaration was kept: ${clusters.conflicts.slice(0, 3).join(", ")}${clusters.conflicts.length > 3 ? `, +${clusters.conflicts.length - 3} more` : ""}.`
|
|
3740
|
+
);
|
|
3741
|
+
}
|
|
3602
3742
|
const holes = localeAxis.multilingual ? deriveTranslationHoles(matrix, options.ignoreHoles) : [];
|
|
3603
3743
|
const ignoredHoles = localeAxis.multilingual && (options.ignoreHoles?.length ?? 0) > 0 ? deriveTranslationHoles(matrix).length - holes.length : 0;
|
|
3604
3744
|
const reciprocity = reciprocityIssues(htmlPages).map((issue) => ({
|
|
@@ -3743,6 +3883,15 @@ async function runAudit(inputUrl, options = {}) {
|
|
|
3743
3883
|
},
|
|
3744
3884
|
...ignoredHoles > 0 ? { ignoredHoles } : {},
|
|
3745
3885
|
...duplicatePages > 0 ? { duplicatePages } : {},
|
|
3886
|
+
// Only when the site declared clusters. A merge that changes which rows
|
|
3887
|
+
// exist must be visible: the alternative is a hole count that moved for
|
|
3888
|
+
// a reason nothing in the report explains.
|
|
3889
|
+
...clusters.size > 0 ? {
|
|
3890
|
+
declaredClusters: {
|
|
3891
|
+
count: clusters.size,
|
|
3892
|
+
...clusters.conflicts.length > 0 ? { conflicts: clusters.conflicts.length } : {}
|
|
3893
|
+
}
|
|
3894
|
+
} : {},
|
|
3746
3895
|
...discovery ? {
|
|
3747
3896
|
sitemap: {
|
|
3748
3897
|
found: discovery.diagnostics.found,
|
|
@@ -4399,9 +4548,12 @@ var Logger = class {
|
|
|
4399
4548
|
};
|
|
4400
4549
|
|
|
4401
4550
|
export {
|
|
4551
|
+
RULES,
|
|
4552
|
+
SITE_RULES,
|
|
4402
4553
|
DEFAULT_PROFILE,
|
|
4403
4554
|
PROFILES,
|
|
4404
4555
|
PROFILE_NAMES,
|
|
4556
|
+
PROSE_RULES,
|
|
4405
4557
|
routeKey,
|
|
4406
4558
|
targetKey,
|
|
4407
4559
|
fingerprint,
|
|
@@ -4414,4 +4566,4 @@ export {
|
|
|
4414
4566
|
summarize,
|
|
4415
4567
|
Logger
|
|
4416
4568
|
};
|
|
4417
|
-
//# sourceMappingURL=chunk-
|
|
4569
|
+
//# sourceMappingURL=chunk-UOKYUKHQ.js.map
|