bfsg-scanner 0.2.2 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +31 -1
- package/README.md +15 -0
- package/dist/cli/run.js +10 -1
- package/dist/cli/run.js.map +1 -1
- package/dist/config/schema.d.ts +1 -0
- package/dist/config/schema.js +10 -0
- package/dist/config/schema.js.map +1 -1
- package/dist/discovery/discover-site.d.ts +12 -1
- package/dist/discovery/discover-site.js +28 -4
- package/dist/discovery/discover-site.js.map +1 -1
- package/dist/discovery/discover.d.ts +16 -4
- package/dist/discovery/discover.js +37 -10
- package/dist/discovery/discover.js.map +1 -1
- package/docs/adr/0001-page-discovery-strategy.md +2 -1
- package/docs/adr/0012-off-origin-sitemap-entries.md +54 -0
- package/package.json +8 -8
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,35 @@ follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.3.0] - 2026-10-17
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `sitemapOrigin` config key / `--sitemap-origin <origin>` flag — the origin
|
|
14
|
+
a sitemap was built for, when it is not `baseUrl`'s. Its entries and
|
|
15
|
+
sub-sitemaps are rewritten onto `baseUrl`'s origin, so a local or staging
|
|
16
|
+
build can be scanned against a sitemap that lists production URLs. See
|
|
17
|
+
ADR 0012.
|
|
18
|
+
|
|
19
|
+
### Fixed
|
|
20
|
+
|
|
21
|
+
- Sitemap entries on another origin than `baseUrl` were dropped silently,
|
|
22
|
+
so a local or staging build whose sitemap lists production URLs scanned
|
|
23
|
+
only the home page and could exit `0`. They now produce a warning with
|
|
24
|
+
the number skipped, and a sitemap that leaves no page on `baseUrl`'s
|
|
25
|
+
origin falls back to the crawl; if that finds nothing either, the run
|
|
26
|
+
exits `3`. ([#58](https://github.com/MihaelaAghirculesei/bfsg-scanner/issues/58))
|
|
27
|
+
|
|
28
|
+
### Changed
|
|
29
|
+
|
|
30
|
+
- The scan engine moves to `axe-core` 4.14.0 (with `@axe-core/playwright`
|
|
31
|
+
4.14.0) and `playwright` 1.64.0. axe-core 4.14 fixes false positives and
|
|
32
|
+
sharpens several rules, so the same site can report a slightly different
|
|
33
|
+
set of findings than under 0.2.x.
|
|
34
|
+
- Dependency bump: `fast-xml-parser` 5.11.1 → 5.11.2, `@biomejs/biome`
|
|
35
|
+
2.5.14 → 2.5.15, `@types/node` 24.19.0 → 24.19.2, `vitest` 5.0.2 → 5.0.3.
|
|
36
|
+
No behavior change.
|
|
37
|
+
|
|
9
38
|
## [0.2.2] - 2026-09-29
|
|
10
39
|
|
|
11
40
|
### Fixed
|
|
@@ -77,7 +106,8 @@ First release.
|
|
|
77
106
|
- Distributable package: a `bfsg-scanner` bin and a `dist` build that
|
|
78
107
|
excludes tests.
|
|
79
108
|
|
|
80
|
-
[Unreleased]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.
|
|
109
|
+
[Unreleased]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.3.0...HEAD
|
|
110
|
+
[0.3.0]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.2.2...v0.3.0
|
|
81
111
|
[0.2.2]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.2.1...v0.2.2
|
|
82
112
|
[0.2.1]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.2.0...v0.2.1
|
|
83
113
|
[0.2.0]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.1.0...v0.2.0
|
package/README.md
CHANGED
|
@@ -113,6 +113,7 @@ bfsg-scanner [url] [options]
|
|
|
113
113
|
| `--output-dir <dir>` | Directory for the report files (default: `reports`). |
|
|
114
114
|
| `--format <list>` | Comma-separated subset of `json,html,pdf` (default: all three). |
|
|
115
115
|
| `--settle <ms>` | Pause after `load`, before the scan, for a client-rendered page to hydrate (default: `0`). |
|
|
116
|
+
| `--sitemap-origin <o>` | Origin the sitemap was built for, rewritten onto the `url`'s origin — for a local or staging build whose sitemap lists production URLs (default: none). |
|
|
116
117
|
| `-h, --help` | Show help and exit. |
|
|
117
118
|
| `-V, --version` | Print the version and exit. |
|
|
118
119
|
|
|
@@ -127,6 +128,7 @@ A YAML file — `bfsg.config.yaml` by default, or `--config <path>`. Only
|
|
|
127
128
|
```yaml
|
|
128
129
|
baseUrl: "https://example.de" # required
|
|
129
130
|
|
|
131
|
+
# sitemapOrigin: "https://example.de" # origin the sitemap was built for (see below)
|
|
130
132
|
maxPages: 50 # sitemap + crawl combined
|
|
131
133
|
settleMs: 0 # pause after load before the scan (see below)
|
|
132
134
|
wcagTags: # axe rule sets to run
|
|
@@ -154,6 +156,19 @@ would flag controls that populate a moment later. Set `settleMs` to a bit
|
|
|
154
156
|
more than that site's slowest post-load render (`1000`–`3000` is typical);
|
|
155
157
|
it is added to every page's scan time.
|
|
156
158
|
|
|
159
|
+
**`sitemapOrigin`** — a sitemap lists absolute URLs, and only entries on
|
|
160
|
+
`baseUrl`'s origin are scanned. Scanning a local or staging build whose
|
|
161
|
+
sitemap still lists production URLs would otherwise drop every entry: the
|
|
162
|
+
scanner warns with the number skipped, and when none are left it crawls
|
|
163
|
+
from `baseUrl` instead (exit `3` if that finds nothing either). Set
|
|
164
|
+
`sitemapOrigin` (or `--sitemap-origin`) to the production origin and its
|
|
165
|
+
entries — and sub-sitemaps — are rewritten onto `baseUrl`'s origin, path
|
|
166
|
+
and query kept:
|
|
167
|
+
|
|
168
|
+
```sh
|
|
169
|
+
npx bfsg-scanner http://localhost:8080/ --sitemap-origin https://example.de
|
|
170
|
+
```
|
|
171
|
+
|
|
157
172
|
## Output
|
|
158
173
|
|
|
159
174
|
Written to `outputDir`, overwritten each run:
|
package/dist/cli/run.js
CHANGED
|
@@ -23,7 +23,10 @@ Options:
|
|
|
23
23
|
--format <list> Comma-separated subset of json,html,pdf (default: all three).
|
|
24
24
|
--settle <ms> Pause after load, before the scan, for client-rendered
|
|
25
25
|
pages to hydrate (default: 0).
|
|
26
|
-
|
|
26
|
+
--sitemap-origin <o> Origin the sitemap was built for, e.g. production while
|
|
27
|
+
scanning a local or staging build; its entries are
|
|
28
|
+
rewritten onto the url's origin (default: none).
|
|
29
|
+
-h, --help Show this help and exit.
|
|
27
30
|
-V, --version Print the version and exit.
|
|
28
31
|
|
|
29
32
|
Exit codes:
|
|
@@ -42,6 +45,7 @@ const OPTIONS = {
|
|
|
42
45
|
'output-dir': { type: 'string' },
|
|
43
46
|
format: { type: 'string' },
|
|
44
47
|
settle: { type: 'string' },
|
|
48
|
+
'sitemap-origin': { type: 'string' },
|
|
45
49
|
};
|
|
46
50
|
function safeParseArgs(argv) {
|
|
47
51
|
try {
|
|
@@ -89,6 +93,9 @@ export function parseCliArgs(argv) {
|
|
|
89
93
|
const parsed = Number(values.settle);
|
|
90
94
|
overrides.settleMs = Number.isNaN(parsed) ? values.settle : parsed;
|
|
91
95
|
}
|
|
96
|
+
if (values['sitemap-origin'] !== undefined) {
|
|
97
|
+
overrides.sitemapOrigin = values['sitemap-origin'];
|
|
98
|
+
}
|
|
92
99
|
if (values.format !== undefined) {
|
|
93
100
|
overrides.reportFormats = values.format
|
|
94
101
|
.split(',')
|
|
@@ -162,6 +169,8 @@ async function runUnguarded(argv, deps) {
|
|
|
162
169
|
baseUrl: config.baseUrl,
|
|
163
170
|
maxPages: config.maxPages,
|
|
164
171
|
excludePaths: config.excludePaths,
|
|
172
|
+
...(config.sitemapOrigin !== undefined ? { sitemapOrigin: config.sitemapOrigin } : {}),
|
|
173
|
+
onWarning: (message) => console.warn(message),
|
|
165
174
|
});
|
|
166
175
|
if (urls.length === 0) {
|
|
167
176
|
console.error(`No scannable pages discovered for ${config.baseUrl}`);
|
package/dist/cli/run.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"run.js","sourceRoot":"","sources":["../../src/cli/run.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAgB,QAAQ,EAAE,MAAM,YAAY,CAAC;AACpD,OAAO,EAAe,WAAW,EAAE,UAAU,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACvF,OAAO,EAAE,YAAY,EAAE,MAAM,uBAAuB,CAAC;AACrD,OAAO,EACL,WAAW,EACX,gBAAgB,EAChB,eAAe,EACf,eAAe,EACf,cAAc,EACd,WAAW,GACZ,MAAM,oBAAoB,CAAC;AAC5B,OAAO,EAAiB,IAAI,EAAE,MAAM,kBAAkB,CAAC;AACvD,OAAO,EAAE,QAAQ,EAAE,MAAM,oBAAoB,CAAC;AAE9C,MAAM,mBAAmB,GAAG,kBAAkB,CAAC;AAE/C,MAAM,SAAS,GAAG;;;;;;;;;0DASwC,mBAAmB
|
|
1
|
+
{"version":3,"file":"run.js","sourceRoot":"","sources":["../../src/cli/run.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAgB,QAAQ,EAAE,MAAM,YAAY,CAAC;AACpD,OAAO,EAAe,WAAW,EAAE,UAAU,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACvF,OAAO,EAAE,YAAY,EAAE,MAAM,uBAAuB,CAAC;AACrD,OAAO,EACL,WAAW,EACX,gBAAgB,EAChB,eAAe,EACf,eAAe,EACf,cAAc,EACd,WAAW,GACZ,MAAM,oBAAoB,CAAC;AAC5B,OAAO,EAAiB,IAAI,EAAE,MAAM,kBAAkB,CAAC;AACvD,OAAO,EAAE,QAAQ,EAAE,MAAM,oBAAoB,CAAC;AAE9C,MAAM,mBAAmB,GAAG,kBAAkB,CAAC;AAE/C,MAAM,SAAS,GAAG;;;;;;;;;0DASwC,mBAAmB;;;;;;;;;;;;;;;;;;;;CAoB5E,CAAC;AAgBF,MAAM,OAAO,GAAG;IACd,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;IACrC,OAAO,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;IACxC,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,KAAK,EAAE,GAAG,EAAE;IACtC,SAAS,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IAC7B,iBAAiB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IACrC,YAAY,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IAChC,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IAC1B,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IAC1B,gBAAgB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;CAC5B,CAAC;AAEX,SAAS,aAAa,CAAC,IAAuB;IAC5C,IAAI,CAAC;QACH,OAAO,SAAS,CAAC,EAAE,IAAI,EAAE,CAAC,GAAG,IAAI,CAAC,EAAE,OAAO,EAAE,OAAO,EAAE,gBAAgB,EAAE,IAAI,EAAE,CAAC,CAAC;IAClF,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,IAAI,WAAW,CAAE,KAAe,CAAC,OAAO,CAAC,CAAC;IAClD,CAAC;AACH,CAAC;AAED;;;;;;;;;;;;GAYG;AACH,MAAM,UAAU,YAAY,CAAC,IAAuB;IAClD,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,CAAC,CAAC;IAEpD,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3B,MAAM,IAAI,WAAW,CAAC,0CAA0C,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC;IACzF,CAAC;IACD,MAAM,GAAG,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;IAC3B,IAAI,GAAG,KAAK,SAAS,IAAI,MAAM,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QACrD,MAAM,IAAI,WAAW,CAAC,4CAA4C,CAAC,CAAC;IACtE,CAAC;IAED,MAAM,SAAS,GAA4B,EAAE,CAAC;IAC9C,IAAI,MAAM,CAAC,SAAS,CAAC,KAAK,SAAS,EAAE,CAAC;QACpC,SAAS,CAAC,MAAM,GAAG,MAAM,CAAC,SAAS,CAAC,CAAC;IACvC,CAAC;IACD,IAAI,MAAM,CAAC,iBAAiB,CAAC,KAAK,SAAS,EAAE,CAAC;QAC5C,SAAS,CAAC,cAAc,GAAG,MAAM,CAAC,iBAAiB,CAAC,CAAC;IACvD,CAAC;IACD,IAAI,MAAM,CAAC,YAAY,CAAC,KAAK,SAAS,EAAE,CAAC;QACvC,SAAS,CAAC,SAAS,GAAG,MAAM,CAAC,YAAY,CAAC,CAAC;IAC7C,CAAC;IACD,IAAI,MAAM,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QAChC,gEAAgE;QAChE,6DAA6D;QAC7D,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QACrC,SAAS,CAAC,QAAQ,GAAG,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC;IACrE,CAAC;IACD,IAAI,MAAM,CAAC,gBAAgB,CAAC,KAAK,SAAS,EAAE,CAAC;QAC3C,SAAS,CAAC,aAAa,GAAG,MAAM,CAAC,gBAAgB,CAAC,CAAC;IACrD,CAAC;IACD,IAAI,MAAM,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QAChC,SAAS,CAAC,aAAa,GAAG,MAAM,CAAC,MAAM;aACpC,KAAK,CAAC,GAAG,CAAC;aACV,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;aAC1B,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IACvC,CAAC;IAED,OAAO;QACL,IAAI,EAAE,MAAM,CAAC,IAAI,IAAI,KAAK;QAC1B,OAAO,EAAE,MAAM,CAAC,OAAO,IAAI,KAAK;QAChC,SAAS;QACT,GAAG,CAAC,MAAM,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACrE,GAAG,CAAC,GAAG,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,GAAG,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACtC,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,GAAY;IACjC,IAAI,GAAG,CAAC,GAAG,KAAK,SAAS,EAAE,CAAC;QAC1B,OAAO,WAAW,CAAC,EAAE,OAAO,EAAE,GAAG,CAAC,GAAG,EAAE,GAAG,GAAG,CAAC,SAAS,EAAE,EAAE,wBAAwB,CAAC,CAAC;IACvF,CAAC;IACD,MAAM,QAAQ,GAAG,UAAU,CAAC,GAAG,CAAC,UAAU,IAAI,mBAAmB,CAAC,CAAC;IACnE,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC5C,OAAO,QAAQ,CAAC;IAClB,CAAC;IACD,OAAO,WAAW,CAAC,EAAE,GAAG,QAAQ,EAAE,GAAG,GAAG,CAAC,SAAS,EAAE,EAAE,wBAAwB,CAAC,CAAC;AAClF,CAAC;AAED;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,GAAG,CAAC,IAAuB,EAAE,IAAI,GAAa,EAAE;IACpE,IAAI,CAAC;QACH,OAAO,MAAM,YAAY,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IACxC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO,CAAC,KAAK,CAAC,qBAAqB,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QAC7F,OAAO,CAAC,CAAC;IACX,CAAC;AACH,CAAC;AAED,KAAK,UAAU,YAAY,CAAC,IAAuB,EAAE,IAAc;IACjE,IAAI,GAAY,CAAC;IACjB,IAAI,CAAC;QACH,GAAG,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IAC3B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO,CAAC,KAAK,CAAC,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;QACtE,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;QACb,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACvB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,GAAG,CAAC,OAAO,EAAE,CAAC;QAChB,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,GAAG,QAAQ,EAAE,CAAC;QACrC,OAAO,CAAC,GAAG,CAAC,GAAG,IAAI,IAAI,OAAO,EAAE,CAAC,CAAC;QAClC,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,MAAc,CAAC;IACnB,IAAI,CAAC;QACH,MAAM,GAAG,aAAa,CAAC,GAAG,CAAC,CAAC;IAC9B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,WAAW,EAAE,CAAC;YACjC,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;YAC7B,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,KAAK,CAAC;IACd,CAAC;IAED,MAAM,IAAI,GAAG,MAAM,YAAY,CAAC;QAC9B,OAAO,EAAE,MAAM,CAAC,OAAO;QACvB,QAAQ,EAAE,MAAM,CAAC,QAAQ;QACzB,YAAY,EAAE,MAAM,CAAC,YAAY;QACjC,GAAG,CAAC,MAAM,CAAC,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,MAAM,CAAC,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACtF,SAAS,EAAE,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC;KAC9C,CAAC,CAAC;IACH,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACtB,OAAO,CAAC,KAAK,CAAC,qCAAqC,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC;QACrE,OAAO,CAAC,CAAC;IACX,CAAC;IACD,OAAO,CAAC,GAAG,CAAC,cAAc,IAAI,CAAC,MAAM,mBAAmB,CAAC,CAAC;IAE1D,0EAA0E;IAC1E,yEAAyE;IACzE,8EAA8E;IAC9E,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,IAAI,CAAC,MAAM,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;IAC1D,MAAM,YAAY,GAAG,IAAI,CAAC,OAAO,KAAK,SAAS,CAAC;IAChD,IAAI,CAAC;QACH,OAAO,MAAM,aAAa,CAAC,MAAM,EAAE,IAAI,EAAE,OAAO,CAAC,CAAC;IACpD,CAAC;YAAS,CAAC;QACT,IAAI,YAAY,EAAE,CAAC;YACjB,MAAM,OAAO,CAAC,KAAK,EAAE,CAAC;QACxB,CAAC;IACH,CAAC;AACH,CAAC;AAED,KAAK,UAAU,aAAa,CAC1B,MAAc,EACd,IAAuB,EACvB,OAAgB;IAEhB,MAAM,MAAM,GAAG,MAAM,IAAI,CACvB,IAAI,EACJ,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE,EACxD,EAAE,OAAO,EAAE,CACZ,CAAC;IAEF,MAAM,MAAM,GAAG,WAAW,CAAC,MAAM,EAAE;QACjC,OAAO,EAAE,MAAM,CAAC,OAAO;QACvB,QAAQ,EAAE,MAAM,CAAC,QAAQ;QACzB,MAAM,EAAE,MAAM,CAAC,MAAM;KACtB,CAAC,CAAC;IAEH,MAAM,OAAO,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,aAAa,CAAC,CAAC;IAC9C,IAAI,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;QACxB,OAAO,CAAC,GAAG,CAAC,qBAAqB,MAAM,WAAW,CAAC,MAAM,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;IAClF,CAAC;IACD,IAAI,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;QAC9C,MAAM,IAAI,GAAG,gBAAgB,CAAC,MAAM,EAAE,MAAM,CAAC,cAAc,CAAC,CAAC;QAC7D,IAAI,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;YACxB,OAAO,CAAC,GAAG,CAAC,0BAA0B,MAAM,eAAe,CAAC,IAAI,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;QACzF,CAAC;QACD,IAAI,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;YACvB,MAAM,GAAG,GAAG,MAAM,eAAe,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;YACjD,OAAO,CAAC,GAAG,CAAC,yBAAyB,MAAM,cAAc,CAAC,GAAG,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;QACtF,CAAC;IACH,CAAC;IAED,MAAM,WAAW,GAAG,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,KAAK,OAAO,CAAC,CAAC;IAC3E,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3B,KAAK,MAAM,IAAI,IAAI,WAAW,EAAE,CAAC;YAC/B,OAAO,CAAC,KAAK,CAAC,kBAAkB,IAAI,CAAC,GAAG,KAAK,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC;QAC7D,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,OAAO,CAAC,GAAG,CACT,WAAW,MAAM,CAAC,OAAO,CAAC,YAAY,aAAa,MAAM,CAAC,OAAO,CAAC,eAAe,oBAAoB,CACtG,CAAC;IAEF,MAAM,EAAE,uBAAuB,EAAE,uBAAuB,EAAE,GAAG,MAAM,CAAC,OAAO,CAAC;IAC5E,IAAI,uBAAuB,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACvC,OAAO,CAAC,GAAG,CAAC,yBAAyB,uBAAuB,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QAC3E,OAAO,CAAC,GAAG,CAAC,gCAAgC,uBAAuB,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACpF,CAAC;IAED,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,CAAC;IAC3B,IAAI,OAAO,CAAC,kBAAkB,GAAG,CAAC,EAAE,CAAC;QACnC,OAAO,CAAC,IAAI,CACV,GAAG,OAAO,CAAC,kBAAkB,mEAAmE,MAAM,CAAC,MAAM,wCAAwC,CACtJ,CAAC;IACJ,CAAC;IAED,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;QACpB,OAAO,CAAC,KAAK,CACX,GAAG,OAAO,CAAC,4BAA4B,8BAA8B,MAAM,CAAC,MAAM,IAAI,CACvF,CAAC;QACF,OAAO,CAAC,CAAC;IACX,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC"}
|
package/dist/config/schema.d.ts
CHANGED
|
@@ -22,6 +22,7 @@ export declare const reportFormatSchema: z.ZodEnum<{
|
|
|
22
22
|
}>;
|
|
23
23
|
export declare const configSchema: z.ZodObject<{
|
|
24
24
|
baseUrl: z.ZodURL;
|
|
25
|
+
sitemapOrigin: z.ZodOptional<z.ZodURL>;
|
|
25
26
|
maxPages: z.ZodDefault<z.ZodNumber>;
|
|
26
27
|
settleMs: z.ZodDefault<z.ZodNumber>;
|
|
27
28
|
wcagTags: z.ZodDefault<z.ZodArray<z.ZodEnum<{
|
package/dist/config/schema.js
CHANGED
|
@@ -3,8 +3,18 @@ export const wcagTagSchema = z.enum(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']
|
|
|
3
3
|
export const failOnSchema = z.enum(['critical', 'serious', 'moderate', 'minor']);
|
|
4
4
|
export const reportLanguageSchema = z.enum(['de', 'en']);
|
|
5
5
|
export const reportFormatSchema = z.enum(['json', 'html', 'pdf']);
|
|
6
|
+
/**
|
|
7
|
+
* A bare origin (`https://example.com`, an optional trailing slash): the
|
|
8
|
+
* part of a sitemap entry that gets swapped for baseUrl's origin. A path
|
|
9
|
+
* would suggest prefix rewriting, which this does not do.
|
|
10
|
+
*/
|
|
11
|
+
const originSchema = z.url().refine((value) => {
|
|
12
|
+
const url = new URL(value);
|
|
13
|
+
return url.pathname === '/' && url.search === '' && url.hash === '';
|
|
14
|
+
}, { message: 'Must be an origin, e.g. "https://example.com", with no path, query or fragment' });
|
|
6
15
|
export const configSchema = z.strictObject({
|
|
7
16
|
baseUrl: z.url(),
|
|
17
|
+
sitemapOrigin: originSchema.optional(),
|
|
8
18
|
maxPages: z.number().int().positive().max(1000).default(50),
|
|
9
19
|
settleMs: z.number().int().min(0).max(60_000).default(0),
|
|
10
20
|
wcagTags: z.array(wcagTagSchema).min(1).default(['wcag2a', 'wcag2aa', 'wcag21aa']),
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"schema.js","sourceRoot":"","sources":["../../src/config/schema.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAExB,MAAM,CAAC,MAAM,aAAa,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,SAAS,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC,CAAC;AAElF,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,UAAU,EAAE,SAAS,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC;AAEjF,MAAM,CAAC,MAAM,oBAAoB,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC;AAEzD,MAAM,CAAC,MAAM,kBAAkB,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC;AAElE,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,CAAC,YAAY,CAAC;IACzC,OAAO,EAAE,CAAC,CAAC,GAAG,EAAE;IAChB,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;IAC3D,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;IACxD,QAAQ,EAAE,CAAC,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;IAClF,YAAY,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;IAC7C,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC;IAC/C,MAAM,EAAE,YAAY,CAAC,OAAO,CAAC,SAAS,CAAC;IACvC,cAAc,EAAE,oBAAoB,CAAC,OAAO,CAAC,IAAI,CAAC;IAClD,aAAa,EAAE,CAAC,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC;CACnF,CAAC,CAAC"}
|
|
1
|
+
{"version":3,"file":"schema.js","sourceRoot":"","sources":["../../src/config/schema.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAExB,MAAM,CAAC,MAAM,aAAa,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,SAAS,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC,CAAC;AAElF,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,UAAU,EAAE,SAAS,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC;AAEjF,MAAM,CAAC,MAAM,oBAAoB,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC;AAEzD,MAAM,CAAC,MAAM,kBAAkB,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC;AAElE;;;;GAIG;AACH,MAAM,YAAY,GAAG,CAAC,CAAC,GAAG,EAAE,CAAC,MAAM,CACjC,CAAC,KAAK,EAAE,EAAE;IACR,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC;IAC3B,OAAO,GAAG,CAAC,QAAQ,KAAK,GAAG,IAAI,GAAG,CAAC,MAAM,KAAK,EAAE,IAAI,GAAG,CAAC,IAAI,KAAK,EAAE,CAAC;AACtE,CAAC,EACD,EAAE,OAAO,EAAE,gFAAgF,EAAE,CAC9F,CAAC;AAEF,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,CAAC,YAAY,CAAC;IACzC,OAAO,EAAE,CAAC,CAAC,GAAG,EAAE;IAChB,aAAa,EAAE,YAAY,CAAC,QAAQ,EAAE;IACtC,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;IAC3D,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;IACxD,QAAQ,EAAE,CAAC,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;IAClF,YAAY,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;IAC7C,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC;IAC/C,MAAM,EAAE,YAAY,CAAC,OAAO,CAAC,SAAS,CAAC;IACvC,cAAc,EAAE,oBAAoB,CAAC,OAAO,CAAC,IAAI,CAAC;IAClD,aAAa,EAAE,CAAC,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC;CACnF,CAAC,CAAC"}
|
|
@@ -5,13 +5,24 @@ export interface DiscoverSiteOptions {
|
|
|
5
5
|
readonly excludePaths?: readonly string[];
|
|
6
6
|
/** Injectable for tests; defaults to the global fetch (with User-Agent). */
|
|
7
7
|
readonly fetchFn?: typeof fetch;
|
|
8
|
+
/** Origin the sitemap was generated for, rewritten onto baseUrl's; see discoverFromSitemap. */
|
|
9
|
+
readonly sitemapOrigin?: string;
|
|
10
|
+
/** Receives one line per discovery problem worth surfacing. Defaults to none. */
|
|
11
|
+
readonly onWarning?: (message: string) => void;
|
|
8
12
|
}
|
|
9
13
|
/**
|
|
10
14
|
* Resolves the list of page URLs to scan for a site.
|
|
11
15
|
*
|
|
12
16
|
* Strategy: prefer the site's sitemap.xml — it is the owner's own
|
|
13
17
|
* declaration of which pages exist. If the root sitemap is missing or
|
|
14
|
-
* unparseable,
|
|
18
|
+
* unparseable, or lists no scannable page on baseUrl's origin, fall back
|
|
19
|
+
* to a breadth-first crawl from baseUrl.
|
|
20
|
+
*
|
|
21
|
+
* Sitemap entries on another origin are dropped with a warning: they are
|
|
22
|
+
* the typical sign of a local or staging build whose sitemap points at
|
|
23
|
+
* production, and dropping them silently would leave a scan of the home
|
|
24
|
+
* page alone looking like a clean site. sitemapOrigin names that other
|
|
25
|
+
* origin, so its entries are rewritten onto baseUrl's and scanned there.
|
|
15
26
|
*
|
|
16
27
|
* Both strategies apply excludePaths and the non-page-asset filter
|
|
17
28
|
* identically. They differ on robots.txt: the crawl path obeys it (a
|
|
@@ -6,7 +6,14 @@ import { SitemapError } from './sitemap.js';
|
|
|
6
6
|
*
|
|
7
7
|
* Strategy: prefer the site's sitemap.xml — it is the owner's own
|
|
8
8
|
* declaration of which pages exist. If the root sitemap is missing or
|
|
9
|
-
* unparseable,
|
|
9
|
+
* unparseable, or lists no scannable page on baseUrl's origin, fall back
|
|
10
|
+
* to a breadth-first crawl from baseUrl.
|
|
11
|
+
*
|
|
12
|
+
* Sitemap entries on another origin are dropped with a warning: they are
|
|
13
|
+
* the typical sign of a local or staging build whose sitemap points at
|
|
14
|
+
* production, and dropping them silently would leave a scan of the home
|
|
15
|
+
* page alone looking like a clean site. sitemapOrigin names that other
|
|
16
|
+
* origin, so its entries are rewritten onto baseUrl's and scanned there.
|
|
10
17
|
*
|
|
11
18
|
* Both strategies apply excludePaths and the non-page-asset filter
|
|
12
19
|
* identically. They differ on robots.txt: the crawl path obeys it (a
|
|
@@ -19,11 +26,19 @@ import { SitemapError } from './sitemap.js';
|
|
|
19
26
|
* from baseUrl. The result never exceeds maxPages.
|
|
20
27
|
*/
|
|
21
28
|
export async function discoverSite(options) {
|
|
22
|
-
const { baseUrl, maxPages, excludePaths = [], fetchFn } = options;
|
|
29
|
+
const { baseUrl, maxPages, excludePaths = [], fetchFn, sitemapOrigin, onWarning } = options;
|
|
23
30
|
const shared = { baseUrl, maxPages, excludePaths, ...(fetchFn ? { fetchFn } : {}) };
|
|
31
|
+
const origin = new URL(baseUrl).origin;
|
|
24
32
|
try {
|
|
25
|
-
const pages = await discoverFromSitemap(
|
|
26
|
-
|
|
33
|
+
const pages = await discoverFromSitemap({
|
|
34
|
+
...shared,
|
|
35
|
+
...(sitemapOrigin !== undefined ? { sitemapOrigin } : {}),
|
|
36
|
+
onOffOrigin: (skipped) => onWarning?.(describeOffOrigin(skipped, origin, sitemapOrigin === undefined)),
|
|
37
|
+
});
|
|
38
|
+
if (pages.length > 0) {
|
|
39
|
+
return withBaseUrl(pages, baseUrl, maxPages);
|
|
40
|
+
}
|
|
41
|
+
onWarning?.(`The sitemap lists no scannable page on ${origin}; crawling from ${baseUrl}.`);
|
|
27
42
|
}
|
|
28
43
|
catch (error) {
|
|
29
44
|
if (!(error instanceof SitemapError)) {
|
|
@@ -32,6 +47,15 @@ export async function discoverSite(options) {
|
|
|
32
47
|
}
|
|
33
48
|
return crawlSite(shared);
|
|
34
49
|
}
|
|
50
|
+
function describeOffOrigin(skipped, origin, withHint) {
|
|
51
|
+
const origins = [...new Set(skipped.map((url) => new URL(url).origin))];
|
|
52
|
+
const shown = origins.length > 3 ? [...origins.slice(0, 3), '…'] : origins;
|
|
53
|
+
const entries = skipped.length === 1 ? 'entry' : 'entries';
|
|
54
|
+
const message = `Skipped ${skipped.length} sitemap ${entries} on another origin than ${origin} (${shown.join(', ')}).`;
|
|
55
|
+
return withHint
|
|
56
|
+
? `${message} Set sitemapOrigin (--sitemap-origin) to scan them on ${origin}.`
|
|
57
|
+
: message;
|
|
58
|
+
}
|
|
35
59
|
function withBaseUrl(pages, baseUrl, maxPages) {
|
|
36
60
|
const normalized = new URL(baseUrl).toString();
|
|
37
61
|
const alreadyListed = pages.some((page) => {
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"discover-site.js","sourceRoot":"","sources":["../../src/discovery/discover-site.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AACvC,OAAO,EAAE,mBAAmB,EAAE,MAAM,eAAe,CAAC;AACpD,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;
|
|
1
|
+
{"version":3,"file":"discover-site.js","sourceRoot":"","sources":["../../src/discovery/discover-site.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AACvC,OAAO,EAAE,mBAAmB,EAAE,MAAM,eAAe,CAAC;AACpD,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAe5C;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAAC,OAA4B;IAC7D,MAAM,EAAE,OAAO,EAAE,QAAQ,EAAE,YAAY,GAAG,EAAE,EAAE,OAAO,EAAE,aAAa,EAAE,SAAS,EAAE,GAAG,OAAO,CAAC;IAC5F,MAAM,MAAM,GAAG,EAAE,OAAO,EAAE,QAAQ,EAAE,YAAY,EAAE,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC;IACpF,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC;IAEvC,IAAI,CAAC;QACH,MAAM,KAAK,GAAG,MAAM,mBAAmB,CAAC;YACtC,GAAG,MAAM;YACT,GAAG,CAAC,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACzD,WAAW,EAAE,CAAC,OAAO,EAAE,EAAE,CACvB,SAAS,EAAE,CAAC,iBAAiB,CAAC,OAAO,EAAE,MAAM,EAAE,aAAa,KAAK,SAAS,CAAC,CAAC;SAC/E,CAAC,CAAC;QACH,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACrB,OAAO,WAAW,CAAC,KAAK,EAAE,OAAO,EAAE,QAAQ,CAAC,CAAC;QAC/C,CAAC;QACD,SAAS,EAAE,CAAC,0CAA0C,MAAM,mBAAmB,OAAO,GAAG,CAAC,CAAC;IAC7F,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,CAAC,CAAC,KAAK,YAAY,YAAY,CAAC,EAAE,CAAC;YACrC,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED,OAAO,SAAS,CAAC,MAAM,CAAC,CAAC;AAC3B,CAAC;AAED,SAAS,iBAAiB,CAAC,OAA0B,EAAE,MAAc,EAAE,QAAiB;IACtF,MAAM,OAAO,GAAG,CAAC,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;IACxE,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC;IAC3E,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;IAC3D,MAAM,OAAO,GAAG,WAAW,OAAO,CAAC,MAAM,YAAY,OAAO,2BAA2B,MAAM,KAAK,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC;IACvH,OAAO,QAAQ;QACb,CAAC,CAAC,GAAG,OAAO,yDAAyD,MAAM,GAAG;QAC9E,CAAC,CAAC,OAAO,CAAC;AACd,CAAC;AAED,SAAS,WAAW,CAAC,KAAwB,EAAE,OAAe,EAAE,QAAgB;IAC9E,MAAM,UAAU,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC;IAC/C,MAAM,aAAa,GAAG,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE;QACxC,IAAI,CAAC;YACH,OAAO,IAAI,GAAG,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,KAAK,UAAU,CAAC;QACjD,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,KAAK,CAAC;QACf,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,IAAI,aAAa,EAAE,CAAC;QAClB,OAAO,CAAC,GAAG,KAAK,CAAC,CAAC;IACpB,CAAC;IACD,OAAO,CAAC,UAAU,EAAE,GAAG,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,QAAQ,CAAC,CAAC;AACnD,CAAC"}
|
|
@@ -9,14 +9,26 @@ export interface DiscoverFromSitemapOptions {
|
|
|
9
9
|
readonly fetchFn?: typeof fetch;
|
|
10
10
|
/** Guards against pathological or cyclic sitemap indexes. Defaults to 3. */
|
|
11
11
|
readonly maxSitemapDepth?: number;
|
|
12
|
+
/**
|
|
13
|
+
* Origin the sitemap was generated for, when it is not baseUrl's — e.g.
|
|
14
|
+
* production, while baseUrl is a local or staging build. Entries (and
|
|
15
|
+
* sub-sitemaps) on this origin are rewritten onto baseUrl's origin,
|
|
16
|
+
* keeping path and query. Defaults to none.
|
|
17
|
+
*/
|
|
18
|
+
readonly sitemapOrigin?: string;
|
|
19
|
+
/**
|
|
20
|
+
* Called once, after discovery, with the distinct page entries dropped
|
|
21
|
+
* for being on another origin than baseUrl. Not called when there were none.
|
|
22
|
+
*/
|
|
23
|
+
readonly onOffOrigin?: (skippedUrls: readonly string[]) => void;
|
|
12
24
|
}
|
|
13
25
|
/**
|
|
14
26
|
* Discovers page URLs from a site's sitemap, following a sitemap index to
|
|
15
27
|
* its sub-sitemaps as needed. Off-origin entries are dropped (a sitemap
|
|
16
|
-
* should only ever list its own site, but nothing enforces that)
|
|
17
|
-
* assets and excludePaths matches
|
|
18
|
-
* crawlSite), duplicate URLs are
|
|
19
|
-
* maxPages is reached.
|
|
28
|
+
* should only ever list its own site, but nothing enforces that) and
|
|
29
|
+
* reported through onOffOrigin, non-page assets and excludePaths matches
|
|
30
|
+
* are filtered out (identically to crawlSite), duplicate URLs are
|
|
31
|
+
* collapsed, and discovery stops as soon as maxPages is reached.
|
|
20
32
|
*
|
|
21
33
|
* A failure to load or parse the root sitemap is fatal and always surfaces
|
|
22
34
|
* as a SitemapError (the caller should fall back to crawling). A failure on
|
|
@@ -4,10 +4,10 @@ import { parseSitemapXml, SitemapError } from './sitemap.js';
|
|
|
4
4
|
/**
|
|
5
5
|
* Discovers page URLs from a site's sitemap, following a sitemap index to
|
|
6
6
|
* its sub-sitemaps as needed. Off-origin entries are dropped (a sitemap
|
|
7
|
-
* should only ever list its own site, but nothing enforces that)
|
|
8
|
-
* assets and excludePaths matches
|
|
9
|
-
* crawlSite), duplicate URLs are
|
|
10
|
-
* maxPages is reached.
|
|
7
|
+
* should only ever list its own site, but nothing enforces that) and
|
|
8
|
+
* reported through onOffOrigin, non-page assets and excludePaths matches
|
|
9
|
+
* are filtered out (identically to crawlSite), duplicate URLs are
|
|
10
|
+
* collapsed, and discovery stops as soon as maxPages is reached.
|
|
11
11
|
*
|
|
12
12
|
* A failure to load or parse the root sitemap is fatal and always surfaces
|
|
13
13
|
* as a SitemapError (the caller should fall back to crawling). A failure on
|
|
@@ -15,13 +15,18 @@ import { parseSitemapXml, SitemapError } from './sitemap.js';
|
|
|
15
15
|
* skipped and discovery continues with the rest.
|
|
16
16
|
*/
|
|
17
17
|
export async function discoverFromSitemap(options) {
|
|
18
|
-
const { baseUrl, maxPages, sitemapPath = '/sitemap.xml', excludePaths = [], fetchFn = fetchWithUserAgent, maxSitemapDepth = 3, } = options;
|
|
18
|
+
const { baseUrl, maxPages, sitemapPath = '/sitemap.xml', excludePaths = [], fetchFn = fetchWithUserAgent, maxSitemapDepth = 3, sitemapOrigin, onOffOrigin, } = options;
|
|
19
19
|
const origin = new URL(baseUrl).origin;
|
|
20
|
+
const rewriteFrom = sitemapOrigin === undefined ? undefined : new URL(sitemapOrigin).origin;
|
|
20
21
|
const rootSitemapUrl = new URL(sitemapPath, baseUrl).toString();
|
|
21
22
|
const collected = [];
|
|
23
|
+
const offOrigin = [];
|
|
22
24
|
const seenPages = new Set();
|
|
23
25
|
const visitedSitemaps = new Set();
|
|
24
26
|
await visitSitemap(rootSitemapUrl, 0);
|
|
27
|
+
if (offOrigin.length > 0) {
|
|
28
|
+
onOffOrigin?.(offOrigin);
|
|
29
|
+
}
|
|
25
30
|
return collected.slice(0, maxPages);
|
|
26
31
|
async function visitSitemap(sitemapUrl, depth) {
|
|
27
32
|
if (collected.length >= maxPages) {
|
|
@@ -54,14 +59,17 @@ export async function discoverFromSitemap(options) {
|
|
|
54
59
|
if (collected.length >= maxPages) {
|
|
55
60
|
return;
|
|
56
61
|
}
|
|
57
|
-
await visitSitemap(childUrl, depth + 1);
|
|
62
|
+
await visitSitemap(rewriteOrigin(childUrl), depth + 1);
|
|
58
63
|
}
|
|
59
64
|
return;
|
|
60
65
|
}
|
|
61
|
-
for (const
|
|
66
|
+
for (const listedUrl of doc.urls) {
|
|
62
67
|
if (collected.length >= maxPages) {
|
|
63
68
|
return;
|
|
64
69
|
}
|
|
70
|
+
// Deduplicated after the rewrite, so a sitemap listing a page on both
|
|
71
|
+
// origins yields it once.
|
|
72
|
+
const pageUrl = rewriteOrigin(listedUrl);
|
|
65
73
|
if (seenPages.has(pageUrl)) {
|
|
66
74
|
continue;
|
|
67
75
|
}
|
|
@@ -73,13 +81,32 @@ export async function discoverFromSitemap(options) {
|
|
|
73
81
|
catch {
|
|
74
82
|
continue;
|
|
75
83
|
}
|
|
76
|
-
if (parsed.origin !== origin
|
|
77
|
-
|
|
78
|
-
|
|
84
|
+
if (parsed.origin !== origin) {
|
|
85
|
+
offOrigin.push(pageUrl);
|
|
86
|
+
continue;
|
|
87
|
+
}
|
|
88
|
+
if (!isCrawlableUrl(parsed) || matchesExcludePattern(parsed.pathname, excludePaths)) {
|
|
79
89
|
continue;
|
|
80
90
|
}
|
|
81
91
|
collected.push(pageUrl);
|
|
82
92
|
}
|
|
83
93
|
}
|
|
94
|
+
/** Moves a URL on sitemapOrigin onto baseUrl's origin; any other URL is returned verbatim. */
|
|
95
|
+
function rewriteOrigin(url) {
|
|
96
|
+
if (rewriteFrom === undefined) {
|
|
97
|
+
return url;
|
|
98
|
+
}
|
|
99
|
+
let parsed;
|
|
100
|
+
try {
|
|
101
|
+
parsed = new URL(url);
|
|
102
|
+
}
|
|
103
|
+
catch {
|
|
104
|
+
return url;
|
|
105
|
+
}
|
|
106
|
+
if (parsed.origin !== rewriteFrom) {
|
|
107
|
+
return url;
|
|
108
|
+
}
|
|
109
|
+
return new URL(`${parsed.pathname}${parsed.search}${parsed.hash}`, origin).toString();
|
|
110
|
+
}
|
|
84
111
|
}
|
|
85
112
|
//# sourceMappingURL=discover.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"discover.js","sourceRoot":"","sources":["../../src/discovery/discover.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,kBAAkB,EAAE,MAAM,yBAAyB,CAAC;AAC7D,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,MAAM,YAAY,CAAC;AACnE,OAAO,EAAE,eAAe,EAAwB,YAAY,EAAE,MAAM,cAAc,CAAC;
|
|
1
|
+
{"version":3,"file":"discover.js","sourceRoot":"","sources":["../../src/discovery/discover.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,kBAAkB,EAAE,MAAM,yBAAyB,CAAC;AAC7D,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,MAAM,YAAY,CAAC;AACnE,OAAO,EAAE,eAAe,EAAwB,YAAY,EAAE,MAAM,cAAc,CAAC;AA2BnF;;;;;;;;;;;;GAYG;AACH,MAAM,CAAC,KAAK,UAAU,mBAAmB,CAAC,OAAmC;IAC3E,MAAM,EACJ,OAAO,EACP,QAAQ,EACR,WAAW,GAAG,cAAc,EAC5B,YAAY,GAAG,EAAE,EACjB,OAAO,GAAG,kBAAkB,EAC5B,eAAe,GAAG,CAAC,EACnB,aAAa,EACb,WAAW,GACZ,GAAG,OAAO,CAAC;IAEZ,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC;IACvC,MAAM,WAAW,GAAG,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,GAAG,CAAC,aAAa,CAAC,CAAC,MAAM,CAAC;IAC5F,MAAM,cAAc,GAAG,IAAI,GAAG,CAAC,WAAW,EAAE,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC;IAEhE,MAAM,SAAS,GAAa,EAAE,CAAC;IAC/B,MAAM,SAAS,GAAa,EAAE,CAAC;IAC/B,MAAM,SAAS,GAAG,IAAI,GAAG,EAAU,CAAC;IACpC,MAAM,eAAe,GAAG,IAAI,GAAG,EAAU,CAAC;IAE1C,MAAM,YAAY,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC;IAEtC,IAAI,SAAS,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACzB,WAAW,EAAE,CAAC,SAAS,CAAC,CAAC;IAC3B,CAAC;IACD,OAAO,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,QAAQ,CAAC,CAAC;IAEpC,KAAK,UAAU,YAAY,CAAC,UAAkB,EAAE,KAAa;QAC3D,IAAI,SAAS,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;YACjC,OAAO;QACT,CAAC;QACD,IAAI,eAAe,CAAC,GAAG,CAAC,UAAU,CAAC,IAAI,KAAK,GAAG,eAAe,EAAE,CAAC;YAC/D,OAAO;QACT,CAAC;QACD,eAAe,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QAEhC,sEAAsE;QACtE,mEAAmE;QACnE,sEAAsE;QACtE,4DAA4D;QAC5D,IAAI,GAAoB,CAAC;QACzB,IAAI,CAAC;YACH,MAAM,QAAQ,GAAG,MAAM,OAAO,CAAC,UAAU,CAAC,CAAC;YAC3C,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,MAAM,IAAI,KAAK,CAAC,QAAQ,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;YAC7C,CAAC;YACD,GAAG,GAAG,eAAe,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC,CAAC;QAC/C,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;gBAChB,MAAM,IAAI,YAAY,CACpB,2BAA2B,UAAU,MAAM,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CACpG,CAAC;YACJ,CAAC;YACD,OAAO;QACT,CAAC;QAED,IAAI,GAAG,CAAC,IAAI,KAAK,cAAc,EAAE,CAAC;YAChC,KAAK,MAAM,QAAQ,IAAI,GAAG,CAAC,QAAQ,EAAE,CAAC;gBACpC,IAAI,SAAS,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;oBACjC,OAAO;gBACT,CAAC;gBACD,MAAM,YAAY,CAAC,aAAa,CAAC,QAAQ,CAAC,EAAE,KAAK,GAAG,CAAC,CAAC,CAAC;YACzD,CAAC;YACD,OAAO;QACT,CAAC;QAED,KAAK,MAAM,SAAS,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;YACjC,IAAI,SAAS,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;gBACjC,OAAO;YACT,CAAC;YACD,sEAAsE;YACtE,0BAA0B;YAC1B,MAAM,OAAO,GAAG,aAAa,CAAC,SAAS,CAAC,CAAC;YACzC,IAAI,SAAS,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC;gBAC3B,SAAS;YACX,CAAC;YACD,SAAS,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;YAEvB,IAAI,MAAW,CAAC;YAChB,IAAI,CAAC;gBACH,MAAM,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC;YAC5B,CAAC;YAAC,MAAM,CAAC;gBACP,SAAS;YACX,CAAC;YACD,IAAI,MAAM,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;gBAC7B,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;gBACxB,SAAS;YACX,CAAC;YACD,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,IAAI,qBAAqB,CAAC,MAAM,CAAC,QAAQ,EAAE,YAAY,CAAC,EAAE,CAAC;gBACpF,SAAS;YACX,CAAC;YACD,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;QAC1B,CAAC;IACH,CAAC;IAED,8FAA8F;IAC9F,SAAS,aAAa,CAAC,GAAW;QAChC,IAAI,WAAW,KAAK,SAAS,EAAE,CAAC;YAC9B,OAAO,GAAG,CAAC;QACb,CAAC;QACD,IAAI,MAAW,CAAC;QAChB,IAAI,CAAC;YACH,MAAM,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC;QACxB,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,GAAG,CAAC;QACb,CAAC;QACD,IAAI,MAAM,CAAC,MAAM,KAAK,WAAW,EAAE,CAAC;YAClC,OAAO,GAAG,CAAC;QACb,CAAC;QACD,OAAO,IAAI,GAAG,CAAC,GAAG,MAAM,CAAC,QAAQ,GAAG,MAAM,CAAC,MAAM,GAAG,MAAM,CAAC,IAAI,EAAE,EAAE,MAAM,CAAC,CAAC,QAAQ,EAAE,CAAC;IACxF,CAAC;AACH,CAAC"}
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# 12. Off-origin sitemap entries
|
|
2
|
+
|
|
3
|
+
Date: 2026-10-17
|
|
4
|
+
|
|
5
|
+
## Status
|
|
6
|
+
|
|
7
|
+
Accepted
|
|
8
|
+
|
|
9
|
+
## Context
|
|
10
|
+
|
|
11
|
+
ADR 0001 collects only same-origin `<loc>` entries from the sitemap and
|
|
12
|
+
falls back to crawling only when the root sitemap is missing or unparseable.
|
|
13
|
+
Dropping off-origin entries was silent.
|
|
14
|
+
|
|
15
|
+
The common way to hit it is a local or staging build whose `sitemap.xml`
|
|
16
|
+
was generated for production: every entry is off-origin, the sitemap still
|
|
17
|
+
parses, so no crawl runs, `baseUrl` is prepended, and the run scans the home
|
|
18
|
+
page alone. With a clean home page that exits `0` — a CI gate reporting a
|
|
19
|
+
clean site it never looked at (#58).
|
|
20
|
+
|
|
21
|
+
## Decision
|
|
22
|
+
|
|
23
|
+
**Warn.** `discoverFromSitemap` reports the distinct off-origin entries it
|
|
24
|
+
dropped; the CLI prints one line with the count, the origins involved, and
|
|
25
|
+
a hint to set `sitemapOrigin`.
|
|
26
|
+
|
|
27
|
+
**A sitemap with no usable page falls back to the crawl.** When the sitemap
|
|
28
|
+
yields no scannable page on `baseUrl`'s origin — all off-origin, all
|
|
29
|
+
excluded, or simply empty — `discoverSite` treats it like a missing sitemap
|
|
30
|
+
and crawls from `baseUrl`, with a warning that it did. A sitemap that leaves
|
|
31
|
+
at least one page is used as before: a partial sitemap is still the owner's
|
|
32
|
+
own list (ADR 0001). If the crawl finds nothing either, discovery returns no
|
|
33
|
+
pages and the run exits `3`, the existing "no pages discovered" code.
|
|
34
|
+
|
|
35
|
+
**`sitemapOrigin` rewrites one origin onto `baseUrl`'s.** A bare origin, in
|
|
36
|
+
config or as `--sitemap-origin`. Page entries and sub-sitemap URLs on that
|
|
37
|
+
origin get `baseUrl`'s origin, path, query and fragment kept, so a staging
|
|
38
|
+
build is scanned page for page against the production sitemap, and the
|
|
39
|
+
build's own copies of sub-sitemaps are read. Entries on any third origin
|
|
40
|
+
are still dropped with the warning. Deduplication runs after the rewrite.
|
|
41
|
+
|
|
42
|
+
It gets a CLI flag, unlike `maxPages` or `excludePaths` (ADR 0007): the case
|
|
43
|
+
it serves — a one-off scan of `http://localhost:…` in a build job — is the
|
|
44
|
+
one a positional URL exists for.
|
|
45
|
+
|
|
46
|
+
## Consequences
|
|
47
|
+
|
|
48
|
+
- No exit code changes meaning. A run that used to pass on the home page
|
|
49
|
+
alone now either scans what the crawl reaches, or exits `3`.
|
|
50
|
+
- A sitemap that lists nothing scannable now costs a crawl. That is the
|
|
51
|
+
same cost as a site with no sitemap.
|
|
52
|
+
- Only the origin is rewritten, never a path prefix. A staging copy served
|
|
53
|
+
under a sub-path (`https://staging/app/` for `https://prod/`) is not
|
|
54
|
+
covered; that would need a prefix mapping, deferred until asked for.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "bfsg-scanner",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "WCAG 2.1 AA / BFSG compliance scanner built on Playwright and axe-core",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"accessibility",
|
|
@@ -54,20 +54,20 @@
|
|
|
54
54
|
"prepublishOnly": "npm run verify && npm run build"
|
|
55
55
|
},
|
|
56
56
|
"dependencies": {
|
|
57
|
-
"@axe-core/playwright": "4.
|
|
58
|
-
"axe-core": "4.
|
|
59
|
-
"fast-xml-parser": "5.11.
|
|
60
|
-
"playwright": "1.
|
|
57
|
+
"@axe-core/playwright": "4.14.0",
|
|
58
|
+
"axe-core": "4.14.0",
|
|
59
|
+
"fast-xml-parser": "5.11.2",
|
|
60
|
+
"playwright": "1.64.0",
|
|
61
61
|
"yaml": "2.9.1",
|
|
62
62
|
"zod": "4.6.5"
|
|
63
63
|
},
|
|
64
64
|
"devDependencies": {
|
|
65
|
-
"@biomejs/biome": "2.5.
|
|
66
|
-
"@types/node": "24.19.
|
|
65
|
+
"@biomejs/biome": "2.5.15",
|
|
66
|
+
"@types/node": "24.19.2",
|
|
67
67
|
"ajv": "8.20.0",
|
|
68
68
|
"rimraf": "6.1.3",
|
|
69
69
|
"tsx": "4.23.15",
|
|
70
70
|
"typescript": "7.0.2",
|
|
71
|
-
"vitest": "5.0.
|
|
71
|
+
"vitest": "5.0.3"
|
|
72
72
|
}
|
|
73
73
|
}
|