bfsg-scanner 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +51 -1
- package/README.md +20 -3
- package/dist/cli/run.d.ts +8 -0
- package/dist/cli/run.js +28 -1
- package/dist/cli/run.js.map +1 -1
- package/dist/config/schema.d.ts +1 -0
- package/dist/config/schema.js +10 -0
- package/dist/config/schema.js.map +1 -1
- package/dist/discovery/discover-site.d.ts +12 -1
- package/dist/discovery/discover-site.js +28 -4
- package/dist/discovery/discover-site.js.map +1 -1
- package/dist/discovery/discover.d.ts +20 -7
- package/dist/discovery/discover.js +47 -16
- package/dist/discovery/discover.js.map +1 -1
- package/docs/adr/0001-page-discovery-strategy.md +2 -1
- package/docs/adr/0011-exit-code-for-unexpected-errors.md +48 -0
- package/docs/adr/0012-off-origin-sitemap-entries.md +54 -0
- package/package.json +11 -11
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,54 @@ follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.3.0] - 2026-10-17
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `sitemapOrigin` config key / `--sitemap-origin <origin>` flag — the origin
|
|
14
|
+
a sitemap was built for, when it is not `baseUrl`'s. Its entries and
|
|
15
|
+
sub-sitemaps are rewritten onto `baseUrl`'s origin, so a local or staging
|
|
16
|
+
build can be scanned against a sitemap that lists production URLs. See
|
|
17
|
+
ADR 0012.
|
|
18
|
+
|
|
19
|
+
### Fixed
|
|
20
|
+
|
|
21
|
+
- Sitemap entries on another origin than `baseUrl` were dropped silently,
|
|
22
|
+
so a local or staging build whose sitemap lists production URLs scanned
|
|
23
|
+
only the home page and could exit `0`. They now produce a warning with
|
|
24
|
+
the number skipped, and a sitemap that leaves no page on `baseUrl`'s
|
|
25
|
+
origin falls back to the crawl; if that finds nothing either, the run
|
|
26
|
+
exits `3`. ([#58](https://github.com/MihaelaAghirculesei/bfsg-scanner/issues/58))
|
|
27
|
+
|
|
28
|
+
### Changed
|
|
29
|
+
|
|
30
|
+
- The scan engine moves to `axe-core` 4.14.0 (with `@axe-core/playwright`
|
|
31
|
+
4.14.0) and `playwright` 1.64.0. axe-core 4.14 fixes false positives and
|
|
32
|
+
sharpens several rules, so the same site can report a slightly different
|
|
33
|
+
set of findings than under 0.2.x.
|
|
34
|
+
- Dependency bump: `fast-xml-parser` 5.11.1 → 5.11.2, `@biomejs/biome`
|
|
35
|
+
2.5.14 → 2.5.15, `@types/node` 24.19.0 → 24.19.2, `vitest` 5.0.2 → 5.0.3.
|
|
36
|
+
No behavior change.
|
|
37
|
+
|
|
38
|
+
## [0.2.2] - 2026-09-29
|
|
39
|
+
|
|
40
|
+
### Fixed
|
|
41
|
+
|
|
42
|
+
- A sub-sitemap that answers `200` with something other than a sitemap (a
|
|
43
|
+
soft-404 HTML page, a truncated file) is now skipped like an unreachable
|
|
44
|
+
one. Previously it discarded the whole sitemap and fell back to crawling.
|
|
45
|
+
- A root sitemap that is not well-formed XML now falls back to crawling
|
|
46
|
+
instead of aborting the run.
|
|
47
|
+
- An unexpected failure (Chromium not installed, an unwritable output
|
|
48
|
+
directory) now prints a one-line message and exits `4`. Previously it
|
|
49
|
+
crashed with a stack trace and exit code `1`, the code reserved for
|
|
50
|
+
"violations found". See ADR 0011.
|
|
51
|
+
|
|
52
|
+
### Changed
|
|
53
|
+
|
|
54
|
+
- Dependency bump: `yaml` 2.9.0 → 2.9.1, `zod` 4.6.2 → 4.6.5. No behavior
|
|
55
|
+
change.
|
|
56
|
+
|
|
9
57
|
## [0.2.1] - 2026-09-14
|
|
10
58
|
|
|
11
59
|
### Changed
|
|
@@ -58,7 +106,9 @@ First release.
|
|
|
58
106
|
- Distributable package: a `bfsg-scanner` bin and a `dist` build that
|
|
59
107
|
excludes tests.
|
|
60
108
|
|
|
61
|
-
[Unreleased]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.
|
|
109
|
+
[Unreleased]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.3.0...HEAD
|
|
110
|
+
[0.3.0]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.2.2...v0.3.0
|
|
111
|
+
[0.2.2]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.2.1...v0.2.2
|
|
62
112
|
[0.2.1]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.2.0...v0.2.1
|
|
63
113
|
[0.2.0]: https://github.com/MihaelaAghirculesei/bfsg-scanner/compare/v0.1.0...v0.2.0
|
|
64
114
|
[0.1.0]: https://github.com/MihaelaAghirculesei/bfsg-scanner/releases/tag/v0.1.0
|
package/README.md
CHANGED
|
@@ -113,6 +113,7 @@ bfsg-scanner [url] [options]
|
|
|
113
113
|
| `--output-dir <dir>` | Directory for the report files (default: `reports`). |
|
|
114
114
|
| `--format <list>` | Comma-separated subset of `json,html,pdf` (default: all three). |
|
|
115
115
|
| `--settle <ms>` | Pause after `load`, before the scan, for a client-rendered page to hydrate (default: `0`). |
|
|
116
|
+
| `--sitemap-origin <o>` | Origin the sitemap was built for, rewritten onto the `url`'s origin — for a local or staging build whose sitemap lists production URLs (default: none). |
|
|
116
117
|
| `-h, --help` | Show help and exit. |
|
|
117
118
|
| `-V, --version` | Print the version and exit. |
|
|
118
119
|
|
|
@@ -127,6 +128,7 @@ A YAML file — `bfsg.config.yaml` by default, or `--config <path>`. Only
|
|
|
127
128
|
```yaml
|
|
128
129
|
baseUrl: "https://example.de" # required
|
|
129
130
|
|
|
131
|
+
# sitemapOrigin: "https://example.de" # origin the sitemap was built for (see below)
|
|
130
132
|
maxPages: 50 # sitemap + crawl combined
|
|
131
133
|
settleMs: 0 # pause after load before the scan (see below)
|
|
132
134
|
wcagTags: # axe rule sets to run
|
|
@@ -154,6 +156,19 @@ would flag controls that populate a moment later. Set `settleMs` to a bit
|
|
|
154
156
|
more than that site's slowest post-load render (`1000`–`3000` is typical);
|
|
155
157
|
it is added to every page's scan time.
|
|
156
158
|
|
|
159
|
+
**`sitemapOrigin`** — a sitemap lists absolute URLs, and only entries on
|
|
160
|
+
`baseUrl`'s origin are scanned. Scanning a local or staging build whose
|
|
161
|
+
sitemap still lists production URLs would otherwise drop every entry: the
|
|
162
|
+
scanner warns with the number skipped, and when none are left it crawls
|
|
163
|
+
from `baseUrl` instead (exit `3` if that finds nothing either). Set
|
|
164
|
+
`sitemapOrigin` (or `--sitemap-origin`) to the production origin and its
|
|
165
|
+
entries — and sub-sitemaps — are rewritten onto `baseUrl`'s origin, path
|
|
166
|
+
and query kept:
|
|
167
|
+
|
|
168
|
+
```sh
|
|
169
|
+
npx bfsg-scanner http://localhost:8080/ --sitemap-origin https://example.de
|
|
170
|
+
```
|
|
171
|
+
|
|
157
172
|
## Output
|
|
158
173
|
|
|
159
174
|
Written to `outputDir`, overwritten each run:
|
|
@@ -177,9 +192,11 @@ The terminal prints a summary and the distinct breached clauses.
|
|
|
177
192
|
| `1` | Scan completed; violations at or above `--fail-on`. |
|
|
178
193
|
| `2` | Invalid arguments or configuration. |
|
|
179
194
|
| `3` | No pages discovered, or a page could not be scanned. |
|
|
195
|
+
| `4` | Unexpected error — e.g. Chromium not installed, output directory not writable. |
|
|
180
196
|
|
|
181
197
|
`3` outranks `1`: a run with unreachable pages scanned an incomplete site, so
|
|
182
|
-
"no violations found" would be a claim the data cannot support.
|
|
198
|
+
"no violations found" would be a claim the data cannot support. `4` means
|
|
199
|
+
the run itself broke, not the site: fix the environment and run again.
|
|
183
200
|
|
|
184
201
|
## Use in CI
|
|
185
202
|
|
|
@@ -187,13 +204,13 @@ The non-zero exit at or above `--fail-on` lets a CI job block a merge on
|
|
|
187
204
|
accessibility. A GitHub Actions step:
|
|
188
205
|
|
|
189
206
|
```yaml
|
|
190
|
-
- uses: actions/setup-node@
|
|
207
|
+
- uses: actions/setup-node@v7
|
|
191
208
|
with:
|
|
192
209
|
node-version: 22
|
|
193
210
|
- run: npx playwright install --with-deps chromium
|
|
194
211
|
- run: npx bfsg-scanner https://staging.example.de --fail-on serious
|
|
195
212
|
- if: always()
|
|
196
|
-
uses: actions/upload-artifact@
|
|
213
|
+
uses: actions/upload-artifact@v7
|
|
197
214
|
with:
|
|
198
215
|
name: accessibility-report
|
|
199
216
|
path: reports/
|
package/dist/cli/run.d.ts
CHANGED
|
@@ -18,6 +18,7 @@ interface CliArgs {
|
|
|
18
18
|
* 1 - scan completed, violations at or above `failOn` were found
|
|
19
19
|
* 2 - invalid arguments or configuration
|
|
20
20
|
* 3 - no pages were discovered, or one or more pages could not be scanned
|
|
21
|
+
* 4 - unexpected error, see `run`
|
|
21
22
|
*
|
|
22
23
|
* 3 outranks 1: a run with unreachable pages scanned an incomplete site, so
|
|
23
24
|
* "no violations found" would be a claim the data cannot support. The
|
|
@@ -25,5 +26,12 @@ interface CliArgs {
|
|
|
25
26
|
* survive either way.
|
|
26
27
|
*/
|
|
27
28
|
export declare function parseCliArgs(argv: readonly string[]): CliArgs;
|
|
29
|
+
/**
|
|
30
|
+
* Anything that escapes the handled paths — Chromium not installed, an
|
|
31
|
+
* unwritable output directory — ends here as exit code 4. Left to crash, it
|
|
32
|
+
* would surface as Node's default exit code 1, which the contract reserves
|
|
33
|
+
* for "violations found": a CI log would read a broken runner as an
|
|
34
|
+
* inaccessible site.
|
|
35
|
+
*/
|
|
28
36
|
export declare function run(argv: readonly string[], deps?: ScanDeps): Promise<number>;
|
|
29
37
|
export {};
|
package/dist/cli/run.js
CHANGED
|
@@ -23,7 +23,10 @@ Options:
|
|
|
23
23
|
--format <list> Comma-separated subset of json,html,pdf (default: all three).
|
|
24
24
|
--settle <ms> Pause after load, before the scan, for client-rendered
|
|
25
25
|
pages to hydrate (default: 0).
|
|
26
|
-
|
|
26
|
+
--sitemap-origin <o> Origin the sitemap was built for, e.g. production while
|
|
27
|
+
scanning a local or staging build; its entries are
|
|
28
|
+
rewritten onto the url's origin (default: none).
|
|
29
|
+
-h, --help Show this help and exit.
|
|
27
30
|
-V, --version Print the version and exit.
|
|
28
31
|
|
|
29
32
|
Exit codes:
|
|
@@ -31,6 +34,7 @@ Exit codes:
|
|
|
31
34
|
1 scan completed, violations at or above fail-on
|
|
32
35
|
2 invalid arguments or configuration
|
|
33
36
|
3 no pages discovered, or a page could not be scanned
|
|
37
|
+
4 unexpected error (e.g. Chromium not installed, output not writable)
|
|
34
38
|
`;
|
|
35
39
|
const OPTIONS = {
|
|
36
40
|
help: { type: 'boolean', short: 'h' },
|
|
@@ -41,6 +45,7 @@ const OPTIONS = {
|
|
|
41
45
|
'output-dir': { type: 'string' },
|
|
42
46
|
format: { type: 'string' },
|
|
43
47
|
settle: { type: 'string' },
|
|
48
|
+
'sitemap-origin': { type: 'string' },
|
|
44
49
|
};
|
|
45
50
|
function safeParseArgs(argv) {
|
|
46
51
|
try {
|
|
@@ -56,6 +61,7 @@ function safeParseArgs(argv) {
|
|
|
56
61
|
* 1 - scan completed, violations at or above `failOn` were found
|
|
57
62
|
* 2 - invalid arguments or configuration
|
|
58
63
|
* 3 - no pages were discovered, or one or more pages could not be scanned
|
|
64
|
+
* 4 - unexpected error, see `run`
|
|
59
65
|
*
|
|
60
66
|
* 3 outranks 1: a run with unreachable pages scanned an incomplete site, so
|
|
61
67
|
* "no violations found" would be a claim the data cannot support. The
|
|
@@ -87,6 +93,9 @@ export function parseCliArgs(argv) {
|
|
|
87
93
|
const parsed = Number(values.settle);
|
|
88
94
|
overrides.settleMs = Number.isNaN(parsed) ? values.settle : parsed;
|
|
89
95
|
}
|
|
96
|
+
if (values['sitemap-origin'] !== undefined) {
|
|
97
|
+
overrides.sitemapOrigin = values['sitemap-origin'];
|
|
98
|
+
}
|
|
90
99
|
if (values.format !== undefined) {
|
|
91
100
|
overrides.reportFormats = values.format
|
|
92
101
|
.split(',')
|
|
@@ -111,7 +120,23 @@ function resolveConfig(cli) {
|
|
|
111
120
|
}
|
|
112
121
|
return parseConfig({ ...fromFile, ...cli.overrides }, 'command-line arguments');
|
|
113
122
|
}
|
|
123
|
+
/**
|
|
124
|
+
* Anything that escapes the handled paths — Chromium not installed, an
|
|
125
|
+
* unwritable output directory — ends here as exit code 4. Left to crash, it
|
|
126
|
+
* would surface as Node's default exit code 1, which the contract reserves
|
|
127
|
+
* for "violations found": a CI log would read a broken runner as an
|
|
128
|
+
* inaccessible site.
|
|
129
|
+
*/
|
|
114
130
|
export async function run(argv, deps = {}) {
|
|
131
|
+
try {
|
|
132
|
+
return await runUnguarded(argv, deps);
|
|
133
|
+
}
|
|
134
|
+
catch (error) {
|
|
135
|
+
console.error(`Unexpected error: ${error instanceof Error ? error.message : String(error)}`);
|
|
136
|
+
return 4;
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
async function runUnguarded(argv, deps) {
|
|
115
140
|
let cli;
|
|
116
141
|
try {
|
|
117
142
|
cli = parseCliArgs(argv);
|
|
@@ -144,6 +169,8 @@ export async function run(argv, deps = {}) {
|
|
|
144
169
|
baseUrl: config.baseUrl,
|
|
145
170
|
maxPages: config.maxPages,
|
|
146
171
|
excludePaths: config.excludePaths,
|
|
172
|
+
...(config.sitemapOrigin !== undefined ? { sitemapOrigin: config.sitemapOrigin } : {}),
|
|
173
|
+
onWarning: (message) => console.warn(message),
|
|
147
174
|
});
|
|
148
175
|
if (urls.length === 0) {
|
|
149
176
|
console.error(`No scannable pages discovered for ${config.baseUrl}`);
|
package/dist/cli/run.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"run.js","sourceRoot":"","sources":["../../src/cli/run.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAgB,QAAQ,EAAE,MAAM,YAAY,CAAC;AACpD,OAAO,EAAe,WAAW,EAAE,UAAU,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACvF,OAAO,EAAE,YAAY,EAAE,MAAM,uBAAuB,CAAC;AACrD,OAAO,EACL,WAAW,EACX,gBAAgB,EAChB,eAAe,EACf,eAAe,EACf,cAAc,EACd,WAAW,GACZ,MAAM,oBAAoB,CAAC;AAC5B,OAAO,EAAiB,IAAI,EAAE,MAAM,kBAAkB,CAAC;AACvD,OAAO,EAAE,QAAQ,EAAE,MAAM,oBAAoB,CAAC;AAE9C,MAAM,mBAAmB,GAAG,kBAAkB,CAAC;AAE/C,MAAM,SAAS,GAAG;;;;;;;;;0DASwC,mBAAmB
|
|
1
|
+
{"version":3,"file":"run.js","sourceRoot":"","sources":["../../src/cli/run.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAgB,QAAQ,EAAE,MAAM,YAAY,CAAC;AACpD,OAAO,EAAe,WAAW,EAAE,UAAU,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACvF,OAAO,EAAE,YAAY,EAAE,MAAM,uBAAuB,CAAC;AACrD,OAAO,EACL,WAAW,EACX,gBAAgB,EAChB,eAAe,EACf,eAAe,EACf,cAAc,EACd,WAAW,GACZ,MAAM,oBAAoB,CAAC;AAC5B,OAAO,EAAiB,IAAI,EAAE,MAAM,kBAAkB,CAAC;AACvD,OAAO,EAAE,QAAQ,EAAE,MAAM,oBAAoB,CAAC;AAE9C,MAAM,mBAAmB,GAAG,kBAAkB,CAAC;AAE/C,MAAM,SAAS,GAAG;;;;;;;;;0DASwC,mBAAmB;;;;;;;;;;;;;;;;;;;;CAoB5E,CAAC;AAgBF,MAAM,OAAO,GAAG;IACd,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;IACrC,OAAO,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;IACxC,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,KAAK,EAAE,GAAG,EAAE;IACtC,SAAS,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IAC7B,iBAAiB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IACrC,YAAY,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IAChC,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IAC1B,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;IAC1B,gBAAgB,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;CAC5B,CAAC;AAEX,SAAS,aAAa,CAAC,IAAuB;IAC5C,IAAI,CAAC;QACH,OAAO,SAAS,CAAC,EAAE,IAAI,EAAE,CAAC,GAAG,IAAI,CAAC,EAAE,OAAO,EAAE,OAAO,EAAE,gBAAgB,EAAE,IAAI,EAAE,CAAC,CAAC;IAClF,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,IAAI,WAAW,CAAE,KAAe,CAAC,OAAO,CAAC,CAAC;IAClD,CAAC;AACH,CAAC;AAED;;;;;;;;;;;;GAYG;AACH,MAAM,UAAU,YAAY,CAAC,IAAuB;IAClD,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,CAAC,CAAC;IAEpD,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3B,MAAM,IAAI,WAAW,CAAC,0CAA0C,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC;IACzF,CAAC;IACD,MAAM,GAAG,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;IAC3B,IAAI,GAAG,KAAK,SAAS,IAAI,MAAM,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QACrD,MAAM,IAAI,WAAW,CAAC,4CAA4C,CAAC,CAAC;IACtE,CAAC;IAED,MAAM,SAAS,GAA4B,EAAE,CAAC;IAC9C,IAAI,MAAM,CAAC,SAAS,CAAC,KAAK,SAAS,EAAE,CAAC;QACpC,SAAS,CAAC,MAAM,GAAG,MAAM,CAAC,SAAS,CAAC,CAAC;IACvC,CAAC;IACD,IAAI,MAAM,CAAC,iBAAiB,CAAC,KAAK,SAAS,EAAE,CAAC;QAC5C,SAAS,CAAC,cAAc,GAAG,MAAM,CAAC,iBAAiB,CAAC,CAAC;IACvD,CAAC;IACD,IAAI,MAAM,CAAC,YAAY,CAAC,KAAK,SAAS,EAAE,CAAC;QACvC,SAAS,CAAC,SAAS,GAAG,MAAM,CAAC,YAAY,CAAC,CAAC;IAC7C,CAAC;IACD,IAAI,MAAM,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QAChC,gEAAgE;QAChE,6DAA6D;QAC7D,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QACrC,SAAS,CAAC,QAAQ,GAAG,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC;IACrE,CAAC;IACD,IAAI,MAAM,CAAC,gBAAgB,CAAC,KAAK,SAAS,EAAE,CAAC;QAC3C,SAAS,CAAC,aAAa,GAAG,MAAM,CAAC,gBAAgB,CAAC,CAAC;IACrD,CAAC;IACD,IAAI,MAAM,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;QAChC,SAAS,CAAC,aAAa,GAAG,MAAM,CAAC,MAAM;aACpC,KAAK,CAAC,GAAG,CAAC;aACV,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;aAC1B,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IACvC,CAAC;IAED,OAAO;QACL,IAAI,EAAE,MAAM,CAAC,IAAI,IAAI,KAAK;QAC1B,OAAO,EAAE,MAAM,CAAC,OAAO,IAAI,KAAK;QAChC,SAAS;QACT,GAAG,CAAC,MAAM,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACrE,GAAG,CAAC,GAAG,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,GAAG,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACtC,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,GAAY;IACjC,IAAI,GAAG,CAAC,GAAG,KAAK,SAAS,EAAE,CAAC;QAC1B,OAAO,WAAW,CAAC,EAAE,OAAO,EAAE,GAAG,CAAC,GAAG,EAAE,GAAG,GAAG,CAAC,SAAS,EAAE,EAAE,wBAAwB,CAAC,CAAC;IACvF,CAAC;IACD,MAAM,QAAQ,GAAG,UAAU,CAAC,GAAG,CAAC,UAAU,IAAI,mBAAmB,CAAC,CAAC;IACnE,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC5C,OAAO,QAAQ,CAAC;IAClB,CAAC;IACD,OAAO,WAAW,CAAC,EAAE,GAAG,QAAQ,EAAE,GAAG,GAAG,CAAC,SAAS,EAAE,EAAE,wBAAwB,CAAC,CAAC;AAClF,CAAC;AAED;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,GAAG,CAAC,IAAuB,EAAE,IAAI,GAAa,EAAE;IACpE,IAAI,CAAC;QACH,OAAO,MAAM,YAAY,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IACxC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO,CAAC,KAAK,CAAC,qBAAqB,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QAC7F,OAAO,CAAC,CAAC;IACX,CAAC;AACH,CAAC;AAED,KAAK,UAAU,YAAY,CAAC,IAAuB,EAAE,IAAc;IACjE,IAAI,GAAY,CAAC;IACjB,IAAI,CAAC;QACH,GAAG,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IAC3B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO,CAAC,KAAK,CAAC,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;QACtE,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;QACb,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACvB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,GAAG,CAAC,OAAO,EAAE,CAAC;QAChB,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,GAAG,QAAQ,EAAE,CAAC;QACrC,OAAO,CAAC,GAAG,CAAC,GAAG,IAAI,IAAI,OAAO,EAAE,CAAC,CAAC;QAClC,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,MAAc,CAAC;IACnB,IAAI,CAAC;QACH,MAAM,GAAG,aAAa,CAAC,GAAG,CAAC,CAAC;IAC9B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,WAAW,EAAE,CAAC;YACjC,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;YAC7B,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,KAAK,CAAC;IACd,CAAC;IAED,MAAM,IAAI,GAAG,MAAM,YAAY,CAAC;QAC9B,OAAO,EAAE,MAAM,CAAC,OAAO;QACvB,QAAQ,EAAE,MAAM,CAAC,QAAQ;QACzB,YAAY,EAAE,MAAM,CAAC,YAAY;QACjC,GAAG,CAAC,MAAM,CAAC,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,MAAM,CAAC,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACtF,SAAS,EAAE,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC;KAC9C,CAAC,CAAC;IACH,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACtB,OAAO,CAAC,KAAK,CAAC,qCAAqC,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC;QACrE,OAAO,CAAC,CAAC;IACX,CAAC;IACD,OAAO,CAAC,GAAG,CAAC,cAAc,IAAI,CAAC,MAAM,mBAAmB,CAAC,CAAC;IAE1D,0EAA0E;IAC1E,yEAAyE;IACzE,8EAA8E;IAC9E,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,IAAI,CAAC,MAAM,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;IAC1D,MAAM,YAAY,GAAG,IAAI,CAAC,OAAO,KAAK,SAAS,CAAC;IAChD,IAAI,CAAC;QACH,OAAO,MAAM,aAAa,CAAC,MAAM,EAAE,IAAI,EAAE,OAAO,CAAC,CAAC;IACpD,CAAC;YAAS,CAAC;QACT,IAAI,YAAY,EAAE,CAAC;YACjB,MAAM,OAAO,CAAC,KAAK,EAAE,CAAC;QACxB,CAAC;IACH,CAAC;AACH,CAAC;AAED,KAAK,UAAU,aAAa,CAC1B,MAAc,EACd,IAAuB,EACvB,OAAgB;IAEhB,MAAM,MAAM,GAAG,MAAM,IAAI,CACvB,IAAI,EACJ,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE,EACxD,EAAE,OAAO,EAAE,CACZ,CAAC;IAEF,MAAM,MAAM,GAAG,WAAW,CAAC,MAAM,EAAE;QACjC,OAAO,EAAE,MAAM,CAAC,OAAO;QACvB,QAAQ,EAAE,MAAM,CAAC,QAAQ;QACzB,MAAM,EAAE,MAAM,CAAC,MAAM;KACtB,CAAC,CAAC;IAEH,MAAM,OAAO,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,aAAa,CAAC,CAAC;IAC9C,IAAI,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;QACxB,OAAO,CAAC,GAAG,CAAC,qBAAqB,MAAM,WAAW,CAAC,MAAM,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;IAClF,CAAC;IACD,IAAI,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;QAC9C,MAAM,IAAI,GAAG,gBAAgB,CAAC,MAAM,EAAE,MAAM,CAAC,cAAc,CAAC,CAAC;QAC7D,IAAI,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;YACxB,OAAO,CAAC,GAAG,CAAC,0BAA0B,MAAM,eAAe,CAAC,IAAI,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;QACzF,CAAC;QACD,IAAI,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;YACvB,MAAM,GAAG,GAAG,MAAM,eAAe,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;YACjD,OAAO,CAAC,GAAG,CAAC,yBAAyB,MAAM,cAAc,CAAC,GAAG,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;QACtF,CAAC;IACH,CAAC;IAED,MAAM,WAAW,GAAG,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,KAAK,OAAO,CAAC,CAAC;IAC3E,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3B,KAAK,MAAM,IAAI,IAAI,WAAW,EAAE,CAAC;YAC/B,OAAO,CAAC,KAAK,CAAC,kBAAkB,IAAI,CAAC,GAAG,KAAK,IAAI,CAAC,KAAK,EAAE,CAAC,CAAC;QAC7D,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,OAAO,CAAC,GAAG,CACT,WAAW,MAAM,CAAC,OAAO,CAAC,YAAY,aAAa,MAAM,CAAC,OAAO,CAAC,eAAe,oBAAoB,CACtG,CAAC;IAEF,MAAM,EAAE,uBAAuB,EAAE,uBAAuB,EAAE,GAAG,MAAM,CAAC,OAAO,CAAC;IAC5E,IAAI,uBAAuB,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACvC,OAAO,CAAC,GAAG,CAAC,yBAAyB,uBAAuB,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QAC3E,OAAO,CAAC,GAAG,CAAC,gCAAgC,uBAAuB,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IACpF,CAAC;IAED,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,CAAC;IAC3B,IAAI,OAAO,CAAC,kBAAkB,GAAG,CAAC,EAAE,CAAC;QACnC,OAAO,CAAC,IAAI,CACV,GAAG,OAAO,CAAC,kBAAkB,mEAAmE,MAAM,CAAC,MAAM,wCAAwC,CACtJ,CAAC;IACJ,CAAC;IAED,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;QACpB,OAAO,CAAC,KAAK,CACX,GAAG,OAAO,CAAC,4BAA4B,8BAA8B,MAAM,CAAC,MAAM,IAAI,CACvF,CAAC;QACF,OAAO,CAAC,CAAC;IACX,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC"}
|
package/dist/config/schema.d.ts
CHANGED
|
@@ -22,6 +22,7 @@ export declare const reportFormatSchema: z.ZodEnum<{
|
|
|
22
22
|
}>;
|
|
23
23
|
export declare const configSchema: z.ZodObject<{
|
|
24
24
|
baseUrl: z.ZodURL;
|
|
25
|
+
sitemapOrigin: z.ZodOptional<z.ZodURL>;
|
|
25
26
|
maxPages: z.ZodDefault<z.ZodNumber>;
|
|
26
27
|
settleMs: z.ZodDefault<z.ZodNumber>;
|
|
27
28
|
wcagTags: z.ZodDefault<z.ZodArray<z.ZodEnum<{
|
package/dist/config/schema.js
CHANGED
|
@@ -3,8 +3,18 @@ export const wcagTagSchema = z.enum(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']
|
|
|
3
3
|
export const failOnSchema = z.enum(['critical', 'serious', 'moderate', 'minor']);
|
|
4
4
|
export const reportLanguageSchema = z.enum(['de', 'en']);
|
|
5
5
|
export const reportFormatSchema = z.enum(['json', 'html', 'pdf']);
|
|
6
|
+
/**
|
|
7
|
+
* A bare origin (`https://example.com`, an optional trailing slash): the
|
|
8
|
+
* part of a sitemap entry that gets swapped for baseUrl's origin. A path
|
|
9
|
+
* would suggest prefix rewriting, which this does not do.
|
|
10
|
+
*/
|
|
11
|
+
const originSchema = z.url().refine((value) => {
|
|
12
|
+
const url = new URL(value);
|
|
13
|
+
return url.pathname === '/' && url.search === '' && url.hash === '';
|
|
14
|
+
}, { message: 'Must be an origin, e.g. "https://example.com", with no path, query or fragment' });
|
|
6
15
|
export const configSchema = z.strictObject({
|
|
7
16
|
baseUrl: z.url(),
|
|
17
|
+
sitemapOrigin: originSchema.optional(),
|
|
8
18
|
maxPages: z.number().int().positive().max(1000).default(50),
|
|
9
19
|
settleMs: z.number().int().min(0).max(60_000).default(0),
|
|
10
20
|
wcagTags: z.array(wcagTagSchema).min(1).default(['wcag2a', 'wcag2aa', 'wcag21aa']),
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"schema.js","sourceRoot":"","sources":["../../src/config/schema.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAExB,MAAM,CAAC,MAAM,aAAa,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,SAAS,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC,CAAC;AAElF,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,UAAU,EAAE,SAAS,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC;AAEjF,MAAM,CAAC,MAAM,oBAAoB,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC;AAEzD,MAAM,CAAC,MAAM,kBAAkB,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC;AAElE,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,CAAC,YAAY,CAAC;IACzC,OAAO,EAAE,CAAC,CAAC,GAAG,EAAE;IAChB,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;IAC3D,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;IACxD,QAAQ,EAAE,CAAC,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;IAClF,YAAY,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;IAC7C,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC;IAC/C,MAAM,EAAE,YAAY,CAAC,OAAO,CAAC,SAAS,CAAC;IACvC,cAAc,EAAE,oBAAoB,CAAC,OAAO,CAAC,IAAI,CAAC;IAClD,aAAa,EAAE,CAAC,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC;CACnF,CAAC,CAAC"}
|
|
1
|
+
{"version":3,"file":"schema.js","sourceRoot":"","sources":["../../src/config/schema.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAExB,MAAM,CAAC,MAAM,aAAa,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,SAAS,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC,CAAC;AAElF,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,UAAU,EAAE,SAAS,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC;AAEjF,MAAM,CAAC,MAAM,oBAAoB,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC;AAEzD,MAAM,CAAC,MAAM,kBAAkB,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC;AAElE;;;;GAIG;AACH,MAAM,YAAY,GAAG,CAAC,CAAC,GAAG,EAAE,CAAC,MAAM,CACjC,CAAC,KAAK,EAAE,EAAE;IACR,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC;IAC3B,OAAO,GAAG,CAAC,QAAQ,KAAK,GAAG,IAAI,GAAG,CAAC,MAAM,KAAK,EAAE,IAAI,GAAG,CAAC,IAAI,KAAK,EAAE,CAAC;AACtE,CAAC,EACD,EAAE,OAAO,EAAE,gFAAgF,EAAE,CAC9F,CAAC;AAEF,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,CAAC,YAAY,CAAC;IACzC,OAAO,EAAE,CAAC,CAAC,GAAG,EAAE;IAChB,aAAa,EAAE,YAAY,CAAC,QAAQ,EAAE;IACtC,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;IAC3D,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;IACxD,QAAQ,EAAE,CAAC,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;IAClF,YAAY,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC;IAC7C,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC;IAC/C,MAAM,EAAE,YAAY,CAAC,OAAO,CAAC,SAAS,CAAC;IACvC,cAAc,EAAE,oBAAoB,CAAC,OAAO,CAAC,IAAI,CAAC;IAClD,aAAa,EAAE,CAAC,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC;CACnF,CAAC,CAAC"}
|
|
@@ -5,13 +5,24 @@ export interface DiscoverSiteOptions {
|
|
|
5
5
|
readonly excludePaths?: readonly string[];
|
|
6
6
|
/** Injectable for tests; defaults to the global fetch (with User-Agent). */
|
|
7
7
|
readonly fetchFn?: typeof fetch;
|
|
8
|
+
/** Origin the sitemap was generated for, rewritten onto baseUrl's; see discoverFromSitemap. */
|
|
9
|
+
readonly sitemapOrigin?: string;
|
|
10
|
+
/** Receives one line per discovery problem worth surfacing. Defaults to none. */
|
|
11
|
+
readonly onWarning?: (message: string) => void;
|
|
8
12
|
}
|
|
9
13
|
/**
|
|
10
14
|
* Resolves the list of page URLs to scan for a site.
|
|
11
15
|
*
|
|
12
16
|
* Strategy: prefer the site's sitemap.xml — it is the owner's own
|
|
13
17
|
* declaration of which pages exist. If the root sitemap is missing or
|
|
14
|
-
* unparseable,
|
|
18
|
+
* unparseable, or lists no scannable page on baseUrl's origin, fall back
|
|
19
|
+
* to a breadth-first crawl from baseUrl.
|
|
20
|
+
*
|
|
21
|
+
* Sitemap entries on another origin are dropped with a warning: they are
|
|
22
|
+
* the typical sign of a local or staging build whose sitemap points at
|
|
23
|
+
* production, and dropping them silently would leave a scan of the home
|
|
24
|
+
* page alone looking like a clean site. sitemapOrigin names that other
|
|
25
|
+
* origin, so its entries are rewritten onto baseUrl's and scanned there.
|
|
15
26
|
*
|
|
16
27
|
* Both strategies apply excludePaths and the non-page-asset filter
|
|
17
28
|
* identically. They differ on robots.txt: the crawl path obeys it (a
|
|
@@ -6,7 +6,14 @@ import { SitemapError } from './sitemap.js';
|
|
|
6
6
|
*
|
|
7
7
|
* Strategy: prefer the site's sitemap.xml — it is the owner's own
|
|
8
8
|
* declaration of which pages exist. If the root sitemap is missing or
|
|
9
|
-
* unparseable,
|
|
9
|
+
* unparseable, or lists no scannable page on baseUrl's origin, fall back
|
|
10
|
+
* to a breadth-first crawl from baseUrl.
|
|
11
|
+
*
|
|
12
|
+
* Sitemap entries on another origin are dropped with a warning: they are
|
|
13
|
+
* the typical sign of a local or staging build whose sitemap points at
|
|
14
|
+
* production, and dropping them silently would leave a scan of the home
|
|
15
|
+
* page alone looking like a clean site. sitemapOrigin names that other
|
|
16
|
+
* origin, so its entries are rewritten onto baseUrl's and scanned there.
|
|
10
17
|
*
|
|
11
18
|
* Both strategies apply excludePaths and the non-page-asset filter
|
|
12
19
|
* identically. They differ on robots.txt: the crawl path obeys it (a
|
|
@@ -19,11 +26,19 @@ import { SitemapError } from './sitemap.js';
|
|
|
19
26
|
* from baseUrl. The result never exceeds maxPages.
|
|
20
27
|
*/
|
|
21
28
|
export async function discoverSite(options) {
|
|
22
|
-
const { baseUrl, maxPages, excludePaths = [], fetchFn } = options;
|
|
29
|
+
const { baseUrl, maxPages, excludePaths = [], fetchFn, sitemapOrigin, onWarning } = options;
|
|
23
30
|
const shared = { baseUrl, maxPages, excludePaths, ...(fetchFn ? { fetchFn } : {}) };
|
|
31
|
+
const origin = new URL(baseUrl).origin;
|
|
24
32
|
try {
|
|
25
|
-
const pages = await discoverFromSitemap(
|
|
26
|
-
|
|
33
|
+
const pages = await discoverFromSitemap({
|
|
34
|
+
...shared,
|
|
35
|
+
...(sitemapOrigin !== undefined ? { sitemapOrigin } : {}),
|
|
36
|
+
onOffOrigin: (skipped) => onWarning?.(describeOffOrigin(skipped, origin, sitemapOrigin === undefined)),
|
|
37
|
+
});
|
|
38
|
+
if (pages.length > 0) {
|
|
39
|
+
return withBaseUrl(pages, baseUrl, maxPages);
|
|
40
|
+
}
|
|
41
|
+
onWarning?.(`The sitemap lists no scannable page on ${origin}; crawling from ${baseUrl}.`);
|
|
27
42
|
}
|
|
28
43
|
catch (error) {
|
|
29
44
|
if (!(error instanceof SitemapError)) {
|
|
@@ -32,6 +47,15 @@ export async function discoverSite(options) {
|
|
|
32
47
|
}
|
|
33
48
|
return crawlSite(shared);
|
|
34
49
|
}
|
|
50
|
+
function describeOffOrigin(skipped, origin, withHint) {
|
|
51
|
+
const origins = [...new Set(skipped.map((url) => new URL(url).origin))];
|
|
52
|
+
const shown = origins.length > 3 ? [...origins.slice(0, 3), '…'] : origins;
|
|
53
|
+
const entries = skipped.length === 1 ? 'entry' : 'entries';
|
|
54
|
+
const message = `Skipped ${skipped.length} sitemap ${entries} on another origin than ${origin} (${shown.join(', ')}).`;
|
|
55
|
+
return withHint
|
|
56
|
+
? `${message} Set sitemapOrigin (--sitemap-origin) to scan them on ${origin}.`
|
|
57
|
+
: message;
|
|
58
|
+
}
|
|
35
59
|
function withBaseUrl(pages, baseUrl, maxPages) {
|
|
36
60
|
const normalized = new URL(baseUrl).toString();
|
|
37
61
|
const alreadyListed = pages.some((page) => {
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"discover-site.js","sourceRoot":"","sources":["../../src/discovery/discover-site.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AACvC,OAAO,EAAE,mBAAmB,EAAE,MAAM,eAAe,CAAC;AACpD,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;
|
|
1
|
+
{"version":3,"file":"discover-site.js","sourceRoot":"","sources":["../../src/discovery/discover-site.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AACvC,OAAO,EAAE,mBAAmB,EAAE,MAAM,eAAe,CAAC;AACpD,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAe5C;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAAC,OAA4B;IAC7D,MAAM,EAAE,OAAO,EAAE,QAAQ,EAAE,YAAY,GAAG,EAAE,EAAE,OAAO,EAAE,aAAa,EAAE,SAAS,EAAE,GAAG,OAAO,CAAC;IAC5F,MAAM,MAAM,GAAG,EAAE,OAAO,EAAE,QAAQ,EAAE,YAAY,EAAE,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC;IACpF,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC;IAEvC,IAAI,CAAC;QACH,MAAM,KAAK,GAAG,MAAM,mBAAmB,CAAC;YACtC,GAAG,MAAM;YACT,GAAG,CAAC,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACzD,WAAW,EAAE,CAAC,OAAO,EAAE,EAAE,CACvB,SAAS,EAAE,CAAC,iBAAiB,CAAC,OAAO,EAAE,MAAM,EAAE,aAAa,KAAK,SAAS,CAAC,CAAC;SAC/E,CAAC,CAAC;QACH,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACrB,OAAO,WAAW,CAAC,KAAK,EAAE,OAAO,EAAE,QAAQ,CAAC,CAAC;QAC/C,CAAC;QACD,SAAS,EAAE,CAAC,0CAA0C,MAAM,mBAAmB,OAAO,GAAG,CAAC,CAAC;IAC7F,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,CAAC,CAAC,KAAK,YAAY,YAAY,CAAC,EAAE,CAAC;YACrC,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED,OAAO,SAAS,CAAC,MAAM,CAAC,CAAC;AAC3B,CAAC;AAED,SAAS,iBAAiB,CAAC,OAA0B,EAAE,MAAc,EAAE,QAAiB;IACtF,MAAM,OAAO,GAAG,CAAC,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;IACxE,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC;IAC3E,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;IAC3D,MAAM,OAAO,GAAG,WAAW,OAAO,CAAC,MAAM,YAAY,OAAO,2BAA2B,MAAM,KAAK,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC;IACvH,OAAO,QAAQ;QACb,CAAC,CAAC,GAAG,OAAO,yDAAyD,MAAM,GAAG;QAC9E,CAAC,CAAC,OAAO,CAAC;AACd,CAAC;AAED,SAAS,WAAW,CAAC,KAAwB,EAAE,OAAe,EAAE,QAAgB;IAC9E,MAAM,UAAU,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC;IAC/C,MAAM,aAAa,GAAG,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE;QACxC,IAAI,CAAC;YACH,OAAO,IAAI,GAAG,CAAC,IAAI,CAAC,CAAC,QAAQ,EAAE,KAAK,UAAU,CAAC;QACjD,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,KAAK,CAAC;QACf,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,IAAI,aAAa,EAAE,CAAC;QAClB,OAAO,CAAC,GAAG,KAAK,CAAC,CAAC;IACpB,CAAC;IACD,OAAO,CAAC,UAAU,EAAE,GAAG,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,QAAQ,CAAC,CAAC;AACnD,CAAC"}
|
|
@@ -9,17 +9,30 @@ export interface DiscoverFromSitemapOptions {
|
|
|
9
9
|
readonly fetchFn?: typeof fetch;
|
|
10
10
|
/** Guards against pathological or cyclic sitemap indexes. Defaults to 3. */
|
|
11
11
|
readonly maxSitemapDepth?: number;
|
|
12
|
+
/**
|
|
13
|
+
* Origin the sitemap was generated for, when it is not baseUrl's — e.g.
|
|
14
|
+
* production, while baseUrl is a local or staging build. Entries (and
|
|
15
|
+
* sub-sitemaps) on this origin are rewritten onto baseUrl's origin,
|
|
16
|
+
* keeping path and query. Defaults to none.
|
|
17
|
+
*/
|
|
18
|
+
readonly sitemapOrigin?: string;
|
|
19
|
+
/**
|
|
20
|
+
* Called once, after discovery, with the distinct page entries dropped
|
|
21
|
+
* for being on another origin than baseUrl. Not called when there were none.
|
|
22
|
+
*/
|
|
23
|
+
readonly onOffOrigin?: (skippedUrls: readonly string[]) => void;
|
|
12
24
|
}
|
|
13
25
|
/**
|
|
14
26
|
* Discovers page URLs from a site's sitemap, following a sitemap index to
|
|
15
27
|
* its sub-sitemaps as needed. Off-origin entries are dropped (a sitemap
|
|
16
|
-
* should only ever list its own site, but nothing enforces that)
|
|
17
|
-
* assets and excludePaths matches
|
|
18
|
-
* crawlSite), duplicate URLs are
|
|
19
|
-
* maxPages is reached.
|
|
28
|
+
* should only ever list its own site, but nothing enforces that) and
|
|
29
|
+
* reported through onOffOrigin, non-page assets and excludePaths matches
|
|
30
|
+
* are filtered out (identically to crawlSite), duplicate URLs are
|
|
31
|
+
* collapsed, and discovery stops as soon as maxPages is reached.
|
|
20
32
|
*
|
|
21
|
-
* A failure to load the root sitemap is fatal
|
|
22
|
-
*
|
|
23
|
-
*
|
|
33
|
+
* A failure to load or parse the root sitemap is fatal and always surfaces
|
|
34
|
+
* as a SitemapError (the caller should fall back to crawling). A failure on
|
|
35
|
+
* a sub-sitemap referenced by an index is not: that one sub-sitemap is
|
|
36
|
+
* skipped and discovery continues with the rest.
|
|
24
37
|
*/
|
|
25
38
|
export declare function discoverFromSitemap(options: DiscoverFromSitemapOptions): Promise<string[]>;
|
|
@@ -4,23 +4,29 @@ import { parseSitemapXml, SitemapError } from './sitemap.js';
|
|
|
4
4
|
/**
|
|
5
5
|
* Discovers page URLs from a site's sitemap, following a sitemap index to
|
|
6
6
|
* its sub-sitemaps as needed. Off-origin entries are dropped (a sitemap
|
|
7
|
-
* should only ever list its own site, but nothing enforces that)
|
|
8
|
-
* assets and excludePaths matches
|
|
9
|
-
* crawlSite), duplicate URLs are
|
|
10
|
-
* maxPages is reached.
|
|
7
|
+
* should only ever list its own site, but nothing enforces that) and
|
|
8
|
+
* reported through onOffOrigin, non-page assets and excludePaths matches
|
|
9
|
+
* are filtered out (identically to crawlSite), duplicate URLs are
|
|
10
|
+
* collapsed, and discovery stops as soon as maxPages is reached.
|
|
11
11
|
*
|
|
12
|
-
* A failure to load the root sitemap is fatal
|
|
13
|
-
*
|
|
14
|
-
*
|
|
12
|
+
* A failure to load or parse the root sitemap is fatal and always surfaces
|
|
13
|
+
* as a SitemapError (the caller should fall back to crawling). A failure on
|
|
14
|
+
* a sub-sitemap referenced by an index is not: that one sub-sitemap is
|
|
15
|
+
* skipped and discovery continues with the rest.
|
|
15
16
|
*/
|
|
16
17
|
export async function discoverFromSitemap(options) {
|
|
17
|
-
const { baseUrl, maxPages, sitemapPath = '/sitemap.xml', excludePaths = [], fetchFn = fetchWithUserAgent, maxSitemapDepth = 3, } = options;
|
|
18
|
+
const { baseUrl, maxPages, sitemapPath = '/sitemap.xml', excludePaths = [], fetchFn = fetchWithUserAgent, maxSitemapDepth = 3, sitemapOrigin, onOffOrigin, } = options;
|
|
18
19
|
const origin = new URL(baseUrl).origin;
|
|
20
|
+
const rewriteFrom = sitemapOrigin === undefined ? undefined : new URL(sitemapOrigin).origin;
|
|
19
21
|
const rootSitemapUrl = new URL(sitemapPath, baseUrl).toString();
|
|
20
22
|
const collected = [];
|
|
23
|
+
const offOrigin = [];
|
|
21
24
|
const seenPages = new Set();
|
|
22
25
|
const visitedSitemaps = new Set();
|
|
23
26
|
await visitSitemap(rootSitemapUrl, 0);
|
|
27
|
+
if (offOrigin.length > 0) {
|
|
28
|
+
onOffOrigin?.(offOrigin);
|
|
29
|
+
}
|
|
24
30
|
return collected.slice(0, maxPages);
|
|
25
31
|
async function visitSitemap(sitemapUrl, depth) {
|
|
26
32
|
if (collected.length >= maxPages) {
|
|
@@ -30,13 +36,17 @@ export async function discoverFromSitemap(options) {
|
|
|
30
36
|
return;
|
|
31
37
|
}
|
|
32
38
|
visitedSitemaps.add(sitemapUrl);
|
|
33
|
-
|
|
39
|
+
// Parsing sits inside the same guard as the fetch: a URL that answers
|
|
40
|
+
// 200 with something other than a sitemap (a soft-404 HTML page, a
|
|
41
|
+
// truncated file) is as unusable as one that answers 404, and must be
|
|
42
|
+
// skipped the same way rather than abort the whole sitemap.
|
|
43
|
+
let doc;
|
|
34
44
|
try {
|
|
35
45
|
const response = await fetchFn(sitemapUrl);
|
|
36
46
|
if (!response.ok) {
|
|
37
47
|
throw new Error(`HTTP ${response.status}`);
|
|
38
48
|
}
|
|
39
|
-
|
|
49
|
+
doc = parseSitemapXml(await response.text());
|
|
40
50
|
}
|
|
41
51
|
catch (cause) {
|
|
42
52
|
if (depth === 0) {
|
|
@@ -44,20 +54,22 @@ export async function discoverFromSitemap(options) {
|
|
|
44
54
|
}
|
|
45
55
|
return;
|
|
46
56
|
}
|
|
47
|
-
const doc = parseSitemapXml(xml);
|
|
48
57
|
if (doc.kind === 'sitemapindex') {
|
|
49
58
|
for (const childUrl of doc.sitemaps) {
|
|
50
59
|
if (collected.length >= maxPages) {
|
|
51
60
|
return;
|
|
52
61
|
}
|
|
53
|
-
await visitSitemap(childUrl, depth + 1);
|
|
62
|
+
await visitSitemap(rewriteOrigin(childUrl), depth + 1);
|
|
54
63
|
}
|
|
55
64
|
return;
|
|
56
65
|
}
|
|
57
|
-
for (const
|
|
66
|
+
for (const listedUrl of doc.urls) {
|
|
58
67
|
if (collected.length >= maxPages) {
|
|
59
68
|
return;
|
|
60
69
|
}
|
|
70
|
+
// Deduplicated after the rewrite, so a sitemap listing a page on both
|
|
71
|
+
// origins yields it once.
|
|
72
|
+
const pageUrl = rewriteOrigin(listedUrl);
|
|
61
73
|
if (seenPages.has(pageUrl)) {
|
|
62
74
|
continue;
|
|
63
75
|
}
|
|
@@ -69,13 +81,32 @@ export async function discoverFromSitemap(options) {
|
|
|
69
81
|
catch {
|
|
70
82
|
continue;
|
|
71
83
|
}
|
|
72
|
-
if (parsed.origin !== origin
|
|
73
|
-
|
|
74
|
-
|
|
84
|
+
if (parsed.origin !== origin) {
|
|
85
|
+
offOrigin.push(pageUrl);
|
|
86
|
+
continue;
|
|
87
|
+
}
|
|
88
|
+
if (!isCrawlableUrl(parsed) || matchesExcludePattern(parsed.pathname, excludePaths)) {
|
|
75
89
|
continue;
|
|
76
90
|
}
|
|
77
91
|
collected.push(pageUrl);
|
|
78
92
|
}
|
|
79
93
|
}
|
|
94
|
+
/** Moves a URL on sitemapOrigin onto baseUrl's origin; any other URL is returned verbatim. */
|
|
95
|
+
function rewriteOrigin(url) {
|
|
96
|
+
if (rewriteFrom === undefined) {
|
|
97
|
+
return url;
|
|
98
|
+
}
|
|
99
|
+
let parsed;
|
|
100
|
+
try {
|
|
101
|
+
parsed = new URL(url);
|
|
102
|
+
}
|
|
103
|
+
catch {
|
|
104
|
+
return url;
|
|
105
|
+
}
|
|
106
|
+
if (parsed.origin !== rewriteFrom) {
|
|
107
|
+
return url;
|
|
108
|
+
}
|
|
109
|
+
return new URL(`${parsed.pathname}${parsed.search}${parsed.hash}`, origin).toString();
|
|
110
|
+
}
|
|
80
111
|
}
|
|
81
112
|
//# sourceMappingURL=discover.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"discover.js","sourceRoot":"","sources":["../../src/discovery/discover.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,kBAAkB,EAAE,MAAM,yBAAyB,CAAC;AAC7D,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,MAAM,YAAY,CAAC;AACnE,OAAO,EAAE,eAAe,
|
|
1
|
+
{"version":3,"file":"discover.js","sourceRoot":"","sources":["../../src/discovery/discover.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,kBAAkB,EAAE,MAAM,yBAAyB,CAAC;AAC7D,OAAO,EAAE,cAAc,EAAE,qBAAqB,EAAE,MAAM,YAAY,CAAC;AACnE,OAAO,EAAE,eAAe,EAAwB,YAAY,EAAE,MAAM,cAAc,CAAC;AA2BnF;;;;;;;;;;;;GAYG;AACH,MAAM,CAAC,KAAK,UAAU,mBAAmB,CAAC,OAAmC;IAC3E,MAAM,EACJ,OAAO,EACP,QAAQ,EACR,WAAW,GAAG,cAAc,EAC5B,YAAY,GAAG,EAAE,EACjB,OAAO,GAAG,kBAAkB,EAC5B,eAAe,GAAG,CAAC,EACnB,aAAa,EACb,WAAW,GACZ,GAAG,OAAO,CAAC;IAEZ,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC;IACvC,MAAM,WAAW,GAAG,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,GAAG,CAAC,aAAa,CAAC,CAAC,MAAM,CAAC;IAC5F,MAAM,cAAc,GAAG,IAAI,GAAG,CAAC,WAAW,EAAE,OAAO,CAAC,CAAC,QAAQ,EAAE,CAAC;IAEhE,MAAM,SAAS,GAAa,EAAE,CAAC;IAC/B,MAAM,SAAS,GAAa,EAAE,CAAC;IAC/B,MAAM,SAAS,GAAG,IAAI,GAAG,EAAU,CAAC;IACpC,MAAM,eAAe,GAAG,IAAI,GAAG,EAAU,CAAC;IAE1C,MAAM,YAAY,CAAC,cAAc,EAAE,CAAC,CAAC,CAAC;IAEtC,IAAI,SAAS,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACzB,WAAW,EAAE,CAAC,SAAS,CAAC,CAAC;IAC3B,CAAC;IACD,OAAO,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,QAAQ,CAAC,CAAC;IAEpC,KAAK,UAAU,YAAY,CAAC,UAAkB,EAAE,KAAa;QAC3D,IAAI,SAAS,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;YACjC,OAAO;QACT,CAAC;QACD,IAAI,eAAe,CAAC,GAAG,CAAC,UAAU,CAAC,IAAI,KAAK,GAAG,eAAe,EAAE,CAAC;YAC/D,OAAO;QACT,CAAC;QACD,eAAe,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QAEhC,sEAAsE;QACtE,mEAAmE;QACnE,sEAAsE;QACtE,4DAA4D;QAC5D,IAAI,GAAoB,CAAC;QACzB,IAAI,CAAC;YACH,MAAM,QAAQ,GAAG,MAAM,OAAO,CAAC,UAAU,CAAC,CAAC;YAC3C,IAAI,CAAC,QAAQ,CAAC,EAAE,EAAE,CAAC;gBACjB,MAAM,IAAI,KAAK,CAAC,QAAQ,QAAQ,CAAC,MAAM,EAAE,CAAC,CAAC;YAC7C,CAAC;YACD,GAAG,GAAG,eAAe,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC,CAAC;QAC/C,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;gBAChB,MAAM,IAAI,YAAY,CACpB,2BAA2B,UAAU,MAAM,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CACpG,CAAC;YACJ,CAAC;YACD,OAAO;QACT,CAAC;QAED,IAAI,GAAG,CAAC,IAAI,KAAK,cAAc,EAAE,CAAC;YAChC,KAAK,MAAM,QAAQ,IAAI,GAAG,CAAC,QAAQ,EAAE,CAAC;gBACpC,IAAI,SAAS,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;oBACjC,OAAO;gBACT,CAAC;gBACD,MAAM,YAAY,CAAC,aAAa,CAAC,QAAQ,CAAC,EAAE,KAAK,GAAG,CAAC,CAAC,CAAC;YACzD,CAAC;YACD,OAAO;QACT,CAAC;QAED,KAAK,MAAM,SAAS,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;YACjC,IAAI,SAAS,CAAC,MAAM,IAAI,QAAQ,EAAE,CAAC;gBACjC,OAAO;YACT,CAAC;YACD,sEAAsE;YACtE,0BAA0B;YAC1B,MAAM,OAAO,GAAG,aAAa,CAAC,SAAS,CAAC,CAAC;YACzC,IAAI,SAAS,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC;gBAC3B,SAAS;YACX,CAAC;YACD,SAAS,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;YAEvB,IAAI,MAAW,CAAC;YAChB,IAAI,CAAC;gBACH,MAAM,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC;YAC5B,CAAC;YAAC,MAAM,CAAC;gBACP,SAAS;YACX,CAAC;YACD,IAAI,MAAM,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;gBAC7B,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;gBACxB,SAAS;YACX,CAAC;YACD,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,IAAI,qBAAqB,CAAC,MAAM,CAAC,QAAQ,EAAE,YAAY,CAAC,EAAE,CAAC;gBACpF,SAAS;YACX,CAAC;YACD,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;QAC1B,CAAC;IACH,CAAC;IAED,8FAA8F;IAC9F,SAAS,aAAa,CAAC,GAAW;QAChC,IAAI,WAAW,KAAK,SAAS,EAAE,CAAC;YAC9B,OAAO,GAAG,CAAC;QACb,CAAC;QACD,IAAI,MAAW,CAAC;QAChB,IAAI,CAAC;YACH,MAAM,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC;QACxB,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,GAAG,CAAC;QACb,CAAC;QACD,IAAI,MAAM,CAAC,MAAM,KAAK,WAAW,EAAE,CAAC;YAClC,OAAO,GAAG,CAAC;QACb,CAAC;QACD,OAAO,IAAI,GAAG,CAAC,GAAG,MAAM,CAAC,QAAQ,GAAG,MAAM,CAAC,MAAM,GAAG,MAAM,CAAC,IAAI,EAAE,EAAE,MAAM,CAAC,CAAC,QAAQ,EAAE,CAAC;IACxF,CAAC;AACH,CAAC"}
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# 11. An exit code for unexpected errors
|
|
2
|
+
|
|
3
|
+
Date: 2026-09-29
|
|
4
|
+
|
|
5
|
+
## Status
|
|
6
|
+
|
|
7
|
+
Accepted. Extends the exit-code table of ADR 0003.
|
|
8
|
+
|
|
9
|
+
## Context
|
|
10
|
+
|
|
11
|
+
ADR 0003 defines `0` (pass), `1` (violations at or above `failOn`), `2`
|
|
12
|
+
(bad arguments or configuration) and `3` (incomplete scan). Every path that
|
|
13
|
+
`run` anticipates maps to one of them. A failure it does not anticipate —
|
|
14
|
+
Chromium missing on a first `npx` run, an output directory that cannot be
|
|
15
|
+
created — escaped as an unhandled rejection. Node then printed a stack
|
|
16
|
+
trace and exited `1`.
|
|
17
|
+
|
|
18
|
+
`1` is the one code a CI gate reads as a verdict on the site. A broken
|
|
19
|
+
runner therefore looked like an inaccessible site, and the person reading
|
|
20
|
+
the log went looking for violations that were never measured.
|
|
21
|
+
|
|
22
|
+
## Decision
|
|
23
|
+
|
|
24
|
+
`run` catches anything that escapes the handled paths, prints a one-line
|
|
25
|
+
`Unexpected error: <message>` to stderr, and returns **`4`**.
|
|
26
|
+
|
|
27
|
+
Alternatives considered:
|
|
28
|
+
|
|
29
|
+
- **Reuse `3`.** `3` already means "no trustworthy result". But it points
|
|
30
|
+
at the *site* (unreachable pages, nothing discoverable), and the fix is
|
|
31
|
+
on the site's side. A missing browser or a read-only disk is fixed on
|
|
32
|
+
the *runner's* side. Separate codes keep that distinction machine-readable.
|
|
33
|
+
- **Leave the crash, document it.** Cheapest, but the collision with `1`
|
|
34
|
+
is the defect being fixed; documenting it does not remove it.
|
|
35
|
+
|
|
36
|
+
The message is kept to the error's own text. Playwright's launch error
|
|
37
|
+
already names the `npx playwright install` command, so no hint is added
|
|
38
|
+
on top of it.
|
|
39
|
+
|
|
40
|
+
## Consequences
|
|
41
|
+
|
|
42
|
+
- The code is additive: a CI job that treats any non-zero code as failure
|
|
43
|
+
behaves as before; one that tests for `1` specifically no longer fires on
|
|
44
|
+
a broken runner.
|
|
45
|
+
- The stack trace is no longer printed. The message carries the cause;
|
|
46
|
+
a developer debugging the tool itself runs it from source.
|
|
47
|
+
- Any future failure mode that deserves its own code must be caught
|
|
48
|
+
explicitly before it reaches this net.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# 12. Off-origin sitemap entries
|
|
2
|
+
|
|
3
|
+
Date: 2026-10-17
|
|
4
|
+
|
|
5
|
+
## Status
|
|
6
|
+
|
|
7
|
+
Accepted
|
|
8
|
+
|
|
9
|
+
## Context
|
|
10
|
+
|
|
11
|
+
ADR 0001 collects only same-origin `<loc>` entries from the sitemap and
|
|
12
|
+
falls back to crawling only when the root sitemap is missing or unparseable.
|
|
13
|
+
Dropping off-origin entries was silent.
|
|
14
|
+
|
|
15
|
+
The common way to hit it is a local or staging build whose `sitemap.xml`
|
|
16
|
+
was generated for production: every entry is off-origin, the sitemap still
|
|
17
|
+
parses, so no crawl runs, `baseUrl` is prepended, and the run scans the home
|
|
18
|
+
page alone. With a clean home page that exits `0` — a CI gate reporting a
|
|
19
|
+
clean site it never looked at (#58).
|
|
20
|
+
|
|
21
|
+
## Decision
|
|
22
|
+
|
|
23
|
+
**Warn.** `discoverFromSitemap` reports the distinct off-origin entries it
|
|
24
|
+
dropped; the CLI prints one line with the count, the origins involved, and
|
|
25
|
+
a hint to set `sitemapOrigin`.
|
|
26
|
+
|
|
27
|
+
**A sitemap with no usable page falls back to the crawl.** When the sitemap
|
|
28
|
+
yields no scannable page on `baseUrl`'s origin — all off-origin, all
|
|
29
|
+
excluded, or simply empty — `discoverSite` treats it like a missing sitemap
|
|
30
|
+
and crawls from `baseUrl`, with a warning that it did. A sitemap that leaves
|
|
31
|
+
at least one page is used as before: a partial sitemap is still the owner's
|
|
32
|
+
own list (ADR 0001). If the crawl finds nothing either, discovery returns no
|
|
33
|
+
pages and the run exits `3`, the existing "no pages discovered" code.
|
|
34
|
+
|
|
35
|
+
**`sitemapOrigin` rewrites one origin onto `baseUrl`'s.** A bare origin, in
|
|
36
|
+
config or as `--sitemap-origin`. Page entries and sub-sitemap URLs on that
|
|
37
|
+
origin get `baseUrl`'s origin, path, query and fragment kept, so a staging
|
|
38
|
+
build is scanned page for page against the production sitemap, and the
|
|
39
|
+
build's own copies of sub-sitemaps are read. Entries on any third origin
|
|
40
|
+
are still dropped with the warning. Deduplication runs after the rewrite.
|
|
41
|
+
|
|
42
|
+
It gets a CLI flag, unlike `maxPages` or `excludePaths` (ADR 0007): the case
|
|
43
|
+
it serves — a one-off scan of `http://localhost:…` in a build job — is the
|
|
44
|
+
one a positional URL exists for.
|
|
45
|
+
|
|
46
|
+
## Consequences
|
|
47
|
+
|
|
48
|
+
- No exit code changes meaning. A run that used to pass on the home page
|
|
49
|
+
alone now either scans what the crawl reaches, or exits `3`.
|
|
50
|
+
- A sitemap that lists nothing scannable now costs a crawl. That is the
|
|
51
|
+
same cost as a site with no sitemap.
|
|
52
|
+
- Only the origin is rewritten, never a path prefix. A staging copy served
|
|
53
|
+
under a sub-path (`https://staging/app/` for `https://prod/`) is not
|
|
54
|
+
covered; that would need a prefix mapping, deferred until asked for.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "bfsg-scanner",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "WCAG 2.1 AA / BFSG compliance scanner built on Playwright and axe-core",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"accessibility",
|
|
@@ -54,20 +54,20 @@
|
|
|
54
54
|
"prepublishOnly": "npm run verify && npm run build"
|
|
55
55
|
},
|
|
56
56
|
"dependencies": {
|
|
57
|
-
"@axe-core/playwright": "4.
|
|
58
|
-
"axe-core": "4.
|
|
59
|
-
"fast-xml-parser": "5.11.
|
|
60
|
-
"playwright": "1.
|
|
61
|
-
"yaml": "2.9.
|
|
62
|
-
"zod": "4.6.
|
|
57
|
+
"@axe-core/playwright": "4.14.0",
|
|
58
|
+
"axe-core": "4.14.0",
|
|
59
|
+
"fast-xml-parser": "5.11.2",
|
|
60
|
+
"playwright": "1.64.0",
|
|
61
|
+
"yaml": "2.9.1",
|
|
62
|
+
"zod": "4.6.5"
|
|
63
63
|
},
|
|
64
64
|
"devDependencies": {
|
|
65
|
-
"@biomejs/biome": "2.5.
|
|
66
|
-
"@types/node": "24.
|
|
65
|
+
"@biomejs/biome": "2.5.15",
|
|
66
|
+
"@types/node": "24.19.2",
|
|
67
67
|
"ajv": "8.20.0",
|
|
68
68
|
"rimraf": "6.1.3",
|
|
69
|
-
"tsx": "4.23.
|
|
69
|
+
"tsx": "4.23.15",
|
|
70
70
|
"typescript": "7.0.2",
|
|
71
|
-
"vitest": "
|
|
71
|
+
"vitest": "5.0.3"
|
|
72
72
|
}
|
|
73
73
|
}
|