lighthouse 13.5.0-dev.20261008 → 13.5.0-dev.20261009
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/update-ard-port/SKILL.md +170 -0
- package/cli/test/smokehouse/config/exclusions.js +1 -1
- package/core/audits/agentic/ard-schema.d.ts +13 -0
- package/core/audits/agentic/ard-schema.js +81 -33
- package/core/audits/agentic/llms-txt.js +5 -2
- package/core/audits/long-tasks.js +2 -1
- package/core/audits/seo/robots-txt.js +3 -8
- package/core/computed/page-dependency-graph.js +0 -4
- package/core/gather/gatherers/agentic/ard.d.ts +59 -4
- package/core/gather/gatherers/agentic/ard.js +218 -56
- package/package.json +5 -6
- package/shared/localization/locales/en-US.json +13 -4
- package/shared/localization/locales/en-XL.json +13 -4
- package/third-party/ard/README.md +18 -9
- package/third-party/ard/ard-test.js +183 -266
- package/third-party/ard/ard.d.ts +17 -1
- package/third-party/ard/ard.js +276 -45
- package/third-party/ard/schema-validator.d.ts +14 -5
- package/third-party/ard/schema-validator.js +921 -1177
- package/third-party/ard/spec/schemas/ard-entry.schema.json +176 -0
- package/types/artifacts.d.ts +29 -5
- package/build-tracker.config.js +0 -16
- package/third-party/ard/spec/schemas/ai-catalog.schema.json +0 -292
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: update-ard-port
|
|
3
|
+
description: Sync Lighthouse's Agentic Resource Discovery (ARD) port with upstream ards-project/ard-spec. Invoke when `node core/scripts/update-ard-spec.js --check` reports the ARD spec is out of sync, when the weekly cron CI job fails on it, or when asked to update the ARD conformance script, schema, or discovery rules.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Update the ARD port
|
|
7
|
+
|
|
8
|
+
Lighthouse vendors a JavaScript port of the upstream ARD conformance tester. When
|
|
9
|
+
upstream changes, the port must be brought back to **1:1 parity** while preserving the
|
|
10
|
+
Lighthouse-specific adaptations listed in `third-party/ard/README.md`.
|
|
11
|
+
|
|
12
|
+
## What lives where
|
|
13
|
+
|
|
14
|
+
| Upstream (ards-project/ard-spec) | Lighthouse | Notes |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| `conformance/bin/conformance-test` → `validate_manifest` and helpers | `third-party/ard/ard.js` | `ConformanceTester` (`validate_manifest`, `run_json_schema_validation`, `add_error/warning/info`), `parse_media_type`, `classify_media_type`, `format_schema_error`. Keep upstream `snake_case` names and **word-for-word message text**. |
|
|
17
|
+
| `spec/schemas/ard-entry.schema.json` | `third-party/ard/spec/schemas/ard-entry.schema.json` | Must stay byte-identical to upstream. |
|
|
18
|
+
| (generated) | `third-party/ard/schema-validator.js` | Precompiled Ajv standalone validator, built by `build/build-ard-schema.js` (`yarn build-ard-schema`). Never hand-edit. |
|
|
19
|
+
| `spec/ard.md` §5.1 Discovery Mechanisms (+ `resolve_publisher` in the script) | `core/gather/gatherers/agentic/ard.js` | Candidate order: robots.txt `agentmap` → `<link rel="ard">` → `Link` header → `/.well-known/ard.json` → legacy `ai-catalog` equivalents. Source names in `Artifacts.ArdDiscoverySource` (`types/artifacts.d.ts`). |
|
|
20
|
+
| severity / scoring (Lighthouse-only) | `core/audits/agentic/ard-schema.js` | errors → score 0; warnings/legacy/broken-link → 0.9; clean → 1. `infos` are not surfaced. |
|
|
21
|
+
| pinned SHA + reviewed hashes + deviation list | `third-party/ard/README.md` | `**Pinned Commit SHA**` and the `**Reviewed Upstream Files**` SHA-256 lines are parsed/written by the update script. Never edit the hashes by hand — they are the proof that the port was reviewed against that exact upstream content. |
|
|
22
|
+
|
|
23
|
+
Tests:
|
|
24
|
+
- `third-party/ard/ard-test.js` — port unit tests
|
|
25
|
+
- `core/test/gather/gatherers/agentic/ard-test.js` — gatherer
|
|
26
|
+
- `core/test/audits/agentic/ard-schema-test.js` — audit
|
|
27
|
+
- `cli/test/smokehouse/test-definitions/ard.js`, `ard-invalid.js` + fixtures in `cli/test/fixtures/agentic/`
|
|
28
|
+
- `core/test/results/artifacts/artifacts.json` — sample `AgentResourceDiscovery` artifact
|
|
29
|
+
|
|
30
|
+
## Procedure
|
|
31
|
+
|
|
32
|
+
### 1. See what changed
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
node core/scripts/update-ard-spec.js --check
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
"In sync" means upstream HEAD matches the vendored schema byte-for-byte **and** the
|
|
39
|
+
SHA-256 of `conformance/bin/conformance-test` and `spec/ard.md` match the hashes recorded
|
|
40
|
+
in the README. The output lists which of the three differ, plus old/new SHA and a compare
|
|
41
|
+
link. Pull the upstream files at **both** SHAs for side-by-side review (the compare API
|
|
42
|
+
patch is often truncated):
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
https://raw.githubusercontent.com/ards-project/ard-spec/<sha>/conformance/bin/conformance-test
|
|
46
|
+
https://raw.githubusercontent.com/ards-project/ard-spec/<sha>/spec/schemas/ard-entry.schema.json
|
|
47
|
+
https://raw.githubusercontent.com/ards-project/ard-spec/<sha>/spec/ard.md
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Save them in the conversation scratch dir and diff old vs new. Also check
|
|
51
|
+
`conformance/` for new test fixtures/expected outputs and `docs/adr/` for ADRs explaining
|
|
52
|
+
renamed or removed fields (e.g. `collections`, media type renames).
|
|
53
|
+
|
|
54
|
+
### 2. Sync the schema
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
yarn update:ard-spec
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
This writes `third-party/ard/spec/schemas/ard-entry.schema.json` verbatim from upstream
|
|
61
|
+
and prints the `conformance-test` diff. If a hand-ported file changed it **exits 1 and
|
|
62
|
+
does not bump the pinned SHA** — that is expected; the SHA is bumped in step 8 after
|
|
63
|
+
porting. Do not pass `--ack-port` yet.
|
|
64
|
+
|
|
65
|
+
### 3. Port conformance-test changes into `third-party/ard/ard.js`
|
|
66
|
+
|
|
67
|
+
For every hunk in the upstream `conformance-test` diff decide whether it touches:
|
|
68
|
+
- **Validation rules** (new/removed checks, changed thresholds such as the
|
|
69
|
+
`representativeQueries` 2–5 range, new deprecated media types, URN regex) → port 1:1.
|
|
70
|
+
- **Severity** (error ↔ warning ↔ info) → mirror it; this changes audit scoring.
|
|
71
|
+
- **Message text** → copy exactly; the audit shows these strings to users and tests
|
|
72
|
+
assert on them.
|
|
73
|
+
- **CLI/ANSI output, argparse, `main()`, `resolve_publisher`** → not ported (see README
|
|
74
|
+
deviations); skip unless the discovery logic itself changed (then see step 4).
|
|
75
|
+
- **`format_schema_error`** → only adapt if upstream changes how schema errors are
|
|
76
|
+
phrased. Ajv's messages differ from Python `jsonschema`; keep the
|
|
77
|
+
`'x' is a required property` mapping.
|
|
78
|
+
|
|
79
|
+
Rules to preserve:
|
|
80
|
+
- Treat non-object roots/entries as empty instead of throwing.
|
|
81
|
+
- Push `{element, message}` objects; never return formatted strings.
|
|
82
|
+
- Log through `lighthouse-logger`, not `console`.
|
|
83
|
+
- Only `ConformanceTester` and `classify_media_type` are exported; add exports only if
|
|
84
|
+
tests need them.
|
|
85
|
+
|
|
86
|
+
If `classify_media_type` / `parse_media_type` changed, run a parity check against the
|
|
87
|
+
real Python script on a list of tricky inputs (params, quoted strings, casing,
|
|
88
|
+
deprecated/unknown `application/*` types) and fix any mismatch.
|
|
89
|
+
|
|
90
|
+
### 4. If `spec/ard.md` changed
|
|
91
|
+
|
|
92
|
+
Read §5.1 (and any section the script's `resolve_publisher` references). If the
|
|
93
|
+
mechanisms, well-known path, `rel` names, header names, or precedence changed:
|
|
94
|
+
- Update the candidate list/order in `core/gather/gatherers/agentic/ard.js`
|
|
95
|
+
(`resolveManifest`) and the `getArdLinksInDOM` rel names.
|
|
96
|
+
- Update `Artifacts.ArdDiscoverySource` and `discoverySignals` in `types/artifacts.d.ts`.
|
|
97
|
+
- Update `LEGACY_SOURCES` / UIStrings in `core/audits/agentic/ard-schema.js` if what counts
|
|
98
|
+
as legacy changed.
|
|
99
|
+
|
|
100
|
+
### 5. Rebuild the validator (only if the schema changed)
|
|
101
|
+
|
|
102
|
+
```
|
|
103
|
+
yarn build-ard-schema
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Review the `schema-validator.js` diff: it should only reflect schema changes. The build
|
|
107
|
+
throws if any `require(` survives or a `$id` is missing — fix `build/build-ard-schema.js`
|
|
108
|
+
rather than the output.
|
|
109
|
+
|
|
110
|
+
### 6. Update tests and fixtures
|
|
111
|
+
|
|
112
|
+
- `third-party/ard/ard-test.js`: add a case for every new/changed rule and message.
|
|
113
|
+
- Gatherer/audit tests if discovery or scoring changed.
|
|
114
|
+
- Smoke: `cli/test/fixtures/agentic/ard.json` / `ai-catalog-invalid.json` and the
|
|
115
|
+
expectations in `cli/test/smokehouse/test-definitions/ard-invalid.js`. Smokehouse
|
|
116
|
+
asserts the **exact count and order** of `details.items`.
|
|
117
|
+
- `core/test/results/artifacts/artifacts.json` if the artifact shape changed.
|
|
118
|
+
|
|
119
|
+
### 7. Update docs
|
|
120
|
+
|
|
121
|
+
- README "Modifications for Lighthouse" if a new deviation was introduced, and the
|
|
122
|
+
"Updating Conformance Script" steps if the process changed.
|
|
123
|
+
- `@fileoverview` comments in `ard.js`, the gatherer and the audit that mention the spec
|
|
124
|
+
version.
|
|
125
|
+
- Audit `UIStrings` if user-facing wording or the spec link changed.
|
|
126
|
+
|
|
127
|
+
### 8. Verify, then acknowledge
|
|
128
|
+
|
|
129
|
+
```
|
|
130
|
+
yarn mocha third-party/ard/ard-test.js core/test/gather/gatherers/agentic/ard-test.js core/test/audits/agentic/ard-schema-test.js
|
|
131
|
+
yarn smoke ard ardInvalid
|
|
132
|
+
yarn type-check
|
|
133
|
+
yarn lint --fix
|
|
134
|
+
yarn update:sample-json
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
`yarn update:sample-json` also runs `i18n:collect-strings`; audit tests resolve
|
|
138
|
+
UIStrings through `shared/localization/locales/en-US.json`, so new strings fail tests
|
|
139
|
+
until this runs. Do not run `yarn unit`.
|
|
140
|
+
|
|
141
|
+
Only once everything above is green:
|
|
142
|
+
|
|
143
|
+
```
|
|
144
|
+
yarn update:ard-spec --ack-port # records reviewed hashes + bumps pinned SHA
|
|
145
|
+
node core/scripts/update-ard-spec.js --check # must now report in sync
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
`--ack-port` is the single statement that "the port matches this upstream content". Never
|
|
149
|
+
run it to silence the check without having ported the diff.
|
|
150
|
+
|
|
151
|
+
### 9. Report
|
|
152
|
+
|
|
153
|
+
Summarize for the user: upstream SHA range, each rule/message/severity change and its
|
|
154
|
+
effect on audit scoring, any discovery changes, and any deviation added to the README.
|
|
155
|
+
Do not commit or open a PR unless asked.
|
|
156
|
+
|
|
157
|
+
## Gotchas
|
|
158
|
+
|
|
159
|
+
- The check compares *content*, not commit IDs. A green check after a deps upgrade means
|
|
160
|
+
nothing changed upstream, not that someone looked. `upgrade-deps.sh` only runs
|
|
161
|
+
`--check`; it cannot acknowledge changes.
|
|
162
|
+
- `run_command` rejects `grep`; use `grep_search`, `awk`, or `sed`.
|
|
163
|
+
- Upstream v0.91 has no `specVersion` check, logs unknown root members only, warns on
|
|
164
|
+
`collections` (removed by ADR-0003), does not check HTTPS, and does not validate
|
|
165
|
+
`capabilities`. Do not add Lighthouse-only checks to the port — put them in the audit
|
|
166
|
+
if ever wanted.
|
|
167
|
+
- The `[label]` prefix upstream prepends to entry messages is carried in `element`, not
|
|
168
|
+
in `message`.
|
|
169
|
+
- The gatherer, not the port, decides which manifest URL is validated; the port only
|
|
170
|
+
receives `raw_content` and `source_label`.
|
|
@@ -23,7 +23,7 @@ const exclusions = {
|
|
|
23
23
|
'redirects-http',
|
|
24
24
|
// Disabled because these tests use settings that cannot be fully configured in
|
|
25
25
|
// DevTools (e.g. throttling method "provided").
|
|
26
|
-
'metrics-tricky-tti', 'metrics-tricky-tti-late-fcp', 'screenshot',
|
|
26
|
+
'metrics-delayed-lcp', 'metrics-tricky-tti', 'metrics-tricky-tti-late-fcp', 'screenshot',
|
|
27
27
|
// Disabled because of differences that need further investigation
|
|
28
28
|
'byte-efficiency', 'byte-gzip', 'perf-preload',
|
|
29
29
|
// Disabled because a renderer crash also breaks devtools frontend
|
|
@@ -1,5 +1,15 @@
|
|
|
1
1
|
export default ArdSchema;
|
|
2
2
|
declare class ArdSchema extends Audit {
|
|
3
|
+
/**
|
|
4
|
+
* @param {LH.Artifacts['AgentResourceDiscovery']['failedSources'][number]} failedSource
|
|
5
|
+
* @param {LH.IcuMessage} severity
|
|
6
|
+
* @return {{element: string, issue: LH.IcuMessage, severity: LH.IcuMessage}}
|
|
7
|
+
*/
|
|
8
|
+
static makeFailedSourceItem({ url, status }: LH.Artifacts["AgentResourceDiscovery"]["failedSources"][number], severity: LH.IcuMessage): {
|
|
9
|
+
element: string;
|
|
10
|
+
issue: LH.IcuMessage;
|
|
11
|
+
severity: LH.IcuMessage;
|
|
12
|
+
};
|
|
3
13
|
/**
|
|
4
14
|
* @param {LH.Artifacts} artifacts
|
|
5
15
|
* @return {LH.Audit.Product}
|
|
@@ -11,6 +21,9 @@ export namespace UIStrings {
|
|
|
11
21
|
let failureTitle: string;
|
|
12
22
|
let description: string;
|
|
13
23
|
let explanation: string;
|
|
24
|
+
let legacyDiscovery: string;
|
|
25
|
+
let sourceUnavailable: string;
|
|
26
|
+
let sourceFetchFailed: string;
|
|
14
27
|
let columnIssue: string;
|
|
15
28
|
let columnSeverity: string;
|
|
16
29
|
let itemSeverityError: string;
|
|
@@ -5,29 +5,48 @@
|
|
|
5
5
|
*/
|
|
6
6
|
|
|
7
7
|
/**
|
|
8
|
-
* @fileoverview Audit that validates
|
|
8
|
+
* @fileoverview Audit that validates a site's Agentic Resource Discovery (ARD) manifest.
|
|
9
9
|
*
|
|
10
|
-
*
|
|
10
|
+
* Validation itself is done by `third-party/ard/ard.js`, a port of `validate_manifest` from the
|
|
11
|
+
* official ARD conformance test (see that directory's README for the pinned upstream commit).
|
|
12
|
+
* This audit maps its errors and warnings onto a score, and adds discovery-related items the
|
|
13
|
+
* conformance test reports in its publisher-resolution mode.
|
|
11
14
|
* @see https://github.com/ards-project/ard-spec/blob/main/conformance/bin/conformance-test
|
|
12
15
|
* @see https://agenticresourcediscovery.org/spec/
|
|
13
|
-
* @version ARD Spec 1.0 / ADR-0003
|
|
14
16
|
*/
|
|
15
17
|
|
|
16
18
|
import {Audit} from '../audit.js';
|
|
17
19
|
import * as i18n from '../../lib/i18n/i18n.js';
|
|
18
20
|
import {ConformanceTester} from '../../../third-party/ard/ard.js';
|
|
19
21
|
|
|
22
|
+
const HTTP_CLIENT_ERROR_CODE_LOW = 400;
|
|
23
|
+
const HTTP_SERVER_ERROR_CODE_LOW = 500;
|
|
24
|
+
|
|
20
25
|
const UIStrings = {
|
|
21
|
-
/** Title of a Lighthouse audit that evaluates whether
|
|
22
|
-
title: '
|
|
23
|
-
/** Title of a Lighthouse audit that evaluates whether
|
|
24
|
-
failureTitle: '
|
|
25
|
-
/** Description of a Lighthouse audit that tells the user why
|
|
26
|
-
description: '
|
|
27
|
-
'
|
|
26
|
+
/** Title of a Lighthouse audit that evaluates whether the site's Agentic Resource Discovery (ARD) manifest conforms to the ARD specification. Shown when valid. "ARD" should not be translated. */
|
|
27
|
+
title: 'ARD manifest is valid',
|
|
28
|
+
/** Title of a Lighthouse audit that evaluates whether the site's Agentic Resource Discovery (ARD) manifest conforms to the ARD specification. Shown when invalid. "ARD" should not be translated. */
|
|
29
|
+
failureTitle: 'ARD manifest is invalid',
|
|
30
|
+
/** Description of a Lighthouse audit that tells the user why their Agentic Resource Discovery (ARD) manifest must match the ARD specification. "ARD", "/.well-known/ard.json" and "rel="ard"" should not be translated. This is displayed after a user expands the section to see more. No character length limits. The last sentence starting with 'Learn' becomes link text to additional documentation. */
|
|
31
|
+
description: 'A valid Agentic Resource Discovery (ARD) manifest, served at ' +
|
|
32
|
+
'`/.well-known/ard.json` or linked with `rel="ard"`, lets AI agents and registries ' +
|
|
33
|
+
'discover and verify your resources. ' +
|
|
28
34
|
'[Learn more about the ARD specification](https://agenticresourcediscovery.org/spec/).',
|
|
29
|
-
/** Explanatory message stating that
|
|
30
|
-
explanation: '
|
|
35
|
+
/** Explanatory message stating that the Agentic Resource Discovery (ARD) manifest could not be loaded for schema validation. "ARD" should not be translated. */
|
|
36
|
+
explanation: 'ARD manifest could not be loaded for schema validation.',
|
|
37
|
+
/** Table item describing that the Agentic Resource Discovery (ARD) manifest was only found at a legacy location that ARD consumers are not required to check. "ARD", "/.well-known/ai-catalog.json", "rel="ai-catalog"", "/.well-known/ard.json" and "rel="ard"" should not be translated. */
|
|
38
|
+
legacyDiscovery: 'Manifest was only found through a legacy location ' +
|
|
39
|
+
'(`/.well-known/ai-catalog.json` or `rel="ai-catalog"`). ARD consumers are not ' +
|
|
40
|
+
'required to check these, so ARD consumers may not discover your resources. Serve the ' +
|
|
41
|
+
'manifest at `/.well-known/ard.json` or link it with `rel="ard"`.',
|
|
42
|
+
/**
|
|
43
|
+
* @description Table item describing that a location the site advertises for its Agentic Resource Discovery (ARD) manifest (for example, a link on the page) returned an unsuccessful HTTP status code. "ARD" should not be translated.
|
|
44
|
+
* @example {404} statusCode
|
|
45
|
+
*/
|
|
46
|
+
sourceUnavailable: 'Advertised ARD manifest location could not be loaded ' +
|
|
47
|
+
'(HTTP status {statusCode}).',
|
|
48
|
+
/** Table item describing that a location the site advertises for its Agentic Resource Discovery (ARD) manifest (for example, a link on the page) could not be fetched at all. "ARD" should not be translated. */
|
|
49
|
+
sourceFetchFailed: 'Advertised ARD manifest location could not be loaded.',
|
|
31
50
|
/** Header of the table column which displays the issue. */
|
|
32
51
|
columnIssue: 'Issue',
|
|
33
52
|
/** Header of the table column which displays the severity. */
|
|
@@ -36,6 +55,9 @@ const UIStrings = {
|
|
|
36
55
|
itemSeverityError: 'Error',
|
|
37
56
|
};
|
|
38
57
|
|
|
58
|
+
/** @type {Set<LH.Artifacts.ArdDiscoverySource>} */
|
|
59
|
+
const LEGACY_SOURCES = new Set(['legacyHtmlLink', 'legacyHttpHeaderLink', 'legacyWellKnown']);
|
|
60
|
+
|
|
39
61
|
const str_ = i18n.createIcuMessageFn(import.meta.url, UIStrings);
|
|
40
62
|
|
|
41
63
|
class ArdSchema extends Audit {
|
|
@@ -53,6 +75,18 @@ class ArdSchema extends Audit {
|
|
|
53
75
|
};
|
|
54
76
|
}
|
|
55
77
|
|
|
78
|
+
/**
|
|
79
|
+
* @param {LH.Artifacts['AgentResourceDiscovery']['failedSources'][number]} failedSource
|
|
80
|
+
* @param {LH.IcuMessage} severity
|
|
81
|
+
* @return {{element: string, issue: LH.IcuMessage, severity: LH.IcuMessage}}
|
|
82
|
+
*/
|
|
83
|
+
static makeFailedSourceItem({url, status}, severity) {
|
|
84
|
+
const issue = status === null ?
|
|
85
|
+
str_(UIStrings.sourceFetchFailed) :
|
|
86
|
+
str_(UIStrings.sourceUnavailable, {statusCode: status});
|
|
87
|
+
return {element: url, issue, severity};
|
|
88
|
+
}
|
|
89
|
+
|
|
56
90
|
/**
|
|
57
91
|
* @param {LH.Artifacts} artifacts
|
|
58
92
|
* @return {LH.Audit.Product}
|
|
@@ -64,30 +98,26 @@ class ArdSchema extends Audit {
|
|
|
64
98
|
const hasExplicitSignal = Boolean(
|
|
65
99
|
signals.robotsTxtAgentmap ||
|
|
66
100
|
signals.htmlLink ||
|
|
67
|
-
signals.httpHeaderLink
|
|
101
|
+
signals.httpHeaderLink ||
|
|
102
|
+
signals.legacyHtmlLink ||
|
|
103
|
+
signals.legacyHttpHeaderLink
|
|
104
|
+
);
|
|
105
|
+
const isClientError = Boolean(
|
|
106
|
+
ard.status &&
|
|
107
|
+
ard.status >= HTTP_CLIENT_ERROR_CODE_LOW &&
|
|
108
|
+
ard.status < HTTP_SERVER_ERROR_CODE_LOW
|
|
68
109
|
);
|
|
69
|
-
const hasCatalog = hasExplicitSignal || ard.status === 200;
|
|
70
110
|
|
|
71
|
-
if (!
|
|
111
|
+
if (!hasExplicitSignal && isClientError) {
|
|
72
112
|
return {
|
|
73
113
|
score: 1,
|
|
74
114
|
notApplicable: true,
|
|
75
115
|
};
|
|
76
116
|
}
|
|
77
117
|
|
|
78
|
-
if (ard.status !== 200 || !ard.content) {
|
|
79
|
-
return {
|
|
80
|
-
score: 0,
|
|
81
|
-
explanation: str_(UIStrings.explanation),
|
|
82
|
-
};
|
|
83
|
-
}
|
|
84
|
-
|
|
85
118
|
const itemSeverityError = str_(UIStrings.itemSeverityError);
|
|
86
119
|
const itemSeverityLow = str_(i18n.UIStrings.itemSeverityLow);
|
|
87
120
|
|
|
88
|
-
/** @type {Array<{element: string, issue: string | LH.IcuMessage, severity: LH.IcuMessage}>} */
|
|
89
|
-
const issues = [];
|
|
90
|
-
|
|
91
121
|
/** @type {LH.Audit.Details.Table['headings']} */
|
|
92
122
|
const headings = [
|
|
93
123
|
{key: 'element', valueType: 'text', label: str_(i18n.UIStrings.columnElement)},
|
|
@@ -95,8 +125,21 @@ class ArdSchema extends Audit {
|
|
|
95
125
|
{key: 'severity', valueType: 'text', label: str_(UIStrings.columnSeverity)},
|
|
96
126
|
];
|
|
97
127
|
|
|
128
|
+
if (!ard.status || ard.status >= HTTP_CLIENT_ERROR_CODE_LOW || !ard.content) {
|
|
129
|
+
const failedItems = ard.failedSources.map(source =>
|
|
130
|
+
ArdSchema.makeFailedSourceItem(source, itemSeverityError));
|
|
131
|
+
return {
|
|
132
|
+
score: 0,
|
|
133
|
+
explanation: str_(UIStrings.explanation),
|
|
134
|
+
details: failedItems.length ? Audit.makeTableDetails(headings, failedItems) : undefined,
|
|
135
|
+
};
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
/** @type {Array<{element: string, issue: string | LH.IcuMessage, severity: LH.IcuMessage}>} */
|
|
139
|
+
const issues = [];
|
|
140
|
+
|
|
98
141
|
const tester = new ConformanceTester();
|
|
99
|
-
tester.validate_manifest(ard.content,
|
|
142
|
+
tester.validate_manifest(ard.content, ard.catalogUrl);
|
|
100
143
|
|
|
101
144
|
for (const err of tester.errors) {
|
|
102
145
|
issues.push({
|
|
@@ -114,16 +157,21 @@ class ArdSchema extends Audit {
|
|
|
114
157
|
});
|
|
115
158
|
}
|
|
116
159
|
|
|
117
|
-
const
|
|
118
|
-
|
|
160
|
+
for (const failedSource of ard.failedSources) {
|
|
161
|
+
issues.push(ArdSchema.makeFailedSourceItem(failedSource, itemSeverityLow));
|
|
162
|
+
}
|
|
119
163
|
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
164
|
+
if (LEGACY_SOURCES.has(ard.discoverySource)) {
|
|
165
|
+
issues.push({
|
|
166
|
+
element: ard.catalogUrl,
|
|
167
|
+
issue: str_(UIStrings.legacyDiscovery),
|
|
168
|
+
severity: itemSeverityLow,
|
|
169
|
+
});
|
|
125
170
|
}
|
|
126
171
|
|
|
172
|
+
const hasErrors = tester.errors.length > 0;
|
|
173
|
+
const score = hasErrors ? 0 : (issues.length ? 0.9 : 1);
|
|
174
|
+
|
|
127
175
|
return {
|
|
128
176
|
score,
|
|
129
177
|
details: issues.length ? Audit.makeTableDetails(headings, issues) : undefined,
|
|
@@ -82,7 +82,7 @@ class LlmsTxt extends Audit {
|
|
|
82
82
|
};
|
|
83
83
|
}
|
|
84
84
|
|
|
85
|
-
if (status >= HTTP_SERVER_ERROR_CODE_LOW) {
|
|
85
|
+
if (status >= HTTP_SERVER_ERROR_CODE_LOW || status === 429) {
|
|
86
86
|
return {
|
|
87
87
|
score: 0,
|
|
88
88
|
displayValue: str_(UIStrings.displayValueHttpBadCode, {statusCode: status}),
|
|
@@ -95,7 +95,10 @@ class LlmsTxt extends Audit {
|
|
|
95
95
|
}
|
|
96
96
|
|
|
97
97
|
if (content === null) {
|
|
98
|
-
|
|
98
|
+
return {
|
|
99
|
+
score: 0,
|
|
100
|
+
explanation: str_(UIStrings.explanation),
|
|
101
|
+
};
|
|
99
102
|
}
|
|
100
103
|
|
|
101
104
|
const hasH1 = /^\s*#\s+.+/m.test(content);
|
|
@@ -192,9 +192,10 @@ class LongTasks extends Audit {
|
|
|
192
192
|
taskTimingsByEvent = new Map();
|
|
193
193
|
|
|
194
194
|
const simulatorOptions = {devtoolsLog, settings: context.settings};
|
|
195
|
+
const fromTrace = process.env.INTERNAL_LANTERN_USE_TRACE !== undefined;
|
|
195
196
|
const pageGraph =
|
|
196
197
|
// eslint-disable-next-line max-len
|
|
197
|
-
await PageDependencyGraph.request({settings, trace, devtoolsLog, URL, SourceMaps, HostDPR, fromTrace
|
|
198
|
+
await PageDependencyGraph.request({settings, trace, devtoolsLog, URL, SourceMaps, HostDPR, fromTrace}, context);
|
|
198
199
|
const simulator = await LoadSimulator.request(simulatorOptions, context);
|
|
199
200
|
const simulation = simulator.simulate(pageGraph, {label: 'long-tasks-diagnostic'});
|
|
200
201
|
for (const [node, timing] of simulation.nodeTimings.entries()) {
|
|
@@ -27,7 +27,7 @@ const DIRECTIVE_SAFELIST = new Set([
|
|
|
27
27
|
DIRECTIVE_ALLOW, DIRECTIVE_SITEMAP, // universally supported
|
|
28
28
|
'crawl-delay', // yahoo, bing, yandex
|
|
29
29
|
'clean-param', 'host', // yandex
|
|
30
|
-
'request-rate', 'visit-time', 'noindex', 'content-signal', // not officially supported, but used in the wild
|
|
30
|
+
'request-rate', 'visit-time', 'noindex', 'content-signal', 'license', // not officially supported, but used in the wild
|
|
31
31
|
]);
|
|
32
32
|
const SITEMAP_VALID_PROTOCOLS = new Set(['https:', 'http:', 'ftp:']);
|
|
33
33
|
|
|
@@ -209,7 +209,7 @@ class RobotsTxt extends Audit {
|
|
|
209
209
|
} = artifacts.RobotsTxt;
|
|
210
210
|
|
|
211
211
|
// Do specific error messages first.
|
|
212
|
-
if (status && status >= HTTP_SERVER_ERROR_CODE_LOW) {
|
|
212
|
+
if (status && (status >= HTTP_SERVER_ERROR_CODE_LOW || status === 429)) {
|
|
213
213
|
return {
|
|
214
214
|
score: 0,
|
|
215
215
|
displayValue: str_(UIStrings.displayValueHttpBadCode, {statusCode: status}),
|
|
@@ -228,18 +228,13 @@ class RobotsTxt extends Audit {
|
|
|
228
228
|
};
|
|
229
229
|
}
|
|
230
230
|
|
|
231
|
-
if (!status) {
|
|
231
|
+
if (!status || content === null) {
|
|
232
232
|
return {
|
|
233
233
|
score: 0,
|
|
234
234
|
explanation: str_(UIStrings.explanation),
|
|
235
235
|
};
|
|
236
236
|
}
|
|
237
237
|
|
|
238
|
-
// If status is good, content must be not null.
|
|
239
|
-
if (content === null) {
|
|
240
|
-
throw new Error(`Status ${status} was valid, but content was null`);
|
|
241
|
-
}
|
|
242
|
-
|
|
243
238
|
const validationErrors = validateRobots(content);
|
|
244
239
|
|
|
245
240
|
/** @type {LH.Audit.Details.Table['headings']} */
|
|
@@ -36,10 +36,6 @@ class PageDependencyGraph {
|
|
|
36
36
|
return graph;
|
|
37
37
|
}
|
|
38
38
|
|
|
39
|
-
// TODO: currently the trace version has no requests that failed, or requests that have "Preflight".
|
|
40
|
-
// so the following gets the devtools log version _closer_ to the exact same results as the trace.
|
|
41
|
-
// const lanternRequests = networkRecords.map(NetworkRequest.asLanternNetworkRequest).filter(r => !r.failed && r.resourceType !== 'Preflight');
|
|
42
|
-
|
|
43
39
|
const lanternRequests = networkRecords.map(NetworkRequest.asLanternNetworkRequest);
|
|
44
40
|
return Lantern.Graph.PageDependencyGraph.createGraph(mainThreadEvents, lanternRequests, URL);
|
|
45
41
|
}
|
|
@@ -1,26 +1,81 @@
|
|
|
1
1
|
export default AgentResourceDiscovery;
|
|
2
|
+
export type ArdLinks = {
|
|
3
|
+
ard: string | null;
|
|
4
|
+
legacy: string | null;
|
|
5
|
+
};
|
|
6
|
+
export type DiscoverySignals = LH.Artifacts["AgentResourceDiscovery"]["discoverySignals"];
|
|
7
|
+
export type FetchResult = {
|
|
8
|
+
status: number | null;
|
|
9
|
+
content: string | null;
|
|
10
|
+
headers: Record<string, string> | null;
|
|
11
|
+
errorMessage?: string | undefined;
|
|
12
|
+
};
|
|
13
|
+
/**
|
|
14
|
+
* A location to try when resolving the manifest. `advertised` locations are ones the site
|
|
15
|
+
* points at (robots.txt, link tags, Link headers); the well-known paths are probed by
|
|
16
|
+
* Lighthouse, so a failure there is not the site's fault and is not reported.
|
|
17
|
+
*/
|
|
18
|
+
export type ManifestCandidate = {
|
|
19
|
+
source: LH.Artifacts.ArdDiscoverySource;
|
|
20
|
+
url: string | null;
|
|
21
|
+
advertised: boolean;
|
|
22
|
+
};
|
|
2
23
|
declare class AgentResourceDiscovery extends BaseGatherer {
|
|
3
24
|
/**
|
|
25
|
+
* Returns the absolute URL from the first `Agentmap:` directive in robots.txt, if any.
|
|
4
26
|
* @param {LH.Artifacts['RobotsTxt']|null|undefined} robotsTxt
|
|
5
27
|
* @param {string} finalDisplayedUrl
|
|
6
28
|
* @return {string|null}
|
|
7
29
|
*/
|
|
8
30
|
static getRobotsTxtAgentmap(robotsTxt: LH.Artifacts["RobotsTxt"] | null | undefined, finalDisplayedUrl: string): string | null;
|
|
9
31
|
/**
|
|
32
|
+
* Finds `<link rel="ard">` and `<link rel="ai-catalog">` in the page.
|
|
10
33
|
* @param {LH.Gatherer.Context} context
|
|
11
34
|
* @param {string} finalDisplayedUrl
|
|
12
|
-
* @return {Promise<
|
|
35
|
+
* @return {Promise<ArdLinks>}
|
|
13
36
|
*/
|
|
14
|
-
static
|
|
37
|
+
static getHtmlLinksFromDom(context: LH.Gatherer.Context, finalDisplayedUrl: string): Promise<ArdLinks>;
|
|
15
38
|
/**
|
|
39
|
+
* Finds `rel="ard"` and `rel="ai-catalog"` in the main document's HTTP `Link` header.
|
|
16
40
|
* @param {LH.Gatherer.Context} context
|
|
17
41
|
* @param {string} finalDisplayedUrl
|
|
18
|
-
* @return {Promise<
|
|
42
|
+
* @return {Promise<ArdLinks>}
|
|
19
43
|
*/
|
|
20
|
-
static
|
|
44
|
+
static getHttpHeaderLinks(context: LH.Gatherer.Context, finalDisplayedUrl: string): Promise<ArdLinks>;
|
|
45
|
+
/**
|
|
46
|
+
* Fetches a candidate manifest URL. Never throws; network errors are returned as a
|
|
47
|
+
* null status with an `errorMessage`.
|
|
48
|
+
* @param {LH.Gatherer.Context} context
|
|
49
|
+
* @param {string} url
|
|
50
|
+
* @return {Promise<FetchResult>}
|
|
51
|
+
*/
|
|
52
|
+
static fetchManifest(context: LH.Gatherer.Context, url: string): Promise<FetchResult>;
|
|
53
|
+
/**
|
|
54
|
+
* Tries each candidate in order and returns the first that loads. Advertised candidates
|
|
55
|
+
* that fail are recorded in `failedSources` and skipped. If nothing loads, the result
|
|
56
|
+
* describes the first advertised location that failed, or else the `/.well-known/ard.json`
|
|
57
|
+
* path consumers are required to fetch.
|
|
58
|
+
* @param {LH.Gatherer.Context} context
|
|
59
|
+
* @param {ManifestCandidate[]} candidates
|
|
60
|
+
* @return {Promise<{
|
|
61
|
+
* discoverySource: LH.Artifacts.ArdDiscoverySource,
|
|
62
|
+
* catalogUrl: string,
|
|
63
|
+
* fetchResult: FetchResult,
|
|
64
|
+
* failedSources: LH.Artifacts['AgentResourceDiscovery']['failedSources'],
|
|
65
|
+
* }>}
|
|
66
|
+
*/
|
|
67
|
+
static resolveManifest(context: LH.Gatherer.Context, candidates: ManifestCandidate[]): Promise<{
|
|
68
|
+
discoverySource: LH.Artifacts.ArdDiscoverySource;
|
|
69
|
+
catalogUrl: string;
|
|
70
|
+
fetchResult: FetchResult;
|
|
71
|
+
failedSources: LH.Artifacts["AgentResourceDiscovery"]["failedSources"];
|
|
72
|
+
}>;
|
|
21
73
|
/** @type {LH.Gatherer.GathererMeta<'RobotsTxt'>} */
|
|
22
74
|
meta: LH.Gatherer.GathererMeta<"RobotsTxt">;
|
|
23
75
|
/**
|
|
76
|
+
* Resolves the manifest following ARD §5.1: explicit ARD signals first, then the
|
|
77
|
+
* `/.well-known/ard.json` path consumers MUST fetch, and only then the predecessor
|
|
78
|
+
* `ai-catalog` relation and well-known path, which consumers MAY consult.
|
|
24
79
|
* @param {LH.Gatherer.Context<'RobotsTxt'>} context
|
|
25
80
|
* @return {Promise<LH.Artifacts['AgentResourceDiscovery']>}
|
|
26
81
|
*/
|