lighthouse 13.5.0-dev.20261008 → 13.5.0-dev.20261009

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,170 @@
1
+ ---
2
+ name: update-ard-port
3
+ description: Sync Lighthouse's Agentic Resource Discovery (ARD) port with upstream ards-project/ard-spec. Invoke when `node core/scripts/update-ard-spec.js --check` reports the ARD spec is out of sync, when the weekly cron CI job fails on it, or when asked to update the ARD conformance script, schema, or discovery rules.
4
+ ---
5
+
6
+ # Update the ARD port
7
+
8
+ Lighthouse vendors a JavaScript port of the upstream ARD conformance tester. When
9
+ upstream changes, the port must be brought back to **1:1 parity** while preserving the
10
+ Lighthouse-specific adaptations listed in `third-party/ard/README.md`.
11
+
12
+ ## What lives where
13
+
14
+ | Upstream (ards-project/ard-spec) | Lighthouse | Notes |
15
+ |---|---|---|
16
+ | `conformance/bin/conformance-test` → `validate_manifest` and helpers | `third-party/ard/ard.js` | `ConformanceTester` (`validate_manifest`, `run_json_schema_validation`, `add_error/warning/info`), `parse_media_type`, `classify_media_type`, `format_schema_error`. Keep upstream `snake_case` names and **word-for-word message text**. |
17
+ | `spec/schemas/ard-entry.schema.json` | `third-party/ard/spec/schemas/ard-entry.schema.json` | Must stay byte-identical to upstream. |
18
+ | (generated) | `third-party/ard/schema-validator.js` | Precompiled Ajv standalone validator, built by `build/build-ard-schema.js` (`yarn build-ard-schema`). Never hand-edit. |
19
+ | `spec/ard.md` §5.1 Discovery Mechanisms (+ `resolve_publisher` in the script) | `core/gather/gatherers/agentic/ard.js` | Candidate order: robots.txt `agentmap` → `<link rel="ard">` → `Link` header → `/.well-known/ard.json` → legacy `ai-catalog` equivalents. Source names in `Artifacts.ArdDiscoverySource` (`types/artifacts.d.ts`). |
20
+ | severity / scoring (Lighthouse-only) | `core/audits/agentic/ard-schema.js` | errors → score 0; warnings/legacy/broken-link → 0.9; clean → 1. `infos` are not surfaced. |
21
+ | pinned SHA + reviewed hashes + deviation list | `third-party/ard/README.md` | `**Pinned Commit SHA**` and the `**Reviewed Upstream Files**` SHA-256 lines are parsed/written by the update script. Never edit the hashes by hand — they are the proof that the port was reviewed against that exact upstream content. |
22
+
23
+ Tests:
24
+ - `third-party/ard/ard-test.js` — port unit tests
25
+ - `core/test/gather/gatherers/agentic/ard-test.js` — gatherer
26
+ - `core/test/audits/agentic/ard-schema-test.js` — audit
27
+ - `cli/test/smokehouse/test-definitions/ard.js`, `ard-invalid.js` + fixtures in `cli/test/fixtures/agentic/`
28
+ - `core/test/results/artifacts/artifacts.json` — sample `AgentResourceDiscovery` artifact
29
+
30
+ ## Procedure
31
+
32
+ ### 1. See what changed
33
+
34
+ ```
35
+ node core/scripts/update-ard-spec.js --check
36
+ ```
37
+
38
+ "In sync" means upstream HEAD matches the vendored schema byte-for-byte **and** the
39
+ SHA-256 of `conformance/bin/conformance-test` and `spec/ard.md` match the hashes recorded
40
+ in the README. The output lists which of the three differ, plus old/new SHA and a compare
41
+ link. Pull the upstream files at **both** SHAs for side-by-side review (the compare API
42
+ patch is often truncated):
43
+
44
+ ```
45
+ https://raw.githubusercontent.com/ards-project/ard-spec/<sha>/conformance/bin/conformance-test
46
+ https://raw.githubusercontent.com/ards-project/ard-spec/<sha>/spec/schemas/ard-entry.schema.json
47
+ https://raw.githubusercontent.com/ards-project/ard-spec/<sha>/spec/ard.md
48
+ ```
49
+
50
+ Save them in the conversation scratch dir and diff old vs new. Also check
51
+ `conformance/` for new test fixtures/expected outputs and `docs/adr/` for ADRs explaining
52
+ renamed or removed fields (e.g. `collections`, media type renames).
53
+
54
+ ### 2. Sync the schema
55
+
56
+ ```
57
+ yarn update:ard-spec
58
+ ```
59
+
60
+ This writes `third-party/ard/spec/schemas/ard-entry.schema.json` verbatim from upstream
61
+ and prints the `conformance-test` diff. If a hand-ported file changed it **exits 1 and
62
+ does not bump the pinned SHA** — that is expected; the SHA is bumped in step 8 after
63
+ porting. Do not pass `--ack-port` yet.
64
+
65
+ ### 3. Port conformance-test changes into `third-party/ard/ard.js`
66
+
67
+ For every hunk in the upstream `conformance-test` diff decide whether it touches:
68
+ - **Validation rules** (new/removed checks, changed thresholds such as the
69
+ `representativeQueries` 2–5 range, new deprecated media types, URN regex) → port 1:1.
70
+ - **Severity** (error ↔ warning ↔ info) → mirror it; this changes audit scoring.
71
+ - **Message text** → copy exactly; the audit shows these strings to users and tests
72
+ assert on them.
73
+ - **CLI/ANSI output, argparse, `main()`, `resolve_publisher`** → not ported (see README
74
+ deviations); skip unless the discovery logic itself changed (then see step 4).
75
+ - **`format_schema_error`** → only adapt if upstream changes how schema errors are
76
+ phrased. Ajv's messages differ from Python `jsonschema`; keep the
77
+ `'x' is a required property` mapping.
78
+
79
+ Rules to preserve:
80
+ - Treat non-object roots/entries as empty instead of throwing.
81
+ - Push `{element, message}` objects; never return formatted strings.
82
+ - Log through `lighthouse-logger`, not `console`.
83
+ - Only `ConformanceTester` and `classify_media_type` are exported; add exports only if
84
+ tests need them.
85
+
86
+ If `classify_media_type` / `parse_media_type` changed, run a parity check against the
87
+ real Python script on a list of tricky inputs (params, quoted strings, casing,
88
+ deprecated/unknown `application/*` types) and fix any mismatch.
89
+
90
+ ### 4. If `spec/ard.md` changed
91
+
92
+ Read §5.1 (and any section the script's `resolve_publisher` references). If the
93
+ mechanisms, well-known path, `rel` names, header names, or precedence changed:
94
+ - Update the candidate list/order in `core/gather/gatherers/agentic/ard.js`
95
+ (`resolveManifest`) and the `getArdLinksInDOM` rel names.
96
+ - Update `Artifacts.ArdDiscoverySource` and `discoverySignals` in `types/artifacts.d.ts`.
97
+ - Update `LEGACY_SOURCES` / UIStrings in `core/audits/agentic/ard-schema.js` if what counts
98
+ as legacy changed.
99
+
100
+ ### 5. Rebuild the validator (only if the schema changed)
101
+
102
+ ```
103
+ yarn build-ard-schema
104
+ ```
105
+
106
+ Review the `schema-validator.js` diff: it should only reflect schema changes. The build
107
+ throws if any `require(` survives or a `$id` is missing — fix `build/build-ard-schema.js`
108
+ rather than the output.
109
+
110
+ ### 6. Update tests and fixtures
111
+
112
+ - `third-party/ard/ard-test.js`: add a case for every new/changed rule and message.
113
+ - Gatherer/audit tests if discovery or scoring changed.
114
+ - Smoke: `cli/test/fixtures/agentic/ard.json` / `ai-catalog-invalid.json` and the
115
+ expectations in `cli/test/smokehouse/test-definitions/ard-invalid.js`. Smokehouse
116
+ asserts the **exact count and order** of `details.items`.
117
+ - `core/test/results/artifacts/artifacts.json` if the artifact shape changed.
118
+
119
+ ### 7. Update docs
120
+
121
+ - README "Modifications for Lighthouse" if a new deviation was introduced, and the
122
+ "Updating Conformance Script" steps if the process changed.
123
+ - `@fileoverview` comments in `ard.js`, the gatherer and the audit that mention the spec
124
+ version.
125
+ - Audit `UIStrings` if user-facing wording or the spec link changed.
126
+
127
+ ### 8. Verify, then acknowledge
128
+
129
+ ```
130
+ yarn mocha third-party/ard/ard-test.js core/test/gather/gatherers/agentic/ard-test.js core/test/audits/agentic/ard-schema-test.js
131
+ yarn smoke ard ardInvalid
132
+ yarn type-check
133
+ yarn lint --fix
134
+ yarn update:sample-json
135
+ ```
136
+
137
+ `yarn update:sample-json` also runs `i18n:collect-strings`; audit tests resolve
138
+ UIStrings through `shared/localization/locales/en-US.json`, so new strings fail tests
139
+ until this runs. Do not run `yarn unit`.
140
+
141
+ Only once everything above is green:
142
+
143
+ ```
144
+ yarn update:ard-spec --ack-port # records reviewed hashes + bumps pinned SHA
145
+ node core/scripts/update-ard-spec.js --check # must now report in sync
146
+ ```
147
+
148
+ `--ack-port` is the single statement that "the port matches this upstream content". Never
149
+ run it to silence the check without having ported the diff.
150
+
151
+ ### 9. Report
152
+
153
+ Summarize for the user: upstream SHA range, each rule/message/severity change and its
154
+ effect on audit scoring, any discovery changes, and any deviation added to the README.
155
+ Do not commit or open a PR unless asked.
156
+
157
+ ## Gotchas
158
+
159
+ - The check compares *content*, not commit IDs. A green check after a deps upgrade means
160
+ nothing changed upstream, not that someone looked. `upgrade-deps.sh` only runs
161
+ `--check`; it cannot acknowledge changes.
162
+ - `run_command` rejects `grep`; use `grep_search`, `awk`, or `sed`.
163
+ - Upstream v0.91 has no `specVersion` check, logs unknown root members only, warns on
164
+ `collections` (removed by ADR-0003), does not check HTTPS, and does not validate
165
+ `capabilities`. Do not add Lighthouse-only checks to the port — put them in the audit
166
+ if ever wanted.
167
+ - The `[label]` prefix upstream prepends to entry messages is carried in `element`, not
168
+ in `message`.
169
+ - The gatherer, not the port, decides which manifest URL is validated; the port only
170
+ receives `raw_content` and `source_label`.
@@ -23,7 +23,7 @@ const exclusions = {
23
23
  'redirects-http',
24
24
  // Disabled because these tests use settings that cannot be fully configured in
25
25
  // DevTools (e.g. throttling method "provided").
26
- 'metrics-tricky-tti', 'metrics-tricky-tti-late-fcp', 'screenshot',
26
+ 'metrics-delayed-lcp', 'metrics-tricky-tti', 'metrics-tricky-tti-late-fcp', 'screenshot',
27
27
  // Disabled because of differences that need further investigation
28
28
  'byte-efficiency', 'byte-gzip', 'perf-preload',
29
29
  // Disabled because a renderer crash also breaks devtools frontend
@@ -1,5 +1,15 @@
1
1
  export default ArdSchema;
2
2
  declare class ArdSchema extends Audit {
3
+ /**
4
+ * @param {LH.Artifacts['AgentResourceDiscovery']['failedSources'][number]} failedSource
5
+ * @param {LH.IcuMessage} severity
6
+ * @return {{element: string, issue: LH.IcuMessage, severity: LH.IcuMessage}}
7
+ */
8
+ static makeFailedSourceItem({ url, status }: LH.Artifacts["AgentResourceDiscovery"]["failedSources"][number], severity: LH.IcuMessage): {
9
+ element: string;
10
+ issue: LH.IcuMessage;
11
+ severity: LH.IcuMessage;
12
+ };
3
13
  /**
4
14
  * @param {LH.Artifacts} artifacts
5
15
  * @return {LH.Audit.Product}
@@ -11,6 +21,9 @@ export namespace UIStrings {
11
21
  let failureTitle: string;
12
22
  let description: string;
13
23
  let explanation: string;
24
+ let legacyDiscovery: string;
25
+ let sourceUnavailable: string;
26
+ let sourceFetchFailed: string;
14
27
  let columnIssue: string;
15
28
  let columnSeverity: string;
16
29
  let itemSeverityError: string;
@@ -5,29 +5,48 @@
5
5
  */
6
6
 
7
7
  /**
8
- * @fileoverview Audit that validates ai-catalog.json manifests against the Agentic Resource Discovery (ARD) specification.
8
+ * @fileoverview Audit that validates a site's Agentic Resource Discovery (ARD) manifest.
9
9
  *
10
- * This implementation is a direct JavaScript port of `validate_manifest` from the official ARD Conformance Test suite:
10
+ * Validation itself is done by `third-party/ard/ard.js`, a port of `validate_manifest` from the
11
+ * official ARD conformance test (see that directory's README for the pinned upstream commit).
12
+ * This audit maps its errors and warnings onto a score, and adds discovery-related items the
13
+ * conformance test reports in its publisher-resolution mode.
11
14
  * @see https://github.com/ards-project/ard-spec/blob/main/conformance/bin/conformance-test
12
15
  * @see https://agenticresourcediscovery.org/spec/
13
- * @version ARD Spec 1.0 / ADR-0003
14
16
  */
15
17
 
16
18
  import {Audit} from '../audit.js';
17
19
  import * as i18n from '../../lib/i18n/i18n.js';
18
20
  import {ConformanceTester} from '../../../third-party/ard/ard.js';
19
21
 
22
+ const HTTP_CLIENT_ERROR_CODE_LOW = 400;
23
+ const HTTP_SERVER_ERROR_CODE_LOW = 500;
24
+
20
25
  const UIStrings = {
21
- /** Title of a Lighthouse audit that evaluates whether ai-catalog.json conforms to the ARD specification. Shown when valid. */
22
- title: '`ai-catalog.json` schema is valid',
23
- /** Title of a Lighthouse audit that evaluates whether ai-catalog.json conforms to the ARD specification. Shown when invalid. */
24
- failureTitle: '`ai-catalog.json` schema is invalid',
25
- /** Description of a Lighthouse audit that tells the user why ai-catalog.json must match the ARD specification. */
26
- description: 'Valid `ai-catalog.json` manifests are required for autonomous ' +
27
- 'AI agents and registries to discover and verify your resources. ' +
26
+ /** Title of a Lighthouse audit that evaluates whether the site's Agentic Resource Discovery (ARD) manifest conforms to the ARD specification. Shown when valid. "ARD" should not be translated. */
27
+ title: 'ARD manifest is valid',
28
+ /** Title of a Lighthouse audit that evaluates whether the site's Agentic Resource Discovery (ARD) manifest conforms to the ARD specification. Shown when invalid. "ARD" should not be translated. */
29
+ failureTitle: 'ARD manifest is invalid',
30
+ /** Description of a Lighthouse audit that tells the user why their Agentic Resource Discovery (ARD) manifest must match the ARD specification. "ARD", "/.well-known/ard.json" and "rel="ard"" should not be translated. This is displayed after a user expands the section to see more. No character length limits. The last sentence starting with 'Learn' becomes link text to additional documentation. */
31
+ description: 'A valid Agentic Resource Discovery (ARD) manifest, served at ' +
32
+ '`/.well-known/ard.json` or linked with `rel="ard"`, lets AI agents and registries ' +
33
+ 'discover and verify your resources. ' +
28
34
  '[Learn more about the ARD specification](https://agenticresourcediscovery.org/spec/).',
29
- /** Explanatory message stating that ai-catalog.json could not be loaded for schema validation. */
30
- explanation: 'Catalog file could not be loaded for schema validation.',
35
+ /** Explanatory message stating that the Agentic Resource Discovery (ARD) manifest could not be loaded for schema validation. "ARD" should not be translated. */
36
+ explanation: 'ARD manifest could not be loaded for schema validation.',
37
+ /** Table item describing that the Agentic Resource Discovery (ARD) manifest was only found at a legacy location that ARD consumers are not required to check. "ARD", "/.well-known/ai-catalog.json", "rel="ai-catalog"", "/.well-known/ard.json" and "rel="ard"" should not be translated. */
38
+ legacyDiscovery: 'Manifest was only found through a legacy location ' +
39
+ '(`/.well-known/ai-catalog.json` or `rel="ai-catalog"`). ARD consumers are not ' +
40
+ 'required to check these, so ARD consumers may not discover your resources. Serve the ' +
41
+ 'manifest at `/.well-known/ard.json` or link it with `rel="ard"`.',
42
+ /**
43
+ * @description Table item describing that a location the site advertises for its Agentic Resource Discovery (ARD) manifest (for example, a link on the page) returned an unsuccessful HTTP status code. "ARD" should not be translated.
44
+ * @example {404} statusCode
45
+ */
46
+ sourceUnavailable: 'Advertised ARD manifest location could not be loaded ' +
47
+ '(HTTP status {statusCode}).',
48
+ /** Table item describing that a location the site advertises for its Agentic Resource Discovery (ARD) manifest (for example, a link on the page) could not be fetched at all. "ARD" should not be translated. */
49
+ sourceFetchFailed: 'Advertised ARD manifest location could not be loaded.',
31
50
  /** Header of the table column which displays the issue. */
32
51
  columnIssue: 'Issue',
33
52
  /** Header of the table column which displays the severity. */
@@ -36,6 +55,9 @@ const UIStrings = {
36
55
  itemSeverityError: 'Error',
37
56
  };
38
57
 
58
+ /** @type {Set<LH.Artifacts.ArdDiscoverySource>} */
59
+ const LEGACY_SOURCES = new Set(['legacyHtmlLink', 'legacyHttpHeaderLink', 'legacyWellKnown']);
60
+
39
61
  const str_ = i18n.createIcuMessageFn(import.meta.url, UIStrings);
40
62
 
41
63
  class ArdSchema extends Audit {
@@ -53,6 +75,18 @@ class ArdSchema extends Audit {
53
75
  };
54
76
  }
55
77
 
78
+ /**
79
+ * @param {LH.Artifacts['AgentResourceDiscovery']['failedSources'][number]} failedSource
80
+ * @param {LH.IcuMessage} severity
81
+ * @return {{element: string, issue: LH.IcuMessage, severity: LH.IcuMessage}}
82
+ */
83
+ static makeFailedSourceItem({url, status}, severity) {
84
+ const issue = status === null ?
85
+ str_(UIStrings.sourceFetchFailed) :
86
+ str_(UIStrings.sourceUnavailable, {statusCode: status});
87
+ return {element: url, issue, severity};
88
+ }
89
+
56
90
  /**
57
91
  * @param {LH.Artifacts} artifacts
58
92
  * @return {LH.Audit.Product}
@@ -64,30 +98,26 @@ class ArdSchema extends Audit {
64
98
  const hasExplicitSignal = Boolean(
65
99
  signals.robotsTxtAgentmap ||
66
100
  signals.htmlLink ||
67
- signals.httpHeaderLink
101
+ signals.httpHeaderLink ||
102
+ signals.legacyHtmlLink ||
103
+ signals.legacyHttpHeaderLink
104
+ );
105
+ const isClientError = Boolean(
106
+ ard.status &&
107
+ ard.status >= HTTP_CLIENT_ERROR_CODE_LOW &&
108
+ ard.status < HTTP_SERVER_ERROR_CODE_LOW
68
109
  );
69
- const hasCatalog = hasExplicitSignal || ard.status === 200;
70
110
 
71
- if (!hasCatalog) {
111
+ if (!hasExplicitSignal && isClientError) {
72
112
  return {
73
113
  score: 1,
74
114
  notApplicable: true,
75
115
  };
76
116
  }
77
117
 
78
- if (ard.status !== 200 || !ard.content) {
79
- return {
80
- score: 0,
81
- explanation: str_(UIStrings.explanation),
82
- };
83
- }
84
-
85
118
  const itemSeverityError = str_(UIStrings.itemSeverityError);
86
119
  const itemSeverityLow = str_(i18n.UIStrings.itemSeverityLow);
87
120
 
88
- /** @type {Array<{element: string, issue: string | LH.IcuMessage, severity: LH.IcuMessage}>} */
89
- const issues = [];
90
-
91
121
  /** @type {LH.Audit.Details.Table['headings']} */
92
122
  const headings = [
93
123
  {key: 'element', valueType: 'text', label: str_(i18n.UIStrings.columnElement)},
@@ -95,8 +125,21 @@ class ArdSchema extends Audit {
95
125
  {key: 'severity', valueType: 'text', label: str_(UIStrings.columnSeverity)},
96
126
  ];
97
127
 
128
+ if (!ard.status || ard.status >= HTTP_CLIENT_ERROR_CODE_LOW || !ard.content) {
129
+ const failedItems = ard.failedSources.map(source =>
130
+ ArdSchema.makeFailedSourceItem(source, itemSeverityError));
131
+ return {
132
+ score: 0,
133
+ explanation: str_(UIStrings.explanation),
134
+ details: failedItems.length ? Audit.makeTableDetails(headings, failedItems) : undefined,
135
+ };
136
+ }
137
+
138
+ /** @type {Array<{element: string, issue: string | LH.IcuMessage, severity: LH.IcuMessage}>} */
139
+ const issues = [];
140
+
98
141
  const tester = new ConformanceTester();
99
- tester.validate_manifest(ard.content, 'ai-catalog.json');
142
+ tester.validate_manifest(ard.content, ard.catalogUrl);
100
143
 
101
144
  for (const err of tester.errors) {
102
145
  issues.push({
@@ -114,16 +157,21 @@ class ArdSchema extends Audit {
114
157
  });
115
158
  }
116
159
 
117
- const hasErrors = issues.some(i => i.severity === itemSeverityError);
118
- const hasLowSeverity = issues.some(i => i.severity === itemSeverityLow);
160
+ for (const failedSource of ard.failedSources) {
161
+ issues.push(ArdSchema.makeFailedSourceItem(failedSource, itemSeverityLow));
162
+ }
119
163
 
120
- let score = 1;
121
- if (hasErrors) {
122
- score = 0;
123
- } else if (hasLowSeverity) {
124
- score = 0.9;
164
+ if (LEGACY_SOURCES.has(ard.discoverySource)) {
165
+ issues.push({
166
+ element: ard.catalogUrl,
167
+ issue: str_(UIStrings.legacyDiscovery),
168
+ severity: itemSeverityLow,
169
+ });
125
170
  }
126
171
 
172
+ const hasErrors = tester.errors.length > 0;
173
+ const score = hasErrors ? 0 : (issues.length ? 0.9 : 1);
174
+
127
175
  return {
128
176
  score,
129
177
  details: issues.length ? Audit.makeTableDetails(headings, issues) : undefined,
@@ -82,7 +82,7 @@ class LlmsTxt extends Audit {
82
82
  };
83
83
  }
84
84
 
85
- if (status >= HTTP_SERVER_ERROR_CODE_LOW) {
85
+ if (status >= HTTP_SERVER_ERROR_CODE_LOW || status === 429) {
86
86
  return {
87
87
  score: 0,
88
88
  displayValue: str_(UIStrings.displayValueHttpBadCode, {statusCode: status}),
@@ -95,7 +95,10 @@ class LlmsTxt extends Audit {
95
95
  }
96
96
 
97
97
  if (content === null) {
98
- throw new Error(`Status ${status} was valid, but content was null`);
98
+ return {
99
+ score: 0,
100
+ explanation: str_(UIStrings.explanation),
101
+ };
99
102
  }
100
103
 
101
104
  const hasH1 = /^\s*#\s+.+/m.test(content);
@@ -192,9 +192,10 @@ class LongTasks extends Audit {
192
192
  taskTimingsByEvent = new Map();
193
193
 
194
194
  const simulatorOptions = {devtoolsLog, settings: context.settings};
195
+ const fromTrace = process.env.INTERNAL_LANTERN_USE_TRACE !== undefined;
195
196
  const pageGraph =
196
197
  // eslint-disable-next-line max-len
197
- await PageDependencyGraph.request({settings, trace, devtoolsLog, URL, SourceMaps, HostDPR, fromTrace: false}, context);
198
+ await PageDependencyGraph.request({settings, trace, devtoolsLog, URL, SourceMaps, HostDPR, fromTrace}, context);
198
199
  const simulator = await LoadSimulator.request(simulatorOptions, context);
199
200
  const simulation = simulator.simulate(pageGraph, {label: 'long-tasks-diagnostic'});
200
201
  for (const [node, timing] of simulation.nodeTimings.entries()) {
@@ -27,7 +27,7 @@ const DIRECTIVE_SAFELIST = new Set([
27
27
  DIRECTIVE_ALLOW, DIRECTIVE_SITEMAP, // universally supported
28
28
  'crawl-delay', // yahoo, bing, yandex
29
29
  'clean-param', 'host', // yandex
30
- 'request-rate', 'visit-time', 'noindex', 'content-signal', // not officially supported, but used in the wild
30
+ 'request-rate', 'visit-time', 'noindex', 'content-signal', 'license', // not officially supported, but used in the wild
31
31
  ]);
32
32
  const SITEMAP_VALID_PROTOCOLS = new Set(['https:', 'http:', 'ftp:']);
33
33
 
@@ -209,7 +209,7 @@ class RobotsTxt extends Audit {
209
209
  } = artifacts.RobotsTxt;
210
210
 
211
211
  // Do specific error messages first.
212
- if (status && status >= HTTP_SERVER_ERROR_CODE_LOW) {
212
+ if (status && (status >= HTTP_SERVER_ERROR_CODE_LOW || status === 429)) {
213
213
  return {
214
214
  score: 0,
215
215
  displayValue: str_(UIStrings.displayValueHttpBadCode, {statusCode: status}),
@@ -228,18 +228,13 @@ class RobotsTxt extends Audit {
228
228
  };
229
229
  }
230
230
 
231
- if (!status) {
231
+ if (!status || content === null) {
232
232
  return {
233
233
  score: 0,
234
234
  explanation: str_(UIStrings.explanation),
235
235
  };
236
236
  }
237
237
 
238
- // If status is good, content must be not null.
239
- if (content === null) {
240
- throw new Error(`Status ${status} was valid, but content was null`);
241
- }
242
-
243
238
  const validationErrors = validateRobots(content);
244
239
 
245
240
  /** @type {LH.Audit.Details.Table['headings']} */
@@ -36,10 +36,6 @@ class PageDependencyGraph {
36
36
  return graph;
37
37
  }
38
38
 
39
- // TODO: currently the trace version has no requests that failed, or requests that have "Preflight".
40
- // so the following gets the devtools log version _closer_ to the exact same results as the trace.
41
- // const lanternRequests = networkRecords.map(NetworkRequest.asLanternNetworkRequest).filter(r => !r.failed && r.resourceType !== 'Preflight');
42
-
43
39
  const lanternRequests = networkRecords.map(NetworkRequest.asLanternNetworkRequest);
44
40
  return Lantern.Graph.PageDependencyGraph.createGraph(mainThreadEvents, lanternRequests, URL);
45
41
  }
@@ -1,26 +1,81 @@
1
1
  export default AgentResourceDiscovery;
2
+ export type ArdLinks = {
3
+ ard: string | null;
4
+ legacy: string | null;
5
+ };
6
+ export type DiscoverySignals = LH.Artifacts["AgentResourceDiscovery"]["discoverySignals"];
7
+ export type FetchResult = {
8
+ status: number | null;
9
+ content: string | null;
10
+ headers: Record<string, string> | null;
11
+ errorMessage?: string | undefined;
12
+ };
13
+ /**
14
+ * A location to try when resolving the manifest. `advertised` locations are ones the site
15
+ * points at (robots.txt, link tags, Link headers); the well-known paths are probed by
16
+ * Lighthouse, so a failure there is not the site's fault and is not reported.
17
+ */
18
+ export type ManifestCandidate = {
19
+ source: LH.Artifacts.ArdDiscoverySource;
20
+ url: string | null;
21
+ advertised: boolean;
22
+ };
2
23
  declare class AgentResourceDiscovery extends BaseGatherer {
3
24
  /**
25
+ * Returns the absolute URL from the first `Agentmap:` directive in robots.txt, if any.
4
26
  * @param {LH.Artifacts['RobotsTxt']|null|undefined} robotsTxt
5
27
  * @param {string} finalDisplayedUrl
6
28
  * @return {string|null}
7
29
  */
8
30
  static getRobotsTxtAgentmap(robotsTxt: LH.Artifacts["RobotsTxt"] | null | undefined, finalDisplayedUrl: string): string | null;
9
31
  /**
32
+ * Finds `<link rel="ard">` and `<link rel="ai-catalog">` in the page.
10
33
  * @param {LH.Gatherer.Context} context
11
34
  * @param {string} finalDisplayedUrl
12
- * @return {Promise<string|null>}
35
+ * @return {Promise<ArdLinks>}
13
36
  */
14
- static getHtmlLinkFromDom(context: LH.Gatherer.Context, finalDisplayedUrl: string): Promise<string | null>;
37
+ static getHtmlLinksFromDom(context: LH.Gatherer.Context, finalDisplayedUrl: string): Promise<ArdLinks>;
15
38
  /**
39
+ * Finds `rel="ard"` and `rel="ai-catalog"` in the main document's HTTP `Link` header.
16
40
  * @param {LH.Gatherer.Context} context
17
41
  * @param {string} finalDisplayedUrl
18
- * @return {Promise<string|null>}
42
+ * @return {Promise<ArdLinks>}
19
43
  */
20
- static getHttpHeaderLink(context: LH.Gatherer.Context, finalDisplayedUrl: string): Promise<string | null>;
44
+ static getHttpHeaderLinks(context: LH.Gatherer.Context, finalDisplayedUrl: string): Promise<ArdLinks>;
45
+ /**
46
+ * Fetches a candidate manifest URL. Never throws; network errors are returned as a
47
+ * null status with an `errorMessage`.
48
+ * @param {LH.Gatherer.Context} context
49
+ * @param {string} url
50
+ * @return {Promise<FetchResult>}
51
+ */
52
+ static fetchManifest(context: LH.Gatherer.Context, url: string): Promise<FetchResult>;
53
+ /**
54
+ * Tries each candidate in order and returns the first that loads. Advertised candidates
55
+ * that fail are recorded in `failedSources` and skipped. If nothing loads, the result
56
+ * describes the first advertised location that failed, or else the `/.well-known/ard.json`
57
+ * path consumers are required to fetch.
58
+ * @param {LH.Gatherer.Context} context
59
+ * @param {ManifestCandidate[]} candidates
60
+ * @return {Promise<{
61
+ * discoverySource: LH.Artifacts.ArdDiscoverySource,
62
+ * catalogUrl: string,
63
+ * fetchResult: FetchResult,
64
+ * failedSources: LH.Artifacts['AgentResourceDiscovery']['failedSources'],
65
+ * }>}
66
+ */
67
+ static resolveManifest(context: LH.Gatherer.Context, candidates: ManifestCandidate[]): Promise<{
68
+ discoverySource: LH.Artifacts.ArdDiscoverySource;
69
+ catalogUrl: string;
70
+ fetchResult: FetchResult;
71
+ failedSources: LH.Artifacts["AgentResourceDiscovery"]["failedSources"];
72
+ }>;
21
73
  /** @type {LH.Gatherer.GathererMeta<'RobotsTxt'>} */
22
74
  meta: LH.Gatherer.GathererMeta<"RobotsTxt">;
23
75
  /**
76
+ * Resolves the manifest following ARD §5.1: explicit ARD signals first, then the
77
+ * `/.well-known/ard.json` path consumers MUST fetch, and only then the predecessor
78
+ * `ai-catalog` relation and well-known path, which consumers MAY consult.
24
79
  * @param {LH.Gatherer.Context<'RobotsTxt'>} context
25
80
  * @return {Promise<LH.Artifacts['AgentResourceDiscovery']>}
26
81
  */