wiki-formant 0.4.1 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +159 -2
- package/dist/analytics.d.ts +22 -0
- package/dist/analytics.d.ts.map +1 -1
- package/dist/analytics.js +31 -1
- package/dist/analytics.js.map +1 -1
- package/dist/combobox.d.ts +50 -0
- package/dist/combobox.d.ts.map +1 -0
- package/dist/combobox.js +59 -0
- package/dist/combobox.js.map +1 -0
- package/dist/conformance.d.ts.map +1 -1
- package/dist/conformance.js +8 -0
- package/dist/conformance.js.map +1 -1
- package/dist/crawlers.d.ts +45 -0
- package/dist/crawlers.d.ts.map +1 -0
- package/dist/crawlers.js +80 -0
- package/dist/crawlers.js.map +1 -0
- package/dist/dom.d.ts +69 -0
- package/dist/dom.d.ts.map +1 -0
- package/dist/dom.js +167 -0
- package/dist/dom.js.map +1 -0
- package/dist/index.d.ts +2 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +2 -0
- package/dist/index.js.map +1 -1
- package/dist/pagination.d.ts +10 -1
- package/dist/pagination.d.ts.map +1 -1
- package/dist/pagination.js +13 -5
- package/dist/pagination.js.map +1 -1
- package/dist/react.d.ts +226 -0
- package/dist/react.d.ts.map +1 -0
- package/dist/react.js +476 -0
- package/dist/react.js.map +1 -0
- package/dist/sidebar.d.ts +38 -0
- package/dist/sidebar.d.ts.map +1 -0
- package/dist/sidebar.js +43 -0
- package/dist/sidebar.js.map +1 -0
- package/dist/taxonomy.d.ts +52 -0
- package/dist/taxonomy.d.ts.map +1 -1
- package/dist/taxonomy.js +95 -42
- package/dist/taxonomy.js.map +1 -1
- package/dist/tiptap.d.ts +78 -0
- package/dist/tiptap.d.ts.map +1 -0
- package/dist/tiptap.js +344 -0
- package/dist/tiptap.js.map +1 -0
- package/package.json +62 -3
package/README.md
CHANGED
|
@@ -47,6 +47,23 @@ Four behaviours worth knowing, because each replaces a plausible wrong answer:
|
|
|
47
47
|
|
|
48
48
|
One `href` builder is passed in and used by every chip, letter and sort button. A sort button that drops the active filters is the tell that a project grew a second one.
|
|
49
49
|
|
|
50
|
+
### The controls, not just the counts
|
|
51
|
+
|
|
52
|
+
`buildFacets` and `alphaIndex` hand back numbers; something still has to turn each one into a pressable thing with a destination. That step was rebuilt in all three wikis, and by the time this was written they no longer agreed on what the row *contains*: one shipped no A–Z index at all, one had no way back to the unfiltered set, and two marked the active chip with `aria-pressed` on an `<a>` — which is not a toggle button and does not take that attribute.
|
|
53
|
+
|
|
54
|
+
```ts
|
|
55
|
+
const state = { sort, filters, letter: taxonomy.resolveLetter(all, filters, query.letter) };
|
|
56
|
+
const groups = taxonomy.facetControls('ecosystem', all, state); // [{key, label, options}]
|
|
57
|
+
const letters = taxonomy.alphaControls('ecosystem', all, state); // [] below the threshold
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Every control arrives as `{value, label, count, href, active}`, its `href` already built through your one builder with **every other axis carried along** — the arithmetic that, done by hand, drops the reader's letter the first time they press a chip.
|
|
61
|
+
|
|
62
|
+
- **The reset control leads the row**, flagged `reset`, so a consumer that maps the array cannot ship a narrowed view with nothing to press to widen it.
|
|
63
|
+
- **`alphaControls` returns `[]` below the index threshold**, folding in `needsAlphaIndex` — one call is either the index you needed or nothing, rather than a decision each caller makes and one caller forgets.
|
|
64
|
+
- **`resolveLetter` drops a letter no page starts with.** A stray `?letter=Q` would otherwise empty the grid with no control marked to explain why.
|
|
65
|
+
|
|
66
|
+
|
|
50
67
|
## Headings
|
|
51
68
|
|
|
52
69
|
Heading ids, permalink anchors, and the list an "on this page" rail renders — one pass over the HTML you already have.
|
|
@@ -148,6 +165,139 @@ export async function GET(request: Request) {
|
|
|
148
165
|
|
|
149
166
|
`parseVersion` / `bump` / `compareVersions` handle revision semver tolerantly — a page always has a version, even when the column holds `null` or junk.
|
|
150
167
|
|
|
168
|
+
## React
|
|
169
|
+
|
|
170
|
+
`wiki-formant/react` is the one part that needs React, so it is the one part behind its own subpath, and React is an *optional* peer. What is shared is behaviour; every class name and icon is passed in, because these wikis style their rails differently and always will.
|
|
171
|
+
|
|
172
|
+
`SidebarProvider` / `useSidebar` hold the rail's collapse state — remembered across loads, defaulted from the viewport only when the reader has never chosen. Pair it with `sidebarBootScript` from `wiki-formant/sidebar` (framework-free, so a server component can stamp it into `<head>`) or a remembered-closed rail paints open and animates shut on every load. `TableOfContents` is the "on this page" list, with scroll-spy, reading either headings you already know or the rendered article.
|
|
173
|
+
|
|
174
|
+
`useTypeahead` is the search field's state machine. Five surfaces across the three wikis had three implementations and no two agreed on what a search field does; this is their union, because each had a piece the others lacked:
|
|
175
|
+
|
|
176
|
+
```ts
|
|
177
|
+
const { query, setQuery, items, highlight, setHighlight, onKeyDown, reset } =
|
|
178
|
+
useTypeahead({ fetch: searchPages, onPick: page => router.push(page.href) });
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
- **A request-id guard**, so a slow response cannot paint over a newer one. Two of the five surfaces had none: type fast enough and you are reading results for a query you have already replaced.
|
|
182
|
+
- **Keyboard navigation.** Two surfaces had none — including the header dropdown of a wiki whose search *is* its primary navigation.
|
|
183
|
+
- **Abort and a per-query cache**, which only the third had. The cache is per mount, so a stale result cannot outlive the page.
|
|
184
|
+
|
|
185
|
+
`fetch` must be referentially stable; it is an effect dependency. Its `signal` is optional to implement — forward it from a REST fetcher, ignore it in a server action.
|
|
186
|
+
|
|
187
|
+
`useLinkPreview` is the Wikipedia-style hover card: one delegated listener rather than a component per link, which is what makes it viable over an article holding hundreds of anchors. Two wikis had written the same ninety lines — same intent delay, same grace period for the cursor to cross into the card, same clamp arithmetic, same cache. They differed in three things, and those three are the options: which anchors are eligible, what fetches a preview, and how tall the card is.
|
|
188
|
+
|
|
189
|
+
`useClickOutside` is fifteen lines and was in all three. Only one had the `offsetParent` check, and without it a container hidden at the current breakpoint still answers outside-clicks — so on a phone, a tap anywhere dismisses the popover the reader is looking at, because the hidden desktop copy got there first.
|
|
190
|
+
|
|
191
|
+
### The listbox contract
|
|
192
|
+
|
|
193
|
+
`useTypeahead` shared the state machine across five surfaces and left the ARIA
|
|
194
|
+
behind, so all five drifted into different wrongness: one put `aria-selected` on
|
|
195
|
+
a plain `<button>`, which is not a role that takes it; one gave the rows
|
|
196
|
+
`role="option"` and the container no `role="listbox"`, so the options had no
|
|
197
|
+
owner; two had no roles at all; one used a `data-highlighted` attribute, which
|
|
198
|
+
nothing reads. None of the five set `aria-activedescendant`, which is the
|
|
199
|
+
attribute that announces the highlighted row as the reader arrows through it.
|
|
200
|
+
|
|
201
|
+
```tsx
|
|
202
|
+
const { items, highlight, setHighlight, combobox } = useTypeahead({ fetch, onPick });
|
|
203
|
+
const { inputProps, listProps, optionProps } = combobox();
|
|
204
|
+
|
|
205
|
+
<input {...inputProps} />
|
|
206
|
+
<ul {...listProps}>
|
|
207
|
+
{items.map((item, i) => (
|
|
208
|
+
<li key={item.id} {...optionProps(i)} onMouseEnter={() => setHighlight(i)}>…</li>
|
|
209
|
+
))}
|
|
210
|
+
</ul>
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
- **`aria-activedescendant` is omitted, never emptied.** An id resolving to no
|
|
214
|
+
element is worse than no attribute: the field claims an active option and the
|
|
215
|
+
reader announces nothing.
|
|
216
|
+
- **`combobox(listKey)` takes a key** because one hook often feeds two rendered
|
|
217
|
+
lists — a header with a desktop field and a mobile one puts both in the same
|
|
218
|
+
document, and a single id set duplicates every option id.
|
|
219
|
+
|
|
220
|
+
The arithmetic is `wiki-formant/combobox`, which imports nothing, so the rules
|
|
221
|
+
are unit-tested without a DOM.
|
|
222
|
+
|
|
223
|
+
## AI crawlers
|
|
224
|
+
|
|
225
|
+
Every wiki kept this roster twice — once in the proxy, keyed by user-agent, to
|
|
226
|
+
count an `AI Bot Visit`; once in `robots.ts`, as the agents that get their own
|
|
227
|
+
group. Both copies were byte-identical across three repos, and they were
|
|
228
|
+
different lists.
|
|
229
|
+
|
|
230
|
+
```ts
|
|
231
|
+
import { detectAiBot, aiCrawlerRules } from 'wiki-formant/crawlers';
|
|
232
|
+
|
|
233
|
+
const bot = detectAiBot(request.headers.get('user-agent')); // label, or null
|
|
234
|
+
const rules = aiCrawlerRules({ allow: '/', disallow, aiAllow });
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
Only one direction of that difference was deliberate: `Applebot-Extended` never
|
|
238
|
+
fetches a page, so it belongs in robots and not in the matcher. The other
|
|
239
|
+
direction was not. Bytespider, CCBot, cohere-ai, Claude-Web and
|
|
240
|
+
Meta-ExternalFetcher were matched by every proxy and named by no robots.txt —
|
|
241
|
+
and a crawler obeys only its most-specific matching group, so an agent with no
|
|
242
|
+
group of its own falls through to `*`. Three wikis were measuring five crawlers
|
|
243
|
+
they had never addressed.
|
|
244
|
+
|
|
245
|
+
## Rendered-article passes
|
|
246
|
+
|
|
247
|
+
`wiki-formant/dom` holds what runs against an article element after it is in the document. Not React, so not in `react.tsx`.
|
|
248
|
+
|
|
249
|
+
```ts
|
|
250
|
+
addCopyButtons(el); // every <pre> gets one, once
|
|
251
|
+
hydrateTweetEmbeds(el); // placeholders get a live src
|
|
252
|
+
const off = onTweetResize(h => sizeTweetEmbeds(el, h));
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
`addCopyButton` was **byte-identical** in two BlockRenderers, down to the SVG path data. Its idempotence guard now lives inside the function rather than in a `pre:not(:has(…))` at the call site, where it can be — and was — retyped.
|
|
256
|
+
|
|
257
|
+
`activateTabGroups` turns stored `[data-tabs]` markup into a working tab group. The editor persists tabs as nested divs, which is the right thing to store — it survives a markdown twin, a plain HTML render and a reader with JavaScript off, all of which show every tab in order. Making one of them pressable is a reader-side job, and it sits beside the other passes rather than inside a component.
|
|
258
|
+
|
|
259
|
+
`TWITTER_ORIGIN` is written down once. It is both the embed host and the allow-list `onTweetResize` checks before believing a posted height, and it had been spelled out at four call sites across two repos. Any page can `postMessage`; only the embed host may size the embed.
|
|
260
|
+
|
|
261
|
+
## Editor nodes
|
|
262
|
+
|
|
263
|
+
`wiki-formant/tiptap` carries the custom nodes both wiki editors had written twice: `Iframe`, `YouTube` (the stock extension plus the paste rule it does not ship with), `TwitterEmbed`, `createMapEmbed`, `createCodeBlock` and `createTabs`. The four `@tiptap/*` packages are optional peers, so a consumer taking only the taxonomy still installs a package with no runtime dependencies.
|
|
264
|
+
|
|
265
|
+
```ts
|
|
266
|
+
const CodeBlock = createCodeBlock({
|
|
267
|
+
langs: CODE_LANGS, defaultLang: DEFAULT_LANG,
|
|
268
|
+
classNames: { button: 'lang-btn', option: 'lang-option', optionActive: 'text-accent' },
|
|
269
|
+
icons: { chevron: open => <ChevronDown className={open ? 'rotate-180' : ''} /> },
|
|
270
|
+
});
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
The ones that take config take it because that is exactly where the two copies differed — class tokens, the language list, and the API route a shortened map URL has to be resolved through. Injecting them is what lets one wiki keep `text-jupiter` and the other `text-accent` without either forking the node, and it keeps this file from dragging an icon library in behind it.
|
|
274
|
+
|
|
275
|
+
`createTabs` returns `TabGroup` and `TabItem` together: `tabGroup`'s content expression is `tabItem+`, so registering one without the other leaves a node type the schema cannot satisfy. A pasted short map link inserts immediately with `about:blank` and swaps its `src` when the redirect resolves — pasting must not block on a network hop, and the node has to exist for the reader to see anything happen.
|
|
276
|
+
|
|
277
|
+
## Analytics
|
|
278
|
+
|
|
279
|
+
Two lanes, one events helper. `mcpCallProps` reads a tool name out of a JSON-RPC
|
|
280
|
+
envelope that is untrusted and may be a batch; `searchQueryProps` normalises a
|
|
281
|
+
search box's free text so the same question aggregates as one row. Sending stays
|
|
282
|
+
with the caller — `plausibleEvent` builds the request, and the framework's own
|
|
283
|
+
deferral (`after()` in a route or server action, `event.waitUntil` in a proxy)
|
|
284
|
+
decides when it goes, so this package keeps its zero-dependency guarantee.
|
|
285
|
+
|
|
286
|
+
```ts
|
|
287
|
+
const props = searchQueryProps({ query, results: rows.length, surface: 'wiki' });
|
|
288
|
+
// null for an empty field, so a blank search cannot fire an event
|
|
289
|
+
if (props) after(() => plausibleEvent({ domain }, 'Search Query', url, props, headers, {
|
|
290
|
+
// a person triggered this, so it joins their session rather than landing
|
|
291
|
+
// as the fixed bot-tracker pseudo-visitor
|
|
292
|
+
userAgent: headers.get('user-agent') ?? undefined,
|
|
293
|
+
}));
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
Instrumenting the agent lane and not the human one is the easy mistake: it
|
|
297
|
+
leaves a wiki able to say what every crawler asked for and nothing about what
|
|
298
|
+
its readers asked for. The queries that return zero rows are the valuable ones —
|
|
299
|
+
they name a gap in the corpus in the reader's own words.
|
|
300
|
+
|
|
151
301
|
## API
|
|
152
302
|
|
|
153
303
|
| Export | From |
|
|
@@ -159,8 +309,15 @@ export async function GET(request: Request) {
|
|
|
159
309
|
| `corpusEtag`, `notModified`, `textHeaders`, `markdownHeaders`, `cleanSnippet`, `pageLine` | `wiki-formant/http` |
|
|
160
310
|
| `parsePagination`, `paginatedResponse`, `toOffset` | `wiki-formant/pagination` |
|
|
161
311
|
| `parseVersion`, `formatVersion`, `incrementVersion`, `bump`, `compareVersions` | `wiki-formant/versioning` |
|
|
162
|
-
|
|
163
|
-
|
|
312
|
+
| `plausibleEvent`, `mcpCallProps`, `searchQueryProps`, `plausibleDomain` | `wiki-formant/analytics` |
|
|
313
|
+
| `comboboxAria`, `listId`, `optionId` | `wiki-formant/combobox` |
|
|
314
|
+
| `AI_CRAWLERS`, `detectAiBot`, `aiCrawlerTokens`, `aiCrawlerRules` | `wiki-formant/crawlers` |
|
|
315
|
+
| `useCollapsibleSidebar`, `SidebarProvider`, `useSidebar`, `TableOfContents`, `useTypeahead`, `useLinkPreview`, `useClickOutside` | `wiki-formant/react` |
|
|
316
|
+
| `resolveSidebarOpen`, `sidebarBootScript`, `SIDEBAR_ATTRIBUTE` | `wiki-formant/sidebar` |
|
|
317
|
+
| `addCopyButtons`, `activateTabGroups`, `tweetEmbedSrc`, `onTweetResize`, `hydrateTweetEmbeds`, `sizeTweetEmbeds`, `TWITTER_ORIGIN` | `wiki-formant/dom` |
|
|
318
|
+
| `Iframe`, `YouTube`, `TwitterEmbed`, `createMapEmbed`, `createCodeBlock`, `createTabs` | `wiki-formant/tiptap` |
|
|
319
|
+
|
|
320
|
+
Everything above `wiki-formant/react` is also re-exported from the package root. The React, sidebar, DOM and tiptap subpaths are not: they carry `'use client'` or reach for a browser global, and the root has to stay importable from a route handler.
|
|
164
321
|
|
|
165
322
|
## Licence
|
|
166
323
|
|
package/dist/analytics.d.ts
CHANGED
|
@@ -38,6 +38,28 @@ export declare function mcpCallProps(request: {
|
|
|
38
38
|
headers: HeaderReader;
|
|
39
39
|
url: string;
|
|
40
40
|
}, body: unknown, server?: string): Record<string, string>;
|
|
41
|
+
export interface SearchQueryInput {
|
|
42
|
+
/** The settled query the typeahead debounce actually dispatched. */
|
|
43
|
+
query: string;
|
|
44
|
+
/** How many rows came back. Zero is the interesting case. */
|
|
45
|
+
results: number;
|
|
46
|
+
/** Which search box, when a site has more than one. */
|
|
47
|
+
surface?: string;
|
|
48
|
+
}
|
|
49
|
+
/**
|
|
50
|
+
* Props for a "Search Query" event — the human-side twin of `mcpCallProps`.
|
|
51
|
+
*
|
|
52
|
+
* A wiki instruments its agent surface and then counts every question an agent
|
|
53
|
+
* asks while recording none of the questions a person asks. The queries that
|
|
54
|
+
* return nothing are the valuable ones: they name a gap in the corpus in the
|
|
55
|
+
* reader's own words, which is otherwise only ever guessed at.
|
|
56
|
+
*
|
|
57
|
+
* Returns `null` for a query with no content, so an empty field cannot fire an
|
|
58
|
+
* event. The text is untrusted, so it is collapsed, lowercased for aggregation
|
|
59
|
+
* and truncated — the same 64-character bound search itself applies, which
|
|
60
|
+
* keeps the prop and the query that produced it the same string.
|
|
61
|
+
*/
|
|
62
|
+
export declare function searchQueryProps({ query, results, surface, }: SearchQueryInput): Record<string, string> | null;
|
|
41
63
|
/** The hostname of a site URL, for the `domain` a Plausible property is filed under. */
|
|
42
64
|
export declare function plausibleDomain(siteUrl: string | undefined, fallback: string): string;
|
|
43
65
|
//# sourceMappingURL=analytics.d.ts.map
|
package/dist/analytics.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"analytics.d.ts","sourceRoot":"","sources":["../src/analytics.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"analytics.d.ts","sourceRoot":"","sources":["../src/analytics.ts"],"names":[],"mappings":"AAiBA,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAEpD,MAAM,WAAW,eAAe;IAC9B,+DAA+D;IAC/D,MAAM,EAAE,MAAM,CAAC;IACf,0CAA0C;IAC1C,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,cAAc;IAC7B,iFAAiF;IACjF,OAAO,CAAC,EAAE;QAAE,QAAQ,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC;IAC/C;;;;OAIG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB;AAID;;;;;GAKG;AACH,wBAAgB,cAAc,CAC5B,MAAM,EAAE,eAAe,EACvB,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,MAAM,EACX,KAAK,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,EAC7B,OAAO,EAAE,YAAY,EACrB,KAAK,GAAE,cAAmB,GACzB,OAAO,CAAC,OAAO,CAAC,CAiBlB;AAED;;;;;;;;GAQG;AACH,wBAAgB,YAAY,CAC1B,OAAO,EAAE;IAAE,OAAO,EAAE,YAAY,CAAC;IAAC,GAAG,EAAE,MAAM,CAAA;CAAE,EAC/C,IAAI,EAAE,OAAO,EACb,MAAM,CAAC,EAAE,MAAM,GACd,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAQxB;AAED,MAAM,WAAW,gBAAgB;IAC/B,oEAAoE;IACpE,KAAK,EAAE,MAAM,CAAC;IACd,6DAA6D;IAC7D,OAAO,EAAE,MAAM,CAAC;IAChB,uDAAuD;IACvD,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED;;;;;;;;;;;;GAYG;AACH,wBAAgB,gBAAgB,CAAC,EAC/B,KAAK,EACL,OAAO,EACP,OAAO,GACR,EAAE,gBAAgB,GAAG,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,IAAI,CAUlD;AAED,wFAAwF;AACxF,wBAAgB,eAAe,CAAC,OAAO,EAAE,MAAM,GAAG,SAAS,EAAE,QAAQ,EAAE,MAAM,GAAG,MAAM,CAMrF"}
|
package/dist/analytics.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// analytics.ts — server-side Plausible events for agent
|
|
1
|
+
// analytics.ts — server-side Plausible events for a wiki's agent and human lanes.
|
|
2
2
|
//
|
|
3
3
|
// The proxy counts an "AI Bot Visit" by matching user agents, which cannot see
|
|
4
4
|
// inside a JSON-RPC envelope. So an MCP call is only countable by tool name at
|
|
@@ -6,6 +6,11 @@
|
|
|
6
6
|
// batch — is the part worth sharing. Every wiki in the workspace had its own
|
|
7
7
|
// copy of it, differing only in the default hostname.
|
|
8
8
|
//
|
|
9
|
+
// `searchQueryProps` is the human-side twin: the same shape problem, from the
|
|
10
|
+
// other direction. A search box is untrusted free text on every wiki, and the
|
|
11
|
+
// normalisation that makes it aggregatable — and keeps an empty field from
|
|
12
|
+
// firing an event — is the part worth sharing rather than re-deriving.
|
|
13
|
+
//
|
|
9
14
|
// Deferral stays with the caller: `after()` in a route handler,
|
|
10
15
|
// `event.waitUntil` in a proxy. Both are framework calls, and importing one
|
|
11
16
|
// here would cost this package its zero-dependency guarantee.
|
|
@@ -49,6 +54,31 @@ export function mcpCallProps(request, body, server) {
|
|
|
49
54
|
const ua = (request.headers.get('user-agent') || 'unknown').slice(0, 80);
|
|
50
55
|
return { ...(server ? { server } : {}), method, ...(tool ? { tool } : {}), ua };
|
|
51
56
|
}
|
|
57
|
+
/**
|
|
58
|
+
* Props for a "Search Query" event — the human-side twin of `mcpCallProps`.
|
|
59
|
+
*
|
|
60
|
+
* A wiki instruments its agent surface and then counts every question an agent
|
|
61
|
+
* asks while recording none of the questions a person asks. The queries that
|
|
62
|
+
* return nothing are the valuable ones: they name a gap in the corpus in the
|
|
63
|
+
* reader's own words, which is otherwise only ever guessed at.
|
|
64
|
+
*
|
|
65
|
+
* Returns `null` for a query with no content, so an empty field cannot fire an
|
|
66
|
+
* event. The text is untrusted, so it is collapsed, lowercased for aggregation
|
|
67
|
+
* and truncated — the same 64-character bound search itself applies, which
|
|
68
|
+
* keeps the prop and the query that produced it the same string.
|
|
69
|
+
*/
|
|
70
|
+
export function searchQueryProps({ query, results, surface, }) {
|
|
71
|
+
const q = (query ?? '').replace(/\s+/g, ' ').trim().toLowerCase().slice(0, 64);
|
|
72
|
+
if (!q)
|
|
73
|
+
return null;
|
|
74
|
+
return {
|
|
75
|
+
q,
|
|
76
|
+
// A string because Plausible props are strings; `results == "0"` is the
|
|
77
|
+
// filter that yields the gap list.
|
|
78
|
+
results: String(Math.max(0, Math.trunc(results) || 0)),
|
|
79
|
+
...(surface ? { surface } : {}),
|
|
80
|
+
};
|
|
81
|
+
}
|
|
52
82
|
/** The hostname of a site URL, for the `domain` a Plausible property is filed under. */
|
|
53
83
|
export function plausibleDomain(siteUrl, fallback) {
|
|
54
84
|
try {
|
package/dist/analytics.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"analytics.js","sourceRoot":"","sources":["../src/analytics.ts"],"names":[],"mappings":"AAAA,
|
|
1
|
+
{"version":3,"file":"analytics.js","sourceRoot":"","sources":["../src/analytics.ts"],"names":[],"mappings":"AAAA,kFAAkF;AAClF,EAAE;AACF,+EAA+E;AAC/E,+EAA+E;AAC/E,8EAA8E;AAC9E,6EAA6E;AAC7E,sDAAsD;AACtD,EAAE;AACF,8EAA8E;AAC9E,8EAA8E;AAC9E,2EAA2E;AAC3E,uEAAuE;AACvE,EAAE;AACF,gEAAgE;AAChE,4EAA4E;AAC5E,8DAA8D;AAsB9D,MAAM,gBAAgB,GAAG,gCAAgC,CAAC;AAE1D;;;;;GAKG;AACH,MAAM,UAAU,cAAc,CAC5B,MAAuB,EACvB,IAAY,EACZ,GAAW,EACX,KAA6B,EAC7B,OAAqB,EACrB,QAAwB,EAAE;IAE1B,OAAO,KAAK,CAAC,MAAM,CAAC,QAAQ,IAAI,gBAAgB,EAAE;QAChD,MAAM,EAAE,MAAM;QACd,OAAO,EAAE;YACP,cAAc,EAAE,kBAAkB;YAClC,YAAY,EAAE,KAAK,CAAC,SAAS,IAAI,0CAA0C;YAC3E,iBAAiB,EACf,OAAO,CAAC,GAAG,CAAC,iBAAiB,CAAC,IAAI,OAAO,CAAC,GAAG,CAAC,WAAW,CAAC,IAAI,WAAW;SAC5E;QACD,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,IAAI;YACJ,GAAG;YACH,MAAM,EAAE,MAAM,CAAC,MAAM;YACrB,KAAK,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC;YAC5B,GAAG,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,KAAK,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SACrD,CAAC;KACH,CAAC,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,SAAS,CAAC,CAAC;AAC5B,CAAC;AAED;;;;;;;;GAQG;AACH,MAAM,UAAU,YAAY,CAC1B,OAA+C,EAC/C,IAAa,EACb,MAAe;IAEf,MAAM,KAAK,GAAG,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAEtC,CAAC;IACd,MAAM,MAAM,GAAG,OAAO,KAAK,EAAE,MAAM,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,SAAS,CAAC;IAC5E,MAAM,IAAI,GAAG,OAAO,KAAK,EAAE,MAAM,EAAE,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;IACrF,MAAM,EAAE,GAAG,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,YAAY,CAAC,IAAI,SAAS,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IACzE,OAAO,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,MAAM,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,EAAE,CAAC;AAClF,CAAC;AAWD;;;;;;;;;;;;GAYG;AACH,MAAM,UAAU,gBAAgB,CAAC,EAC/B,KAAK,EACL,OAAO,EACP,OAAO,GACU;IACjB,MAAM,CAAC,GAAG,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IAC/E,IAAI,CAAC,CAAC;QAAE,OAAO,IAAI,CAAC;IACpB,OAAO;QACL,CAAC;QACD,wEAAwE;QACxE,mCAAmC;QACnC,OAAO,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC;QACtD,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAChC,CAAC;AACJ,CAAC;AAED,wFAAwF;AACxF,MAAM,UAAU,eAAe,CAAC,OAA2B,EAAE,QAAgB;IAC3E,IAAI,CAAC;QACH,OAAO,IAAI,GAAG,CAAC,OAAO,IAAI,QAAQ,CAAC,CAAC,QAAQ,CAAC;IAC/C,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,QAAQ,CAAC,OAAO,CAAC,cAAc,EAAE,EAAE,CAAC,CAAC,OAAO,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;IACnE,CAAC;AACH,CAAC"}
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
/** Which surface a set of controls belongs to, when one hook feeds several. */
|
|
2
|
+
export interface ComboboxInput {
|
|
3
|
+
/** Stable per-mount prefix. React callers pass `useId()`. */
|
|
4
|
+
baseId: string;
|
|
5
|
+
/** Distinguishes surfaces sharing one hook. A header with a desktop and a
|
|
6
|
+
* mobile list renders both into the same document; one id set for both
|
|
7
|
+
* would duplicate every option id and leave `aria-activedescendant`
|
|
8
|
+
* pointing at whichever copy the parser reached first. */
|
|
9
|
+
listKey?: string;
|
|
10
|
+
/** How many rows are currently rendered. */
|
|
11
|
+
count: number;
|
|
12
|
+
/** Index of the highlighted row. */
|
|
13
|
+
highlight: number;
|
|
14
|
+
}
|
|
15
|
+
export interface ComboboxInputProps {
|
|
16
|
+
role: 'combobox';
|
|
17
|
+
'aria-expanded': boolean;
|
|
18
|
+
'aria-controls': string;
|
|
19
|
+
'aria-autocomplete': 'list';
|
|
20
|
+
'aria-activedescendant': string | undefined;
|
|
21
|
+
}
|
|
22
|
+
export interface ComboboxListProps {
|
|
23
|
+
id: string;
|
|
24
|
+
role: 'listbox';
|
|
25
|
+
}
|
|
26
|
+
export interface ComboboxOptionProps {
|
|
27
|
+
id: string;
|
|
28
|
+
role: 'option';
|
|
29
|
+
'aria-selected': boolean;
|
|
30
|
+
}
|
|
31
|
+
export interface ComboboxAria {
|
|
32
|
+
inputProps: ComboboxInputProps;
|
|
33
|
+
listProps: ComboboxListProps;
|
|
34
|
+
optionProps: (index: number) => ComboboxOptionProps;
|
|
35
|
+
}
|
|
36
|
+
/** The id of one option row. Exported because a caller that scrolls the
|
|
37
|
+
* highlighted row into view has to find it by the same id the input points at. */
|
|
38
|
+
export declare function optionId(baseId: string, listKey: string | undefined, index: number): string;
|
|
39
|
+
export declare function listId(baseId: string, listKey?: string): string;
|
|
40
|
+
/**
|
|
41
|
+
* Every attribute the three parts of a combobox need, derived from what the
|
|
42
|
+
* typeahead already knows.
|
|
43
|
+
*
|
|
44
|
+
* `aria-activedescendant` is omitted rather than emptied when nothing is
|
|
45
|
+
* highlighted or the list is closed. An id that resolves to no element is worse
|
|
46
|
+
* than no attribute: the reader announces nothing and the field still claims to
|
|
47
|
+
* have an active option.
|
|
48
|
+
*/
|
|
49
|
+
export declare function comboboxAria({ baseId, listKey, count, highlight }: ComboboxInput): ComboboxAria;
|
|
50
|
+
//# sourceMappingURL=combobox.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"combobox.d.ts","sourceRoot":"","sources":["../src/combobox.ts"],"names":[],"mappings":"AAsBA,+EAA+E;AAC/E,MAAM,WAAW,aAAa;IAC5B,6DAA6D;IAC7D,MAAM,EAAE,MAAM,CAAC;IACf;;;+DAG2D;IAC3D,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,4CAA4C;IAC5C,KAAK,EAAE,MAAM,CAAC;IACd,oCAAoC;IACpC,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,kBAAkB;IACjC,IAAI,EAAE,UAAU,CAAC;IACjB,eAAe,EAAE,OAAO,CAAC;IACzB,eAAe,EAAE,MAAM,CAAC;IACxB,mBAAmB,EAAE,MAAM,CAAC;IAC5B,uBAAuB,EAAE,MAAM,GAAG,SAAS,CAAC;CAC7C;AAED,MAAM,WAAW,iBAAiB;IAChC,EAAE,EAAE,MAAM,CAAC;IACX,IAAI,EAAE,SAAS,CAAC;CACjB;AAED,MAAM,WAAW,mBAAmB;IAClC,EAAE,EAAE,MAAM,CAAC;IACX,IAAI,EAAE,QAAQ,CAAC;IACf,eAAe,EAAE,OAAO,CAAC;CAC1B;AAED,MAAM,WAAW,YAAY;IAC3B,UAAU,EAAE,kBAAkB,CAAC;IAC/B,SAAS,EAAE,iBAAiB,CAAC;IAC7B,WAAW,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,mBAAmB,CAAC;CACrD;AAED;mFACmF;AACnF,wBAAgB,QAAQ,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,GAAG,SAAS,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAE3F;AAED,wBAAgB,MAAM,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,MAAM,GAAG,MAAM,CAE/D;AAED;;;;;;;;GAQG;AACH,wBAAgB,YAAY,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,KAAK,EAAE,SAAS,EAAE,EAAE,aAAa,GAAG,YAAY,CAmB/F"}
|
package/dist/combobox.js
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
// combobox.ts — the listbox contract a typeahead owes a screen reader.
|
|
2
|
+
//
|
|
3
|
+
// Framework-free on purpose, the same way `sidebar.ts` is: the arithmetic is
|
|
4
|
+
// pure, so it can be tested without a DOM, and `react.tsx` is left holding only
|
|
5
|
+
// the hook that calls it.
|
|
6
|
+
//
|
|
7
|
+
// `useTypeahead` already shared the state machine across five surfaces in three
|
|
8
|
+
// wikis. The ARIA did not travel with it, and all five had drifted into
|
|
9
|
+
// different wrongness:
|
|
10
|
+
//
|
|
11
|
+
// - one put `aria-selected` on a plain `<button>`, which is not a role that
|
|
12
|
+
// takes it, and gave the container no `role="listbox"` at all;
|
|
13
|
+
// - one gave the rows `role="option"` but still no listbox, so the options
|
|
14
|
+
// had no owner;
|
|
15
|
+
// - two had no roles whatsoever;
|
|
16
|
+
// - one used a `data-highlighted` attribute, which no assistive technology
|
|
17
|
+
// reads.
|
|
18
|
+
//
|
|
19
|
+
// None of the five set `aria-activedescendant`, which is the attribute that
|
|
20
|
+
// actually announces the highlighted row as the reader arrows through it. A
|
|
21
|
+
// combobox without it is a text field that silently changes what Enter does.
|
|
22
|
+
/** The id of one option row. Exported because a caller that scrolls the
|
|
23
|
+
* highlighted row into view has to find it by the same id the input points at. */
|
|
24
|
+
export function optionId(baseId, listKey, index) {
|
|
25
|
+
return `${listId(baseId, listKey)}-opt-${index}`;
|
|
26
|
+
}
|
|
27
|
+
export function listId(baseId, listKey) {
|
|
28
|
+
return listKey ? `${baseId}-${listKey}` : `${baseId}-list`;
|
|
29
|
+
}
|
|
30
|
+
/**
|
|
31
|
+
* Every attribute the three parts of a combobox need, derived from what the
|
|
32
|
+
* typeahead already knows.
|
|
33
|
+
*
|
|
34
|
+
* `aria-activedescendant` is omitted rather than emptied when nothing is
|
|
35
|
+
* highlighted or the list is closed. An id that resolves to no element is worse
|
|
36
|
+
* than no attribute: the reader announces nothing and the field still claims to
|
|
37
|
+
* have an active option.
|
|
38
|
+
*/
|
|
39
|
+
export function comboboxAria({ baseId, listKey, count, highlight }) {
|
|
40
|
+
const list = listId(baseId, listKey);
|
|
41
|
+
const open = count > 0;
|
|
42
|
+
const active = open && highlight >= 0 && highlight < count;
|
|
43
|
+
return {
|
|
44
|
+
inputProps: {
|
|
45
|
+
role: 'combobox',
|
|
46
|
+
'aria-expanded': open,
|
|
47
|
+
'aria-controls': list,
|
|
48
|
+
'aria-autocomplete': 'list',
|
|
49
|
+
'aria-activedescendant': active ? optionId(baseId, listKey, highlight) : undefined,
|
|
50
|
+
},
|
|
51
|
+
listProps: { id: list, role: 'listbox' },
|
|
52
|
+
optionProps: (index) => ({
|
|
53
|
+
id: optionId(baseId, listKey, index),
|
|
54
|
+
role: 'option',
|
|
55
|
+
'aria-selected': index === highlight,
|
|
56
|
+
}),
|
|
57
|
+
};
|
|
58
|
+
}
|
|
59
|
+
//# sourceMappingURL=combobox.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"combobox.js","sourceRoot":"","sources":["../src/combobox.ts"],"names":[],"mappings":"AAAA,uEAAuE;AACvE,EAAE;AACF,6EAA6E;AAC7E,gFAAgF;AAChF,0BAA0B;AAC1B,EAAE;AACF,gFAAgF;AAChF,wEAAwE;AACxE,uBAAuB;AACvB,EAAE;AACF,8EAA8E;AAC9E,mEAAmE;AACnE,6EAA6E;AAC7E,oBAAoB;AACpB,mCAAmC;AACnC,6EAA6E;AAC7E,aAAa;AACb,EAAE;AACF,4EAA4E;AAC5E,4EAA4E;AAC5E,6EAA6E;AA0C7E;mFACmF;AACnF,MAAM,UAAU,QAAQ,CAAC,MAAc,EAAE,OAA2B,EAAE,KAAa;IACjF,OAAO,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,QAAQ,KAAK,EAAE,CAAC;AACnD,CAAC;AAED,MAAM,UAAU,MAAM,CAAC,MAAc,EAAE,OAAgB;IACrD,OAAO,OAAO,CAAC,CAAC,CAAC,GAAG,MAAM,IAAI,OAAO,EAAE,CAAC,CAAC,CAAC,GAAG,MAAM,OAAO,CAAC;AAC7D,CAAC;AAED;;;;;;;;GAQG;AACH,MAAM,UAAU,YAAY,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,KAAK,EAAE,SAAS,EAAiB;IAC/E,MAAM,IAAI,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IACrC,MAAM,IAAI,GAAG,KAAK,GAAG,CAAC,CAAC;IACvB,MAAM,MAAM,GAAG,IAAI,IAAI,SAAS,IAAI,CAAC,IAAI,SAAS,GAAG,KAAK,CAAC;IAC3D,OAAO;QACL,UAAU,EAAE;YACV,IAAI,EAAE,UAAU;YAChB,eAAe,EAAE,IAAI;YACrB,eAAe,EAAE,IAAI;YACrB,mBAAmB,EAAE,MAAM;YAC3B,uBAAuB,EAAE,MAAM,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,EAAE,OAAO,EAAE,SAAS,CAAC,CAAC,CAAC,CAAC,SAAS;SACnF;QACD,SAAS,EAAE,EAAE,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,SAAS,EAAE;QACxC,WAAW,EAAE,CAAC,KAAa,EAAE,EAAE,CAAC,CAAC;YAC/B,EAAE,EAAE,QAAQ,CAAC,MAAM,EAAE,OAAO,EAAE,KAAK,CAAC;YACpC,IAAI,EAAE,QAAQ;YACd,eAAe,EAAE,KAAK,KAAK,SAAS;SACrC,CAAC;KACH,CAAC;AACJ,CAAC"}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"conformance.d.ts","sourceRoot":"","sources":["../src/conformance.ts"],"names":[],"mappings":"AAcA,MAAM,WAAW,GAAG;IAClB,MAAM,CAAC,EAAE;QACP,OAAO,CAAC,EAAE,KAAK,CAAC;YAAE,IAAI,CAAC,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QACnC,OAAO,CAAC,EAAE,OAAO,CAAC;QAClB,KAAK,CAAC,EAAE,KAAK,CAAC;YAAE,IAAI,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QAChC,SAAS,CAAC,EAAE,KAAK,CAAC;YAAE,GAAG,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QACnC,QAAQ,CAAC,EAAE,KAAK,CAAC;YAAE,IAAI,CAAC,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QACpC,UAAU,CAAC,EAAE;YAAE,OAAO,CAAC,EAAE,MAAM,CAAA;SAAE,CAAC;QAClC,YAAY,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;QACvC,YAAY,CAAC,EAAE,MAAM,CAAC;KACvB,CAAC;IACF,KAAK,CAAC,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,CAAC,EAAE;YAAE,cAAc,CAAC,EAAE,MAAM,EAAE,CAAA;SAAE,CAAA;KAAE,CAAC;CACjF;AAED,MAAM,WAAW,WAAW;IAC1B,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,OAAO,CAAC;IACZ,IAAI,EAAE,MAAM,CAAC;CACd;AAED,MAAM,WAAW,aAAa;IAC5B,4CAA4C;IAC5C,IAAI,EAAE,MAAM,CAAC;IACb,mDAAmD;IACnD,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,oEAAoE;IACpE,UAAU,EAAE,MAAM,CAAC;CACpB;AAED,MAAM,WAAW,MAAM;IACrB,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EAAE,MAAM,CAAC;IACjB,8DAA8D;IAC9D,GAAG,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC;IACpD,8BAA8B;IAC9B,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC;IACjE,8EAA8E;IAC9E,OAAO,CAAC,CAAC,EAAE,GAAG,GAAG,OAAO,CAAC;IACzB,sDAAsD;IACtD,KAAK,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,GAAG,IAAI,CAAC;IACtD,OAAO,EAAE,WAAW,EAAE,CAAC;IACvB,kCAAkC;IAClC,SAAS,IAAI,MAAM,CAAC;IACpB;;;OAGG;IACH,UAAU,IAAI,IAAI,CAAC;IACnB,wDAAwD;IACxD,OAAO,IAAI,MAAM,CAAC;CACnB;AAED,wBAAgB,YAAY,CAAC,IAAI,EAAE,aAAa,GAAG,MAAM,CAkDxD;AAED;;;;GAIG;AACH,wBAAsB,eAAe,CACnC,CAAC,EAAE,MAAM,EACT,UAAU,EAAE,MAAM;AAClB;;;;GAIG;AACH,oBAAoB,GAAE,SAAS,MAAM,EAAc,GAClD,OAAO,CAAC,GAAG,CAAC,CAyDd;AAED;;;;GAIG;AACH,wBAAsB,gBAAgB,CACpC,CAAC,EAAE,MAAM,EACT,QAAQ,EAAE,MAAM,EAChB,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE;IAAE,GAAG,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,CAAC,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,OAAO,CAAA;CAAE,CAAC,EACxF,WAAW,CAAC,EAAE,MAAM,GACnB,OAAO,CAAC,IAAI,CAAC,CAef;AAED;;;;;GAKG;AACH,wBAAsB,eAAe,CAAC,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAM9D;AAED;;;GAGG;AACH,wBAAsB,oBAAoB,CAAC,CAAC,EAAE,MAAM,EAAE,KAAK,EAAE,SAAS,MAAM,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC,
|
|
1
|
+
{"version":3,"file":"conformance.d.ts","sourceRoot":"","sources":["../src/conformance.ts"],"names":[],"mappings":"AAcA,MAAM,WAAW,GAAG;IAClB,MAAM,CAAC,EAAE;QACP,OAAO,CAAC,EAAE,KAAK,CAAC;YAAE,IAAI,CAAC,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QACnC,OAAO,CAAC,EAAE,OAAO,CAAC;QAClB,KAAK,CAAC,EAAE,KAAK,CAAC;YAAE,IAAI,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QAChC,SAAS,CAAC,EAAE,KAAK,CAAC;YAAE,GAAG,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QACnC,QAAQ,CAAC,EAAE,KAAK,CAAC;YAAE,IAAI,CAAC,EAAE,MAAM,CAAA;SAAE,CAAC,CAAC;QACpC,UAAU,CAAC,EAAE;YAAE,OAAO,CAAC,EAAE,MAAM,CAAA;SAAE,CAAC;QAClC,YAAY,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;QACvC,YAAY,CAAC,EAAE,MAAM,CAAC;KACvB,CAAC;IACF,KAAK,CAAC,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,CAAC,EAAE;YAAE,cAAc,CAAC,EAAE,MAAM,EAAE,CAAA;SAAE,CAAA;KAAE,CAAC;CACjF;AAED,MAAM,WAAW,WAAW;IAC1B,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,OAAO,CAAC;IACZ,IAAI,EAAE,MAAM,CAAC;CACd;AAED,MAAM,WAAW,aAAa;IAC5B,4CAA4C;IAC5C,IAAI,EAAE,MAAM,CAAC;IACb,mDAAmD;IACnD,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,oEAAoE;IACpE,UAAU,EAAE,MAAM,CAAC;CACpB;AAED,MAAM,WAAW,MAAM;IACrB,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EAAE,MAAM,CAAC;IACjB,8DAA8D;IAC9D,GAAG,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC;IACpD,8BAA8B;IAC9B,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC;IACjE,8EAA8E;IAC9E,OAAO,CAAC,CAAC,EAAE,GAAG,GAAG,OAAO,CAAC;IACzB,sDAAsD;IACtD,KAAK,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,MAAM,GAAG,IAAI,CAAC;IACtD,OAAO,EAAE,WAAW,EAAE,CAAC;IACvB,kCAAkC;IAClC,SAAS,IAAI,MAAM,CAAC;IACpB;;;OAGG;IACH,UAAU,IAAI,IAAI,CAAC;IACnB,wDAAwD;IACxD,OAAO,IAAI,MAAM,CAAC;CACnB;AAED,wBAAgB,YAAY,CAAC,IAAI,EAAE,aAAa,GAAG,MAAM,CAkDxD;AAED;;;;GAIG;AACH,wBAAsB,eAAe,CACnC,CAAC,EAAE,MAAM,EACT,UAAU,EAAE,MAAM;AAClB;;;;GAIG;AACH,oBAAoB,GAAE,SAAS,MAAM,EAAc,GAClD,OAAO,CAAC,GAAG,CAAC,CAyDd;AAED;;;;GAIG;AACH,wBAAsB,gBAAgB,CACpC,CAAC,EAAE,MAAM,EACT,QAAQ,EAAE,MAAM,EAChB,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE;IAAE,GAAG,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,CAAC,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,OAAO,CAAA;CAAE,CAAC,EACxF,WAAW,CAAC,EAAE,MAAM,GACnB,OAAO,CAAC,IAAI,CAAC,CAef;AAED;;;;;GAKG;AACH,wBAAsB,eAAe,CAAC,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAM9D;AAED;;;GAGG;AACH,wBAAsB,oBAAoB,CAAC,CAAC,EAAE,MAAM,EAAE,KAAK,EAAE,SAAS,MAAM,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC,CAe7F"}
|
package/dist/conformance.js
CHANGED
|
@@ -151,8 +151,16 @@ export async function conditionalGetChecks(t, paths) {
|
|
|
151
151
|
for (const path of paths) {
|
|
152
152
|
const fresh = await fetch(`${t.base}/${path}`);
|
|
153
153
|
const etag = fresh.headers.get('etag');
|
|
154
|
+
// Both validators, not just the one the 304 is driven by. A crawler that
|
|
155
|
+
// sends If-Modified-Since rather than If-None-Match — and several do — gets
|
|
156
|
+
// a full render on every pass from a response that carries no Last-Modified,
|
|
157
|
+
// and the ETag check alone would call that surface conformant. acuiq2's
|
|
158
|
+
// suite asserted this locally; adopting the shared check should not have
|
|
159
|
+
// been a trade.
|
|
160
|
+
const lastModified = fresh.headers.get('last-modified');
|
|
154
161
|
const revalidated = etag ? await fetch(`${t.base}/${path}`, { headers: { 'If-None-Match': etag } }) : null;
|
|
155
162
|
t.check(`${path} 304`, !!etag && revalidated?.status === 304, `etag=${etag} revalidate=${revalidated?.status}`);
|
|
163
|
+
t.check(`${path} last-modified`, !!lastModified, `${fresh.status} ${lastModified}`);
|
|
156
164
|
}
|
|
157
165
|
}
|
|
158
166
|
//# sourceMappingURL=conformance.js.map
|
package/dist/conformance.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"conformance.js","sourceRoot":"","sources":["../src/conformance.ts"],"names":[],"mappings":"AAAA,gFAAgF;AAChF,kBAAkB;AAClB,EAAE;AACF,iFAAiF;AACjF,mFAAmF;AACnF,gFAAgF;AAChF,+EAA+E;AAC/E,6DAA6D;AAC7D,EAAE;AACF,kFAAkF;AAClF,+EAA+E;AAC/E,kFAAkF;AAClF,oDAAoD;AAsDpD,MAAM,UAAU,YAAY,CAAC,IAAmB;IAC9C,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;IAC1C,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,GAAG,IAAI,UAAU,CAAC;IACpD,MAAM,OAAO,GAAkB,EAAE,CAAC;IAClC,IAAI,KAAK,GAAG,CAAC,CAAC;IAEd,MAAM,GAAG,GAAG,KAAK,EAAE,MAAc,EAAE,MAAgB,EAAgB,EAAE;QACnE,KAAK,EAAE,CAAC;QACR,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;YAChC,MAAM,EAAE,MAAM;YACd,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,YAAY,EAAE,IAAI,CAAC,UAAU,EAAE;YAC9E,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC;SACpE,CAAC,CAAC;QACH,4EAA4E;QAC5E,qEAAqE;QACrE,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YACvB,MAAM,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,UAAU,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC;YAC5C,OAAO,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC7B,CAAC;QACD,OAAO,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAQ,CAAC;IACnC,CAAC,CAAC;IAEF,OAAO;QACL,IAAI;QACJ,QAAQ;QACR,GAAG;QACH,IAAI,EAAE,CAAC,IAAI,EAAE,IAAI,GAAG,EAAE,EAAE,EAAE,CAAC,GAAG,CAAC,YAAY,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC;QACvE,OAAO,CAAC,CAAC;YACP,MAAM,IAAI,GAAG,CAAC,CAAC,MAAM,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC;YAC1C,IAAI,IAAI,IAAI,IAAI;gBAAE,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC;YAClE,IAAI,CAAC;gBACH,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;YAC1B,CAAC;YAAC,MAAM,CAAC;gBACP,OAAO,IAAI,CAAC;YACd,CAAC;QACH,CAAC;QACD,KAAK,CAAC,IAAI,EAAE,EAAE,EAAE,IAAI;YAClB,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC;YACjC,OAAO,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,QAAQ,KAAK,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC;QACzE,CAAC;QACD,OAAO;QACP,SAAS,EAAE,GAAG,EAAE,CAAC,KAAK;QACtB,UAAU,EAAE,GAAG,EAAE,GAAG,KAAK,EAAE,CAAC,CAAC,CAAC;QAC9B,OAAO;YACL,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;YAC1C,OAAO,CAAC,GAAG,CAAC,cAAc,OAAO,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,IAAI,OAAO,CAAC,MAAM,YAAY,KAAK,0BAA0B,CAAC,CAAC;YACvH,KAAK,MAAM,CAAC,IAAI,MAAM;gBAAE,OAAO,CAAC,GAAG,CAAC,WAAW,CAAC,CAAC,IAAI,MAAM,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;YACrE,OAAO,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QAC/B,CAAC;KACF,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,eAAe,CACnC,CAAS,EACT,UAAkB;AAClB;;;;GAIG;AACH,uBAA0C,CAAC,OAAO,CAAC;IAEnD,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,CAAC,QAAQ,MAAM,CAAC,CAAC;IACnD,MAAM,IAAI,GAAG,MAAM,CAAC,CAAC,GAAG,CAAC,YAAY,EAAE;QACrC,eAAe,EAAE,YAAY;QAC7B,YAAY,EAAE,EAAE;QAChB,UAAU,EAAE,EAAE,IAAI,EAAE,UAAU,EAAE,OAAO,EAAE,GAAG,EAAE;KAC/C,CAAC,CAAC;IACH,CAAC,CAAC,KAAK,CAAC,YAAY,EAAE,CAAC,CAAC,IAAI,CAAC,MAAM,EAAE,UAAU,IAAI,CAAC,CAAC,IAAI,CAAC,MAAM,EAAE,YAAY,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,MAAM,EAAE,UAAU,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;IAEvI,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,CAAC;IAEnC,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE,EAAE,MAAM,EAAE,SAAS,EAAE,CAAC,CAAC;IAC3D,CAAC,CAAC,KAAK,CACL,mBAAmB,EACnB,GAAG,CAAC,MAAM,KAAK,GAAG;QAChB,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,KAAK,GAAG;QACtD,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,8BAA8B,CAAC,IAAI,EAAE,CAAC,CAAC,QAAQ,CAAC,sBAAsB,CAAC,EAC1F,GAAG,GAAG,CAAC,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,EAAE,CACvE,CAAC;IAEF,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC;IACpC,CAAC,CAAC,KAAK,CACL,SAAS,EACT,GAAG,CAAC,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,KAAK,GAAG,EAC5E,GAAG,GAAG,CAAC,MAAM,UAAU,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,SAAS,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,EAAE,CACzG,CAAC;IAEF,6EAA6E;IAC7E,kDAAkD;IAClD,MAAM,KAAK,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE;QACpC,MAAM,EAAE,MAAM;QACd,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE;QAC/C,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,2BAA2B,EAAE,CAAC;KAC9E,CAAC,CAAC;IACH,MAAM,SAAS,GAAG,MAAM,KAAK,CAAC,IAAI,EAAE,CAAC;IACrC,CAAC,CAAC,KAAK,CAAC,kBAAkB,EAAE,KAAK,CAAC,MAAM,KAAK,GAAG,IAAI,SAAS,KAAK,EAAE,EAAE,GAAG,KAAK,CAAC,MAAM,UAAU,SAAS,GAAG,CAAC,CAAC;IAE7G,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC,CAAC;IAC3H,MAAM,OAAO,GAAG,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAQ,CAAC;IAC1C,CAAC,CAAC,KAAK,CAAC,iBAAiB,EAAE,GAAG,CAAC,MAAM,KAAK,GAAG,IAAI,OAAO,CAAC,KAAK,EAAE,IAAI,KAAK,CAAC,KAAK,EAAE,GAAG,GAAG,CAAC,MAAM,SAAS,OAAO,CAAC,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;IAE9H,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE;QACnC,MAAM,EAAE,MAAM;QACd,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE;QAC/C,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC;KACxG,CAAC,CAAC;IACH,MAAM,QAAQ,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAQ,CAAC;IAC5C,CAAC,CAAC,KAAK,CAAC,gBAAgB,EAAE,QAAQ,CAAC,KAAK,EAAE,IAAI,KAAK,CAAC,KAAK,EAAE,QAAQ,QAAQ,CAAC,KAAK,EAAE,IAAI,KAAK,CAAC,QAAQ,CAAC,KAAK,EAAE,OAAO,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;IAE5I,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,EAAE,YAAY,IAAI,EAAE,CAAC;IAC7C,CAAC,CAAC,KAAK,CACL,qBAAqB,EACrB,oBAAoB,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,EAC1C,GAAG,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,oBAAoB,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CACxE,CAAC;IAEF,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,gBAAgB,CACpC,CAAS,EACT,QAAgB,EAChB,OAAwF,EACxF,WAAoB;IAEpB,MAAM,KAAK,GAA4B,EAAE,IAAI,EAAE,WAAW,EAAE,CAAC;IAC7D,KAAK,MAAM,CAAC,KAAK,EAAE,EAAE,GAAG,EAAE,EAAE,EAAE,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;QAC3D,IAAI,CAAC;YACH,KAAK,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,CAAC,MAAM,CAAC,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,CAA4B,CAAC,CAAC;QAClF,CAAC;QAAC,MAAM,CAAC;YACP,KAAK,CAAC,KAAK,CAAC,GAAG,eAAe,CAAC;QACjC,CAAC;IACH,CAAC;IACD,MAAM,GAAG,GAAG,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,KAAK,SAAS,CAAC,CAAC;IAC9D,CAAC,CAAC,KAAK,CACL,mBAAmB,EACnB,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,KAAK,QAAQ,CAAC,EAC9B,YAAY,QAAQ,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CACvF,CAAC;AACJ,CAAC;AAED;;;;;GAKG;AACH,MAAM,CAAC,KAAK,UAAU,eAAe,CAAC,CAAS;IAC7C,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC;QAC/B,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,8BAA8B,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QAClE,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,yBAAyB,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;KAC9D,CAAC,CAAC;IACH,CAAC,CAAC,KAAK,CAAC,mBAAmB,EAAE,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,KAAK,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,EAAE,gCAAgC,CAAC,CAAC;AAC1G,CAAC;AAED;;;GAGG;AACH,MAAM,CAAC,KAAK,UAAU,oBAAoB,CAAC,CAAS,EAAE,KAAwB;IAC5E,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,MAAM,KAAK,GAAG,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,IAAI,IAAI,EAAE,CAAC,CAAC;QAC/C,MAAM,IAAI,GAAG,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QACvC,MAAM,WAAW,GAAG,IAAI,CAAC,CAAC,CAAC,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,IAAI,IAAI,EAAE,EAAE,EAAE,OAAO,EAAE,EAAE,eAAe,EAAE,IAAI,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QAC3G,CAAC,CAAC,KAAK,CAAC,GAAG,IAAI,MAAM,EAAE,CAAC,CAAC,IAAI,IAAI,WAAW,EAAE,MAAM,KAAK,GAAG,EAAE,QAAQ,IAAI,eAAe,WAAW,EAAE,MAAM,EAAE,CAAC,CAAC;
|
|
1
|
+
{"version":3,"file":"conformance.js","sourceRoot":"","sources":["../src/conformance.ts"],"names":[],"mappings":"AAAA,gFAAgF;AAChF,kBAAkB;AAClB,EAAE;AACF,iFAAiF;AACjF,mFAAmF;AACnF,gFAAgF;AAChF,+EAA+E;AAC/E,6DAA6D;AAC7D,EAAE;AACF,kFAAkF;AAClF,+EAA+E;AAC/E,kFAAkF;AAClF,oDAAoD;AAsDpD,MAAM,UAAU,YAAY,CAAC,IAAmB;IAC9C,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;IAC1C,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,GAAG,IAAI,UAAU,CAAC;IACpD,MAAM,OAAO,GAAkB,EAAE,CAAC;IAClC,IAAI,KAAK,GAAG,CAAC,CAAC;IAEd,MAAM,GAAG,GAAG,KAAK,EAAE,MAAc,EAAE,MAAgB,EAAgB,EAAE;QACnE,KAAK,EAAE,CAAC;QACR,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;YAChC,MAAM,EAAE,MAAM;YACd,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,YAAY,EAAE,IAAI,CAAC,UAAU,EAAE;YAC9E,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC;SACpE,CAAC,CAAC;QACH,4EAA4E;QAC5E,qEAAqE;QACrE,IAAI,GAAG,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;YACvB,MAAM,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,UAAU,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC;YAC5C,OAAO,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC7B,CAAC;QACD,OAAO,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAQ,CAAC;IACnC,CAAC,CAAC;IAEF,OAAO;QACL,IAAI;QACJ,QAAQ;QACR,GAAG;QACH,IAAI,EAAE,CAAC,IAAI,EAAE,IAAI,GAAG,EAAE,EAAE,EAAE,CAAC,GAAG,CAAC,YAAY,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC;QACvE,OAAO,CAAC,CAAC;YACP,MAAM,IAAI,GAAG,CAAC,CAAC,MAAM,EAAE,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC;YAC1C,IAAI,IAAI,IAAI,IAAI;gBAAE,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC;YAClE,IAAI,CAAC;gBACH,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;YAC1B,CAAC;YAAC,MAAM,CAAC;gBACP,OAAO,IAAI,CAAC;YACd,CAAC;QACH,CAAC;QACD,KAAK,CAAC,IAAI,EAAE,EAAE,EAAE,IAAI;YAClB,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC;YACjC,OAAO,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,QAAQ,KAAK,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC;QACzE,CAAC;QACD,OAAO;QACP,SAAS,EAAE,GAAG,EAAE,CAAC,KAAK;QACtB,UAAU,EAAE,GAAG,EAAE,GAAG,KAAK,EAAE,CAAC,CAAC,CAAC;QAC9B,OAAO;YACL,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;YAC1C,OAAO,CAAC,GAAG,CAAC,cAAc,OAAO,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,IAAI,OAAO,CAAC,MAAM,YAAY,KAAK,0BAA0B,CAAC,CAAC;YACvH,KAAK,MAAM,CAAC,IAAI,MAAM;gBAAE,OAAO,CAAC,GAAG,CAAC,WAAW,CAAC,CAAC,IAAI,MAAM,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;YACrE,OAAO,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QAC/B,CAAC;KACF,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,eAAe,CACnC,CAAS,EACT,UAAkB;AAClB;;;;GAIG;AACH,uBAA0C,CAAC,OAAO,CAAC;IAEnD,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,CAAC,QAAQ,MAAM,CAAC,CAAC;IACnD,MAAM,IAAI,GAAG,MAAM,CAAC,CAAC,GAAG,CAAC,YAAY,EAAE;QACrC,eAAe,EAAE,YAAY;QAC7B,YAAY,EAAE,EAAE;QAChB,UAAU,EAAE,EAAE,IAAI,EAAE,UAAU,EAAE,OAAO,EAAE,GAAG,EAAE;KAC/C,CAAC,CAAC;IACH,CAAC,CAAC,KAAK,CAAC,YAAY,EAAE,CAAC,CAAC,IAAI,CAAC,MAAM,EAAE,UAAU,IAAI,CAAC,CAAC,IAAI,CAAC,MAAM,EAAE,YAAY,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,MAAM,EAAE,UAAU,IAAI,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;IAEvI,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,CAAC;IAEnC,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE,EAAE,MAAM,EAAE,SAAS,EAAE,CAAC,CAAC;IAC3D,CAAC,CAAC,KAAK,CACL,mBAAmB,EACnB,GAAG,CAAC,MAAM,KAAK,GAAG;QAChB,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,KAAK,GAAG;QACtD,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,8BAA8B,CAAC,IAAI,EAAE,CAAC,CAAC,QAAQ,CAAC,sBAAsB,CAAC,EAC1F,GAAG,GAAG,CAAC,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,EAAE,CACvE,CAAC;IAEF,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC;IACpC,CAAC,CAAC,KAAK,CACL,SAAS,EACT,GAAG,CAAC,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,KAAK,GAAG,EAC5E,GAAG,GAAG,CAAC,MAAM,UAAU,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,SAAS,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,6BAA6B,CAAC,EAAE,CACzG,CAAC;IAEF,6EAA6E;IAC7E,kDAAkD;IAClD,MAAM,KAAK,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE;QACpC,MAAM,EAAE,MAAM;QACd,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE;QAC/C,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,2BAA2B,EAAE,CAAC;KAC9E,CAAC,CAAC;IACH,MAAM,SAAS,GAAG,MAAM,KAAK,CAAC,IAAI,EAAE,CAAC;IACrC,CAAC,CAAC,KAAK,CAAC,kBAAkB,EAAE,KAAK,CAAC,MAAM,KAAK,GAAG,IAAI,SAAS,KAAK,EAAE,EAAE,GAAG,KAAK,CAAC,MAAM,UAAU,SAAS,GAAG,CAAC,CAAC;IAE7G,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC,CAAC;IAC3H,MAAM,OAAO,GAAG,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAAQ,CAAC;IAC1C,CAAC,CAAC,KAAK,CAAC,iBAAiB,EAAE,GAAG,CAAC,MAAM,KAAK,GAAG,IAAI,OAAO,CAAC,KAAK,EAAE,IAAI,KAAK,CAAC,KAAK,EAAE,GAAG,GAAG,CAAC,MAAM,SAAS,OAAO,CAAC,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;IAE9H,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,CAAC,CAAC,QAAQ,EAAE;QACnC,MAAM,EAAE,MAAM;QACd,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE;QAC/C,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC;KACxG,CAAC,CAAC;IACH,MAAM,QAAQ,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAQ,CAAC;IAC5C,CAAC,CAAC,KAAK,CAAC,gBAAgB,EAAE,QAAQ,CAAC,KAAK,EAAE,IAAI,KAAK,CAAC,KAAK,EAAE,QAAQ,QAAQ,CAAC,KAAK,EAAE,IAAI,KAAK,CAAC,QAAQ,CAAC,KAAK,EAAE,OAAO,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;IAE5I,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,EAAE,YAAY,IAAI,EAAE,CAAC;IAC7C,CAAC,CAAC,KAAK,CACL,qBAAqB,EACrB,oBAAoB,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,EAC1C,GAAG,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,oBAAoB,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CACxE,CAAC;IAEF,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,gBAAgB,CACpC,CAAS,EACT,QAAgB,EAChB,OAAwF,EACxF,WAAoB;IAEpB,MAAM,KAAK,GAA4B,EAAE,IAAI,EAAE,WAAW,EAAE,CAAC;IAC7D,KAAK,MAAM,CAAC,KAAK,EAAE,EAAE,GAAG,EAAE,EAAE,EAAE,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;QAC3D,IAAI,CAAC;YACH,KAAK,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,CAAC,MAAM,CAAC,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,CAA4B,CAAC,CAAC;QAClF,CAAC;QAAC,MAAM,CAAC;YACP,KAAK,CAAC,KAAK,CAAC,GAAG,eAAe,CAAC;QACjC,CAAC;IACH,CAAC;IACD,MAAM,GAAG,GAAG,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,KAAK,SAAS,CAAC,CAAC;IAC9D,CAAC,CAAC,KAAK,CACL,mBAAmB,EACnB,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,KAAK,QAAQ,CAAC,EAC9B,YAAY,QAAQ,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CACvF,CAAC;AACJ,CAAC;AAED;;;;;GAKG;AACH,MAAM,CAAC,KAAK,UAAU,eAAe,CAAC,CAAS;IAC7C,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC;QAC/B,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,8BAA8B,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QAClE,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,yBAAyB,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;KAC9D,CAAC,CAAC;IACH,CAAC,CAAC,KAAK,CAAC,mBAAmB,EAAE,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,KAAK,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,EAAE,gCAAgC,CAAC,CAAC;AAC1G,CAAC;AAED;;;GAGG;AACH,MAAM,CAAC,KAAK,UAAU,oBAAoB,CAAC,CAAS,EAAE,KAAwB;IAC5E,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,MAAM,KAAK,GAAG,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,IAAI,IAAI,EAAE,CAAC,CAAC;QAC/C,MAAM,IAAI,GAAG,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QACvC,yEAAyE;QACzE,4EAA4E;QAC5E,6EAA6E;QAC7E,wEAAwE;QACxE,yEAAyE;QACzE,gBAAgB;QAChB,MAAM,YAAY,GAAG,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC;QACxD,MAAM,WAAW,GAAG,IAAI,CAAC,CAAC,CAAC,MAAM,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,IAAI,IAAI,EAAE,EAAE,EAAE,OAAO,EAAE,EAAE,eAAe,EAAE,IAAI,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QAC3G,CAAC,CAAC,KAAK,CAAC,GAAG,IAAI,MAAM,EAAE,CAAC,CAAC,IAAI,IAAI,WAAW,EAAE,MAAM,KAAK,GAAG,EAAE,QAAQ,IAAI,eAAe,WAAW,EAAE,MAAM,EAAE,CAAC,CAAC;QAChH,CAAC,CAAC,KAAK,CAAC,GAAG,IAAI,gBAAgB,EAAE,CAAC,CAAC,YAAY,EAAE,GAAG,KAAK,CAAC,MAAM,IAAI,YAAY,EAAE,CAAC,CAAC;IACtF,CAAC;AACH,CAAC"}
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
export interface AiCrawler {
|
|
2
|
+
/** Matched as a substring of the User-Agent, and the robots.txt token. */
|
|
3
|
+
token: string;
|
|
4
|
+
/** Event label. Several tokens deliberately share one. */
|
|
5
|
+
label: string;
|
|
6
|
+
/**
|
|
7
|
+
* False for a robots.txt-only token that never issues a request. Such a token
|
|
8
|
+
* still needs its group — that is the entire point of it — but matching it in
|
|
9
|
+
* a proxy counts a visit that cannot happen.
|
|
10
|
+
*/
|
|
11
|
+
fetches: boolean;
|
|
12
|
+
}
|
|
13
|
+
/**
|
|
14
|
+
* Order is significant: `detectAiBot` returns the first token the user agent
|
|
15
|
+
* contains, so a token that is a substring of another must come after it.
|
|
16
|
+
*/
|
|
17
|
+
export declare const AI_CRAWLERS: readonly AiCrawler[];
|
|
18
|
+
/** The event label for a user agent, or null when it is not a known crawler. */
|
|
19
|
+
export declare function detectAiBot(userAgent: string | null | undefined): string | null;
|
|
20
|
+
/** Every token that needs its own robots.txt group — which is all of them. */
|
|
21
|
+
export declare function aiCrawlerTokens(): string[];
|
|
22
|
+
/**
|
|
23
|
+
* Structurally what Next's `MetadataRoute.Robots['rules']` wants, without
|
|
24
|
+
* importing Next: this package has no runtime dependencies and is not going to
|
|
25
|
+
* grow one for a shape that is three optional string fields.
|
|
26
|
+
*/
|
|
27
|
+
export interface RobotsGroup {
|
|
28
|
+
userAgent: string;
|
|
29
|
+
allow?: string | string[];
|
|
30
|
+
disallow?: string | string[];
|
|
31
|
+
}
|
|
32
|
+
/**
|
|
33
|
+
* The wildcard group followed by one group per crawler.
|
|
34
|
+
*
|
|
35
|
+
* Every named agent gets an explicit `disallow`. Omitting it is the failure the
|
|
36
|
+
* comment in all three `robots.ts` files warned about and all three then
|
|
37
|
+
* committed for five agents: a named group that disallows nothing grants that
|
|
38
|
+
* agent everything, because it stops matching `*` the moment it matches itself.
|
|
39
|
+
*/
|
|
40
|
+
export declare function aiCrawlerRules(opts: {
|
|
41
|
+
allow: string | string[];
|
|
42
|
+
disallow: string | string[];
|
|
43
|
+
aiAllow: string | string[];
|
|
44
|
+
}): RobotsGroup[];
|
|
45
|
+
//# sourceMappingURL=crawlers.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"crawlers.d.ts","sourceRoot":"","sources":["../src/crawlers.ts"],"names":[],"mappings":"AAoBA,MAAM,WAAW,SAAS;IACxB,0EAA0E;IAC1E,KAAK,EAAE,MAAM,CAAC;IACd,0DAA0D;IAC1D,KAAK,EAAE,MAAM,CAAC;IACd;;;;OAIG;IACH,OAAO,EAAE,OAAO,CAAC;CAClB;AAED;;;GAGG;AACH,eAAO,MAAM,WAAW,EAAE,SAAS,SAAS,EAuB3C,CAAC;AAEF,gFAAgF;AAChF,wBAAgB,WAAW,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,GAAG,MAAM,GAAG,IAAI,CAM/E;AAED,8EAA8E;AAC9E,wBAAgB,eAAe,IAAI,MAAM,EAAE,CAE1C;AAED;;;;GAIG;AACH,MAAM,WAAW,WAAW;IAC1B,SAAS,EAAE,MAAM,CAAC;IAClB,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IAC1B,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;CAC9B;AAED;;;;;;;GAOG;AACH,wBAAgB,cAAc,CAAC,IAAI,EAAE;IACnC,KAAK,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IACzB,QAAQ,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IAC5B,OAAO,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;CAC5B,GAAG,WAAW,EAAE,CAShB"}
|
package/dist/crawlers.js
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
// crawlers.ts — one roster of AI crawler tokens, for both surfaces that need it.
|
|
2
|
+
//
|
|
3
|
+
// Every wiki in the workspace kept this list twice: once in the proxy, keyed by
|
|
4
|
+
// user-agent substring, to count an "AI Bot Visit"; and once in `robots.ts`, as
|
|
5
|
+
// the set of agents that get their own group. The two copies were each
|
|
6
|
+
// byte-identical across all three repos — and they were different lists.
|
|
7
|
+
//
|
|
8
|
+
// Only one direction of that difference was deliberate. `Applebot-Extended`
|
|
9
|
+
// never fetches a page: it is a robots.txt-only token that Applebot consults
|
|
10
|
+
// before using already-crawled data for AI, so counting it would count nothing.
|
|
11
|
+
// That belongs in robots and not in the matcher, and it is declared here.
|
|
12
|
+
//
|
|
13
|
+
// The other direction was not deliberate. Bytespider, CCBot, cohere-ai,
|
|
14
|
+
// Claude-Web and Meta-ExternalFetcher were matched by every proxy and named by
|
|
15
|
+
// no robots.txt — and a crawler obeys only its most-specific matching group, so
|
|
16
|
+
// an agent with no group of its own falls through to `*` and is granted
|
|
17
|
+
// whatever that grants. The wikis were measuring five crawlers they had never
|
|
18
|
+
// addressed. One roster makes that a property of the data rather than of which
|
|
19
|
+
// file you happened to edit.
|
|
20
|
+
/**
|
|
21
|
+
* Order is significant: `detectAiBot` returns the first token the user agent
|
|
22
|
+
* contains, so a token that is a substring of another must come after it.
|
|
23
|
+
*/
|
|
24
|
+
export const AI_CRAWLERS = [
|
|
25
|
+
{ token: 'GPTBot', label: 'GPTBot', fetches: true },
|
|
26
|
+
{ token: 'ChatGPT-User', label: 'ChatGPT', fetches: true },
|
|
27
|
+
{ token: 'OAI-SearchBot', label: 'OAISearchBot', fetches: true },
|
|
28
|
+
{ token: 'ClaudeBot', label: 'ClaudeBot', fetches: true },
|
|
29
|
+
{ token: 'Claude-Web', label: 'ClaudeBot', fetches: true },
|
|
30
|
+
{ token: 'Claude-User', label: 'ClaudeUser', fetches: true },
|
|
31
|
+
{ token: 'Claude-SearchBot', label: 'ClaudeSearchBot', fetches: true },
|
|
32
|
+
{ token: 'PerplexityBot', label: 'PerplexityBot', fetches: true },
|
|
33
|
+
{ token: 'Perplexity-User', label: 'PerplexityUser', fetches: true },
|
|
34
|
+
{ token: 'Amazonbot', label: 'Amazonbot', fetches: true },
|
|
35
|
+
// Like Applebot-Extended, this is a preference token rather than a fetcher —
|
|
36
|
+
// Googlebot does the crawling. It is kept matchable because every proxy in the
|
|
37
|
+
// workspace already matched it, and a token that never arrives costs nothing.
|
|
38
|
+
{ token: 'Google-Extended', label: 'GoogleExtended', fetches: true },
|
|
39
|
+
{ token: 'Bytespider', label: 'Bytespider', fetches: true },
|
|
40
|
+
{ token: 'CCBot', label: 'CCBot', fetches: true },
|
|
41
|
+
{ token: 'cohere-ai', label: 'CohereBot', fetches: true },
|
|
42
|
+
{ token: 'Meta-ExternalAgent', label: 'MetaExternalAgent', fetches: true },
|
|
43
|
+
{ token: 'Meta-ExternalFetcher', label: 'MetaExternalFetcher', fetches: true },
|
|
44
|
+
{ token: 'MistralAI-User', label: 'MistralAI', fetches: true },
|
|
45
|
+
{ token: 'DuckAssistBot', label: 'DuckAssistBot', fetches: true },
|
|
46
|
+
{ token: 'Applebot-Extended', label: 'AppleExtended', fetches: false },
|
|
47
|
+
];
|
|
48
|
+
/** The event label for a user agent, or null when it is not a known crawler. */
|
|
49
|
+
export function detectAiBot(userAgent) {
|
|
50
|
+
if (!userAgent)
|
|
51
|
+
return null;
|
|
52
|
+
for (const crawler of AI_CRAWLERS) {
|
|
53
|
+
if (crawler.fetches && userAgent.includes(crawler.token))
|
|
54
|
+
return crawler.label;
|
|
55
|
+
}
|
|
56
|
+
return null;
|
|
57
|
+
}
|
|
58
|
+
/** Every token that needs its own robots.txt group — which is all of them. */
|
|
59
|
+
export function aiCrawlerTokens() {
|
|
60
|
+
return AI_CRAWLERS.map(c => c.token);
|
|
61
|
+
}
|
|
62
|
+
/**
|
|
63
|
+
* The wildcard group followed by one group per crawler.
|
|
64
|
+
*
|
|
65
|
+
* Every named agent gets an explicit `disallow`. Omitting it is the failure the
|
|
66
|
+
* comment in all three `robots.ts` files warned about and all three then
|
|
67
|
+
* committed for five agents: a named group that disallows nothing grants that
|
|
68
|
+
* agent everything, because it stops matching `*` the moment it matches itself.
|
|
69
|
+
*/
|
|
70
|
+
export function aiCrawlerRules(opts) {
|
|
71
|
+
return [
|
|
72
|
+
{ userAgent: '*', allow: opts.allow, disallow: opts.disallow },
|
|
73
|
+
...aiCrawlerTokens().map(userAgent => ({
|
|
74
|
+
userAgent,
|
|
75
|
+
allow: opts.aiAllow,
|
|
76
|
+
disallow: opts.disallow,
|
|
77
|
+
})),
|
|
78
|
+
];
|
|
79
|
+
}
|
|
80
|
+
//# sourceMappingURL=crawlers.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"crawlers.js","sourceRoot":"","sources":["../src/crawlers.ts"],"names":[],"mappings":"AAAA,iFAAiF;AACjF,EAAE;AACF,gFAAgF;AAChF,gFAAgF;AAChF,uEAAuE;AACvE,yEAAyE;AACzE,EAAE;AACF,4EAA4E;AAC5E,6EAA6E;AAC7E,gFAAgF;AAChF,0EAA0E;AAC1E,EAAE;AACF,wEAAwE;AACxE,+EAA+E;AAC/E,gFAAgF;AAChF,wEAAwE;AACxE,8EAA8E;AAC9E,+EAA+E;AAC/E,6BAA6B;AAe7B;;;GAGG;AACH,MAAM,CAAC,MAAM,WAAW,GAAyB;IAC/C,EAAE,KAAK,EAAE,QAAQ,EAAE,KAAK,EAAE,QAAQ,EAAE,OAAO,EAAE,IAAI,EAAE;IACnD,EAAE,KAAK,EAAE,cAAc,EAAE,KAAK,EAAE,SAAS,EAAE,OAAO,EAAE,IAAI,EAAE;IAC1D,EAAE,KAAK,EAAE,eAAe,EAAE,KAAK,EAAE,cAAc,EAAE,OAAO,EAAE,IAAI,EAAE;IAChE,EAAE,KAAK,EAAE,WAAW,EAAE,KAAK,EAAE,WAAW,EAAE,OAAO,EAAE,IAAI,EAAE;IACzD,EAAE,KAAK,EAAE,YAAY,EAAE,KAAK,EAAE,WAAW,EAAE,OAAO,EAAE,IAAI,EAAE;IAC1D,EAAE,KAAK,EAAE,aAAa,EAAE,KAAK,EAAE,YAAY,EAAE,OAAO,EAAE,IAAI,EAAE;IAC5D,EAAE,KAAK,EAAE,kBAAkB,EAAE,KAAK,EAAE,iBAAiB,EAAE,OAAO,EAAE,IAAI,EAAE;IACtE,EAAE,KAAK,EAAE,eAAe,EAAE,KAAK,EAAE,eAAe,EAAE,OAAO,EAAE,IAAI,EAAE;IACjE,EAAE,KAAK,EAAE,iBAAiB,EAAE,KAAK,EAAE,gBAAgB,EAAE,OAAO,EAAE,IAAI,EAAE;IACpE,EAAE,KAAK,EAAE,WAAW,EAAE,KAAK,EAAE,WAAW,EAAE,OAAO,EAAE,IAAI,EAAE;IACzD,6EAA6E;IAC7E,+EAA+E;IAC/E,8EAA8E;IAC9E,EAAE,KAAK,EAAE,iBAAiB,EAAE,KAAK,EAAE,gBAAgB,EAAE,OAAO,EAAE,IAAI,EAAE;IACpE,EAAE,KAAK,EAAE,YAAY,EAAE,KAAK,EAAE,YAAY,EAAE,OAAO,EAAE,IAAI,EAAE;IAC3D,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE;IACjD,EAAE,KAAK,EAAE,WAAW,EAAE,KAAK,EAAE,WAAW,EAAE,OAAO,EAAE,IAAI,EAAE;IACzD,EAAE,KAAK,EAAE,oBAAoB,EAAE,KAAK,EAAE,mBAAmB,EAAE,OAAO,EAAE,IAAI,EAAE;IAC1E,EAAE,KAAK,EAAE,sBAAsB,EAAE,KAAK,EAAE,qBAAqB,EAAE,OAAO,EAAE,IAAI,EAAE;IAC9E,EAAE,KAAK,EAAE,gBAAgB,EAAE,KAAK,EAAE,WAAW,EAAE,OAAO,EAAE,IAAI,EAAE;IAC9D,EAAE,KAAK,EAAE,eAAe,EAAE,KAAK,EAAE,eAAe,EAAE,OAAO,EAAE,IAAI,EAAE;IACjE,EAAE,KAAK,EAAE,mBAAmB,EAAE,KAAK,EAAE,eAAe,EAAE,OAAO,EAAE,KAAK,EAAE;CACvE,CAAC;AAEF,gFAAgF;AAChF,MAAM,UAAU,WAAW,CAAC,SAAoC;IAC9D,IAAI,CAAC,SAAS;QAAE,OAAO,IAAI,CAAC;IAC5B,KAAK,MAAM,OAAO,IAAI,WAAW,EAAE,CAAC;QAClC,IAAI,OAAO,CAAC,OAAO,IAAI,SAAS,CAAC,QAAQ,CAAC,OAAO,CAAC,KAAK,CAAC;YAAE,OAAO,OAAO,CAAC,KAAK,CAAC;IACjF,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC;AAED,8EAA8E;AAC9E,MAAM,UAAU,eAAe;IAC7B,OAAO,WAAW,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;AACvC,CAAC;AAaD;;;;;;;GAOG;AACH,MAAM,UAAU,cAAc,CAAC,IAI9B;IACC,OAAO;QACL,EAAE,SAAS,EAAE,GAAG,EAAE,KAAK,EAAE,IAAI,CAAC,KAAK,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE;QAC9D,GAAG,eAAe,EAAE,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;YACrC,SAAS;YACT,KAAK,EAAE,IAAI,CAAC,OAAO;YACnB,QAAQ,EAAE,IAAI,CAAC,QAAQ;SACxB,CAAC,CAAC;KACJ,CAAC;AACJ,CAAC"}
|