wiki-formant 0.52.0 → 0.53.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -268,7 +268,7 @@ Four configs had restated these by hand on the grounds that a config could not i
268
268
  `withWikiDefaults(config, { csp })` wraps the whole config: the headers on every route with `csp` enforced in production, AVIF before WebP from the image optimiser, Turbopack's build cache off (Vercel restored a stale one and shipped a stylesheet two commits old), and `distDir` from `STUDIO_DIST_DIR` for radix-studio's second dev server. What the app sets wins, and its own `headers()` follow these.
269
269
 
270
270
  ```ts
271
- export default withWikiDefaults({ async redirects() { /* … */ } }, { csp });
271
+ export default withWikiDefaults<NextConfig>({ async redirects() { /* … */ } }, { csp });
272
272
  ```
273
273
 
274
274
  `contentSecurityPolicy` starts from the policy all four origins shared line for line, and each directive passed replaces its default rather than appending, so the call site reads as the whole list. An empty list drops a directive.
@@ -392,7 +392,7 @@ different lists.
392
392
  ```ts
393
393
  import { trackAiBot, crawlerRules } from 'wiki-formant/crawlers';
394
394
 
395
- trackAiBot(request, event, () => import('@/lib/track')); // in proxy.ts: label, or null
395
+ trackAiBot(request, event, () => import('@/lib/track')); // in proxy.ts: whether it counted
396
396
  const rules = crawlerRules({ allow: '/', disallow, aiAllow });
397
397
  ```
398
398
 
@@ -408,7 +408,7 @@ Every agent group also allows the agent surface — `/api/mcp`, the three `llms`
408
408
 
409
409
  `crawlerRules` returns every group robots.txt needs: the wildcard, one per AI crawler, and one for each search engine. Search engines get the HTML site and none of its twins — `/*.md$` always, plus `searchDisallow` — because the `*` group keeps the machine surface open to agents, and for Google a twin is a page it already has. Two origins wrote that group by hand and the third never did, so Google crawled its twins. The `<Link>` prefetch payload (`?_rsc=`) is refused in every group.
410
410
 
411
- `trackAiBot(request, event, load)` counts an `AI Bot Visit` inside `event.waitUntil`. `load` is a dynamic import because the tracker pulls in a database client, which a static import would load for every request the proxy sees.
411
+ `trackAiBot(request, event, load)` counts an `AI Bot Visit` inside `event.waitUntil`: a crawler on the roster wherever it goes, and anything but a search engine that asks for what only agents read (a `.md` twin or `Accept: text/markdown`, the `llms` exports, `/openapi.json`, `/.well-known/`), since most agents are on no roster. Its user agent is kept, so `digest` can name it. `load` is a dynamic import because the tracker pulls in a database client, which a static import would load for every request the proxy sees.
412
412
 
413
413
  ## IndexNow
414
414
 
@@ -778,10 +778,13 @@ and may be a batch; `trackSearch` normalises a search box's free text so
778
778
  the same question aggregates as one row, and drops an empty one. Pass the request's headers and the
779
779
  event joins whoever sent it, person or agent.
780
780
 
781
- `digest(sql, { days })` reads a site back as one JSON object: visitors, page
782
- views, bounce rate, visit length, sources (a link's `utm_source`, else the
783
- referring host), top and entry pages, countries, devices, visitors by day, MCP
784
- tool calls, and the searches that found nothing. A visit is a run of one
781
+ `digest(sql, { days })` reads a site back as one JSON object: visitors, agents,
782
+ page views, bounce rate, visit length, sources (a link's `utm_source`, else the
783
+ referring host), top and entry pages, countries, devices, visitors and agents by
784
+ day, the agents by name, MCP tool calls, and the searches that found nothing.
785
+ An agent is counted as a visitor is, one address and user agent a day, from its
786
+ `AI Bot Visit` and `MCP Call` events; it runs no page script, so it is never
787
+ among the visitors. `ownAgents` leaves the site's own out. A visit is a run of one
785
788
  visitor's views with no gap over 30 minutes. A multi-tenant app passes a `site`
786
789
  key to every call and a `site` resolver to `collect`.
787
790
 
@@ -831,8 +834,8 @@ headline figures, each with its change from the window before
831
834
  site's stylesheet to colour), one of them per day, every non-empty
832
835
  ranked list, and `?days=` links between windows, all time among them, pinned in
833
836
  the chart's frame as its own buttons are (`TimeChart`'s `controls`). Each
834
- figure is a button that charts it: visitors, page views, bounce rate, visit
835
- length, visitors with agents and clicks to X. `digest`'s `by_day` carries every
837
+ figure is a button that charts it: visitors, agents, page views, bounce rate,
838
+ visit length and clicks to X. `digest`'s `by_day` carries every
836
839
  one of those per UTC day, a visit on the day it began, so the days add up to the
837
840
  totals. A day with no visits has no bounce rate or visit length and is left out
838
841
  of those charts rather than drawn as zero. Everything but the figures and the
@@ -66,22 +66,28 @@ for (const f of cssFiles) {
66
66
  }
67
67
  }
68
68
 
69
- // Classes this project defines itself. Everything else the build emitted is a
70
- // Tailwind utility, which is what makes the composition count derivable rather
71
- // than a maintained prefix list. Read from rule preludes only, so a decimal in a
72
- // value or a class named in a comment is not taken for a selector.
73
- const globalsFile = walk(SRC_DIR, '.css').find(f => f.endsWith('globals.css'));
74
- const named = new Set();
75
- if (globalsFile) {
76
- const css = fs.readFileSync(globalsFile, 'utf8').replace(/\/\*[\s\S]*?\*\//g, '');
69
+ // Classes defined by this project's stylesheets or by wiki-formant's base.css.
70
+ // Everything else the build emitted is a Tailwind utility, which is what makes
71
+ // the composition count derivable rather than a maintained prefix list. Read
72
+ // from rule preludes only, so a decimal in a value or a class named in a
73
+ // comment is not taken for a selector.
74
+ const classesIn = file => {
75
+ const out = new Set();
76
+ const css = fs.readFileSync(file, 'utf8').replace(/\/\*[\s\S]*?\*\//g, '');
77
77
  for (const [, prelude] of css.matchAll(/([^{};]*)\{/g)) {
78
78
  if (prelude.trim().startsWith('@')) continue;
79
79
  for (const m of prelude.matchAll(/\.((?:\\.|[-\w])+)/g)) {
80
80
  const name = m[1].replace(/\\/g, '');
81
- if (/^[A-Za-z_-]/.test(name)) named.add(name);
81
+ if (/^[A-Za-z_-]/.test(name)) out.add(name);
82
82
  }
83
83
  }
84
- }
84
+ return out;
85
+ };
86
+ const ownCss = walk(SRC_DIR, '.css');
87
+ const globalsFile = ownCss.find(f => f.endsWith('globals.css'));
88
+ const own = new Set(ownCss.flatMap(f => [...classesIn(f)]));
89
+ const baseCss = new URL('../dist/base.css', import.meta.url);
90
+ const named = new Set([...own, ...(fs.existsSync(baseCss) ? classesIn(baseCss) : [])]);
85
91
 
86
92
  // Only bare literals: anything interpolated is beyond a static check.
87
93
  const dead = new Map();
@@ -120,20 +126,22 @@ if (compositions.length) {
120
126
  }
121
127
 
122
128
  // Every token in the JavaScript the build ships, server and client. A selector
123
- // is referenced when its name appears whole, or when a prefix of it ending in `-`
124
- // or `_` does, which is how `editorial-banner-${variant}` is written.
129
+ // is referenced when its name appears whole, when a prefix of it ending in `-`
130
+ // or `_` does, which is how `editorial-banner-${variant}` is written, or when it
131
+ // splits at a `-` or `_` into two shipped tokens, which is how `RailShell`'s
132
+ // `${prefix}--open` is written.
125
133
  const shipped = new Set();
126
134
  for (const f of [...walk(STATIC_DIR, '.js'), ...walk('.next/server', '.js')]) {
127
135
  for (const [tok] of fs.readFileSync(f, 'utf8').matchAll(/[\w-]+/g)) shipped.add(tok);
128
136
  }
129
137
  const referenced = name =>
130
138
  shipped.has(name) ||
131
- [...name].some((c, i) => i >= 3 && (c === '-' || c === '_') && shipped.has(name.slice(0, i + 1)));
132
- const unused = [...named].filter(n => !referenced(n)).sort();
139
+ [...name].some((c, i) => i >= 3 && (c === '-' || c === '_') && (shipped.has(name.slice(0, i + 1)) || (shipped.has(name.slice(0, i)) && shipped.has(name.slice(i)))));
140
+ const unused = [...own].filter(n => !referenced(n)).sort();
133
141
 
134
142
  if (unused.length) {
135
143
  const verb = CHECK_UNUSED ? 'must be deleted' : 'advisory';
136
- console.error(`check-classes: ${unused.length} selector(s) in ${globalsFile} referenced by nothing the build ships (${verb}):\n`);
144
+ console.error(`check-classes: ${unused.length} selector(s) in ${ownCss.join(', ')} referenced by nothing the build ships (${verb}):\n`);
137
145
  console.error(` ${unused.join(', ')}\n`);
138
146
  console.error(' Class names inside HTML stored in a database do not count, so check stored content before deleting one.\n');
139
147
  }
@@ -67,8 +67,9 @@ export interface DigestOptions {
67
67
  siblings?: string[];
68
68
  gaps?: Gap[];
69
69
  /**
70
- * MCP callers left out of `agent_tool_calls`, matched inside the user agent.
71
- * The default is the conformance suites (`<repo>-mcp-test`) and the studio's own tools.
70
+ * The site's own agents, left out of `agents`, `top_agents` and
71
+ * `agent_tool_calls`, matched inside the user agent. The default is the
72
+ * conformance suites (`<repo>-mcp-test`) and the studio's own tools.
72
73
  */
73
74
  ownAgents?: string[];
74
75
  /**
@@ -89,6 +90,7 @@ export interface DigestDay {
89
90
  bounces: number;
90
91
  /** Seconds from each visit's first view to its last, summed. */
91
92
  seconds: number;
93
+ agents: number;
92
94
  visitors_incl_agents: number;
93
95
  /** Clicks on links to the site's X handle; zero without a handle. */
94
96
  follow_clicks: number;
@@ -100,6 +102,13 @@ export interface Digest {
100
102
  bounce_rate: number | null;
101
103
  /** Mean seconds from a visit's first view to its last; null with no visits. */
102
104
  visit_duration: number | null;
105
+ /**
106
+ * Agents, counted as visitors are: an AI crawler, an MCP client, or anything
107
+ * else reading the machine surface (`trackAiBot`), one per address and user
108
+ * agent a day, the site's own left out. They run no page script, so they are
109
+ * never among `visitors`.
110
+ */
111
+ agents: number;
103
112
  /** Visitors plus every agent that sent an event, such as an MCP call. */
104
113
  visitors_incl_agents: number;
105
114
  /**
@@ -108,7 +117,7 @@ export interface Digest {
108
117
  * the archive cannot stand in for it: event figures always, all of them
109
118
  * when filtered.
110
119
  */
111
- previous: Record<'visitors' | 'pageviews' | 'bounce_rate' | 'visit_duration' | 'visitors_incl_agents' | 'follow_clicks', number | null>;
120
+ previous: Record<'visitors' | 'pageviews' | 'bounce_rate' | 'visit_duration' | 'agents' | 'visitors_incl_agents' | 'follow_clicks', number | null>;
112
121
  /** The first UTC day this site counted itself, before which only `ViewDay` speaks; null with no views. */
113
122
  counted_from: string | null;
114
123
  top_sources: Ranked<'source', 'visitors'>;
@@ -125,6 +134,8 @@ export interface Digest {
125
134
  events: number;
126
135
  visitors: number;
127
136
  } | null;
137
+ /** Each agent by name: the roster's label, else what its user agent calls itself. */
138
+ top_agents: Ranked<'agent', 'visitors'>;
128
139
  agent_tool_calls: Ranked<'tool', 'events'>;
129
140
  gaps: Ranked<'query', 'events'>;
130
141
  }
package/dist/analytics.js CHANGED
@@ -160,6 +160,11 @@ const VISIT_TEST = {
160
160
  country: v => `bool_or(country = ${v})`,
161
161
  device: v => `bool_or(device = ${v})`,
162
162
  };
163
+ /**
164
+ * An agent's name from its event: the roster's label, else what its user agent
165
+ * calls itself, `compatible; X` or the first product token.
166
+ */
167
+ const AGENT_NAME = `coalesce(props->>'bot', substring(props->>'ua' from 'compatible; ?([^/;) ]+)'), nullif(split_part(split_part(props->>'ua', '/', 1), ' ', 1), ''), '(unknown)')`;
163
168
  /**
164
169
  * Everything a report needs about one site's last `days` days, in one query.
165
170
  * Visitor figures count distinct visitors, so a person who comes back on
@@ -181,6 +186,8 @@ export async function digest(sql, { site = '', days, handle, siblings = [], gaps
181
186
  const follows = handle && `name = 'Outbound Link: Click' AND props->>'url' ILIKE ANY(${$([`%x.com/${handle}%`, `%twitter.com/${handle}%`])}::text[])`;
182
187
  const tests = STATS_FILTERS.flatMap(k => (filter[k] ? [VISIT_TEST[k]($(filter[k]))] : []));
183
188
  const day = (at) => `to_char(${at} AT TIME ZONE 'UTC', 'YYYY-MM-DD')`;
189
+ /** The site's own agents, as a test on an event's user agent. */
190
+ const ours = `coalesce(props->>'ua', '') ILIKE ANY(${$(ownAgents.map(a => `%${a}%`))}::text[])`;
184
191
  const gapSelects = gaps.map(g => {
185
192
  const prop = $(g.prop);
186
193
  return `(SELECT props->>${prop} AS query, count(*) AS events FROM e
@@ -210,6 +217,9 @@ export async function digest(sql, { site = '', days, handle, siblings = [], gaps
210
217
  ), e${s} AS (
211
218
  SELECT name, props, visitor, "createdAt" AS at FROM "Event" WHERE "siteId" = $1 AND ${at}
212
219
  ${tests.length ? `AND visitor IN (SELECT visitor FROM v${s})` : ''}
220
+ ), ag${s} AS (
221
+ SELECT visitor, at, ${AGENT_NAME} AS agent FROM e${s}
222
+ WHERE name IN ('AI Bot Visit', 'MCP Call') AND visitor IS NOT NULL AND NOT ${ours}
213
223
  ), a${s} AS (
214
224
  SELECT * FROM "ViewDay" WHERE "siteId" = $1 AND day >= ${dayAgo(back + 1)}${back ? ` AND day < ${dayAgo(back)}` : ''}${tests.length ? ' AND false' : ''}
215
225
  ), sessions${s} AS (
@@ -226,6 +236,7 @@ export async function digest(sql, { site = '', days, handle, siblings = [], gaps
226
236
  'pageviews', (SELECT count(*) FROM v${s}) + (SELECT pageviews FROM archived${s}),
227
237
  'bounce_rate', (SELECT round(100.0 * bounces / nullif(visits, 0)) FROM sessions${s}),
228
238
  'visit_duration', (SELECT round(seconds / nullif(visits, 0)) FROM sessions${s}),
239
+ 'agents', (SELECT count(DISTINCT visitor) FROM ag${s}),
229
240
  'visitors_incl_agents', (SELECT count(DISTINCT visitor) FROM (SELECT visitor FROM v${s} UNION SELECT visitor FROM e${s}) u) + (SELECT visitors FROM archived${s})`;
230
241
  const [row] = (await sql(`WITH ${chain('', 0)}, ${chain('_p', 1)}, src AS (
231
242
  SELECT coalesce(source, '(none)') AS source, count(DISTINCT visitor) AS visitors FROM visits GROUP BY 1
@@ -243,17 +254,18 @@ export async function digest(sql, { site = '', days, handle, siblings = [], gaps
243
254
  'countries', ${list(merged('country', 'countries', 'SELECT country, count(DISTINCT visitor) FROM v WHERE country IS NOT NULL GROUP BY 1'), 10)},
244
255
  'devices', ${list(merged('device', 'devices', 'SELECT device, count(DISTINCT visitor) FROM v GROUP BY 1'))},
245
256
  'by_day', ${list(`SELECT date, sum(visitors) AS visitors, sum(pageviews) AS pageviews, sum(visits) AS visits, sum(bounces) AS bounces,
246
- sum(seconds) AS seconds, sum(agents) AS visitors_incl_agents, sum(follows) AS follow_clicks FROM (
247
- SELECT ${day('at')}, count(DISTINCT visitor), count(*), 0, 0, 0, 0, 0 FROM v GROUP BY 1
248
- UNION ALL SELECT ${day('at')}, 0, 0, count(*), count(*) FILTER (WHERE views = 1), coalesce(sum(seconds), 0), 0, 0 FROM visits GROUP BY 1
249
- UNION ALL SELECT ${day('at')}, 0, 0, 0, 0, 0, count(DISTINCT visitor), 0 FROM (SELECT at, visitor FROM v UNION ALL SELECT at, visitor FROM e) u GROUP BY 1
250
- ${follows ? `UNION ALL SELECT ${day('at')}, 0, 0, 0, 0, 0, 0, count(*) FROM e WHERE ${follows} GROUP BY 1` : ''}
251
- UNION ALL SELECT to_char(day, 'YYYY-MM-DD'), visitors, pageviews, visits, bounces, seconds, visitors, 0 FROM a
252
- ) t(date, visitors, pageviews, visits, bounces, seconds, agents, follows) GROUP BY 1`, undefined, '1')},
257
+ sum(seconds) AS seconds, sum(agents) AS agents, sum(everyone) AS visitors_incl_agents, sum(follows) AS follow_clicks FROM (
258
+ SELECT ${day('at')}, count(DISTINCT visitor), count(*), 0, 0, 0, 0, 0, 0 FROM v GROUP BY 1
259
+ UNION ALL SELECT ${day('at')}, 0, 0, count(*), count(*) FILTER (WHERE views = 1), coalesce(sum(seconds), 0), 0, 0, 0 FROM visits GROUP BY 1
260
+ UNION ALL SELECT ${day('at')}, 0, 0, 0, 0, 0, count(DISTINCT visitor), 0, 0 FROM ag GROUP BY 1
261
+ UNION ALL SELECT ${day('at')}, 0, 0, 0, 0, 0, 0, count(DISTINCT visitor), 0 FROM (SELECT at, visitor FROM v UNION ALL SELECT at, visitor FROM e) u GROUP BY 1
262
+ ${follows ? `UNION ALL SELECT ${day('at')}, 0, 0, 0, 0, 0, 0, 0, count(*) FROM e WHERE ${follows} GROUP BY 1` : ''}
263
+ UNION ALL SELECT to_char(day, 'YYYY-MM-DD'), visitors, pageviews, visits, bounces, seconds, 0, visitors, 0 FROM a
264
+ ) t(date, visitors, pageviews, visits, bounces, seconds, agents, everyone, follows) GROUP BY 1`, undefined, '1')},
253
265
  'follow_clicks', ${follows ? `(SELECT json_build_object('events', count(*), 'visitors', count(DISTINCT visitor)) FROM e WHERE ${follows})` : 'NULL'},
266
+ 'top_agents', ${list('SELECT agent, count(DISTINCT visitor) AS visitors FROM ag GROUP BY 1', 10)},
254
267
  'agent_tool_calls', ${list(`SELECT coalesce(props->>'tool', '(none)') AS tool, count(*) AS events FROM e
255
- WHERE name = 'MCP Call' AND props->>'method' = 'tools/call'
256
- AND NOT coalesce(props->>'ua', '') ILIKE ANY(${$(ownAgents.map(a => `%${a}%`))}::text[])
268
+ WHERE name = 'MCP Call' AND props->>'method' = 'tools/call' AND NOT ${ours}
257
269
  GROUP BY 1`, 10)},
258
270
  'gaps', ${gapSelects.length ? `(SELECT coalesce(json_agg(t), '[]') FROM (${gapSelects.join(' UNION ALL ')}) t)` : `'[]'::json`}
259
271
  ) AS digest`, ...values));
@@ -267,7 +279,7 @@ export async function digest(sql, { site = '', days, handle, siblings = [], gaps
267
279
  for (const k of Object.keys(p))
268
280
  p[k] = null;
269
281
  else
270
- p.visitors_incl_agents = p.follow_clicks = null;
282
+ p.agents = p.visitors_incl_agents = p.follow_clicks = null;
271
283
  }
272
284
  return d;
273
285
  }
@@ -46,16 +46,20 @@ export declare function crawlerRules(opts: {
46
46
  searchDisallow?: string | string[];
47
47
  }): RobotsGroup[];
48
48
  /**
49
- * Count an AI crawler's visit from a proxy, without holding up its response.
49
+ * Count an agent's visit from a proxy, without holding up its response: a
50
+ * crawler on the roster wherever it goes, and anything but a search engine
51
+ * that asks for what only agents read, since most agents are on no roster.
52
+ * The user agent is kept beside the label, so `digest` can name an agent the
53
+ * roster does not know and leave out the site's own.
50
54
  *
51
55
  * `load` is a dynamic import of the module that records it, because that
52
56
  * module pulls in a database client, and a static import would load it for
53
57
  * every request the proxy sees, bots or not. Three proxies wrote these lines.
54
- * Returns the bot's label, or null.
58
+ * Returns whether the request was counted.
55
59
  */
56
60
  export declare function trackAiBot(request: {
57
61
  url: string;
58
62
  headers: HeaderReader;
59
63
  }, event: {
60
64
  waitUntil(promise: Promise<unknown>): void;
61
- }, load: () => Promise<Pick<Tracker, 'trackEvent'>>): string | null;
65
+ }, load: () => Promise<Pick<Tracker, 'trackEvent'>>): boolean;
package/dist/crawlers.js CHANGED
@@ -1,7 +1,8 @@
1
1
  // crawlers.ts — one roster of AI crawler tokens, for both surfaces that need it.
2
2
  //
3
3
  // The proxy counts an "AI Bot Visit" by user-agent substring, and `robots.ts`
4
- // gives each agent its own group. Both read this one roster.
4
+ // gives each agent its own group. Both read this one roster. The proxy also
5
+ // counts any agent reading the machine surface, roster or not.
5
6
  //
6
7
  // One difference between the two uses is deliberate. `Applebot-Extended`
7
8
  // never fetches a page: it is a robots.txt-only token that Applebot consults
@@ -11,6 +12,7 @@
11
12
  // Every other token appears in both, because a crawler obeys only its
12
13
  // most-specific matching group: an agent with no group of its own falls through
13
14
  // to `*` and is granted whatever that grants.
15
+ import { wantsMarkdown } from './http.js';
14
16
  /**
15
17
  * Order is significant: `detectAiBot` returns the first token the user agent
16
18
  * contains, so a token that is a substring of another must come after it.
@@ -113,16 +115,30 @@ export function crawlerRules(opts) {
113
115
  ];
114
116
  }
115
117
  /**
116
- * Count an AI crawler's visit from a proxy, without holding up its response.
118
+ * What only an agent asks for: a markdown twin, an `llms` export, the OpenAPI
119
+ * document, a well-known descriptor. MCP is left out, since "MCP Call" counts it.
120
+ */
121
+ const AGENT_ONLY = /\.md$|^\/llms[\w-]*\.txt$|^\/openapi\.json$|^\/\.well-known\//;
122
+ /**
123
+ * Count an agent's visit from a proxy, without holding up its response: a
124
+ * crawler on the roster wherever it goes, and anything but a search engine
125
+ * that asks for what only agents read, since most agents are on no roster.
126
+ * The user agent is kept beside the label, so `digest` can name an agent the
127
+ * roster does not know and leave out the site's own.
117
128
  *
118
129
  * `load` is a dynamic import of the module that records it, because that
119
130
  * module pulls in a database client, and a static import would load it for
120
131
  * every request the proxy sees, bots or not. Three proxies wrote these lines.
121
- * Returns the bot's label, or null.
132
+ * Returns whether the request was counted.
122
133
  */
123
134
  export function trackAiBot(request, event, load) {
124
- const bot = detectAiBot(request.headers.get('user-agent'));
125
- if (bot)
126
- event.waitUntil(load().then(m => m.trackEvent('AI Bot Visit', request.url, { bot }, request.headers)));
127
- return bot;
135
+ const ua = request.headers.get('user-agent') ?? '';
136
+ const bot = detectAiBot(ua);
137
+ const agent = bot !== null ||
138
+ (!!ua && !SEARCH_ENGINES.some(s => ua.includes(s)) && (AGENT_ONLY.test(new URL(request.url).pathname) || wantsMarkdown(request)));
139
+ if (agent) {
140
+ const props = { ...(bot ? { bot } : {}), ua: ua.slice(0, 80) };
141
+ event.waitUntil(load().then(m => m.trackEvent('AI Bot Visit', request.url, props, request.headers)));
142
+ }
143
+ return agent;
128
144
  }
package/dist/headers.d.ts CHANGED
@@ -30,7 +30,7 @@ interface WikiNextConfig {
30
30
  distDir?: string;
31
31
  experimental?: object;
32
32
  images?: object;
33
- headers?: () => Promise<unknown[]>;
33
+ headers?: () => Promise<unknown[]> | unknown[];
34
34
  }
35
35
  /**
36
36
  * An app's Next config with the settings every app here wrote out the same way:
@@ -46,7 +46,8 @@ interface WikiNextConfig {
46
46
  * Next's dev runtime is inline and `eval`. An origin that sets its policy
47
47
  * per request (a nonce) passes none.
48
48
  *
49
- * Anything the app sets wins, and its own `headers()` follow these.
49
+ * Anything the app sets wins, and its own `headers()` follow these. Pass
50
+ * `NextConfig` as `T` so the object is typed as Next's own config.
50
51
  */
51
52
  export declare function withWikiDefaults<T extends WikiNextConfig>(config: T, { csp }?: {
52
53
  csp?: string;
package/dist/headers.js CHANGED
@@ -93,7 +93,8 @@ export function securityHeaders(csp) {
93
93
  * Next's dev runtime is inline and `eval`. An origin that sets its policy
94
94
  * per request (a nonce) passes none.
95
95
  *
96
- * Anything the app sets wins, and its own `headers()` follow these.
96
+ * Anything the app sets wins, and its own `headers()` follow these. Pass
97
+ * `NextConfig` as `T` so the object is typed as Next's own config.
97
98
  */
98
99
  export function withWikiDefaults(config, { csp } = {}) {
99
100
  return {
package/dist/http.d.ts CHANGED
@@ -74,7 +74,7 @@ export declare function markdownHeaders(lastModified?: string | null, opts?: {
74
74
  export declare function wantsMarkdown(request: {
75
75
  url: string;
76
76
  headers: {
77
- get(name: string): string | null;
77
+ get(name: string): string | null | undefined;
78
78
  };
79
79
  }): boolean;
80
80
  /** The header every response from a content-negotiated URL carries. */
@@ -24,6 +24,7 @@ function change(now, before, { points = false, lowerIsBetter = false } = {}) {
24
24
  }
25
25
  const figuresOf = ({ previous: p, ...d }) => [
26
26
  { label: 'Visitors', value: count(d.visitors), delta: change(d.visitors, p.visitors), day: r => r.visitors },
27
+ { label: 'Agents', value: count(d.agents), delta: change(d.agents, p.agents), day: r => r.agents },
27
28
  { label: 'Page views', value: count(d.pageviews), delta: change(d.pageviews, p.pageviews), day: r => r.pageviews },
28
29
  {
29
30
  label: 'Bounce rate',
@@ -39,12 +40,6 @@ const figuresOf = ({ previous: p, ...d }) => [
39
40
  day: r => (r.visits ? r.seconds / r.visits : null),
40
41
  format: duration,
41
42
  },
42
- {
43
- label: 'With agents',
44
- value: count(d.visitors_incl_agents),
45
- delta: change(d.visitors_incl_agents, p.visitors_incl_agents),
46
- day: r => r.visitors_incl_agents,
47
- },
48
43
  ...(d.follow_clicks
49
44
  ? [
50
45
  {
@@ -57,7 +52,7 @@ const figuresOf = ({ previous: p, ...d }) => [
57
52
  : []),
58
53
  ];
59
54
  const DAY = 86_400;
60
- const EMPTY = { date: '', visitors: 0, pageviews: 0, visits: 0, bounces: 0, seconds: 0, visitors_incl_agents: 0, follow_clicks: 0 };
55
+ const EMPTY = { date: '', visitors: 0, pageviews: 0, visits: 0, bounces: 0, seconds: 0, agents: 0, visitors_incl_agents: 0, follow_clicks: 0 };
61
56
  /**
62
57
  * The figures as `.stats-figures button[aria-pressed]`, each with its change
63
58
  * as `.stats-delta[data-better]` for a stylesheet to colour, and the pressed one per
package/dist/stats.js CHANGED
@@ -73,7 +73,7 @@ export function Stats({ digest: d, days, filter = {}, ranges = STATS_RANGES }) {
73
73
  // Each window is another digest, so a link rather than a button, pinned
74
74
  // with the chart's own controls.
75
75
  const windows = (_jsx("div", { role: "group", "aria-label": "Window", children: ranges.map(n => (_jsx("a", { href: href(n, filter), "aria-current": n === days ? 'page' : undefined, children: windowLabel(n) }, n))) }));
76
- return (_jsxs("div", { className: "stats", children: [narrowed.length > 0 && (_jsxs("div", { className: "stats-filters", children: [_jsx("ul", { "aria-label": "Narrowed to", children: narrowed.map(k => (_jsx("li", { children: _jsxs("a", { href: href(days, { ...filter, [k]: undefined }), title: "Remove", children: [NAMES[k], ": ", k === 'country' ? country(filter[k]) : filter[k], " \u00D7"] }) }, k))) }), start < counted && d.counted_from && (_jsxs("p", { children: ["Narrowed figures start on ", d.counted_from, ", the first day this site counted its own visits."] }))] })), _jsx(StatsFigures, { digest: d, first: first, windows: windows }), _jsxs("div", { className: "stats-lists", children: [_jsx(StatsList, { title: "Pages", rows: rows(d.top_pages, 'page', 'visitors'), to: to('page') }), _jsx(StatsList, { title: "Entry pages", rows: rows(d.entry_pages, 'page', 'visitors'), to: to('entry') }), _jsx(StatsList, { title: "Sources", rows: rows(d.top_sources, 'source', 'visitors'), to: to('source') }), _jsx(StatsList, { title: "From X posts", rows: rows(d.from_posts, 'page', 'visitors'), to: v => href(days, { ...filter, source: 'x', entry: v }) }), _jsx(StatsList, { title: "Sibling sites", rows: rows(d.sibling_referrals, 'source', 'visitors'), to: to('source') }), _jsx(StatsList, { title: "Countries", rows: rows(d.countries, 'country', 'visitors'), label: country, to: to('country') }), _jsx(StatsList, { title: "Devices", rows: rows(d.devices, 'device', 'visitors'), to: to('device') }), _jsx(StatsList, { title: "Agent tool calls", rows: rows(d.agent_tool_calls, 'tool', 'events') }), _jsx(StatsList, { title: "Searches that found nothing", rows: rows(d.gaps, 'query', 'events') })] })] }));
76
+ return (_jsxs("div", { className: "stats", children: [narrowed.length > 0 && (_jsxs("div", { className: "stats-filters", children: [_jsx("ul", { "aria-label": "Narrowed to", children: narrowed.map(k => (_jsx("li", { children: _jsxs("a", { href: href(days, { ...filter, [k]: undefined }), title: "Remove", children: [NAMES[k], ": ", k === 'country' ? country(filter[k]) : filter[k], " \u00D7"] }) }, k))) }), start < counted && d.counted_from && (_jsxs("p", { children: ["Narrowed figures start on ", d.counted_from, ", the first day this site counted its own visits."] }))] })), _jsx(StatsFigures, { digest: d, first: first, windows: windows }), _jsxs("div", { className: "stats-lists", children: [_jsx(StatsList, { title: "Pages", rows: rows(d.top_pages, 'page', 'visitors'), to: to('page') }), _jsx(StatsList, { title: "Entry pages", rows: rows(d.entry_pages, 'page', 'visitors'), to: to('entry') }), _jsx(StatsList, { title: "Sources", rows: rows(d.top_sources, 'source', 'visitors'), to: to('source') }), _jsx(StatsList, { title: "From X posts", rows: rows(d.from_posts, 'page', 'visitors'), to: v => href(days, { ...filter, source: 'x', entry: v }) }), _jsx(StatsList, { title: "Sibling sites", rows: rows(d.sibling_referrals, 'source', 'visitors'), to: to('source') }), _jsx(StatsList, { title: "Countries", rows: rows(d.countries, 'country', 'visitors'), label: country, to: to('country') }), _jsx(StatsList, { title: "Devices", rows: rows(d.devices, 'device', 'visitors'), to: to('device') }), _jsx(StatsList, { title: "Agents", rows: rows(d.top_agents, 'agent', 'visitors') }), _jsx(StatsList, { title: "Agent tool calls", rows: rows(d.agent_tool_calls, 'tool', 'events') }), _jsx(StatsList, { title: "Searches that found nothing", rows: rows(d.gaps, 'query', 'events') })] })] }));
77
77
  }
78
78
  /**
79
79
  * A stats page's door: `children` for a signed-in reader, else a line saying
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "wiki-formant",
3
- "version": "0.52.0",
3
+ "version": "0.53.0",
4
4
  "description": "The portable half of a wiki: derived taxonomy and facet controls, a version-negotiating MCP transport, markdown twins, block rendering, a rich-text editor engine, conditional-GET plumbing, and the headers and CSP every origin sends.",
5
5
  "type": "module",
6
6
  "license": "MIT",