wiki-formant 0.99.0 → 0.104.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -2
- package/dist/analytics.d.ts +15 -4
- package/dist/analytics.js +79 -21
- package/dist/chart.js +7 -4
- package/dist/react-server.d.ts +2 -0
- package/dist/react-server.js +1 -1
- package/dist/search.d.ts +19 -0
- package/dist/search.js +59 -0
- package/dist/stats-figures.js +14 -3
- package/dist/stats.d.ts +5 -3
- package/dist/stats.js +9 -6
- package/dist/store.d.ts +10 -6
- package/dist/store.js +23 -3
- package/dist/wiki-tools.d.ts +1 -0
- package/dist/wiki-tools.js +5 -4
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -8,6 +8,8 @@ The editor, history, sanitiser, chart and passkey modules bring their libraries
|
|
|
8
8
|
npm install wiki-formant
|
|
9
9
|
```
|
|
10
10
|
|
|
11
|
+
An agent can start a wiki for you, hosted on miow.me or in your own app, with the skill in `skills/wiki-formant`: `npx skills add tutmoses/wiki-formant`.
|
|
12
|
+
|
|
11
13
|
What stays the app's: its design system, its block types beyond the core, its connection, cache and auth. Everything else here is shared.
|
|
12
14
|
|
|
13
15
|
## API
|
|
@@ -563,7 +565,7 @@ const off = onTweetResize((h, source) => sizeTweetEmbed(el, h, source));
|
|
|
563
565
|
|
|
564
566
|
## Search
|
|
565
567
|
|
|
566
|
-
`wiki-formant/search`: what a literal tier and a full-text tier must agree on: `proseSql`, `HEADLINE_OPTIONS`, `FTS_RANK_NORMALIZATION`, and `searchTsvDdl(table)`, the statements that drop and re-add the generated `search_tsv` column and its index in one transaction.
|
|
568
|
+
`wiki-formant/search`: what a literal tier and a full-text tier must agree on: `proseSql`, `HEADLINE_OPTIONS`, `FTS_RANK_NORMALIZATION`, and `searchTsvDdl(table)`, the statements that drop and re-add the generated `search_tsv` column and its index in one transaction. `fullTextQuery` is a query as the full-text tier reads it, a hyphen or underscore inside a word a space; `correctQuery(query, vocabulary)` corrects a query's misspelt words against a site's own, which the store's `search` retries with when its first page is empty, returning the query that found the rows as `corrected`.
|
|
567
569
|
|
|
568
570
|
```ts
|
|
569
571
|
const { column, index } = searchTsvDdl('"WikiPage"');
|
|
@@ -713,7 +715,7 @@ export const POST = viewRoute();
|
|
|
713
715
|
|
|
714
716
|
- `createTracker` records only on a Vercel production deployment unless `enabled` says otherwise. `trackMcpCall` fits `McpServerConfig.onCall`; `trackSearch` normalises a query and drops an empty one; `recordEvent` is the undeferred write.
|
|
715
717
|
- A proxy reaches the tracker through a dynamic `import()` inside `event.waitUntil`, since it imports the database client.
|
|
716
|
-
- `digest(sql, { days })` is the site as one JSON object: visitors, agents, views, bounce rate, visit length, sources, pages, countries, devices, by day, agents by name, MCP calls and searches that found nothing. A visit is a run of views with no gap over 30 minutes; a bounce is a one-view visit with no event of the reader's. `ownAgents` leaves the site's own out. A multi-tenant app passes `site`.
|
|
718
|
+
- `digest(sql, { days })` is the site as one JSON object: visitors, agents, views, visits, bounce rate, visit length, sources, pages, countries, devices, by day, agents by name, MCP calls and searches that found nothing, less a search its visitor went on typing within five minutes and any of one character. A visit is a run of views with no gap over 30 minutes; a bounce is a one-view visit with no event of the reader's. `ownAgents` leaves the site's own out. A multi-tenant app passes `site`.
|
|
717
719
|
- A browser under automation (`navigator.webdriver`) sends no beacon.
|
|
718
720
|
|
|
719
721
|
## Passkeys and the stats page
|
package/dist/analytics.d.ts
CHANGED
|
@@ -78,10 +78,18 @@ export interface DigestOptions {
|
|
|
78
78
|
ownAgents?: string[];
|
|
79
79
|
/**
|
|
80
80
|
* Narrows every figure to the visits that pass, and the events to their
|
|
81
|
-
* visitors
|
|
82
|
-
* and starts at `counted_from`,
|
|
81
|
+
* visitors; an agent passes as `AGENT_TEST` has it. `ViewDay` keeps no
|
|
82
|
+
* visits, so a narrowed digest leaves it out and starts at `counted_from`,
|
|
83
|
+
* and has no `previous_visitors`.
|
|
83
84
|
*/
|
|
84
85
|
filter?: StatsFilter;
|
|
86
|
+
/**
|
|
87
|
+
* People and agents counted apart, as they are by default: the ranked lists
|
|
88
|
+
* count people. False counts agents in the pages, entry pages, sources,
|
|
89
|
+
* countries and devices too, as `visitors_incl_agents` counts them, so a
|
|
90
|
+
* stats page's lists add up to the visitors its chart draws.
|
|
91
|
+
*/
|
|
92
|
+
apart?: boolean;
|
|
85
93
|
}
|
|
86
94
|
type Ranked<K extends string, M extends string> = Array<Record<K, string> & Record<M, number>>;
|
|
87
95
|
/** One UTC day of the figures a digest totals; a visit falls on the day it began. */
|
|
@@ -102,6 +110,8 @@ export interface DigestDay {
|
|
|
102
110
|
export interface Digest {
|
|
103
111
|
visitors: number;
|
|
104
112
|
pageviews: number;
|
|
113
|
+
/** Runs of one visitor's views with no gap over 30 minutes. */
|
|
114
|
+
visits: number;
|
|
105
115
|
/** Percent of visits that are bounces (`DigestDay.bounces`); null with no visits. */
|
|
106
116
|
bounce_rate: number | null;
|
|
107
117
|
/** Mean seconds from a visit's first view to its last; null with no visits. */
|
|
@@ -121,9 +131,10 @@ export interface Digest {
|
|
|
121
131
|
* the archive cannot stand in for it: event figures always, all of them
|
|
122
132
|
* when filtered.
|
|
123
133
|
*/
|
|
124
|
-
previous: Record<'visitors' | 'pageviews' | 'bounce_rate' | 'visit_duration' | 'agents' | 'visitors_incl_agents' | 'follow_clicks', number | null>;
|
|
134
|
+
previous: Record<'visitors' | 'pageviews' | 'visits' | 'bounce_rate' | 'visit_duration' | 'agents' | 'visitors_incl_agents' | 'follow_clicks', number | null>;
|
|
125
135
|
/** The first UTC day this site counted itself, before which only `ViewDay` speaks; null with no views. */
|
|
126
136
|
counted_from: string | null;
|
|
137
|
+
/** `(bot)` stands for each referrer `BOT_VISITORS` takes for a crawler's. */
|
|
127
138
|
top_sources: Ranked<'source', 'visitors'>;
|
|
128
139
|
top_pages: Ranked<'page', 'visitors'>;
|
|
129
140
|
entry_pages: Ranked<'page', 'visitors'>;
|
|
@@ -160,7 +171,7 @@ export interface Digest {
|
|
|
160
171
|
* The window before is the same chain of views, visits and events a second
|
|
161
172
|
* time, `_p` on every name, so its figures are the headline's to the letter.
|
|
162
173
|
*/
|
|
163
|
-
export declare function digest(sql: Sql, { site, days, handle, siblings, gaps, ownAgents, filter }: DigestOptions): Promise<Digest>;
|
|
174
|
+
export declare function digest(sql: Sql, { site, days, handle, siblings, gaps, ownAgents, filter, apart }: DigestOptions): Promise<Digest>;
|
|
164
175
|
/**
|
|
165
176
|
* Props for an "MCP Call" event, read out of the JSON-RPC envelope.
|
|
166
177
|
*
|
package/dist/analytics.js
CHANGED
|
@@ -109,7 +109,9 @@ export function recordEvent(sql, { name, props = {}, url, headers, site = '' })
|
|
|
109
109
|
let path = null;
|
|
110
110
|
if (url)
|
|
111
111
|
path = normalisePath(new URL(url, 'http://x').pathname);
|
|
112
|
-
|
|
112
|
+
// An agent's country is where its requests came from, so the stats page's countries can count it.
|
|
113
|
+
const country = headers?.get('x-vercel-ip-country')?.slice(0, 2);
|
|
114
|
+
return sql('INSERT INTO "Event" ("siteId", name, path, props, visitor) VALUES ($1, $2, $3, $4::jsonb, $5)', site, name.slice(0, 64), path, JSON.stringify(country ? { ...props, country } : props), headers?.get('user-agent') ? await visitorOf(sql, site, headers) : null);
|
|
113
115
|
});
|
|
114
116
|
}
|
|
115
117
|
/** A browser-sent event's props: eight at most, each value a short string or a number. */
|
|
@@ -156,16 +158,44 @@ export const STATS_FILTERS = ['page', 'entry', 'source', 'country', 'device'];
|
|
|
156
158
|
* `Stats` calls it on the server, where a client module's export is only a reference. */
|
|
157
159
|
export const count = (n) => n.toLocaleString('en');
|
|
158
160
|
/**
|
|
159
|
-
*
|
|
160
|
-
*
|
|
161
|
+
* A referrer that sent this many visitors in the window, not one of whom read a
|
|
162
|
+
* second page, is a crawler naming a site it never came from: aocr.org sent
|
|
163
|
+
* radix.wiki 319 one-page visitors in a month while Google's bounced four in five.
|
|
164
|
+
*/
|
|
165
|
+
const BOT_VISITORS = 20;
|
|
166
|
+
const BOT_SOURCE = '(bot)';
|
|
167
|
+
/** A visit's source: its first view's, or `(bot)` where that source is among `bots`. */
|
|
168
|
+
const visitSource = (s) => {
|
|
169
|
+
const first = '(array_agg(source ORDER BY at, id))[1]';
|
|
170
|
+
return `CASE WHEN ${first} IN (SELECT source FROM bots${s}) THEN '${BOT_SOURCE}' ELSE ${first} END`;
|
|
171
|
+
};
|
|
172
|
+
/**
|
|
173
|
+
* Each filter as a test on one visit in the window suffixed `s`. A page,
|
|
174
|
+
* country or device is any of the visit's views; an entry page or a source is
|
|
175
|
+
* its first. `(none)` is no source.
|
|
161
176
|
*/
|
|
162
177
|
const VISIT_TEST = {
|
|
163
178
|
page: v => `bool_or(path = ${v})`,
|
|
164
179
|
entry: v => `(array_agg(path ORDER BY at, id))[1] = ${v}`,
|
|
165
|
-
source: v => `coalesce((
|
|
180
|
+
source: (v, s) => `coalesce(${visitSource(s)}, '(none)') = ${v}`,
|
|
166
181
|
country: v => `bool_or(country = ${v})`,
|
|
167
182
|
device: v => `bool_or(device = ${v})`,
|
|
168
183
|
};
|
|
184
|
+
/** An agent's source and device in the lists, which no request of its carries. */
|
|
185
|
+
const AGENT_SOURCE = '(agent)';
|
|
186
|
+
const AGENT_DEVICE = 'agent';
|
|
187
|
+
/**
|
|
188
|
+
* Each filter as a test on one agent: its reads in the window are its visit,
|
|
189
|
+
* the first its entry page. Its source and device are the two above, and its
|
|
190
|
+
* country the one its requests came from, where they were counted with one.
|
|
191
|
+
*/
|
|
192
|
+
const AGENT_TEST = {
|
|
193
|
+
page: v => `bool_or(path = ${v})`,
|
|
194
|
+
entry: v => `(array_agg(path ORDER BY at))[1] = ${v}`,
|
|
195
|
+
source: v => `${v} = '${AGENT_SOURCE}'`,
|
|
196
|
+
country: v => `bool_or(country = ${v})`,
|
|
197
|
+
device: v => `${v} = '${AGENT_DEVICE}'`,
|
|
198
|
+
};
|
|
169
199
|
/**
|
|
170
200
|
* An agent's name from its event: the roster's label, else what its user agent
|
|
171
201
|
* calls itself, `compatible; X` or the first product token.
|
|
@@ -180,7 +210,7 @@ const AGENT_NAME = `coalesce(props->>'bot', substring(props->>'ua' from 'compati
|
|
|
180
210
|
* The window before is the same chain of views, visits and events a second
|
|
181
211
|
* time, `_p` on every name, so its figures are the headline's to the letter.
|
|
182
212
|
*/
|
|
183
|
-
export async function digest(sql, { site = '', days, handle, siblings = [], gaps = [SEARCH_GAPS], ownAgents = ['-mcp-test', 'radix-studio'], filter = {} }) {
|
|
213
|
+
export async function digest(sql, { site = '', days, handle, siblings = [], gaps = [SEARCH_GAPS], ownAgents = ['-mcp-test', 'radix-studio'], filter = {}, apart = true }) {
|
|
184
214
|
// The window is the database's clock, which stamped the rows; the app's can run behind it.
|
|
185
215
|
// All time is a century: further back than any row, and still an int to Postgres.
|
|
186
216
|
const values = [site, Number.isFinite(days) ? Math.max(1, Math.round(days)) : 36_500];
|
|
@@ -190,14 +220,34 @@ export async function digest(sql, { site = '', days, handle, siblings = [], gaps
|
|
|
190
220
|
const merged = (label, column, live) => `SELECT k AS ${label}, sum(n) AS visitors FROM (${live} UNION ALL SELECT key, value::int FROM a, jsonb_each_text(a."${column}")) t(k, n) GROUP BY 1`;
|
|
191
221
|
/** Clicks on links to the site's own X account, among `e`. */
|
|
192
222
|
const follows = handle && `name = 'Outbound Link: Click' AND props->>'url' ILIKE ANY(${$([`%x.com/${handle}%`, `%twitter.com/${handle}%`])}::text[])`;
|
|
193
|
-
const tests = STATS_FILTERS.flatMap(k =>
|
|
223
|
+
const tests = STATS_FILTERS.flatMap(k => {
|
|
224
|
+
if (!filter[k])
|
|
225
|
+
return [];
|
|
226
|
+
const v = $(filter[k]);
|
|
227
|
+
return [(s) => VISIT_TEST[k](v, s)];
|
|
228
|
+
});
|
|
229
|
+
const agentTests = STATS_FILTERS.flatMap(k => (filter[k] ? [AGENT_TEST[k]($(filter[k]))] : []));
|
|
230
|
+
/** Agents among the rows of a ranked list, unless they are counted apart. */
|
|
231
|
+
const agentsToo = (select) => (apart ? '' : ` UNION ALL ${select}`);
|
|
194
232
|
const day = (at) => `to_char(${at} AT TIME ZONE 'UTC', 'YYYY-MM-DD')`;
|
|
195
233
|
/** The site's own agents, as a test on an event's user agent. */
|
|
196
234
|
const ours = `coalesce(props->>'ua', '') ILIKE ANY(${$(ownAgents.map(a => `%${a}%`))}::text[])`;
|
|
235
|
+
/**
|
|
236
|
+
* A typeahead logs each settled keystroke, so one reader typing a question
|
|
237
|
+
* filled acuiq's list with thirteen of its prefixes. A query is left out when
|
|
238
|
+
* the same visitor's next one, inside five minutes, extends it: only where
|
|
239
|
+
* the reader stopped is a gap. So is a single character.
|
|
240
|
+
*/
|
|
197
241
|
const gapSelects = gaps.map(g => {
|
|
198
242
|
const prop = $(g.prop);
|
|
199
|
-
return `(SELECT
|
|
200
|
-
|
|
243
|
+
return `(SELECT query, count(*) AS events FROM (
|
|
244
|
+
SELECT props, visitor, props->>${prop} AS query,
|
|
245
|
+
lead(props->>${prop}) OVER typed AS next, lead(at) OVER typed - at AS after
|
|
246
|
+
FROM e WHERE name = ${$(g.event)} AND props->>${prop} IS NOT NULL
|
|
247
|
+
WINDOW typed AS (PARTITION BY visitor ORDER BY at)
|
|
248
|
+
) s
|
|
249
|
+
WHERE props @> ${$(JSON.stringify(g.where ?? {}))}::jsonb AND char_length(query) > 1
|
|
250
|
+
AND NOT coalesce(visitor IS NOT NULL AND after <= interval '5 minutes' AND char_length(next) > char_length(query) AND starts_with(next, query), false)
|
|
201
251
|
GROUP BY 1 ORDER BY 2 DESC, 1 LIMIT 15)`;
|
|
202
252
|
});
|
|
203
253
|
/**
|
|
@@ -214,22 +264,29 @@ export async function digest(sql, { site = '', days, handle, siblings = [], gaps
|
|
|
214
264
|
SELECT *, CASE WHEN at - lag(at) OVER (PARTITION BY visitor ORDER BY at, id) <= interval '30 minutes' THEN 0 ELSE 1 END AS starts FROM viewed${s}
|
|
215
265
|
), numbered${s} AS (
|
|
216
266
|
SELECT *, sum(starts) OVER (PARTITION BY visitor ORDER BY at, id) AS visit FROM marked${s}
|
|
267
|
+
), bots${s} AS (
|
|
268
|
+
SELECT source FROM (SELECT visitor, (array_agg(source ORDER BY at, id))[1] AS source, count(*) AS views FROM numbered${s} GROUP BY visitor, visit) f
|
|
269
|
+
WHERE source IS NOT NULL GROUP BY 1 HAVING count(DISTINCT visitor) >= ${BOT_VISITORS} AND max(views) = 1
|
|
217
270
|
), visits${s} AS (
|
|
218
271
|
SELECT visitor, visit, min(at) AS at, count(*) AS views, extract(epoch FROM max(at) - min(at)) AS seconds,
|
|
219
|
-
(array_agg(path ORDER BY at, id))[1] AS entry, (
|
|
220
|
-
FROM numbered${s} GROUP BY visitor, visit${tests.length ? ` HAVING ${tests.join(' AND ')}` : ''}
|
|
272
|
+
(array_agg(path ORDER BY at, id))[1] AS entry, ${visitSource(s)} AS source
|
|
273
|
+
FROM numbered${s} GROUP BY visitor, visit${tests.length ? ` HAVING ${tests.map(t => t(s)).join(' AND ')}` : ''}
|
|
221
274
|
), v${s} AS (
|
|
222
275
|
${tests.length ? `SELECT n.id, n.path, n.source, n.country, n.device, n.visitor, n.at FROM numbered${s} n JOIN visits${s} USING (visitor, visit)` : `SELECT * FROM viewed${s}`}
|
|
276
|
+
), read${s} AS (
|
|
277
|
+
SELECT visitor, "createdAt" AS at, path, props->>'country' AS country, ${AGENT_NAME} AS agent FROM "Event"
|
|
278
|
+
WHERE "siteId" = $1 AND ${at} AND name IN ('AI Bot Visit', 'MCP Call') AND visitor IS NOT NULL AND NOT ${ours}
|
|
279
|
+
), ag${s} AS (
|
|
280
|
+
SELECT * FROM read${s}${agentTests.length ? ` WHERE visitor IN (SELECT visitor FROM read${s} GROUP BY 1 HAVING ${agentTests.join(' AND ')})` : ''}
|
|
223
281
|
), e${s} AS (
|
|
224
282
|
SELECT name, props, visitor, "createdAt" AS at FROM "Event" WHERE "siteId" = $1 AND ${at}
|
|
225
|
-
${tests.length ? `AND visitor IN (SELECT visitor FROM v${s})` : ''}
|
|
283
|
+
${tests.length ? `AND (visitor IN (SELECT visitor FROM v${s}) OR visitor IN (SELECT visitor FROM ag${s}))` : ''}
|
|
284
|
+
), acted${s} AS MATERIALIZED (
|
|
285
|
+
SELECT visitor, at FROM e${s} WHERE name NOT IN ('AI Bot Visit', 'MCP Call') AND visitor IS NOT NULL
|
|
226
286
|
), bounced${s} AS (
|
|
287
|
+
-- Against the few reader events alone: probing every agent event once per visit took seconds.
|
|
227
288
|
SELECT * FROM visits${s} b WHERE views = 1 AND NOT EXISTS (
|
|
228
|
-
SELECT 1 FROM
|
|
229
|
-
AND x.at >= b.at AND x.at < b.at + interval '30 minutes')
|
|
230
|
-
), ag${s} AS (
|
|
231
|
-
SELECT visitor, at, ${AGENT_NAME} AS agent FROM e${s}
|
|
232
|
-
WHERE name IN ('AI Bot Visit', 'MCP Call') AND visitor IS NOT NULL AND NOT ${ours}
|
|
289
|
+
SELECT 1 FROM acted${s} x WHERE x.visitor = b.visitor AND x.at >= b.at AND x.at < b.at + interval '30 minutes')
|
|
233
290
|
), a${s} AS (
|
|
234
291
|
SELECT * FROM "ViewDay" WHERE "siteId" = $1 AND day >= ${dayAgo(back + 1)}${back ? ` AND day < ${dayAgo(back)}` : ''}${tests.length ? ' AND false' : ''}
|
|
235
292
|
), sessions${s} AS (
|
|
@@ -244,25 +301,26 @@ export async function digest(sql, { site = '', days, handle, siblings = [], gaps
|
|
|
244
301
|
/** One window's headline figures, as `json_build_object` arguments. */
|
|
245
302
|
const headline = (s) => `'visitors', (SELECT count(DISTINCT visitor) FROM v${s}) + (SELECT visitors FROM archived${s}),
|
|
246
303
|
'pageviews', (SELECT count(*) FROM v${s}) + (SELECT pageviews FROM archived${s}),
|
|
304
|
+
'visits', (SELECT visits FROM sessions${s}),
|
|
247
305
|
'bounce_rate', (SELECT round(100.0 * bounces / nullif(visits, 0)) FROM sessions${s}),
|
|
248
306
|
'visit_duration', (SELECT round(seconds / nullif(visits, 0)) FROM sessions${s}),
|
|
249
307
|
'agents', (SELECT count(DISTINCT visitor) FROM ag${s}),
|
|
250
308
|
'visitors_incl_agents', (SELECT count(DISTINCT visitor) FROM (SELECT visitor FROM v${s} UNION SELECT visitor FROM ag${s}) u) + (SELECT visitors FROM archived${s})`;
|
|
251
309
|
const [row] = (await sql(`WITH ${chain('', 0)}, ${chain('_p', 1)}, src AS (
|
|
252
310
|
SELECT coalesce(source, '(none)') AS source, count(DISTINCT visitor) AS visitors FROM visits GROUP BY 1
|
|
253
|
-
UNION ALL SELECT key, value::int FROM a, jsonb_each_text(a.sources)
|
|
311
|
+
UNION ALL SELECT key, value::int FROM a, jsonb_each_text(a.sources)${agentsToo(`SELECT '${AGENT_SOURCE}', count(DISTINCT visitor) FROM ag HAVING count(*) > 0`)}
|
|
254
312
|
)
|
|
255
313
|
SELECT json_build_object(
|
|
256
314
|
${headline('')},
|
|
257
315
|
'previous', json_build_object(${headline('_p')}, 'follow_clicks', ${follows ? `(SELECT count(*) FROM e_p WHERE ${follows})` : 'NULL'}),
|
|
258
316
|
'counted_from', (SELECT ${day('min("createdAt")')} FROM "View" WHERE "siteId" = $1),
|
|
259
317
|
'top_sources', ${list('SELECT source, sum(visitors) AS visitors FROM src GROUP BY 1', 6)},
|
|
260
|
-
'top_pages', ${list(merged('page', 'pages',
|
|
261
|
-
'entry_pages', ${list(merged('page', 'entryPages',
|
|
318
|
+
'top_pages', ${list(merged('page', 'pages', `SELECT path, count(DISTINCT visitor) FROM v GROUP BY 1${agentsToo('SELECT path, count(DISTINCT visitor) FROM ag WHERE path IS NOT NULL GROUP BY 1')}`), 6)},
|
|
319
|
+
'entry_pages', ${list(merged('page', 'entryPages', `SELECT entry, count(DISTINCT visitor) FROM visits GROUP BY 1${agentsToo('SELECT entry, count(*) FROM (SELECT (array_agg(path ORDER BY at))[1] AS entry FROM ag GROUP BY visitor) f WHERE entry IS NOT NULL GROUP BY 1')}`), 6)},
|
|
262
320
|
'from_posts', ${list(`SELECT entry AS page, count(DISTINCT visitor) AS visitors FROM visits WHERE source = 'x' GROUP BY 1`, 15)},
|
|
263
321
|
'sibling_referrals', ${list(`SELECT source, sum(visitors) AS visitors FROM src WHERE source = ANY(${$(siblings)}::text[]) GROUP BY 1`, 6)},
|
|
264
|
-
'countries', ${list(merged('country', 'countries',
|
|
265
|
-
'devices', ${list(merged('device', 'devices',
|
|
322
|
+
'countries', ${list(merged('country', 'countries', `SELECT country, count(DISTINCT visitor) FROM v WHERE country IS NOT NULL GROUP BY 1${agentsToo('SELECT country, count(DISTINCT visitor) FROM ag WHERE country IS NOT NULL GROUP BY 1')}`), 10)},
|
|
323
|
+
'devices', ${list(merged('device', 'devices', `SELECT device, count(DISTINCT visitor) FROM v GROUP BY 1${agentsToo(`SELECT '${AGENT_DEVICE}', count(DISTINCT visitor) FROM ag HAVING count(*) > 0`)}`))},
|
|
266
324
|
'by_day', ${list(`SELECT date, sum(visitors) AS visitors, sum(pageviews) AS pageviews, sum(visits) AS visits, sum(bounces) AS bounces,
|
|
267
325
|
sum(seconds) AS seconds, sum(agents) AS agents, sum(everyone) AS visitors_incl_agents, sum(follows) AS follow_clicks FROM (
|
|
268
326
|
SELECT ${day('at')}, count(DISTINCT visitor), count(*), 0, 0, 0, 0, 0, 0 FROM v GROUP BY 1
|
package/dist/chart.js
CHANGED
|
@@ -60,21 +60,24 @@ function bucketStart(time, step) {
|
|
|
60
60
|
* Points, oldest first, one per step. A price keeps the last value in each
|
|
61
61
|
* step, as a candle does its close: the first open, the highest high, the
|
|
62
62
|
* lowest low. A count is the total of its days; a rate, their mean. Volume is
|
|
63
|
-
* summed.
|
|
63
|
+
* summed. The first step sits on its first point when the points start inside
|
|
64
|
+
* it, so the axis never reaches back past the days that were counted.
|
|
64
65
|
*
|
|
65
66
|
* @internal
|
|
66
67
|
*/
|
|
67
68
|
export function bucketChart(points, step, aggregate) {
|
|
68
69
|
const out = [];
|
|
69
70
|
let n = 0;
|
|
71
|
+
let bucket;
|
|
70
72
|
for (const p of points) {
|
|
71
73
|
const start = bucketStart(p.time, step);
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
74
|
+
if (bucket !== start) {
|
|
75
|
+
out.push({ ...p, time: out.length ? start : p.time });
|
|
76
|
+
bucket = start;
|
|
75
77
|
n = 1;
|
|
76
78
|
continue;
|
|
77
79
|
}
|
|
80
|
+
const last = out.at(-1);
|
|
78
81
|
if (aggregate === 'last')
|
|
79
82
|
last.value = p.value;
|
|
80
83
|
else if (aggregate === 'sum')
|
package/dist/react-server.d.ts
CHANGED
|
@@ -22,6 +22,8 @@ export interface WikiLinkProps {
|
|
|
22
22
|
'aria-checked'?: boolean;
|
|
23
23
|
/** The name of a link drawn as an icon. */
|
|
24
24
|
'aria-label'?: string;
|
|
25
|
+
/** False where following the link costs the server a query, as a stats page's links do; a plain anchor ignores it. */
|
|
26
|
+
prefetch?: boolean;
|
|
25
27
|
children?: ReactNode;
|
|
26
28
|
}
|
|
27
29
|
export type WikiLinkComponent = ComponentType<WikiLinkProps>;
|
package/dist/react-server.js
CHANGED
|
@@ -29,7 +29,7 @@ import { formatDay } from './freshness.js';
|
|
|
29
29
|
* Exported because `block-views` renders links too and had declared the same
|
|
30
30
|
* three lines; a default that exists twice is a default that can diverge.
|
|
31
31
|
*/
|
|
32
|
-
export const Anchor = (props) => _jsx("a", { ...props });
|
|
32
|
+
export const Anchor = ({ prefetch: _, ...props }) => _jsx("a", { ...props });
|
|
33
33
|
const pagesPhrase = (n) => `${n} page${n === 1 ? '' : 's'}`;
|
|
34
34
|
/**
|
|
35
35
|
* The section's second axis, as pressable chips.
|
package/dist/search.d.ts
CHANGED
|
@@ -72,3 +72,22 @@ export declare const FTS_RANK_NORMALIZATION = 32;
|
|
|
72
72
|
* rather than as a pattern matching everything.
|
|
73
73
|
*/
|
|
74
74
|
export declare function escapeLikeTerm(query: string): string;
|
|
75
|
+
/**
|
|
76
|
+
* A query as the full-text tier reads it: a hyphen or underscore inside a word
|
|
77
|
+
* is a space. `websearch_to_tsquery` keeps a hyphenated word as a phrase that
|
|
78
|
+
* needs the compound itself, so the slug `radix-pipeline` found nothing where
|
|
79
|
+
* `radix pipeline` found sixteen pages. A leading `-` is websearch's NOT, and
|
|
80
|
+
* stays.
|
|
81
|
+
*/
|
|
82
|
+
export declare function fullTextQuery(query: string): string;
|
|
83
|
+
/**
|
|
84
|
+
* A query with each misspelled word replaced by the nearest word in `vocabulary`
|
|
85
|
+
* (word → how often it occurs), or `null` when no word changes. Called only for
|
|
86
|
+
* a query that found nothing, so a word the wiki already uses is never touched.
|
|
87
|
+
*
|
|
88
|
+
* Tight on purpose: a word of four to seven letters may be one edit out, a
|
|
89
|
+
* longer one two, and a shorter one none. Looser, a name the wiki lacks turns
|
|
90
|
+
* into a word it has – `bondix` into `bonds` – and the reader gets an answer to
|
|
91
|
+
* a question they did not ask. Ties go to the commoner word.
|
|
92
|
+
*/
|
|
93
|
+
export declare function correctQuery(query: string, vocabulary: ReadonlyMap<string, number>): string | null;
|
package/dist/search.js
CHANGED
|
@@ -120,3 +120,62 @@ export const FTS_RANK_NORMALIZATION = 32;
|
|
|
120
120
|
export function escapeLikeTerm(query) {
|
|
121
121
|
return (query ?? '').trim().replace(/[\\%_]/g, char => `\\${char}`);
|
|
122
122
|
}
|
|
123
|
+
/**
|
|
124
|
+
* A query as the full-text tier reads it: a hyphen or underscore inside a word
|
|
125
|
+
* is a space. `websearch_to_tsquery` keeps a hyphenated word as a phrase that
|
|
126
|
+
* needs the compound itself, so the slug `radix-pipeline` found nothing where
|
|
127
|
+
* `radix pipeline` found sixteen pages. A leading `-` is websearch's NOT, and
|
|
128
|
+
* stays.
|
|
129
|
+
*/
|
|
130
|
+
export function fullTextQuery(query) {
|
|
131
|
+
return (query ?? '').trim().replace(/([\p{L}\p{N}])[-_]+(?=[\p{L}\p{N}])/gu, '$1 ');
|
|
132
|
+
}
|
|
133
|
+
/** Edits between two words, a swapped adjacent pair counting as one. */
|
|
134
|
+
function editDistance(a, b) {
|
|
135
|
+
const d = Array.from({ length: a.length + 1 }, (_, i) => [i, ...Array(b.length).fill(0)]);
|
|
136
|
+
for (let j = 1; j <= b.length; j++)
|
|
137
|
+
d[0][j] = j;
|
|
138
|
+
for (let i = 1; i <= a.length; i++)
|
|
139
|
+
for (let j = 1; j <= b.length; j++) {
|
|
140
|
+
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
|
|
141
|
+
d[i][j] = Math.min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + cost);
|
|
142
|
+
if (i > 1 && j > 1 && a[i - 1] === b[j - 2] && a[i - 2] === b[j - 1])
|
|
143
|
+
d[i][j] = Math.min(d[i][j], d[i - 2][j - 2] + 1);
|
|
144
|
+
}
|
|
145
|
+
return d[a.length][b.length];
|
|
146
|
+
}
|
|
147
|
+
/**
|
|
148
|
+
* A query with each misspelled word replaced by the nearest word in `vocabulary`
|
|
149
|
+
* (word → how often it occurs), or `null` when no word changes. Called only for
|
|
150
|
+
* a query that found nothing, so a word the wiki already uses is never touched.
|
|
151
|
+
*
|
|
152
|
+
* Tight on purpose: a word of four to seven letters may be one edit out, a
|
|
153
|
+
* longer one two, and a shorter one none. Looser, a name the wiki lacks turns
|
|
154
|
+
* into a word it has – `bondix` into `bonds` – and the reader gets an answer to
|
|
155
|
+
* a question they did not ask. Ties go to the commoner word.
|
|
156
|
+
*/
|
|
157
|
+
export function correctQuery(query, vocabulary) {
|
|
158
|
+
let changed = false;
|
|
159
|
+
const fixed = query.trim().toLowerCase().split(/\s+/).map(word => {
|
|
160
|
+
const budget = word.length >= 8 ? 2 : word.length >= 4 ? 1 : 0;
|
|
161
|
+
if (!budget || vocabulary.has(word) || !/^[\p{L}\p{N}]+$/u.test(word))
|
|
162
|
+
return word;
|
|
163
|
+
let best = null;
|
|
164
|
+
let bestDistance = Infinity;
|
|
165
|
+
let bestCount = 0;
|
|
166
|
+
for (const [candidate, count] of vocabulary) {
|
|
167
|
+
if (Math.abs(candidate.length - word.length) > budget)
|
|
168
|
+
continue;
|
|
169
|
+
const distance = editDistance(word, candidate);
|
|
170
|
+
if (distance > budget)
|
|
171
|
+
continue;
|
|
172
|
+
if (distance < bestDistance || (distance === bestDistance && count > bestCount))
|
|
173
|
+
[best, bestDistance, bestCount] = [candidate, distance, count];
|
|
174
|
+
}
|
|
175
|
+
if (!best)
|
|
176
|
+
return word;
|
|
177
|
+
changed = true;
|
|
178
|
+
return best;
|
|
179
|
+
});
|
|
180
|
+
return changed ? fixed.join(' ') : null;
|
|
181
|
+
}
|
package/dist/stats-figures.js
CHANGED
|
@@ -13,6 +13,8 @@ import { TimeChart, stepDecimals } from './chart.js';
|
|
|
13
13
|
const places = (step) => (step ? stepDecimals(step) : 0);
|
|
14
14
|
const duration = (s, step) => (s < 60 ? `${s.toFixed(places(step))}s` : `${Math.floor(Math.round(s) / 60)}m ${Math.round(s) % 60}s`);
|
|
15
15
|
const percent = (n, step) => `${n.toFixed(places(step))}%`;
|
|
16
|
+
// Views per visit sit between one and a few, so a tenth tells them apart.
|
|
17
|
+
const perVisit = (n, step) => n.toFixed(step ? stepDecimals(step) : 1);
|
|
16
18
|
/**
|
|
17
19
|
* The change from the window before: a share of the figure before, or for a
|
|
18
20
|
* rate its difference in points, since a bounce rate up 10% of itself reads
|
|
@@ -42,6 +44,14 @@ const figuresOf = ({ previous: p, ...d }, apart) => [
|
|
|
42
44
|
},
|
|
43
45
|
]),
|
|
44
46
|
{ label: 'Page views', value: count(d.pageviews), delta: change(d.pageviews, p.pageviews), day: r => r.pageviews },
|
|
47
|
+
{
|
|
48
|
+
label: 'Views per visit',
|
|
49
|
+
value: d.visits ? perVisit(d.pageviews / d.visits) : '–',
|
|
50
|
+
delta: change(d.visits ? d.pageviews / d.visits : null, p.pageviews != null && p.visits ? p.pageviews / p.visits : null),
|
|
51
|
+
day: r => (r.visits ? r.pageviews / r.visits : null),
|
|
52
|
+
format: perVisit,
|
|
53
|
+
aggregate: 'mean',
|
|
54
|
+
},
|
|
45
55
|
{
|
|
46
56
|
label: 'Bounce rate',
|
|
47
57
|
value: d.bounce_rate == null ? '–' : percent(d.bounce_rate),
|
|
@@ -79,8 +89,9 @@ const EMPTY = { date: '', visitors: 0, pageviews: 0, visits: 0, bounces: 0, seco
|
|
|
79
89
|
*/
|
|
80
90
|
export function StatsFigures({ digest, first, apart, controls }) {
|
|
81
91
|
const figures = useMemo(() => figuresOf(digest, apart), [digest, apart]);
|
|
82
|
-
|
|
83
|
-
const
|
|
92
|
+
// By name, since a window or the agents switch can arrive without a reload and the figures shift.
|
|
93
|
+
const [chosen, choose] = useState(figures[0].label);
|
|
94
|
+
const figure = figures.find(f => f.label === chosen) ?? figures[0];
|
|
84
95
|
const byDay = useMemo(() => new Map(digest.by_day.map(r => [r.date, r])), [digest.by_day]);
|
|
85
96
|
const series = useMemo(() => {
|
|
86
97
|
const today = Math.floor(Date.now() / 1000 / DAY);
|
|
@@ -90,7 +101,7 @@ export function StatsFigures({ digest, first, apart, controls }) {
|
|
|
90
101
|
return value == null ? null : { time, value };
|
|
91
102
|
}).filter(p => p !== null);
|
|
92
103
|
}, [byDay, first, figure]);
|
|
93
|
-
return (_jsxs(_Fragment, { children: [_jsx("div", { className: "stats-figures", role: "group", "aria-label": "Chart", children: figures.map(
|
|
104
|
+
return (_jsxs(_Fragment, { children: [_jsx("div", { className: "stats-figures", role: "group", "aria-label": "Chart", children: figures.map(f => (_jsxs("button", { type: "button", "aria-pressed": f === figure, onClick: () => choose(f.label), children: [_jsx("span", { children: f.label }), _jsxs("strong", { children: [f.value, f.delta && (_jsx("small", { className: "stats-delta", "data-better": f.delta.better, children: f.delta.text }))] })] }, f.label))) }), _jsx(TimeChart, { series: series, label: figure.label, ranges: ['all'], aggregate: figure.aggregate ?? 'sum', fromZero: true, format: figure.format, controls: controls, className: "stats-chart" })] }));
|
|
94
105
|
}
|
|
95
106
|
async function post(endpoint, body) {
|
|
96
107
|
const res = await fetch(endpoint, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });
|
package/dist/stats.d.ts
CHANGED
|
@@ -13,7 +13,7 @@ export interface StatsProps {
|
|
|
13
13
|
apart?: boolean;
|
|
14
14
|
/** The windows offered as `?days=` links on the same page. */
|
|
15
15
|
ranges?: readonly number[];
|
|
16
|
-
/** Every link on the page: `next/link`, say, so a window or a filter does not reload it. */
|
|
16
|
+
/** Every link on the page: `next/link`, say, so a window or a filter does not reload it. Each is given `prefetch={false}`. */
|
|
17
17
|
link?: WikiLinkComponent;
|
|
18
18
|
}
|
|
19
19
|
/**
|
|
@@ -23,14 +23,16 @@ export interface StatsProps {
|
|
|
23
23
|
*/
|
|
24
24
|
export declare function statsQuery(params: Record<string, string | string[] | undefined>, ranges?: readonly number[]): {
|
|
25
25
|
days: number;
|
|
26
|
-
filter: Partial<Record<"page" | "
|
|
26
|
+
filter: Partial<Record<"page" | "country" | "entry" | "source" | "device", string>>;
|
|
27
27
|
apart: boolean;
|
|
28
28
|
};
|
|
29
29
|
/**
|
|
30
30
|
* A site's own analytics on one page: the headline figures against the window
|
|
31
31
|
* before, visitors counting agents unless `apart`, whichever of them is chosen per day, and every ranked list the
|
|
32
32
|
* digest carries, each row a link that narrows the page to it. Lists with
|
|
33
|
-
* nothing in them are left out
|
|
33
|
+
* nothing in them are left out, but for the searches that found nothing,
|
|
34
|
+
* which says it found none, so a quiet window never reads as one not counted. The digest is made with the same `apart`, so
|
|
35
|
+
* the lists count the visitors the chart draws. The chart and the bars are `currentColor`
|
|
34
36
|
* until the importing stylesheet gives them a colour.
|
|
35
37
|
*/
|
|
36
38
|
export declare function Stats({ digest: d, days, filter, apart, ranges, link: Link }: StatsProps): import("react").JSX.Element;
|
package/dist/stats.js
CHANGED
|
@@ -22,6 +22,7 @@ export function statsQuery(params, ranges = STATS_RANGES) {
|
|
|
22
22
|
if (typeof v === 'string' && v)
|
|
23
23
|
filter[k] = v.slice(0, 500);
|
|
24
24
|
}
|
|
25
|
+
// `apart` is `digest`'s option too, so its lists count the visitors the chart draws.
|
|
25
26
|
return { days: statsDays(params.days, ranges), filter, apart: params.agents === 'apart' };
|
|
26
27
|
}
|
|
27
28
|
/** The page at a window, a filter and a count of agents, as a query string. */
|
|
@@ -45,17 +46,19 @@ const country = (code) => {
|
|
|
45
46
|
};
|
|
46
47
|
const NAMES = { page: 'Page', entry: 'Entry page', source: 'Source', country: 'Country', device: 'Device' };
|
|
47
48
|
const DAY_MS = 86_400_000;
|
|
48
|
-
function StatsList({ title, rows, label = v => v, to, link: Link, }) {
|
|
49
|
+
function StatsList({ title, rows, label = v => v, to, empty, link: Link, }) {
|
|
49
50
|
if (!rows.length)
|
|
50
|
-
return null;
|
|
51
|
+
return empty ? (_jsxs("section", { className: "stats-list", children: [_jsx("h2", { children: title }), _jsx("p", { children: empty })] })) : null;
|
|
51
52
|
const max = Math.max(...rows.map(([, n]) => n));
|
|
52
|
-
return (_jsxs("section", { className: "stats-list", children: [_jsx("h2", { children: title }), _jsx("ol", { children: rows.map(([v, n]) => (_jsxs("li", { style: { '--share': n / max }, children: [_jsx("span", { children: to ? _jsx(Link, { href: to(v), children: label(v) }) : label(v) }), _jsx("span", { className: "stats-count", children: count(n) })] }, v))) })] }));
|
|
53
|
+
return (_jsxs("section", { className: "stats-list", children: [_jsx("h2", { children: title }), _jsx("ol", { children: rows.map(([v, n]) => (_jsxs("li", { style: { '--share': n / max }, children: [_jsx("span", { children: to ? (_jsx(Link, { href: to(v), prefetch: false, children: label(v) })) : (label(v)) }), _jsx("span", { className: "stats-count", children: count(n) })] }, v))) })] }));
|
|
53
54
|
}
|
|
54
55
|
/**
|
|
55
56
|
* A site's own analytics on one page: the headline figures against the window
|
|
56
57
|
* before, visitors counting agents unless `apart`, whichever of them is chosen per day, and every ranked list the
|
|
57
58
|
* digest carries, each row a link that narrows the page to it. Lists with
|
|
58
|
-
* nothing in them are left out
|
|
59
|
+
* nothing in them are left out, but for the searches that found nothing,
|
|
60
|
+
* which says it found none, so a quiet window never reads as one not counted. The digest is made with the same `apart`, so
|
|
61
|
+
* the lists count the visitors the chart draws. The chart and the bars are `currentColor`
|
|
59
62
|
* until the importing stylesheet gives them a colour.
|
|
60
63
|
*/
|
|
61
64
|
export function Stats({ digest: d, days, filter = {}, apart = false, ranges = STATS_RANGES, link: Link = Anchor }) {
|
|
@@ -74,8 +77,8 @@ export function Stats({ digest: d, days, filter = {}, apart = false, ranges = ST
|
|
|
74
77
|
const to = (k) => (v) => at({ ...filter, [k]: v });
|
|
75
78
|
// Each window is another digest, so a link rather than a button, pinned
|
|
76
79
|
// with the chart's own controls, and so is the switch that counts agents apart.
|
|
77
|
-
const controls = (_jsxs(_Fragment, { children: [_jsx("div", { role: "group", "aria-label": "Window", children: ranges.map(n => (_jsx(Link, { href: at(filter, n), "aria-current": n === days ? 'page' : undefined, children: windowLabel(n) }, n))) }), _jsx("div", { children: _jsx(Link, { href: at(filter, days, !apart), role: "switch", "aria-checked": !apart, "aria-label": "Agents counted among visitors", title: "Agents counted among visitors", children: _jsxs("svg", { className: "chart-icon", viewBox: "0 0 12 12", "aria-hidden": true, children: [_jsx("path", { d: "M6 1.5v2M1 6.5v2M11 6.5v2", stroke: "currentColor" }), _jsx("rect", { x: "2.5", y: "3.5", width: "7", height: "7", rx: "1.5", fill: "none", stroke: "currentColor" }), _jsx("circle", { cx: "4.75", cy: "7", r: "0.9", fill: "currentColor" }), _jsx("circle", { cx: "7.25", cy: "7", r: "0.9", fill: "currentColor" })] }) }) })] }));
|
|
78
|
-
return (_jsxs("div", { className: "stats", children: [narrowed.length > 0 && (_jsxs("div", { className: "stats-filters", children: [_jsx("ul", { "aria-label": "Narrowed to", children: narrowed.map(k => (_jsx("li", { children: _jsxs(Link, { href: at({ ...filter, [k]: undefined }), title: "Remove", children: [NAMES[k], ": ", k === 'country' ? country(filter[k]) : filter[k], " \u00D7"] }) }, k))) }), start < counted && d.counted_from && (_jsxs("p", { children: ["Narrowed figures start on ", d.counted_from, ", the first day this site counted its own visits."] }))] })), _jsx(StatsFigures, { digest: d, first: first, apart: apart, controls: controls }), _jsxs("div", { className: "stats-lists", children: [_jsx(StatsList, { link: Link, title: "Pages", rows: rows(d.top_pages, 'page', 'visitors'), to: to('page') }), _jsx(StatsList, { link: Link, title: "Entry pages", rows: rows(d.entry_pages, 'page', 'visitors'), to: to('entry') }), _jsx(StatsList, { link: Link, title: "Sources", rows: rows(d.top_sources, 'source', 'visitors'), to: to('source') }), _jsx(StatsList, { link: Link, title: "From X posts", rows: rows(d.from_posts, 'page', 'visitors'), to: v => at({ ...filter, source: 'x', entry: v }) }), _jsx(StatsList, { link: Link, title: "Sibling sites", rows: rows(d.sibling_referrals, 'source', 'visitors'), to: to('source') }), _jsx(StatsList, { link: Link, title: "Countries", rows: rows(d.countries, 'country', 'visitors'), label: country, to: to('country') }), _jsx(StatsList, { link: Link, title: "Devices", rows: rows(d.devices, 'device', 'visitors'), to: to('device') }), _jsx(StatsList, { link: Link, title: "Agents", rows: rows(d.top_agents, 'agent', 'visitors') }), _jsx(StatsList, { link: Link, title: "Agent tool calls", rows: rows(d.agent_tool_calls, 'tool', 'events') }), _jsx(StatsList, { link: Link, title: "Searches that found nothing", rows: rows(d.gaps, 'query', 'events') })] })] }));
|
|
80
|
+
const controls = (_jsxs(_Fragment, { children: [_jsx("div", { role: "group", "aria-label": "Window", children: ranges.map(n => (_jsx(Link, { href: at(filter, n), "aria-current": n === days ? 'page' : undefined, prefetch: false, children: windowLabel(n) }, n))) }), _jsx("div", { children: _jsx(Link, { href: at(filter, days, !apart), role: "switch", "aria-checked": !apart, "aria-label": "Agents counted among visitors", title: "Agents counted among visitors", prefetch: false, children: _jsxs("svg", { className: "chart-icon", viewBox: "0 0 12 12", "aria-hidden": true, children: [_jsx("path", { d: "M6 1.5v2M1 6.5v2M11 6.5v2", stroke: "currentColor" }), _jsx("rect", { x: "2.5", y: "3.5", width: "7", height: "7", rx: "1.5", fill: "none", stroke: "currentColor" }), _jsx("circle", { cx: "4.75", cy: "7", r: "0.9", fill: "currentColor" }), _jsx("circle", { cx: "7.25", cy: "7", r: "0.9", fill: "currentColor" })] }) }) })] }));
|
|
81
|
+
return (_jsxs("div", { className: "stats", children: [narrowed.length > 0 && (_jsxs("div", { className: "stats-filters", children: [_jsx("ul", { "aria-label": "Narrowed to", children: narrowed.map(k => (_jsx("li", { children: _jsxs(Link, { href: at({ ...filter, [k]: undefined }), title: "Remove", prefetch: false, children: [NAMES[k], ": ", k === 'country' ? country(filter[k]) : filter[k], " \u00D7"] }) }, k))) }), start < counted && d.counted_from && (_jsxs("p", { children: ["Narrowed figures start on ", d.counted_from, ", the first day this site counted its own visits."] }))] })), _jsx(StatsFigures, { digest: d, first: first, apart: apart, controls: controls }), _jsxs("div", { className: "stats-lists", children: [_jsx(StatsList, { link: Link, title: "Pages", rows: rows(d.top_pages, 'page', 'visitors'), to: to('page') }), _jsx(StatsList, { link: Link, title: "Entry pages", rows: rows(d.entry_pages, 'page', 'visitors'), to: to('entry') }), _jsx(StatsList, { link: Link, title: "Sources", rows: rows(d.top_sources, 'source', 'visitors'), to: to('source') }), _jsx(StatsList, { link: Link, title: "From X posts", rows: rows(d.from_posts, 'page', 'visitors'), to: v => at({ ...filter, source: 'x', entry: v }) }), _jsx(StatsList, { link: Link, title: "Sibling sites", rows: rows(d.sibling_referrals, 'source', 'visitors'), to: to('source') }), _jsx(StatsList, { link: Link, title: "Countries", rows: rows(d.countries, 'country', 'visitors'), label: country, to: to('country') }), _jsx(StatsList, { link: Link, title: "Devices", rows: rows(d.devices, 'device', 'visitors'), to: to('device') }), _jsx(StatsList, { link: Link, title: "Agents", rows: rows(d.top_agents, 'agent', 'visitors') }), _jsx(StatsList, { link: Link, title: "Agent tool calls", rows: rows(d.agent_tool_calls, 'tool', 'events') }), _jsx(StatsList, { link: Link, title: "Searches that found nothing", rows: rows(d.gaps, 'query', 'events'), empty: "None in this window." })] })] }));
|
|
79
82
|
}
|
|
80
83
|
/**
|
|
81
84
|
* A stats page's door: `children` and a sign-out button for a signed-in
|
package/dist/store.d.ts
CHANGED
|
@@ -47,6 +47,13 @@ export type WikiRevisionEntry = Omit<WikiRevisionRow, 'content'> & {
|
|
|
47
47
|
changes: HistoryChange[];
|
|
48
48
|
};
|
|
49
49
|
export type ListSort = 'title' | 'recent' | 'date';
|
|
50
|
+
/** A search's scope and page. `content` adds each row's blocks. */
|
|
51
|
+
export interface SearchOptions {
|
|
52
|
+
tagPath?: string | null;
|
|
53
|
+
skip?: number;
|
|
54
|
+
take?: number;
|
|
55
|
+
content?: boolean;
|
|
56
|
+
}
|
|
50
57
|
export interface WikiStoreOptions<B extends DiffBlock> {
|
|
51
58
|
sql: Sql;
|
|
52
59
|
/** The site every read and write is scoped to. "" in an app that serves one wiki. */
|
|
@@ -169,17 +176,14 @@ export declare function wikiStore<B extends DiffBlock = DiffBlock>(o: WikiStoreO
|
|
|
169
176
|
pages: number;
|
|
170
177
|
updated: Date | null;
|
|
171
178
|
}>;
|
|
172
|
-
search: (query: string, { tagPath, skip, take, content: withContent }?: {
|
|
173
|
-
tagPath?: string | null;
|
|
174
|
-
skip?: number;
|
|
175
|
-
take?: number;
|
|
176
|
-
content?: boolean;
|
|
177
|
-
}) => Promise<{
|
|
179
|
+
search: (query: string, options?: SearchOptions) => Promise<Awaited<ReturnType<(query: string, { tagPath, skip, take, content: withContent }?: SearchOptions) => Promise<{
|
|
178
180
|
items: (WikiPageListing & {
|
|
179
181
|
score: number;
|
|
180
182
|
content?: B[];
|
|
181
183
|
})[];
|
|
182
184
|
total: number;
|
|
185
|
+
}>>> & {
|
|
186
|
+
corrected?: string;
|
|
183
187
|
}>;
|
|
184
188
|
history: (pageId: string, { limit, offset }?: {
|
|
185
189
|
limit?: number;
|
package/dist/store.js
CHANGED
|
@@ -18,7 +18,7 @@
|
|
|
18
18
|
import { computeRevisionDiff } from './revisions.js';
|
|
19
19
|
import { bump } from './versioning.js';
|
|
20
20
|
import { withChanges } from './history.js';
|
|
21
|
-
import { proseSql, escapeLikeTerm, HEADLINE_OPTIONS, FTS_RANK_NORMALIZATION } from './search.js';
|
|
21
|
+
import { proseSql, escapeLikeTerm, fullTextQuery, correctQuery, HEADLINE_OPTIONS, FTS_RANK_NORMALIZATION } from './search.js';
|
|
22
22
|
import { openingText, matchSnippet } from './text.js';
|
|
23
23
|
import { describe } from './metadata.js';
|
|
24
24
|
import { decodeEntities } from './markdown.js';
|
|
@@ -132,7 +132,7 @@ export function wikiStore(o) {
|
|
|
132
132
|
* page is never a result. A tier-3 row's snippet is Postgres' own headline,
|
|
133
133
|
* because a stem match has no literal substring to window on.
|
|
134
134
|
*/
|
|
135
|
-
const
|
|
135
|
+
const ranked = async (query, { tagPath = null, skip = 0, take = 25, content: withContent = false } = {}) => {
|
|
136
136
|
const term = escapeLikeTerm(query);
|
|
137
137
|
if (!term)
|
|
138
138
|
return { items: [], total: 0 };
|
|
@@ -163,7 +163,7 @@ export function wikiStore(o) {
|
|
|
163
163
|
SELECT ${COLUMNS}, tier, total,
|
|
164
164
|
CASE WHEN tier = 3 THEN ts_headline('english', ${proseSql()}, tsq, '${HEADLINE_OPTIONS}') END AS headline
|
|
165
165
|
FROM (SELECT count(*)::int AS total FROM matched) t LEFT JOIN paged ON true
|
|
166
|
-
ORDER BY tier, CASE WHEN tier = 0 THEN title END, CASE WHEN tier = 3 THEN fts END DESC, "updatedAt" DESC`, query
|
|
166
|
+
ORDER BY tier, CASE WHEN tier = 0 THEN title END, CASE WHEN tier = 3 THEN fts END DESC, "updatedAt" DESC`, fullTextQuery(query), `${term}%`, `%${term}%`, hidden, site, tagPath, take, skip);
|
|
167
167
|
const items = found.filter(hit => hit.id !== null).map(({ tier, total: _, headline, content, ...row }) => ({
|
|
168
168
|
...row,
|
|
169
169
|
...(withContent ? { content: content } : {}),
|
|
@@ -173,6 +173,26 @@ export function wikiStore(o) {
|
|
|
173
173
|
}));
|
|
174
174
|
return { items, total: found[0]?.total ?? 0 };
|
|
175
175
|
};
|
|
176
|
+
/** Every word of the searchable titles, with how many titles use it: what a misspelling is corrected against. */
|
|
177
|
+
const titleWords = async (tagPath) => new Map((await rows(`SELECT word, count(*)::int AS n FROM ${PAGE} p, regexp_split_to_table(lower(p.title), '[^[:alnum:]]+') word
|
|
178
|
+
WHERE p."siteId" = $1 AND word <> '' AND ($2::text IS NULL OR p."tagPath" = $2) AND ($2::text IS NOT NULL OR NOT ${hiddenSql('$3')})
|
|
179
|
+
GROUP BY 1`, site, tagPath, hidden)).map(r => [r.word, r.n]));
|
|
180
|
+
/**
|
|
181
|
+
* {@link ranked}, and when its first page is empty, the same search with each
|
|
182
|
+
* misspelled word corrected against the titles' own words (`correctQuery`).
|
|
183
|
+
* A result found that way carries `corrected`, the query that found it, so a
|
|
184
|
+
* page can say what it searched for instead.
|
|
185
|
+
*/
|
|
186
|
+
const search = async (query, options = {}) => {
|
|
187
|
+
const found = await ranked(query, options);
|
|
188
|
+
if (found.total || options.skip)
|
|
189
|
+
return found;
|
|
190
|
+
const corrected = correctQuery(query, await titleWords(options.tagPath ?? null));
|
|
191
|
+
if (!corrected)
|
|
192
|
+
return found;
|
|
193
|
+
const again = await ranked(corrected, options);
|
|
194
|
+
return again.total ? { ...again, corrected } : found;
|
|
195
|
+
};
|
|
176
196
|
/** A revision query's guard: `$1`, its page, is this site's. Another site's page id finds nothing. */
|
|
177
197
|
const OWN_PAGE = `"pageId" IN (SELECT id FROM ${PAGE} WHERE id = $1 AND "siteId" = $2)`;
|
|
178
198
|
/**
|
package/dist/wiki-tools.d.ts
CHANGED
|
@@ -75,5 +75,6 @@ export declare function searchWiki<B extends DiffBlock>(o: Pick<WikiToolsOptions
|
|
|
75
75
|
}): Promise<{
|
|
76
76
|
items: WikiRow[];
|
|
77
77
|
total: number;
|
|
78
|
+
corrected?: string;
|
|
78
79
|
}>;
|
|
79
80
|
export declare function wikiReadTools<B extends DiffBlock>(o: WikiToolsOptions<B>): WikiTool[];
|
package/dist/wiki-tools.js
CHANGED
|
@@ -54,8 +54,8 @@ const rowFor = (o) => (p) => {
|
|
|
54
54
|
export async function searchWiki(o, query, { tagPath, page = 1, size = 20 } = {}) {
|
|
55
55
|
if (offFor(o)(tagPath))
|
|
56
56
|
return { items: [], total: 0 };
|
|
57
|
-
const { items, total } = await o.store.search(query, { tagPath, skip: (page - 1) * size, take: size, content: true });
|
|
58
|
-
return { items: items.map(rowFor(o)), total };
|
|
57
|
+
const { items, total, corrected } = await o.store.search(query, { tagPath, skip: (page - 1) * size, take: size, content: true });
|
|
58
|
+
return { items: items.map(rowFor(o)), total, ...(corrected ? { corrected } : {}) };
|
|
59
59
|
}
|
|
60
60
|
export function wikiReadTools(o) {
|
|
61
61
|
const { store, tree, noun } = o;
|
|
@@ -116,7 +116,7 @@ export function wikiReadTools(o) {
|
|
|
116
116
|
skill: { id: 'search', tags: ['search', 'wiki'] },
|
|
117
117
|
description: `Keyword search across ${noun}: the first call when you know what you are looking for but not where it lives. ` +
|
|
118
118
|
'Matches titles and body prose, ranked in four tiers: titles starting with the term, titles containing it, literal body hits, then English full-text hits by relevance, which stem words and drop stopwords. ' +
|
|
119
|
-
'Not semantic: one distinctive word beats a sentence. ' +
|
|
119
|
+
'Not semantic: one distinctive word beats a sentence. A query that finds nothing is retried with its misspelled words corrected, and `corrected` then names what was searched. ' +
|
|
120
120
|
'Each row carries a `path`, its size in `chars` and a snippet windowed on the match. Feed the paths to get_pages in one call rather than looping get_page. ' +
|
|
121
121
|
'Narrow with tagPath when a term is common. Defaults: page 1, pageSize 20 (max 50); keep paging while `hasMore` is true.',
|
|
122
122
|
inputSchema: {
|
|
@@ -137,9 +137,10 @@ export function wikiReadTools(o) {
|
|
|
137
137
|
const size = r.num('pageSize', 20, 1, 50);
|
|
138
138
|
if (!query)
|
|
139
139
|
throw new McpToolError('`query` is empty. Pass a keyword to search for, or call list_pages to enumerate a branch instead.');
|
|
140
|
-
const { items, total } = await searchWiki(o, query, { tagPath, page, size });
|
|
140
|
+
const { items, total, corrected } = await searchWiki(o, query, { tagPath, page, size });
|
|
141
141
|
return withAdjustments(r, {
|
|
142
142
|
query,
|
|
143
|
+
...(corrected ? { corrected } : {}),
|
|
143
144
|
...listEnvelope(items, total, page, size, o.noMatch?.(query, tagPath)
|
|
144
145
|
?? `Nothing matches "${query}"${tagPath ? ` under "${tagPath}"` : ''}. Search is keyword, not semantic: try one distinctive word${tagPath ? ', or drop the tagPath filter' : ''}, or call get_categories to see what ${noun} covers.`),
|
|
145
146
|
});
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "wiki-formant",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.104.0",
|
|
4
4
|
"description": "The portable half of a wiki: derived taxonomy and facet controls, a version-negotiating MCP transport, markdown twins, block rendering, a rich-text editor engine, conditional-GET plumbing, and the headers and CSP every origin sends.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|