@bitbaum/searchkit 0.0.0-stage → 0.2.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Cato
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,164 @@
1
- # Temporary Holding Version
1
+ # searchkit
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Search that finds what was meant.
4
+
5
+ You start typing and you get where you need to go. One wrong letter in a
6
+ name — the SoundCloud failure, where an artist is unfindable because you typed
7
+ `kayne` — still finds it. Zero dependencies; React optional.
8
+
9
+ A survey of the fleet on 2026-10-12 found about 25 search boxes, and one with
10
+ any typo tolerance (substrata's own engine). The rest were
11
+ `text.toLowerCase().includes(q)` or an SQL `ILIKE '%q%'`: no typos, no
12
+ ranking, a few without escaping `%`. Every new site would have written a 26th.
13
+
14
+ ## Where it lives, and how a site adopts it
15
+
16
+ This repo, created by bitbaum/fleet's `package-repos` workflow from
17
+ `bitbaum/orangecat` `packages/searchkit` (its history before 0.2.2 is there).
18
+ A site installs a release from npm, which Dependabot moves forward on its own:
19
+
20
+ ```jsonc
21
+ // package.json
22
+ "@bitbaum/searchkit": "^0.2.2"
23
+ ```
24
+
25
+ It ships compiled JavaScript in `dist/`, committed beside the source the way
26
+ `@bitbaum/whenkit` does, so nothing has to transpile it: Next.js, Vitest, and
27
+ Node's own test runner (which refuses TypeScript inside `node_modules`) all
28
+ load it as is. Its tests are in `test/` and run with plain Node (`npm test`);
29
+ one of them fails while `dist/` is not the build of `src/` (`npm run build`).
30
+
31
+ **Every search box in the fleet should go through it.** The first adopter,
32
+ OrangeCat, runs it in three places: the ⌘K palette's own lists (pages, your
33
+ things), `/api/search` (vocabulary corrections handed to Postgres, then
34
+ ranking), and `/discover`.
35
+
36
+ ## What it ships
37
+
38
+ ```ts
39
+ import { createSearch } from '@bitbaum/searchkit';
40
+
41
+ const people = createSearch(rows, {
42
+ fields: [
43
+ { name: 'name', get: p => p.name }, // most important first
44
+ { name: 'handle', get: p => p.username },
45
+ { name: 'bio', get: p => p.bio },
46
+ ],
47
+ rank: p => Math.log1p(p.followers), // last tie-break only
48
+ });
49
+
50
+ people.search('kayne wset'); // → Kanye West, with ranges to mark
51
+ ```
52
+
53
+ ### Matching — what counts as the same word
54
+
55
+ | typed | finds | how |
56
+ | ------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
57
+ | `cqto` | Cato | one wrong letter (budget: 0 typos under 4 letters, 1 from 4, 2 from 8) |
58
+ | `kayne` | Kanye | a swap is ONE typo (optimal string alignment) |
59
+ | `koki` | Loki | a wrong first letter still matches — ranked as two typos |
60
+ | `lo` | Loki | the start of a word, while typing |
61
+ | `bitcon` | Bitcoin | a typo in the start of a word |
62
+ | `zurich`, `mueller` | Zürich, Müller | accents folded; umlauts also indexed spelled out |
63
+ | `orangecat`, `orange cat` | OrangeCat, Orange Cat | camelCase and separators split; adjacent words also indexed joined (ranked as a typo when the text wrote them apart, so `catto` still finds Cato first) |
64
+ | `cato xylophone` | Cato | a word that matches nothing is set aside, not fatal |
65
+ | `jon smith` | Jon Snow, John Smith | when no item has every word, the most words any item has are kept — whichever words they are |
66
+
67
+ The budget is the one Algolia and Typesense use. Meilisearch's (one typo from
68
+ 5 letters, a first-letter typo costs two) cannot find a mistyped 4-letter name
69
+ at all, and a 5–8-letter one whose first letter slipped.
70
+
71
+ ### Ranking — a tie-break, not a weighted sum
72
+
73
+ Each rule only decides between results the rules above it called equal:
74
+ typos → words matched → field → exactness (the whole field typed, then whole
75
+ words, then prefixes) → proximity → position in the field → your `rank` →
76
+ shorter field. Sums need tuning and drift; a long description outscores a
77
+ short exact title. This is Algolia's published order, adjusted for names.
78
+
79
+ ### For a search that runs in a database
80
+
81
+ The forgiving part does not need to live in the database. Keep the
82
+ collection's words (`createVocabulary`), expand what was typed into the words
83
+ it could mean, and send those along:
84
+
85
+ ```ts
86
+ import { corrections, createVocabulary, didYouMean } from '@bitbaum/searchkit';
87
+
88
+ const vocab = createVocabulary(wordsWithCounts); // e.g. from a cached SQL view
89
+ corrections('cqto', vocab); // → ["cato"]: OR these into your tsquery / ILIKE
90
+ didYouMean('cqto', vocab); // → "cato", for the zero-result screen
91
+ ```
92
+
93
+ Then rerank the rows the database returned with `createSearch` — one ranking
94
+ for in-memory lists and database results alike.
95
+
96
+ Why not do it in Postgres: `pg_trgm` scores `kanye`/`kayne` at 0.2 and
97
+ `cqto`/`cato` at 0.4, under any useful threshold, and `fuzzystrmatch`'s
98
+ `levenshtein` counts a swap as two edits. Use trigrams for recall at a low
99
+ threshold if you like; decide typos here.
100
+
101
+ ### Your own spelling rule — `normalize`
102
+
103
+ `SearchSpec.normalize(word)` runs on every word, both sides, after the
104
+ standard folding. Heidi reads Swiss German as people type it: "ie" as "i",
105
+ doubled letters as one, so `gruzi mittenand` finds "Grüezi mitenand". Keep it
106
+ to spelling; meaning belongs elsewhere. `expandQuery`/`corrections` take the
107
+ same function so a vocabulary agrees with the index. The typo budget stays the
108
+ WRITTEN word's: the rule turns "vill" into "vil", and a slip in "vill" is
109
+ still forgiven as in any four-letter word.
110
+
111
+ ### Rows a database already found — `rerank`
112
+
113
+ `rerank(rows, spec, query)` ranks them by what was typed and keeps the rows
114
+ it cannot place (matched on a field you did not pass, or by stemming) after
115
+ the ranked ones, in their original order. OrangeCat's `/api/search` ranks
116
+ Postgres candidates this way.
117
+
118
+ ### Highlighting
119
+
120
+ `hit.matches[field]` holds ranges into the original text (a typo match marks
121
+ the whole word; a prefix only what was typed). `highlightQuery(text, q)` marks
122
+ words that start with a typed word, for text that was not searched. Both
123
+ return plain parts — never HTML. `snippet(text, ranges, { maxLength })` cuts
124
+ a long description to the window around its first match, at word
125
+ boundaries, with "…" where it was cut, and re-bases the ranges onto it.
126
+
127
+ ### React (`@bitbaum/searchkit/react`)
128
+
129
+ - `useSearchAsYouType(text, (q, signal) => fetch(...), { debounceMs: 100 })` —
130
+ each keystroke aborts the request before it; a late answer to an older
131
+ query is dropped.
132
+ - `<Highlight text ranges | query markClassName />` — unstyled `<mark>`s.
133
+
134
+ ### Proving it — `typoCases` and `evaluate`
135
+
136
+ ```ts
137
+ const cases = typoCases(names); // every one-slip misspelling of every name
138
+ const report = await evaluate(cases, q => search(q).map(h => h.id));
139
+ // { successAt1, successAt3, mrr, zeroResults, misses }
140
+ ```
141
+
142
+ Slips are generated, not imagined: a neighbouring key (QWERTY and QWERTZ), a
143
+ dropped letter, a doubled letter, a swapped pair. Put it in CI and "one slip
144
+ still finds it" fails the day it stops being true. This package's own suite
145
+ runs 571 such cases over eight names at success@1 = 1.
146
+
147
+ ## What it does not do
148
+
149
+ - **Meaning.** "someone to fix my bike" finding "Velo repair" is embeddings'
150
+ job (descriptive queries, never names — dense retrieval loses to lexical on
151
+ names by tens of points). Fuse such results with Reciprocal Rank Fusion
152
+ (k = 60) if you add them.
153
+ - **Security.** Whether a row may be shown is decided before it reaches this
154
+ package. An in-memory matcher has no business behind a visibility gate.
155
+ - **Phonetics.** Schmidt/Schmitt, Meier/Mayer. Useful for person names only,
156
+ harmful elsewhere; not yet built.
157
+
158
+ ## Sources
159
+
160
+ Algolia typo tolerance and ranking criteria; Typesense search parameters;
161
+ Meilisearch typo internals; Damerau (1964), Pollock & Zamora (1983) on single
162
+ edits; Schulz & Mihov (2002) on Levenshtein automata; SymSpell; Nielsen's
163
+ response-time limits; Cormack, Clarke & Büttcher (SIGIR 2009) on RRF;
164
+ Sciavolino et al. (EMNLP 2021) on dense retrieval and entity names.
@@ -0,0 +1,24 @@
1
+ /**
2
+ * How far apart two words are, in typos.
3
+ *
4
+ * A typo is one of the four things fingers do: a wrong letter (cqto → cato),
5
+ * a missing letter (cto), an extra letter (caato), or two letters swapped
6
+ * (ctao). Counting a swap as ONE edit — optimal string alignment, the
7
+ * restricted Damerau–Levenshtein distance — matters: plain Levenshtein counts
8
+ * "ctao" as two, which spends a whole short word's budget on one slip.
9
+ *
10
+ * Both functions stop as soon as the answer must exceed `max`, so checking a
11
+ * query word against thousands of vocabulary words stays cheap.
12
+ */
13
+ /**
14
+ * Optimal string alignment distance, or `max + 1` when it is larger than
15
+ * `max`. Strings are compared by code point.
16
+ */
17
+ export declare function editDistance(a: string, b: string, max?: number): number;
18
+ /**
19
+ * How many typos it takes to turn `typed` into the START of `word` — the
20
+ * distance that matters while someone is still typing: "bitcon" is one typo
21
+ * from "bitcoin", and "bitco" is zero typos from it. Returns the smallest
22
+ * distance between `typed` and any prefix of `word`, or `max + 1`.
23
+ */
24
+ export declare function prefixDistance(typed: string, word: string, max?: number): number;
@@ -0,0 +1,91 @@
1
+ /**
2
+ * How far apart two words are, in typos.
3
+ *
4
+ * A typo is one of the four things fingers do: a wrong letter (cqto → cato),
5
+ * a missing letter (cto), an extra letter (caato), or two letters swapped
6
+ * (ctao). Counting a swap as ONE edit — optimal string alignment, the
7
+ * restricted Damerau–Levenshtein distance — matters: plain Levenshtein counts
8
+ * "ctao" as two, which spends a whole short word's budget on one slip.
9
+ *
10
+ * Both functions stop as soon as the answer must exceed `max`, so checking a
11
+ * query word against thousands of vocabulary words stays cheap.
12
+ */
13
+ /**
14
+ * Optimal string alignment distance, or `max + 1` when it is larger than
15
+ * `max`. Strings are compared by code point.
16
+ */
17
+ export function editDistance(a, b, max = Infinity) {
18
+ const s = Array.from(a);
19
+ const t = Array.from(b);
20
+ if (Math.abs(s.length - t.length) > max) {
21
+ return max + 1;
22
+ }
23
+ if (s.length === 0 || t.length === 0) {
24
+ const d = Math.max(s.length, t.length);
25
+ return d > max ? max + 1 : d;
26
+ }
27
+ let prevPrev = [];
28
+ let prev = Array.from({ length: t.length + 1 }, (_, j) => j);
29
+ for (let i = 1; i <= s.length; i++) {
30
+ const row = [i];
31
+ let rowMin = i;
32
+ for (let j = 1; j <= t.length; j++) {
33
+ const cost = s[i - 1] === t[j - 1] ? 0 : 1;
34
+ let d = Math.min(prev[j] + 1, row[j - 1] + 1, prev[j - 1] + cost);
35
+ if (i > 1 && j > 1 && s[i - 1] === t[j - 2] && s[i - 2] === t[j - 1]) {
36
+ d = Math.min(d, prevPrev[j - 2] + 1);
37
+ }
38
+ row.push(d);
39
+ if (d < rowMin) {
40
+ rowMin = d;
41
+ }
42
+ }
43
+ if (rowMin > max) {
44
+ return max + 1;
45
+ }
46
+ prevPrev = prev;
47
+ prev = row;
48
+ }
49
+ const d = prev[t.length];
50
+ return d > max ? max + 1 : d;
51
+ }
52
+ /**
53
+ * How many typos it takes to turn `typed` into the START of `word` — the
54
+ * distance that matters while someone is still typing: "bitcon" is one typo
55
+ * from "bitcoin", and "bitco" is zero typos from it. Returns the smallest
56
+ * distance between `typed` and any prefix of `word`, or `max + 1`.
57
+ */
58
+ export function prefixDistance(typed, word, max = Infinity) {
59
+ const s = Array.from(typed);
60
+ const t = Array.from(word);
61
+ if (s.length === 0) {
62
+ return 0;
63
+ }
64
+ // A prefix longer than typed + max can never be within max; one shorter
65
+ // than typed - max neither. Cap the work at the first typed + max letters.
66
+ const cols = Math.min(t.length, s.length + max);
67
+ let prevPrev = [];
68
+ let prev = Array.from({ length: cols + 1 }, (_, j) => j);
69
+ for (let i = 1; i <= s.length; i++) {
70
+ const row = [i];
71
+ let rowMin = i;
72
+ for (let j = 1; j <= cols; j++) {
73
+ const cost = s[i - 1] === t[j - 1] ? 0 : 1;
74
+ let d = Math.min(prev[j] + 1, row[j - 1] + 1, prev[j - 1] + cost);
75
+ if (i > 1 && j > 1 && s[i - 1] === t[j - 2] && s[i - 2] === t[j - 1]) {
76
+ d = Math.min(d, prevPrev[j - 2] + 1);
77
+ }
78
+ row.push(d);
79
+ if (d < rowMin) {
80
+ rowMin = d;
81
+ }
82
+ }
83
+ if (rowMin > max) {
84
+ return max + 1;
85
+ }
86
+ prevPrev = prev;
87
+ prev = row;
88
+ }
89
+ const best = Math.min(...prev);
90
+ return best > max ? max + 1 : best;
91
+ }
@@ -0,0 +1,76 @@
1
+ import { type TypoPolicy } from './typo.ts';
2
+ import { type Vocabulary } from './vocabulary.ts';
3
+ export interface SearchField<T> {
4
+ /** For your own debugging and for `Hit.matches`. */
5
+ name: string;
6
+ get: (item: T) => string | readonly string[] | null | undefined;
7
+ }
8
+ export interface SearchSpec<T> {
9
+ /** Most important first: a title before a description. */
10
+ fields: readonly SearchField<T>[];
11
+ /** The last tie-break, higher first: popularity, recency. */
12
+ rank?: (item: T) => number;
13
+ typos?: TypoPolicy;
14
+ /**
15
+ * Let typed words match the start of longer words. "all" (default): every
16
+ * word, so "bit pay" finds "Bitcoin payments". "last": only the word being
17
+ * typed. "none": whole words only.
18
+ */
19
+ prefix?: 'all' | 'last' | 'none';
20
+ /**
21
+ * A site's own spelling rule, applied to every word on both sides after
22
+ * the standard folding: Swiss German written as people type it ("ie" and
23
+ * "i", doubled letters), a product's abbreviations. Keep it to how a word
24
+ * is SPELT; meaning belongs elsewhere.
25
+ */
26
+ normalize?: (word: string) => string;
27
+ }
28
+ /** Where a typed word landed in one field, as offsets into the original text. */
29
+ export interface Range {
30
+ start: number;
31
+ end: number;
32
+ }
33
+ export interface Hit<T> {
34
+ item: T;
35
+ /** Index of the item in the list it was searched from. */
36
+ index: number;
37
+ /** The tie-break values, in rule order. Lower sorts first. */
38
+ score: readonly number[];
39
+ /** How many typed words matched. */
40
+ words: number;
41
+ typos: number;
42
+ /** Matched ranges per field name, for highlighting. */
43
+ matches: Record<string, Range[]>;
44
+ }
45
+ export interface SearchOptions {
46
+ limit?: number;
47
+ /**
48
+ * When no record has every typed word, drop the last words until some do.
49
+ * Default true.
50
+ */
51
+ relax?: boolean;
52
+ }
53
+ export interface Searcher<T> {
54
+ search(query: string, options?: SearchOptions): Hit<T>[];
55
+ /** The words of the list, for `didYouMean` and for expanding a query sent elsewhere. */
56
+ readonly vocabulary: Vocabulary;
57
+ }
58
+ /**
59
+ * Index `items` once; search them as often as you like. Rebuild when the list
60
+ * changes — indexing is linear and cheap next to a network round trip.
61
+ */
62
+ export declare function createSearch<T>(items: readonly T[], spec: SearchSpec<T>): Searcher<T>;
63
+ /** Sort order for hits: the tie-break values, compared left to right. */
64
+ export declare function compareHits<T>(a: Hit<T>, b: Hit<T>): number;
65
+ /** Search a list once, without keeping an index. */
66
+ export declare function search<T>(items: readonly T[], spec: SearchSpec<T>, query: string, options?: SearchOptions): Hit<T>[];
67
+ /**
68
+ * Rank rows another engine already found — a database, an API — by what was
69
+ * typed. Rows the ranker cannot place (they matched on a field you did not
70
+ * pass, or through stemming) keep their original order after the ranked
71
+ * ones: still results, just later ones.
72
+ */
73
+ export declare function rerank<T>(rows: readonly T[], spec: SearchSpec<T>, query: string): Array<{
74
+ item: T;
75
+ hit: Hit<T> | null;
76
+ }>;