@dokebi/search 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AI.md +134 -0
- package/LICENSE +21 -0
- package/README.md +8 -2
- package/dist/api.d.ts +78 -0
- package/dist/api.js +1 -0
- package/dist/cursor.d.ts +19 -0
- package/dist/cursor.js +68 -0
- package/dist/dependency.d.ts +4 -0
- package/dist/dependency.js +13 -0
- package/dist/errors.d.ts +6 -0
- package/dist/errors.js +58 -0
- package/dist/index.d.ts +8 -0
- package/dist/index.js +7 -0
- package/dist/indexes.d.ts +27 -0
- package/dist/indexes.js +87 -0
- package/dist/install.d.ts +37 -0
- package/dist/install.js +56 -0
- package/dist/options.d.ts +23 -0
- package/dist/options.js +73 -0
- package/dist/plugin.d.ts +6 -0
- package/dist/plugin.js +671 -0
- package/dist/sql.d.ts +32 -0
- package/dist/sql.js +104 -0
- package/dist/store.d.ts +66 -0
- package/dist/store.js +137 -0
- package/dist/terms.d.ts +31 -0
- package/dist/terms.js +58 -0
- package/options.schema.json +95 -0
- package/package.json +86 -4
- package/templates/indexes.ts +10 -0
- package/templates/migrations/dokebi_search.ts +92 -0
package/AI.md
ADDED
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# AI guide — @dokebi/search
|
|
2
|
+
|
|
3
|
+
Search in the lists of an application's own rows (posts, comments, members, log targets) that stays **bounded on a table that only grows**: @dokebi/db AI.md section 6.10, once. The words of each item are kept in a term table whose key ends in the display order, written **in the transaction of the text**; Korean and other languages that attach endings are found by the stem (word prefixes); one word is one index lookup whatever its frequency; several words are driven by the rarest, with a cap on the rows one call and one whole search examine, kept in a signed cursor. Answers are item ids in display order: your code reads its rows. PostgreSQL, MySQL 8, MariaDB 10.6+ through @dokebi/db (no MongoDB). Node >=24.0.0, core ^0.4.2, @dokebi/db ^0.4.1. Decision: [D-0170](https://git.nyx-zone.com/dokebi/framework/-/blob/main/docs/decisions/D-0170-search.md).
|
|
4
|
+
|
|
5
|
+
## 1. Install and set up
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
npx dokebi add @dokebi/db --adapter pg # the main connection, if the app has none yet
|
|
9
|
+
npx dokebi add @dokebi/search # writes src/infra/search/indexes.ts, the migration and a cursor secret in dokebi.yaml
|
|
10
|
+
npx dokebi-migrate up # creates dokebi_search_terms, dokebi_search_counts, dokebi_search_indexes
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
> `npx dokebi` is the @dokebi/cli installed in the project (`dokebi init` adds it to devDependencies). The npm package named `dokebi` is not this CLI, so never run `npx -y dokebi`. Without a local install use `npx -y @dokebi/cli …`.
|
|
14
|
+
|
|
15
|
+
- **The tables live with your rows** (the dokebi-db connection `connection`, default `main`), so `index` shares the transaction that saves the text. At start the plugin derives its own pool of that connection (`pool.max`, default 4) for `find`, and checks the tables: without the migration the start fails with `TABLE_MISSING` (its fix: `npx dokebi-migrate up`).
|
|
16
|
+
- **The cursor secret**: `dokebi add` writes a generated one into `secrets`; `npx dokebi config secret dokebi-search.secrets` makes one later. Without it the start fails with `SECRET_MISSING`. The variable of `secretsEnv` (default `DOKEBI_SEARCH_SECRETS`, comma-separated) wins over dokebi.yaml.
|
|
17
|
+
- **The indexes** are `src/infra/search/indexes.ts` (yours; `dokebi add` writes an example):
|
|
18
|
+
|
|
19
|
+
```ts snippet
|
|
20
|
+
import { defineSearchIndexes } from '@dokebi/search';
|
|
21
|
+
|
|
22
|
+
export default defineSearchIndexes({
|
|
23
|
+
posts: { fields: { title: {}, body: {}, author: { prefix: 0 } } },
|
|
24
|
+
members: { fields: { name: {}, nickname: {}, email: {} } },
|
|
25
|
+
});
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Index and field names are a-z then a-z, 0-9 or `_` (at most 40); at most 64 indexes of 32 fields. Per field: `prefix` (0–32, default 10) is the longest word prefix kept, so `도깨비` finds `도깨비가` and `도깨비를`, and `note` finds `notebook` <!-- docs-check: allow-hangul --> (0 keeps whole words only: a name list where a prefix would match too much); `minLength` (1–prefix, default 1) the shortest. A bad declaration fails the start (`INVALID_INDEXES`). The declaration's fingerprint is stored: after you change a field's rules the start warns `SEARCH_INDEX_CHANGED` and `status()` says `stale` until you indexed the rows again and called `markCurrent(index)`.
|
|
29
|
+
|
|
30
|
+
## 2. Configuration `options.dokebi-search`
|
|
31
|
+
|
|
32
|
+
| Key | Type | Default | Description |
|
|
33
|
+
|---|---|---|---|
|
|
34
|
+
| `connection` | string | `main` | The dokebi-db connection of the database that holds your rows; `dokebi add` writes the migration into its folder |
|
|
35
|
+
| `pool.max` | integer 0–64 | 4 | Connections of the pool derived from it for `find`; 0 uses the connection pool |
|
|
36
|
+
| `examineLimit` | integer 100–100000 | 2000 | Most rows of the leading word one `find` of several words examines |
|
|
37
|
+
| `searchLimit` | integer 100–10000000 | 50000 | Most rows one search of several words examines over all its pages; at least `examineLimit` |
|
|
38
|
+
| `list.maxLimit` | integer 1–1000 | 100 | Most items one `find` returns (default page 20) |
|
|
39
|
+
| `counts.slots` | integer 1–64 | 4 | Rows a word's item count is split into |
|
|
40
|
+
| `counts.flushMs` | integer 100–60000 | 1000 | How often each worker adds its buffered word counts to the table |
|
|
41
|
+
| `secrets` | array | — | Cursor signing secrets (32+ characters); the first signs, all verify |
|
|
42
|
+
| `secretsEnv` | string | — | The environment variable that replaces `secrets` (default `DOKEBI_SEARCH_SECRETS`) |
|
|
43
|
+
|
|
44
|
+
## 3. Index items
|
|
45
|
+
|
|
46
|
+
```ts snippet
|
|
47
|
+
import { dependency as searchDependency } from '@dokebi/search/dependency';
|
|
48
|
+
|
|
49
|
+
const search = context.getDependency(searchDependency);
|
|
50
|
+
// Saving a post: the row and its terms in one transaction.
|
|
51
|
+
const client = await pool.connect();
|
|
52
|
+
try {
|
|
53
|
+
await client.query('BEGIN');
|
|
54
|
+
const { rows } = await client.query(
|
|
55
|
+
'INSERT INTO posts (board_id, title, body, author_name, created_at) VALUES ($1, $2, $3, $4, $5) RETURNING id',
|
|
56
|
+
[board, title, body, authorName, Date.now()],
|
|
57
|
+
);
|
|
58
|
+
await search.index(client, 'posts', {
|
|
59
|
+
id: String(rows[0].id),
|
|
60
|
+
sortKey: createdAt,
|
|
61
|
+
scopes: ['', `board:${board}`],
|
|
62
|
+
fields: { title, body, author: authorName },
|
|
63
|
+
});
|
|
64
|
+
await client.query('COMMIT');
|
|
65
|
+
} catch (error) {
|
|
66
|
+
await client.query('ROLLBACK');
|
|
67
|
+
throw error;
|
|
68
|
+
} finally {
|
|
69
|
+
client.release();
|
|
70
|
+
}
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
- **`index(tx, index, item)`** writes in the transaction of `tx` (the pg client of `BEGIN`…`COMMIT`, or the mysql2/mariadb connection of the transaction). `item` is the item's **whole** searchable text: `id` (1–128), `sortKey` (a safe integer: the display order, newest first — usually the creation time in ms), `scopes` (the filters a search is combined with, at most 32 of at most 64 characters; default `['']`, everything), `fields` (a text per declared field; absent or null has no terms; at most 1,000,000 characters). It reads the item's terms by the item index and writes only the difference; a new `sortKey` rewrites them. Call it in the transaction that changes the item's row, so that row's lock orders two writers of the same item.
|
|
74
|
+
- **Scopes** are how a search is filtered without reading other rows: an item is indexed under each, so a board and its ancestor categories are `['', 'board:12', 'cat:3', 'cat:3/7']` and a search names one. Each scope adds the item's terms again: keep them few.
|
|
75
|
+
- **`remove(tx, index, id)`** when the item is hidden or deleted → `{ removed }`. Items a viewer may not see must not be in the index (or in a scope that viewer searches): `find` answers ids, it does not check rights.
|
|
76
|
+
- **`indexMany(tx, index, items)`**: a first fill or a re-index, in batches of a few hundred per transaction from a job; then `flush()` and `markCurrent(index)`.
|
|
77
|
+
- **A display order that changes all the time** (views, likes) is not a `sortKey`: every change would rewrite every term row. Search in the index's order and rank a capped set of results yourself, saying so.
|
|
78
|
+
- **Word counts** (which word is rarest) are a hint the plugin adds after the write, outside your transaction: each worker buffers them and writes them every `counts.flushMs`, at drain and on `flush()`. A common word's count is never a row every writer waits on, and a rolled-back write leaves its count changed: the counts may drift, the results never do.
|
|
79
|
+
|
|
80
|
+
## 4. Search
|
|
81
|
+
|
|
82
|
+
```ts snippet
|
|
83
|
+
const page = await search.find('posts', {
|
|
84
|
+
field: 'body',
|
|
85
|
+
q: request.query.q,
|
|
86
|
+
scope: `board:${board}`,
|
|
87
|
+
limit: 20,
|
|
88
|
+
cursor: request.query.cursor,
|
|
89
|
+
});
|
|
90
|
+
const rows = await postsByIds(page.items.map((hit) => hit.id)); // keep the page's order
|
|
91
|
+
return { rows, next: page.next, partial: page.partial, stopped: page.stopped };
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
- **`find(index, { field, q, scope?, limit?, cursor? })`** → `{ items: [{ id, sortKey }], next?, partial, stopped?, examined }`. Every word of `q` is required (AND), 1–8 words after normalisation (NFKC, lower case, split on anything that is not a letter, mark or digit). A word matches the words that start with it up to the field's `prefix`; a longer word matches whole words only. A word shorter than `minLength`, no word or more than 8 is `INVALID_QUERY` (400).
|
|
95
|
+
- **One word** is one lookup in display order: a page reads `limit + 1` rows, whether the word is in three rows or in all of them.
|
|
96
|
+
- **Several words**: the rarest leads in display order and every other word is a key lookup. One call examines at most `examineLimit` rows; when it reaches that before the page is full it answers what it found, `next` and `partial: true`. Never show an empty page on its own: follow `next` (show that the search continues) until there are matches or no `next`. A search that examined `searchLimit` rows over its pages ends with `stopped: 'SEARCH_TOO_BROAD'` and no `next`: show what was found and ask for a narrower word or scope.
|
|
97
|
+
- **`next`** is signed (HMAC-SHA256 with `secrets`) and bound to its query: pass it back unchanged as `cursor` with the same `field`, `q` and `scope`; anything else is `INVALID_CURSOR` (400). The running total of a search is in it, so a client cannot reset it.
|
|
98
|
+
- **No relevance ranking**: results are in the index's display order. `examined` is the number of term rows the call read; keep it as a test of your lists (@dokebi/db AI.md 6.10: measure all five kinds — a word nothing has, a rare one, one in most rows, two characters, two common words that rarely meet).
|
|
99
|
+
- **`terms(index, field, text)`**: the terms a text gets (what can match); `words`, `terms` and `queryTerms` are exported as pure functions.
|
|
100
|
+
|
|
101
|
+
## 5. Logs, health and describe
|
|
102
|
+
|
|
103
|
+
debug ([D-0047](https://git.nyx-zone.com/dokebi/framework/-/blob/main/docs/decisions/D-0047-logging-and-no-silent-errors.md)): `SEARCH_SETUP` (connection, indexes, ready, ms), `SEARCH_SECRETS` (where the secrets came from and how many, never their values), `SEARCH_CONNECTED` (dialect, server version, pool, stale indexes), `SEARCH_STORE_FAILED` (operation, the full error; the caller gets `STORE_FAILED` with it as the cause), `SEARCH_TOO_BROAD`. info: `SEARCH_DB_RECOVERED`. warn: `SEARCH_DB_UNAVAILABLE`, `SEARCH_INDEX_CHANGED` (with its fix). error: `SEARCH_COUNTS_FAILED` (a count flush failed; kept for the next one; repeats counted once a minute). On a `startup: optional` connection whose database is down at start ([D-0127](https://git.nyx-zone.com/dokebi/framework/-/blob/main/docs/decisions/D-0127-optional-db-startup.md)) the app starts; calls answer `NOT_READY` (503) and reach the database again at most once a second. `dokebi describe` connects nothing: section `search` (connection, tables, indexes and fields, limits); calls answer `DESCRIBING`.
|
|
104
|
+
|
|
105
|
+
## 6. Errors
|
|
106
|
+
|
|
107
|
+
`SearchError` (`name`, `code`, `plugin: dokebi-search`, `fix`, `data`).
|
|
108
|
+
|
|
109
|
+
<!-- docs-check: complete-error-codes SearchErrorCode -->
|
|
110
|
+
| Code | When |
|
|
111
|
+
|---|---|
|
|
112
|
+
| `INVALID_OPTIONS` | options the schema refuses, `searchLimit` below `examineLimit`, bad secrets in the environment variable |
|
|
113
|
+
| `INVALID_INDEXES` | the entry has no declaration, or a name or rule outside its rule |
|
|
114
|
+
| `UNKNOWN_INDEX` | an index not declared |
|
|
115
|
+
| `UNKNOWN_FIELD` | a field the index does not declare |
|
|
116
|
+
| `INVALID_ITEM` | an item field outside its rule (`data.field`), a transaction client without `query` |
|
|
117
|
+
| `INVALID_QUERY` | no word, more than 8, one shorter than `minLength`, a bad scope or limit (400) |
|
|
118
|
+
| `INVALID_CURSOR` | a cursor this server did not sign, a changed one, or one of another query (400) |
|
|
119
|
+
| `SECRET_MISSING` | no `secrets` in dokebi.yaml and the variable of `secretsEnv` is empty |
|
|
120
|
+
| `UNKNOWN_CONNECTION` | `connection` is not a dokebi-db connection |
|
|
121
|
+
| `UNSUPPORTED_RESOURCE` | the connection is not a pg, mysql2/promise or mariadb pool (MongoDB) |
|
|
122
|
+
| `TABLE_MISSING` | the tables or their columns are missing at start (run the migration) |
|
|
123
|
+
| `NOT_READY` | the database of an optional connection is down (503) |
|
|
124
|
+
| `STORE_FAILED` | a statement failed (the driver error is the cause) |
|
|
125
|
+
| `DESCRIBING` | a call during `dokebi describe` |
|
|
126
|
+
|
|
127
|
+
## 7. Cautions
|
|
128
|
+
|
|
129
|
+
- Index **in the transaction** that saves the text; an index written outside it can stay when the save fails.
|
|
130
|
+
- Remove what a viewer may not see (`remove`, or a scope they never search), and still check rights on the rows you read by id.
|
|
131
|
+
- A long word is cut to 64 characters; two words that share their first 64 match each other.
|
|
132
|
+
- Changing `prefix` or `minLength` changes what the stored rows mean: index the rows again (`indexMany` in batches from a job), then `markCurrent`.
|
|
133
|
+
- Exports: `createPlugin`, `dependency` (also `@dokebi/search/dependency`), `defineSearchIndexes`, `SearchError`, `options`, `words`, `terms`, `queryTerms`, `TERMS_TABLE`, `COUNTS_TABLE`, `INDEXES_TABLE`, types `SearchApi`, `SearchItem`, `SearchQuery`, `SearchAnswer`, `SearchHit`, `SearchIndexStatus`, `SearchIndexes`, `IndexesDeclaration`, `IndexDeclaration`, `FieldDeclaration`, `TermRules`, `SearchErrorCode`, `SearchOptions`.
|
|
134
|
+
- Example: [examples/66-search](https://git.nyx-zone.com/dokebi/framework/-/blob/main/examples/66-search/README.md).
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Dokebi contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,9 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @dokebi/search
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Search in the lists of an application's own rows that stays bounded on a growing table (@dokebi/db AI.md 6.10), on PostgreSQL, MySQL 8 and MariaDB 10.6+ through @dokebi/db. Version 0.1.0 (unpublished). AI agents start with [AI.md](AI.md); the design is [D-0170](https://git.nyx-zone.com/dokebi/framework/-/blob/main/docs/decisions/D-0170-search.md).
|
|
4
|
+
|
|
5
|
+
- `search.index(tx, 'posts', { id, sortKey, scopes, fields })` writes the item's terms in the transaction that saves its text: every word and its prefixes, so a stem finds the words languages attach endings to.
|
|
6
|
+
- `search.find('posts', { field, q, scope, cursor })`: one word is one index lookup in display order; several words are driven by the rarest, with a cap per call (`examineLimit`) and per search (`searchLimit`) kept in a signed cursor, and `stopped: 'SEARCH_TOO_BROAD'` at the end of a search that is too broad. Answers item ids; your code reads the rows.
|
|
7
|
+
- Indexes come from `src/infra/search/indexes.ts` (`defineSearchIndexes`), which `npx dokebi add @dokebi/search` writes with the table migration and a cursor secret.
|
|
8
|
+
|
|
9
|
+
MIT licensed.
|
package/dist/api.d.ts
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
import type { SearchIndexes } from './indexes.js';
|
|
2
|
+
/** One item of an index: its whole searchable text, the display order and the filters. */
|
|
3
|
+
export interface SearchItem {
|
|
4
|
+
/** The application's id of the row (1–128 characters). */
|
|
5
|
+
readonly id: string;
|
|
6
|
+
/** The display order, newest first: a safe integer (usually the creation time in ms). */
|
|
7
|
+
readonly sortKey: number;
|
|
8
|
+
/**
|
|
9
|
+
* The filters a search is combined with (a board, each ancestor category), at most 32 of at
|
|
10
|
+
* most 64 characters; the item is indexed under each. Default `['']`: everything.
|
|
11
|
+
*/
|
|
12
|
+
readonly scopes?: readonly string[];
|
|
13
|
+
/** The text of each declared field; a field left out or null has no terms. */
|
|
14
|
+
readonly fields: Readonly<Record<string, string | null | undefined>>;
|
|
15
|
+
}
|
|
16
|
+
export interface SearchQuery {
|
|
17
|
+
/** The declared field to search. */
|
|
18
|
+
readonly field: string;
|
|
19
|
+
/** The words, all required (1–8 after normalisation). */
|
|
20
|
+
readonly q: string;
|
|
21
|
+
/** One scope the items were indexed under; default `''` (everything). */
|
|
22
|
+
readonly scope?: string;
|
|
23
|
+
/** Items per page (default 20, at most list.maxLimit). */
|
|
24
|
+
readonly limit?: number;
|
|
25
|
+
/** The `next` of the previous answer of the same search. */
|
|
26
|
+
readonly cursor?: string;
|
|
27
|
+
}
|
|
28
|
+
export interface SearchHit {
|
|
29
|
+
readonly id: string;
|
|
30
|
+
readonly sortKey: number;
|
|
31
|
+
}
|
|
32
|
+
export interface SearchAnswer {
|
|
33
|
+
/** Matches in display order (sortKey, then id, descending). */
|
|
34
|
+
readonly items: readonly SearchHit[];
|
|
35
|
+
/** Present when more matches may follow: pass it back as `cursor`. */
|
|
36
|
+
readonly next?: string;
|
|
37
|
+
/**
|
|
38
|
+
* The request reached examineLimit before the page was full: the matches found so far, and
|
|
39
|
+
* `next` to go on. Follow `next` (showing that the search continues) rather than showing an
|
|
40
|
+
* empty page on its own.
|
|
41
|
+
*/
|
|
42
|
+
readonly partial: boolean;
|
|
43
|
+
/** The search examined searchLimit rows over its pages and ended: ask for a narrower query. */
|
|
44
|
+
readonly stopped?: 'SEARCH_TOO_BROAD';
|
|
45
|
+
/** Rows of the term table this call examined. */
|
|
46
|
+
readonly examined: number;
|
|
47
|
+
}
|
|
48
|
+
export interface SearchIndexStatus {
|
|
49
|
+
readonly name: string;
|
|
50
|
+
readonly fingerprint: string;
|
|
51
|
+
/** The declaration changed since the rows were indexed: index them again, then markCurrent. */
|
|
52
|
+
readonly stale: boolean;
|
|
53
|
+
}
|
|
54
|
+
export interface SearchApi {
|
|
55
|
+
/** The declared indexes (src/infra/search/indexes.ts). */
|
|
56
|
+
readonly indexes: SearchIndexes;
|
|
57
|
+
/** The terms a text gets in a field (what a search can match), sorted. */
|
|
58
|
+
terms(index: string, field: string, text: string): readonly string[];
|
|
59
|
+
/** Writes the item's terms in the caller's transaction, replacing what it had. */
|
|
60
|
+
index(tx: unknown, index: string, item: SearchItem): Promise<void>;
|
|
61
|
+
/** `index` for several items, in one transaction (a first fill or a re-index in batches). */
|
|
62
|
+
indexMany(tx: unknown, index: string, items: readonly SearchItem[]): Promise<void>;
|
|
63
|
+
/** Removes the item's terms (hidden or deleted) in the caller's transaction. */
|
|
64
|
+
remove(tx: unknown, index: string, id: string): Promise<{
|
|
65
|
+
readonly removed: boolean;
|
|
66
|
+
}>;
|
|
67
|
+
/** One page of a search; bounded by examineLimit per call and searchLimit per search. */
|
|
68
|
+
find(index: string, query: SearchQuery): Promise<SearchAnswer>;
|
|
69
|
+
/**
|
|
70
|
+
* Writes the buffered word counts now (each worker does every counts.flushMs and at drain);
|
|
71
|
+
* a re-index job calls it when it finished. Never rejects: a failure is logged and retried.
|
|
72
|
+
*/
|
|
73
|
+
flush(): Promise<void>;
|
|
74
|
+
/** Records the current declaration of an index as indexed (after a re-index finished). */
|
|
75
|
+
markCurrent(index: string): Promise<void>;
|
|
76
|
+
/** Per index: the fingerprint and whether it is stale. No I/O. */
|
|
77
|
+
status(): readonly SearchIndexStatus[];
|
|
78
|
+
}
|
package/dist/api.js
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
export {};
|
package/dist/cursor.d.ts
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
export interface CursorState {
|
|
2
|
+
/** The query fingerprint (index, field, scope, terms). */
|
|
3
|
+
readonly query: string;
|
|
4
|
+
/** The leading word of a search of several words; absent for one word. */
|
|
5
|
+
readonly lead?: string;
|
|
6
|
+
readonly sortKey: number;
|
|
7
|
+
readonly id: string;
|
|
8
|
+
/** Rows of the leading word examined over every page so far. */
|
|
9
|
+
readonly examined: number;
|
|
10
|
+
}
|
|
11
|
+
/** The fingerprint of a query: a cursor of another query is refused. */
|
|
12
|
+
export declare function queryFingerprint(index: string, field: string, scope: string, terms: readonly string[]): string;
|
|
13
|
+
export declare class Cursors {
|
|
14
|
+
#private;
|
|
15
|
+
constructor(secrets: readonly string[]);
|
|
16
|
+
sign(state: CursorState): string;
|
|
17
|
+
/** The state of a cursor this server signed for `query`, or INVALID_CURSOR. */
|
|
18
|
+
open(cursor: unknown, query: string): CursorState;
|
|
19
|
+
}
|
package/dist/cursor.js
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
import { createHash, createHmac, timingSafeEqual } from 'node:crypto';
|
|
2
|
+
import { SearchError } from './errors.js';
|
|
3
|
+
const MAX_CURSOR = 1024;
|
|
4
|
+
/** The fingerprint of a query: a cursor of another query is refused. */
|
|
5
|
+
export function queryFingerprint(index, field, scope, terms) {
|
|
6
|
+
return createHash('sha256')
|
|
7
|
+
.update(JSON.stringify([index, field, scope, [...terms].sort()]))
|
|
8
|
+
.digest('base64url')
|
|
9
|
+
.slice(0, 22);
|
|
10
|
+
}
|
|
11
|
+
export class Cursors {
|
|
12
|
+
#secrets;
|
|
13
|
+
constructor(secrets) {
|
|
14
|
+
this.#secrets = secrets.map((secret) => Buffer.from(secret, 'utf8'));
|
|
15
|
+
}
|
|
16
|
+
#mac(body, secret) {
|
|
17
|
+
return createHmac('sha256', secret).update(body).digest();
|
|
18
|
+
}
|
|
19
|
+
sign(state) {
|
|
20
|
+
const body = Buffer.from(JSON.stringify([state.query, state.lead ?? null, state.sortKey, state.id, state.examined]), 'utf8').toString('base64url');
|
|
21
|
+
const mac = this.#mac(body, this.#secrets[0]).toString('base64url');
|
|
22
|
+
return `${body}.${mac}`;
|
|
23
|
+
}
|
|
24
|
+
/** The state of a cursor this server signed for `query`, or INVALID_CURSOR. */
|
|
25
|
+
open(cursor, query) {
|
|
26
|
+
const refuse = (reason) => new SearchError('INVALID_CURSOR', `The search cursor is not valid: ${reason}`, {
|
|
27
|
+
data: { reason },
|
|
28
|
+
});
|
|
29
|
+
if (typeof cursor !== 'string' || cursor.length > MAX_CURSOR)
|
|
30
|
+
throw refuse('not a cursor');
|
|
31
|
+
const dot = cursor.indexOf('.');
|
|
32
|
+
if (dot < 1)
|
|
33
|
+
throw refuse('not a cursor');
|
|
34
|
+
const body = cursor.slice(0, dot);
|
|
35
|
+
const given = Buffer.from(cursor.slice(dot + 1), 'base64url');
|
|
36
|
+
const signed = this.#secrets.some((secret) => {
|
|
37
|
+
const expected = this.#mac(body, secret);
|
|
38
|
+
return given.length === expected.length && timingSafeEqual(given, expected);
|
|
39
|
+
});
|
|
40
|
+
if (!signed)
|
|
41
|
+
throw refuse('signature');
|
|
42
|
+
let fields;
|
|
43
|
+
try {
|
|
44
|
+
fields = JSON.parse(Buffer.from(body, 'base64url').toString('utf8'));
|
|
45
|
+
}
|
|
46
|
+
catch (cause) {
|
|
47
|
+
// A signed body always parses; this is a cursor of a changed format, refused with its cause.
|
|
48
|
+
throw new SearchError('INVALID_CURSOR', 'The search cursor is not valid: body', { cause });
|
|
49
|
+
}
|
|
50
|
+
if (!Array.isArray(fields) || fields.length !== 5)
|
|
51
|
+
throw refuse('body');
|
|
52
|
+
const [owner, lead, sortKey, id, examined] = fields;
|
|
53
|
+
if (owner !== query)
|
|
54
|
+
throw refuse('another query');
|
|
55
|
+
if ((lead !== null && typeof lead !== 'string') ||
|
|
56
|
+
!Number.isSafeInteger(sortKey) ||
|
|
57
|
+
typeof id !== 'string' ||
|
|
58
|
+
!Number.isSafeInteger(examined))
|
|
59
|
+
throw refuse('body');
|
|
60
|
+
return {
|
|
61
|
+
query,
|
|
62
|
+
...(typeof lead === 'string' ? { lead } : {}),
|
|
63
|
+
sortKey: sortKey,
|
|
64
|
+
id,
|
|
65
|
+
examined: examined,
|
|
66
|
+
};
|
|
67
|
+
}
|
|
68
|
+
}
|
|
@@ -0,0 +1,4 @@
|
|
|
1
|
+
import type { SearchApi } from './api.js';
|
|
2
|
+
export declare const provider: import("@dokebi/core").ProviderDescriptor<SearchApi>;
|
|
3
|
+
/** Required: created from @dokebi/search when no `dokebi-search` plugin is registered. */
|
|
4
|
+
export declare const dependency: import("@dokebi/core").PluginDependency<SearchApi>;
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
import { defineProvider } from '@dokebi/core';
|
|
2
|
+
/*
|
|
3
|
+
* The one place dokebi-search states its name, package and range (D-0063 §6). A module that
|
|
4
|
+
* indexes or searches lists `dependency` from '@dokebi/search/dependency'.
|
|
5
|
+
*/
|
|
6
|
+
export const provider = defineProvider({
|
|
7
|
+
name: 'dokebi-search',
|
|
8
|
+
version: '^0.1.0',
|
|
9
|
+
packageName: '@dokebi/search',
|
|
10
|
+
load: () => import('./index.js'),
|
|
11
|
+
});
|
|
12
|
+
/** Required: created from @dokebi/search when no `dokebi-search` plugin is registered. */
|
|
13
|
+
export const dependency = provider.dependency;
|
package/dist/errors.d.ts
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
import { DokebiError, type DokebiErrorOptions } from '@dokebi/core';
|
|
2
|
+
export type SearchErrorCode = 'INVALID_OPTIONS' | 'INVALID_INDEXES' | 'UNKNOWN_INDEX' | 'UNKNOWN_FIELD' | 'INVALID_ITEM' | 'INVALID_QUERY' | 'INVALID_CURSOR' | 'SECRET_MISSING' | 'UNKNOWN_CONNECTION' | 'UNSUPPORTED_RESOURCE' | 'TABLE_MISSING' | 'NOT_READY' | 'STORE_FAILED' | 'DESCRIBING';
|
|
3
|
+
/** Errors of @dokebi/search (D-0063): a DokebiError with `plugin: 'dokebi-search'`. */
|
|
4
|
+
export declare class SearchError extends DokebiError<SearchErrorCode> {
|
|
5
|
+
constructor(code: SearchErrorCode, message: string, options?: DokebiErrorOptions);
|
|
6
|
+
}
|
package/dist/errors.js
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
import { DokebiError } from '@dokebi/core';
|
|
2
|
+
/** Repair advice (D-0063): `fix` for the developer or AI, `hint` for whoever got the error. */
|
|
3
|
+
const ADVICE = {
|
|
4
|
+
INVALID_OPTIONS: {
|
|
5
|
+
fix: 'Correct options.dokebi-search in dokebi.yaml; options.schema.json of @dokebi/search lists the fields (searchLimit is at least examineLimit).',
|
|
6
|
+
},
|
|
7
|
+
INVALID_INDEXES: {
|
|
8
|
+
fix: 'Export default defineSearchIndexes({ <index>: { fields: { <field>: { prefix?, minLength? } } } }) from src/infra/search/indexes.ts (npx dokebi add @dokebi/search writes it); names are a-z then a-z, 0-9 or _, at most 40 characters.',
|
|
9
|
+
},
|
|
10
|
+
UNKNOWN_INDEX: {
|
|
11
|
+
fix: 'Use an index of src/infra/search/indexes.ts, or declare it there.',
|
|
12
|
+
},
|
|
13
|
+
UNKNOWN_FIELD: {
|
|
14
|
+
fix: 'Use a field the index declares in src/infra/search/indexes.ts, or declare it there (then index the rows again).',
|
|
15
|
+
},
|
|
16
|
+
INVALID_ITEM: {
|
|
17
|
+
fix: 'Pass { id (1-128 characters), sortKey (a safe integer), scopes? (up to 32 strings of at most 64 characters), fields: { <declared field>: string } } and the transaction client.',
|
|
18
|
+
},
|
|
19
|
+
INVALID_QUERY: {
|
|
20
|
+
fix: 'Search with 1 to 8 words (letters or digits), each at least the field minLength, a limit within list.maxLimit and a scope of at most 64 characters.',
|
|
21
|
+
hint: 'Type at least one word to search for.',
|
|
22
|
+
},
|
|
23
|
+
INVALID_CURSOR: {
|
|
24
|
+
fix: 'Pass back the next value of the previous answer of the same search unchanged; a cursor of another query or a changed one is refused.',
|
|
25
|
+
hint: 'Start the search again.',
|
|
26
|
+
},
|
|
27
|
+
SECRET_MISSING: {
|
|
28
|
+
fix: 'Run npx dokebi config secret dokebi-search.secrets (npx dokebi add @dokebi/search does this on install), or set the environment variable of secretsEnv (default DOKEBI_SEARCH_SECRETS).',
|
|
29
|
+
},
|
|
30
|
+
UNKNOWN_CONNECTION: {
|
|
31
|
+
fix: 'Set options.dokebi-search.connection to a dokebi-db connection (default main), or add that connection.',
|
|
32
|
+
},
|
|
33
|
+
UNSUPPORTED_RESOURCE: {
|
|
34
|
+
fix: 'Use a pg, mysql2/promise or mariadb pool (the @dokebi/db templates); MongoDB is not supported in version 1 (D-0170).',
|
|
35
|
+
},
|
|
36
|
+
TABLE_MISSING: {
|
|
37
|
+
fix: 'Run npx dokebi-migrate up: the migration npx dokebi add @dokebi/search wrote creates dokebi_search_terms, dokebi_search_counts and dokebi_search_indexes.',
|
|
38
|
+
},
|
|
39
|
+
NOT_READY: {
|
|
40
|
+
hint: 'Search is not available right now; try again shortly.',
|
|
41
|
+
},
|
|
42
|
+
STORE_FAILED: {
|
|
43
|
+
hint: 'The search could not be run; try again shortly.',
|
|
44
|
+
},
|
|
45
|
+
};
|
|
46
|
+
/** Errors of @dokebi/search (D-0063): a DokebiError with `plugin: 'dokebi-search'`. */
|
|
47
|
+
export class SearchError extends DokebiError {
|
|
48
|
+
constructor(code, message, options = {}) {
|
|
49
|
+
super(code, message, {
|
|
50
|
+
...ADVICE[code],
|
|
51
|
+
...(code === 'NOT_READY' ? { status: 503 } : {}),
|
|
52
|
+
...(code === 'INVALID_QUERY' || code === 'INVALID_CURSOR' ? { status: 400 } : {}),
|
|
53
|
+
...options,
|
|
54
|
+
plugin: 'dokebi-search',
|
|
55
|
+
});
|
|
56
|
+
this.name = 'SearchError';
|
|
57
|
+
}
|
|
58
|
+
}
|
package/dist/index.d.ts
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
export type { SearchAnswer, SearchApi, SearchHit, SearchIndexStatus, SearchItem, SearchQuery, } from './api.js';
|
|
2
|
+
export { dependency } from './dependency.js';
|
|
3
|
+
export { SearchError, type SearchErrorCode } from './errors.js';
|
|
4
|
+
export { defineSearchIndexes, type FieldDeclaration, type IndexDeclaration, type IndexesDeclaration, type SearchIndexes, } from './indexes.js';
|
|
5
|
+
export { options, type SearchOptions } from './options.js';
|
|
6
|
+
export { createPlugin } from './plugin.js';
|
|
7
|
+
export { COUNTS_TABLE, INDEXES_TABLE, TERMS_TABLE } from './store.js';
|
|
8
|
+
export { queryTerms, type TermRules, terms, words } from './terms.js';
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
export { dependency } from './dependency.js';
|
|
2
|
+
export { SearchError } from './errors.js';
|
|
3
|
+
export { defineSearchIndexes, } from './indexes.js';
|
|
4
|
+
export { options } from './options.js';
|
|
5
|
+
export { createPlugin } from './plugin.js';
|
|
6
|
+
export { COUNTS_TABLE, INDEXES_TABLE, TERMS_TABLE } from './store.js';
|
|
7
|
+
export { queryTerms, terms, words } from './terms.js';
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
import { type TermRules } from './terms.js';
|
|
2
|
+
export interface FieldDeclaration {
|
|
3
|
+
/** The longest word prefix indexed (default 10); 0 indexes whole words only. */
|
|
4
|
+
readonly prefix?: number;
|
|
5
|
+
/** The shortest prefix indexed (default 1, at most prefix). */
|
|
6
|
+
readonly minLength?: number;
|
|
7
|
+
}
|
|
8
|
+
export interface IndexDeclaration {
|
|
9
|
+
readonly fields: Readonly<Record<string, FieldDeclaration>>;
|
|
10
|
+
}
|
|
11
|
+
export type IndexesDeclaration = Readonly<Record<string, IndexDeclaration>>;
|
|
12
|
+
export interface SearchIndexes {
|
|
13
|
+
/** The declared index names, in declaration order. */
|
|
14
|
+
readonly names: readonly string[];
|
|
15
|
+
/** The fields of an index, or undefined for an unknown index. */
|
|
16
|
+
fields(index: string): readonly string[] | undefined;
|
|
17
|
+
/** The term rules of a field, or undefined for an unknown index or field. */
|
|
18
|
+
rules(index: string, field: string): TermRules | undefined;
|
|
19
|
+
/** The fingerprint of an index (32 hex characters). */
|
|
20
|
+
fingerprint(index: string): string | undefined;
|
|
21
|
+
}
|
|
22
|
+
export declare const NAME: RegExp;
|
|
23
|
+
export declare const DEFAULT_PREFIX = 10;
|
|
24
|
+
/** Checks the declaration once and returns the indexes the plugin and the application use. */
|
|
25
|
+
export declare function defineSearchIndexes(declaration: IndexesDeclaration): SearchIndexes;
|
|
26
|
+
/** The indexes of the entry module (`export default defineSearchIndexes(…)`), or themselves. */
|
|
27
|
+
export declare function indexesOf(entry: unknown): SearchIndexes;
|
package/dist/indexes.js
ADDED
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
import { createHash } from 'node:crypto';
|
|
2
|
+
import { SearchError } from './errors.js';
|
|
3
|
+
import { MAX_PREFIX, TERMS_VERSION } from './terms.js';
|
|
4
|
+
export const NAME = /^[a-z][a-z0-9_]{0,39}$/;
|
|
5
|
+
const MAX_INDEXES = 64;
|
|
6
|
+
const MAX_FIELDS = 32;
|
|
7
|
+
export const DEFAULT_PREFIX = 10;
|
|
8
|
+
const BRAND = Symbol.for('dokebi.search.indexes');
|
|
9
|
+
function invalid(message) {
|
|
10
|
+
return new SearchError('INVALID_INDEXES', message);
|
|
11
|
+
}
|
|
12
|
+
function plain(value) {
|
|
13
|
+
return value !== null && typeof value === 'object' && !Array.isArray(value);
|
|
14
|
+
}
|
|
15
|
+
function integer(value, fallback, low, high, at) {
|
|
16
|
+
if (value === undefined)
|
|
17
|
+
return fallback;
|
|
18
|
+
if (!Number.isSafeInteger(value) || value < low || value > high)
|
|
19
|
+
throw invalid(`${at} is an integer from ${low} to ${high}`);
|
|
20
|
+
return value;
|
|
21
|
+
}
|
|
22
|
+
/** Checks the declaration once and returns the indexes the plugin and the application use. */
|
|
23
|
+
export function defineSearchIndexes(declaration) {
|
|
24
|
+
if (!plain(declaration))
|
|
25
|
+
throw invalid('The declaration is an object of indexes');
|
|
26
|
+
const names = Object.keys(declaration);
|
|
27
|
+
if (!names.length)
|
|
28
|
+
throw invalid('The declaration has no index');
|
|
29
|
+
if (names.length > MAX_INDEXES)
|
|
30
|
+
throw invalid(`More than ${MAX_INDEXES} indexes`);
|
|
31
|
+
const rules = new Map();
|
|
32
|
+
const prints = new Map();
|
|
33
|
+
for (const index of names) {
|
|
34
|
+
if (!NAME.test(index))
|
|
35
|
+
throw invalid(`Index name ${JSON.stringify(index)} is not a-z, 0-9, _`);
|
|
36
|
+
const entry = declaration[index];
|
|
37
|
+
if (!plain(entry) || !plain(entry.fields))
|
|
38
|
+
throw invalid(`Index ${index} has no fields object`);
|
|
39
|
+
const fields = Object.keys(entry.fields);
|
|
40
|
+
if (!fields.length)
|
|
41
|
+
throw invalid(`Index ${index} has no field`);
|
|
42
|
+
if (fields.length > MAX_FIELDS)
|
|
43
|
+
throw invalid(`Index ${index} has more than ${MAX_FIELDS} fields`);
|
|
44
|
+
const map = new Map();
|
|
45
|
+
for (const field of fields) {
|
|
46
|
+
if (!NAME.test(field))
|
|
47
|
+
throw invalid(`Field name ${JSON.stringify(field)} of ${index} is not a-z, 0-9, _`);
|
|
48
|
+
const given = entry.fields[field];
|
|
49
|
+
if (given !== undefined && !plain(given))
|
|
50
|
+
throw invalid(`Field ${index}.${field} is an object of rules`);
|
|
51
|
+
const unknown = Object.keys(given ?? {}).filter((key) => key !== 'prefix' && key !== 'minLength');
|
|
52
|
+
if (unknown.length)
|
|
53
|
+
throw invalid(`Field ${index}.${field} has unknown rules: ${unknown}`);
|
|
54
|
+
const prefix = integer(given?.prefix, DEFAULT_PREFIX, 0, MAX_PREFIX, `${index}.${field}.prefix`);
|
|
55
|
+
const minLength = integer(given?.minLength, 1, 1, Math.max(1, prefix), `${index}.${field}.minLength`);
|
|
56
|
+
map.set(field, Object.freeze({ prefix, minLength }));
|
|
57
|
+
}
|
|
58
|
+
rules.set(index, map);
|
|
59
|
+
const shape = [...map].sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
|
|
60
|
+
prints.set(index, createHash('sha256')
|
|
61
|
+
.update(JSON.stringify({ version: TERMS_VERSION, fields: shape }))
|
|
62
|
+
.digest('hex')
|
|
63
|
+
.slice(0, 32));
|
|
64
|
+
}
|
|
65
|
+
const api = {
|
|
66
|
+
names: Object.freeze([...names]),
|
|
67
|
+
fields: (index) => {
|
|
68
|
+
const map = typeof index === 'string' ? rules.get(index) : undefined;
|
|
69
|
+
return map ? Object.freeze([...map.keys()]) : undefined;
|
|
70
|
+
},
|
|
71
|
+
rules: (index, field) => typeof index === 'string' && typeof field === 'string'
|
|
72
|
+
? rules.get(index)?.get(field)
|
|
73
|
+
: undefined,
|
|
74
|
+
fingerprint: (index) => prints.get(index),
|
|
75
|
+
};
|
|
76
|
+
Object.defineProperty(api, BRAND, { value: true });
|
|
77
|
+
return Object.freeze(api);
|
|
78
|
+
}
|
|
79
|
+
/** The indexes of the entry module (`export default defineSearchIndexes(…)`), or themselves. */
|
|
80
|
+
export function indexesOf(entry) {
|
|
81
|
+
const candidate = entry !== null && typeof entry === 'object' && !(BRAND in entry)
|
|
82
|
+
? entry.default
|
|
83
|
+
: entry;
|
|
84
|
+
if (candidate !== null && typeof candidate === 'object' && BRAND in candidate)
|
|
85
|
+
return candidate;
|
|
86
|
+
throw new SearchError('INVALID_INDEXES', 'The search entry has no indexes: src/infra/search/indexes.ts exports default defineSearchIndexes({ … })');
|
|
87
|
+
}
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
interface InstallContext {
|
|
2
|
+
readonly options: Readonly<Record<string, unknown>>;
|
|
3
|
+
/** The keys this CLI accepts in a planInstall result (CLI 0.4.3 and later). */
|
|
4
|
+
readonly planKeys?: readonly string[];
|
|
5
|
+
}
|
|
6
|
+
/**
|
|
7
|
+
* The index declaration (src/infra/search/indexes.ts), the table migration in the migrations
|
|
8
|
+
* folder of `connection` (default main, the database of the application's rows) and a generated
|
|
9
|
+
* cursor secret in dokebi.yaml (a CLI without `configSecrets` leaves it to
|
|
10
|
+
* `npx dokebi config secret dokebi-search.secrets`, which the start asks for).
|
|
11
|
+
*/
|
|
12
|
+
export declare function planInstall(context: InstallContext): {
|
|
13
|
+
requires: string[];
|
|
14
|
+
entry: {
|
|
15
|
+
source: string;
|
|
16
|
+
target: string;
|
|
17
|
+
};
|
|
18
|
+
files?: {
|
|
19
|
+
source: string;
|
|
20
|
+
target: string;
|
|
21
|
+
}[];
|
|
22
|
+
configSecrets?: {
|
|
23
|
+
path: string[];
|
|
24
|
+
}[];
|
|
25
|
+
};
|
|
26
|
+
export declare function planUpgrade(): {
|
|
27
|
+
requires: string[];
|
|
28
|
+
entry: {
|
|
29
|
+
source: string;
|
|
30
|
+
target: string;
|
|
31
|
+
};
|
|
32
|
+
};
|
|
33
|
+
export declare function check(): {
|
|
34
|
+
requires: string[];
|
|
35
|
+
diagnostics: [];
|
|
36
|
+
};
|
|
37
|
+
export {};
|