@dokebi/search 0.1.0 → 0.1.2

package/AI.md

@@ -70,12 +70,12 @@ try {
 }
 ```
 
-- **`index(tx, index, item)`** writes in the transaction of `tx` (the pg client of `BEGIN`…`COMMIT`, or the mysql2/mariadb connection of the transaction). `item` is the item's **whole** searchable text: `id` (1–128), `sortKey` (a safe integer: the display order, newest first — usually the creation time in ms), `scopes` (the filters a search is combined with, at most 32 of at most 64 characters; default `['']`, everything), `fields` (a text per declared field; absent or null has no terms; at most 1,000,000 characters). It reads the item's terms by the item index and writes only the difference; a new `sortKey` rewrites them. Call it in the transaction that changes the item's row, so that row's lock orders two writers of the same item.
+- **`index(tx, index, item)`** writes in the transaction of `tx` (the pg client of `BEGIN`…`COMMIT`, or the mysql2/mariadb connection of the transaction). `item` is the item's **whole** searchable text: `id` (1–128), `sortKey` (a safe integer: the display order, newest first — usually the creation time in ms), `scopes` (the filters a search is combined with, at most 32 of at most 64 characters; default `['']`, everything), `fields` (a text per declared field; absent or null has no terms; at most 1,000,000 characters). It reads the item's terms by the item index and writes only the difference; a new `sortKey` rewrites them. Call it in the transaction that changes the item's row, so that row's lock orders two writers of the same item. On MySQL and MariaDB a transaction reads the snapshot of its first plain read, which may predate another writer's commit of the item: after its own inserts the write counts the item's rows as committed now (a locking read) and, when there are more than its new text, removes the others (measured on a 1,000-character edit: p50 7.4 → 9.0 ms on MySQL 8, 6.2 → 7.0 ms on MariaDB; PostgreSQL's READ COMMITTED statements read the committed rows and skip it). A deadlock, lock wait timeout or changed snapshot (MariaDB errno 1020) of your transaction comes back as the driver's own error, not `STORE_FAILED`: run the whole transaction again (@dokebi/db AI.md 6.4); a route that does not answers 503 `DB_BUSY`.
 - **Scopes** are how a search is filtered without reading other rows: an item is indexed under each, so a board and its ancestor categories are `['', 'board:12', 'cat:3', 'cat:3/7']` and a search names one. Each scope adds the item's terms again: keep them few.
-- **`remove(tx, index, id)`** when the item is hidden or deleted → `{ removed }`. Items a viewer may not see must not be in the index (or in a scope that viewer searches): `find` answers ids, it does not check rights.
+- **`remove(tx, index, id)`** when the item is hidden or deleted → `{ removed }` (on MySQL and MariaDB it reads the item's rows as committed now, as above). Items a viewer may not see must not be in the index (or in a scope that viewer searches): `find` answers ids, it does not check rights.
 - **`indexMany(tx, index, items)`**: a first fill or a re-index, in batches of a few hundred per transaction from a job; then `flush()` and `markCurrent(index)`.
 - **A display order that changes all the time** (views, likes) is not a `sortKey`: every change would rewrite every term row. Search in the index's order and rank a capped set of results yourself, saying so.
-- **Word counts** (which word is rarest) are a hint the plugin adds after the write, outside your transaction: each worker buffers them and writes them every `counts.flushMs`, at drain and on `flush()`. A common word's count is never a row every writer waits on, and a rolled-back write leaves its count changed: the counts may drift, the results never do.
+- **Word counts** (which word is rarest) are a hint the plugin adds after the write, outside your transaction: each worker buffers them and writes them every `counts.flushMs`, at drain and on `flush()` (which also writes what was buffered while an earlier flush was running). A common word's count is never a row every writer waits on, and a rolled-back write leaves its count changed: the counts may drift, the results never do.
 
 ## 4. Search
 
@@ -121,7 +121,7 @@ debug ([D-0047](https://git.nyx-zone.com/dokebi/framework/-/blob/main/docs/decis
 | `UNSUPPORTED_RESOURCE` | the connection is not a pg, mysql2/promise or mariadb pool (MongoDB) |
 | `TABLE_MISSING` | the tables or their columns are missing at start (run the migration) |
 | `NOT_READY` | the database of an optional connection is down (503) |
-| `STORE_FAILED` | a statement failed (the driver error is the cause) |
+| `STORE_FAILED` | a statement failed (the driver error is the cause); deadlocks, lock waits and changed snapshots in your transaction come out as the driver's own error instead |
 | `DESCRIBING` | a call during `dokebi describe` |
 
 ## 7. Cautions

package/dist/plugin.js

@@ -4,8 +4,8 @@ import { Cursors, queryFingerprint } from './cursor.js';
 import { SearchError } from './errors.js';
 import { indexesOf } from './indexes.js';
 import { DEFAULT_SECRETS_ENV, MIN_SECRET_LENGTH, options } from './options.js';
-import { detectDriver, isMissingTable, isQueryable, serverDialect, session, } from './sql.js';
-import { COUNTS_TABLE, INDEXES_TABLE, Store, TERMS_TABLE, } from './store.js';
+import { detectDriver, isMissingTable, isQueryable, isRetryable, serverDialect, session, } from './sql.js';
+import { COUNT_BATCH, COUNTS_TABLE, INDEXES_TABLE, Store, TERMS_TABLE, } from './store.js';
 import { MAX_QUERY_WORDS, queryTerms, terms } from './terms.js';
 const db = providerDependency({
     name: 'dokebi-db',
@@ -130,7 +130,7 @@ class SearchRuntime {
         }
         this.#repeats = new diagnostics.Repeats({ logger: context.logger, intervalMs: REPEAT_MS });
         const timer = setInterval(() => {
-            void this.#flush();
+            this.#flushSoon();
         }, settings.counts.flushMs);
         timer.unref();
         context.defer(async () => {
@@ -241,8 +241,11 @@ class SearchRuntime {
             data: { connection: this.#options.connection, retryAfterMs: RETRY_MS },
         });
     }
-    /** A statement failure as a coded error that keeps the driver's cause. */
-    async #run(operation, work) {
+    /**
+     * A statement failure as a coded error that keeps the driver's cause; `inCaller`: the work ran
+     * in the caller's transaction.
+     */
+    async #run(operation, work, inCaller = false) {
         const live = await this.#ready();
         try {
             return await work(live);
@@ -256,6 +259,10 @@ class SearchRuntime {
                     operation,
                     error: diagnostics.describeError(error),
                 });
+            // The caller's transaction ended by a deadlock, a lock wait or a changed snapshot: its own
+            // error, so the caller's retry sees it and @dokebi/db answers 503 DB_BUSY / DB_TIMEOUT.
+            if (inCaller && isRetryable(error))
+                throw error;
             throw new SearchError('STORE_FAILED', `The search could not ${operation}`, {
                 cause: error,
                 data: { operation },
@@ -351,6 +358,23 @@ class SearchRuntime {
                 await live.store.deleteRows(tx, index, item.id, gone);
             if (added.length)
                 await live.store.insertRows(tx, index, item.id, item.sortKey, added);
+            // After the inserts the item holds at least its new rows, once each: only more means rows
+            // this transaction's snapshot did not show (one small locking read, the rows only then).
+            if (live.store.snapshots &&
+                (await live.store.lockedItemCount(tx, index, item.id)) !== rows.length) {
+                // MySQL and MariaDB: `old` came from the transaction's snapshot, which may predate another
+                // writer's commit of this item (its first plain read fixed it). Rows that writer added are
+                // read as committed now and removed (R8-g2-2); their words count as the item's before.
+                const extra = (await live.store.lockedItemRows(tx, index, item.id)).filter((row) => row.sortKey !== item.sortKey || !now.has(key(row)));
+                for (const row of extra)
+                    oldPairs.add(`${row.field}${SEP}${row.term}`);
+                if (extra.some((row) => row.sortKey !== item.sortKey)) {
+                    await live.store.deleteItem(tx, index, item.id);
+                    await live.store.insertRows(tx, index, item.id, item.sortKey, rows);
+                }
+                else if (extra.length)
+                    await live.store.deleteRows(tx, index, item.id, extra);
+            }
         }
         this.#count(index, oldPairs, pairs);
     }
@@ -371,16 +395,26 @@ class SearchRuntime {
             if (!after.has(pair))
                 add(pair, -1);
         if (this.#pending.size >= FLUSH_WORDS)
-            void this.#flush();
+            this.#flushSoon();
     }
     /**
-     * Adds the buffered count changes to the table, one autocommit statement per index on the
-     * plugin's pool, words in sorted order. A failure is logged and the changes are kept for the
+     * Adds the buffered count changes to the table, in autocommit statements of up to COUNT_BATCH
+     * words per index on the plugin's pool, words in sorted order. A failure is logged and the changes are kept for the
      * next flush. Never rejects.
      */
     #flush() {
+        // A flush in flight took the buffer before the newest changes: an explicit flush (flush(),
+        // the drain, the stop) waits for it, then writes what was buffered since.
         if (this.#flushing)
-            return this.#flushing;
+            return this.#flushing.then(() => this.#flush());
+        return this.#flushNow();
+    }
+    /** The timer's and the buffer-size flush: none while one is in flight (it runs again soon). */
+    #flushSoon() {
+        if (!this.#flushing)
+            void this.#flushNow();
+    }
+    #flushNow() {
         if (!this.#pending.size || !this.#live)
             return Promise.resolve();
         const live = this.#live;
@@ -398,14 +432,23 @@ class SearchRuntime {
             }
             for (const [index, deltas] of byIndex) {
                 deltas.sort((a, b) => byKey(a.field, b.field) || byKey(a.term, b.term));
+                // One autocommit statement per COUNT_BATCH words: `written` words are in the table.
+                let written = 0;
                 try {
-                    await live.store.addCounts(live.pool, index, deltas);
+                    for (; written < deltas.length; written += COUNT_BATCH)
+                        await live.store.addCounts(live.pool, index, deltas.slice(written, written + COUNT_BATCH));
                 }
                 catch (error) {
-                    // Kept for the next flush: the counts are late, never lost while the process runs.
-                    for (const row of deltas) {
+                    // Kept for the next flush: the counts are late, never lost while the process runs. Only
+                    // the words from the failed statement on: the statements before it are committed, and
+                    // keeping their words too would add them twice (R9-g14-3).
+                    for (const row of deltas.slice(written)) {
                         const key = `${index}${SEP}${row.field}${SEP}${row.term}`;
-                        this.#pending.set(key, (this.#pending.get(key) ?? 0) + row.delta);
+                        const value = (this.#pending.get(key) ?? 0) + row.delta;
+                        if (value === 0)
+                            this.#pending.delete(key);
+                        else
+                            this.#pending.set(key, value);
                     }
                     this.#repeats?.log('error', 'Search word counts could not be written; they are kept for the next flush.', 'SEARCH_COUNTS_FAILED', error);
                 }
@@ -582,7 +625,7 @@ class SearchRuntime {
             },
             // The item is checked (and its terms made) once, inside #write: a SearchError passes #run.
             index: async (tx, index, item) => {
-                await this.#run('index an item', async (live) => this.#write(live, session(live.driver, tx), index, item));
+                await this.#run('index an item', async (live) => this.#write(live, session(live.driver, tx), index, item), true);
             },
             indexMany: async (tx, index, items) => {
                 if (!Array.isArray(items))
@@ -591,20 +634,23 @@ class SearchRuntime {
                     const client = session(live.driver, tx);
                     for (const item of items)
                         await this.#write(live, client, index, item);
-                });
+                }, true);
             },
             remove: async (tx, index, id) => {
                 this.#fields(index);
                 const item = this.#id(id);
                 return this.#run('remove an item', async (live) => {
                     const client = session(live.driver, tx);
-                    const old = await live.store.itemRows(client, index, item);
+                    // MySQL and MariaDB: as committed now, not as the transaction's snapshot (R8-g2-2).
+                    const old = live.store.snapshots
+                        ? await live.store.lockedItemRows(client, index, item)
+                        : await live.store.itemRows(client, index, item);
                     if (!old.length)
                         return Object.freeze({ removed: false });
                     await live.store.deleteItem(client, index, item);
                     this.#count(index, new Set(old.map((row) => `${row.field}${SEP}${row.term}`)), new Set());
                     return Object.freeze({ removed: true });
-                });
+                }, true);
             },
             find: async (index, query) => {
                 return this.#run('search', (live) => this.#find(live, index, query));
@@ -660,7 +706,7 @@ class SearchRuntime {
 /** Factory input: `{ app, options?, entry }`: `entry` is the module of src/infra/search/indexes.ts. */
 export const createPlugin = definePlugin({
     name: 'dokebi-search',
-    version: '0.1.0',
+    version: '0.1.2',
     core: '^0.4.2',
     options,
     dependencies: () => [db],

package/dist/sql.d.ts

@@ -26,6 +26,11 @@ export declare function serverDialect(driver: Driver, pool: Session): Promise<{
 }>;
 /** A missing table or column (PostgreSQL 42P01, 42703; MySQL and MariaDB 1146, 1054). */
 export declare function isMissingTable(error: unknown): boolean;
+/**
+ * A deadlock, serialization failure or lock wait timeout (PostgreSQL 40001, 40P01, 55P03; MySQL
+ * and MariaDB 1213, 1020, 1205): the caller's transaction is over and is run again whole.
+ */
+export declare function isRetryable(error: unknown): boolean;
 /** A number from a driver value: pg bigint strings, mariadb BigInt, mysql2 numbers. */
 export declare function num(value: unknown): number;
 export declare function text(value: unknown): string | undefined;

package/dist/sql.js

@@ -95,6 +95,16 @@ export function isMissingTable(error) {
     const value = error;
     return MISSING_TABLE.has(value?.code) || MISSING_TABLE_ERRNO.has(Number(value?.errno));
 }
+const RETRY = new Set(['40001', '40P01', '55P03']);
+const RETRY_ERRNO = new Set([1213, 1020, 1205]);
+/**
+ * A deadlock, serialization failure or lock wait timeout (PostgreSQL 40001, 40P01, 55P03; MySQL
+ * and MariaDB 1213, 1020, 1205): the caller's transaction is over and is run again whole.
+ */
+export function isRetryable(error) {
+    const value = error;
+    return RETRY.has(value?.code) || RETRY_ERRNO.has(Number(value?.errno));
+}
 /** A number from a driver value: pg bigint strings, mariadb BigInt, mysql2 numbers. */
 export function num(value) {
     return value === null || value === undefined ? 0 : Number(value);

package/dist/store.d.ts

@@ -3,6 +3,8 @@ import type { Dialect, Session } from './sql.js';
 export declare const TERMS_TABLE = "dokebi_search_terms";
 export declare const COUNTS_TABLE = "dokebi_search_counts";
 export declare const INDEXES_TABLE = "dokebi_search_indexes";
+/** Words of one count statement: a flush hands `addCounts` this many at a time. */
+export declare const COUNT_BATCH = 500;
 export interface TermRow {
     readonly field: string;
     readonly term: string;
@@ -25,18 +27,32 @@ export interface Scanned extends SearchHit {
 }
 export declare class Store {
     #private;
+    /** MySQL and MariaDB read an older snapshot in a REPEATABLE READ transaction. */
+    get snapshots(): boolean;
     constructor(dialect: Dialect, slots: number);
     check(db: Session): Promise<void>;
     fingerprints(db: Session): Promise<Map<string, string>>;
     saveFingerprint(db: Session, name: string, fingerprint: string, at: number): Promise<void>;
     /** The rows of one item (the item index). */
     itemRows(db: Session, index: string, id: string): Promise<StoredRow[]>;
+    /**
+     * MySQL and MariaDB: the item's rows as committed now (a locking read), for a write whose
+     * transaction may read an older snapshot. Called after the write's own inserts: its range then
+     * holds the write's own rows, so the read waits for no other writer (an empty range would take a
+     * gap lock that two writers of new items share and then both insert into).
+     */
+    lockedItemRows(db: Session, index: string, id: string): Promise<StoredRow[]>;
+    /** MySQL and MariaDB: how many rows the item has as committed now (a locking read, as above). */
+    lockedItemCount(db: Session, index: string, id: string): Promise<number>;
     deleteItem(db: Session, index: string, id: string): Promise<number>;
     deleteRows(db: Session, index: string, id: string, rows: readonly TermRow[]): Promise<void>;
     insertRows(db: Session, index: string, id: string, sortKey: number, rows: readonly TermRow[]): Promise<void>;
     /**
      * Moves the item counts of words by their deltas, in one slot chosen at random, in the order
-     * given (the caller sorts them, so two writers lock the same rows in the same order).
+     * given (the caller sorts them, so two writers lock the same rows in the same order). One
+     * statement for up to COUNT_BATCH deltas: all of them are written or none. More are several
+     * statements, and a failure then leaves the earlier ones written, so a caller that keeps what
+     * failed hands over COUNT_BATCH at a time.
      */
     addCounts(db: Session, index: string, deltas: readonly CountDelta[]): Promise<void>;
     /** The item count of each word (its slots summed); a word without rows is absent. */

package/dist/store.js

@@ -10,8 +10,14 @@ export const COUNTS_TABLE = 'dokebi_search_counts';
 export const INDEXES_TABLE = 'dokebi_search_indexes';
 /** Rows of one multi-row INSERT or DELETE (parameters stay far below every driver's limit). */
 const BATCH = 500;
+/** Words of one count statement: a flush hands `addCounts` this many at a time. */
+export const COUNT_BATCH = BATCH;
 export class Store {
     #dialect;
+    /** MySQL and MariaDB read an older snapshot in a REPEATABLE READ transaction. */
+    get snapshots() {
+        return this.#dialect !== 'postgres';
+    }
     #slots;
     constructor(dialect, slots) {
         this.#dialect = dialect;
@@ -43,6 +49,26 @@ export class Store {
             sortKey: num(row.sort_key),
         }));
     }
+    /**
+     * MySQL and MariaDB: the item's rows as committed now (a locking read), for a write whose
+     * transaction may read an older snapshot. Called after the write's own inserts: its range then
+     * holds the write's own rows, so the read waits for no other writer (an empty range would take a
+     * gap lock that two writers of new items share and then both insert into).
+     */
+    async lockedItemRows(db, index, id) {
+        const { rows } = await db.query(`SELECT field, term, scope, sort_key FROM ${TERMS_TABLE} WHERE index_name = ? AND item_id = ? FOR UPDATE`, [index, id]);
+        return rows.map((row) => ({
+            field: String(row.field),
+            term: String(row.term),
+            scope: String(row.scope),
+            sortKey: num(row.sort_key),
+        }));
+    }
+    /** MySQL and MariaDB: how many rows the item has as committed now (a locking read, as above). */
+    async lockedItemCount(db, index, id) {
+        const { rows } = await db.query(`SELECT COUNT(*) AS n FROM ${TERMS_TABLE} WHERE index_name = ? AND item_id = ? FOR UPDATE`, [index, id]);
+        return num(rows[0]?.n);
+    }
     async deleteItem(db, index, id) {
         const result = await db.query(`DELETE FROM ${TERMS_TABLE} WHERE index_name = ? AND item_id = ?`, [index, id]);
         return result.affected;
@@ -66,12 +92,18 @@ export class Store {
                 params.push(index, row.field, row.term, row.scope, sortKey, id);
                 return '(?, ?, ?, ?, ?, ?)';
             });
-            await db.query(`INSERT INTO ${TERMS_TABLE} (index_name, field, term, scope, sort_key, item_id) VALUES ${tuples.join(', ')}`, params);
+            // A row another writer committed after this transaction's snapshot may already be there
+            // (MySQL, MariaDB): it is the same row, kept.
+            const keep = this.#dialect === 'postgres' ? '' : ' ON DUPLICATE KEY UPDATE item_id = item_id';
+            await db.query(`INSERT INTO ${TERMS_TABLE} (index_name, field, term, scope, sort_key, item_id) VALUES ${tuples.join(', ')}${keep}`, params);
         }
     }
     /**
      * Moves the item counts of words by their deltas, in one slot chosen at random, in the order
-     * given (the caller sorts them, so two writers lock the same rows in the same order).
+     * given (the caller sorts them, so two writers lock the same rows in the same order). One
+     * statement for up to COUNT_BATCH deltas: all of them are written or none. More are several
+     * statements, and a failure then leaves the earlier ones written, so a caller that keeps what
+     * failed hands over COUNT_BATCH at a time.
      */
     async addCounts(db, index, deltas) {
         const slot = Math.floor(Math.random() * this.#slots);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@dokebi/search",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "Search in an application's lists for Dokebi by @dokebi/db AI.md 6.10: a term table in display order written in the text's transaction, Korean word prefixes, the rarest word leads, bounded requests and searches with a signed cursor.",
   "keywords": [
     "dokebi",
@@ -45,7 +45,7 @@
     "@dokebi/db": "^0.4.1"
   },
   "devDependencies": {
-    "@dokebi/db": "0.4.1"
+    "@dokebi/db": "0.4.2"
   },
   "dokebi": {
     "schemaVersion": 2,