tr-pg-name-value-store 0.0.0 → 2.0.0

package/CHANGELOG.md

@@ -4,20 +4,53 @@ All notable changes to this project are documented in this file. The format is
 based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
 project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.0.0] - Unreleased
+## [2.0.0] - 2026-09-07
 
-Initial implementation, ready for testing.
+No breaking changes; every existing method, guarantee and the frozen schema are
+untouched.
+
+### Added
+
+- `list(options?)`: pages through the store's names — with their values, or
+  without (`values: false`) for a cheap walk over large documents — in byte
+  order of the name (`COLLATE "C"`, identical on every database), with keyset
+  paging (`after`, `have_more`, `limit` up to 1000) and a literal `prefix`
+  filter in which `%`, `_` and `\` match themselves. Each page is one
+  lock-free statement over its own snapshot; no index is added.
+- `count(options?)`: the number of names, optionally under the same literal
+  prefix.
+- Exported types `ListOptions`, `ListEntry`, `ListResult` and `CountOptions`.
+
+## [1.0.0] - 2026-09-07
+
+Initial release.
 
 ### Added
 
 - `PgNameValueStore` class constructed from a pg `Pool`, with a self-maintaining,
   never-migrated schema (one `name_value_store` table per namespace; pre-existing
-  tables are verified, never altered).
+  tables are verified, never altered; a matching operator-created table is
+  adopted as-is).
 - Async API: `init`, `get`, `set`, `remove`, `update`, `removeAll`.
 - JSONB value storage for any JSON value; `undefined` (no value) is kept distinct
   from a stored JSON `null` via row presence.
-- Atomic `update(name, callback)` read-modify-write, serialized per name by a
-  PostgreSQL advisory lock, with graceful cancel (callback throws `null`/
-  `undefined`), delete (callback returns `undefined`), and error pass-through.
-- `SchemaMismatchError` for a conflicting pre-existing table.
-- Integration test suite (vitest) and documentation.
+- Exact previous-value contract: `set`, `remove` and `update` each resolve to the
+  value they replaced. All writes to the same name run in a transaction on a
+  dedicated connection under a per-`(namespace, name)` advisory lock, so they are
+  strictly serialized across processes and the previous value is always the one
+  left by the immediately preceding write. Writes to different names, and all
+  reads, are never blocked.
+- Atomic `update(name, callback)` read-modify-write with graceful cancel
+  (callback throws `null`/`undefined`), delete (callback returns `undefined`),
+  and error pass-through.
+- Name validation: a non-empty string of at most 1024 bytes of UTF-8 without
+  `U+0000`, rejected with `TypeError` before any I/O.
+- `SchemaMismatchError` for a conflicting pre-existing table. A failed `init()`
+  is not cached; the next call retries.
+- Integration test suite (vitest) against a real PostgreSQL — a throwaway local
+  cluster by default, or any server via `TR_PG_NAME_VALUE_STORE_TEST_URL` (a
+  compose file for PostgreSQL 16 is included). Verified on PostgreSQL 9.6, 16
+  and 18.
+
+[2.0.0]: https://github.com/rinne/node-tr-pg-name-value-store/releases/tag/v2.0.0
+[1.0.0]: https://github.com/rinne/node-tr-pg-name-value-store/releases/tag/v1.0.0

package/README.md

@@ -10,6 +10,13 @@ returned as the corresponding JavaScript value.
   *verified, never altered*.
 - **Multiple named stores.** A namespace gives each store its own table in the
   same database.
+- **Listing.** `list()` pages through the names (with values, or without for a
+  cheap walk) in byte order with keyset paging, and `count()` counts them, both
+  optionally under a literal prefix — so nobody needs to query the table.
+- **Exact previous values.** Every write (`set`, `remove`, `update`) resolves to
+  the value it replaced. Writes to the same name are serialized across all
+  processes sharing the database, so that previous value is always the one left
+  by the immediately preceding write.
 - **Atomic read-modify-write.** `update()` runs a callback inside a transaction,
   serialized per name, so concurrent updates compose correctly.
 - **Presence vs. `null`.** The store distinguishes *no value* (`undefined`) from
@@ -22,7 +29,8 @@ npm install tr-pg-name-value-store pg
 ```
 
 `pg` is a peer dependency (`>= 8`). Requires Node `>= 18` and PostgreSQL `>= 9.5`
-(for `INSERT … ON CONFLICT`).
+(for `INSERT … ON CONFLICT`). The test suite runs against PostgreSQL 9.6, 16 and
+18.
 
 ## Quick start
 
@@ -60,6 +68,16 @@ A stored value is any JSON-serializable JavaScript value: `number`, `string`,
   are reordered, insignificant whitespace is dropped, duplicate keys collapse to
   the last, and numbers are canonicalized. A value read back is *semantically*
   equal but may not be textually identical to the value written (e.g. key order).
+- **No `U+0000` in strings.** PostgreSQL's `jsonb` cannot store the NUL character
+  inside a string; the server's error propagates unchanged and nothing is written.
+
+## Names
+
+A name is any non-empty string of at most **1024 bytes of UTF-8** (so up to 1024
+ASCII characters, fewer for non-ASCII) that does not contain `U+0000`. Anything
+else throws `TypeError` before any I/O. Within those bounds names are opaque:
+case-sensitive, whitespace-significant, any punctuation or Unicode. Listing
+orders them by byte order of their UTF-8 encoding (see [Listing](#listing)).
 
 ## API
 
@@ -76,11 +94,13 @@ Constructs a store over a pg `Pool`. Performs no I/O. Options:
 Idempotently ensures the schema (creating the table if absent, verifying it if
 present). Called automatically on first use of any method; call it explicitly to
 surface schema/connection errors at startup. Safe to call repeatedly and
-concurrently (across processes too).
+concurrently (across processes too). A failed `init()` is not cached; the next
+call retries.
 
 ### `get(name): Promise<value>`
 
-Resolves to the current value, or `undefined` if `name` has no value.
+Resolves to the current value, or `undefined` if `name` has no value. A single
+lock-free `SELECT` that reads the latest committed value.
 
 ### `set(name, value): Promise<previous>`
 
@@ -103,14 +123,9 @@ based on what the callback does:
 | returns a JSON-serializable value (not `undefined`)    | stores it                       | resolves to the previous value     |
 | returns `undefined`                                    | removes the name                | resolves to the previous value     |
 | returns a non-serializable value (function, `BigInt`, …) | nothing (rolled back)         | throws `TypeError`                 |
-| throws `null` or `undefined`                           | nothing (rolled back)           | resolves to the previous value     |
+| throws `null` or `undefined`                           | nothing (graceful cancel)       | resolves to the previous value     |
 | throws anything else                                   | nothing (rolled back)           | re-throws that value unchanged     |
 
-The whole sequence runs in one transaction on a dedicated pooled connection.
-Concurrent updates of the **same** name are serialized by a per-name advisory
-lock, so each callback sees the committed result of the previous one — including
-the create-from-absent case.
-
 ```ts
 // atomic counter (creates from absent, then increments)
 await store.update('hits', (n) => (typeof n === 'number' ? n : 0) + 1);
@@ -125,10 +140,89 @@ await store.update('config', (cfg) => {
 await store.update('stale', () => undefined);
 ```
 
+The callback must not write to the **same name** through this or any other
+store instance — it would wait forever on the lock the callback itself holds.
+Writes to other names, and reads of anything, are fine.
+
 ### `removeAll(): Promise<void>`
 
 Removes every pair from this store's namespace. Other namespaces are untouched.
 
+### Listing
+
+#### `list(options?): Promise<ListResult>`
+
+Lists one page of names, in ascending **byte order** of the name, with their
+values unless `values: false`.
+
+| Option   | Type      | Default | Meaning |
+|----------|-----------|---------|---------|
+| `prefix` | `string`  | none    | Only names starting with this exact string. Literal, not a pattern: `%`, `_` and `\` match themselves. Validated like a name (≤ 1024 bytes, no `U+0000`); empty means no filter. |
+| `after`  | `string`  | none    | Keyset cursor: only names strictly after this one in byte order — normally the last name of the previous page. Need not be an existing name. Validated like `prefix`; empty means start from the beginning. |
+| `limit`  | `number`  | 1000    | Page size, a positive integer. The maximum is 1000; larger values clamp silently (keep paging while `have_more` is true). A non-integer or a value below 1 throws `TypeError`. |
+| `values` | `boolean` | `true`  | With `false`, `value` is omitted from every entry and the value column is not read, so a store of large documents can be walked cheaply. |
+
+The result:
+
+| Field       | Type          | Meaning |
+|-------------|---------------|---------|
+| `entries`   | `ListEntry[]` | The page, each `{ name, value?, updated_at }`. `value` is present (and may be `null`) whenever `values` is on; a name with no value has no entry at all. `updated_at` is a `Date` (millisecond precision) of the last write. |
+| `have_more` | `boolean`     | `true` if more names followed this page in the same snapshot; continue with `after` set to the last entry's name. |
+
+Paging loop:
+
+```ts
+const names: string[] = [];
+let after: string | undefined;
+for (;;) {
+  const page = await store.list({ prefix: 'job:', after, values: false });
+  names.push(...page.entries.map((e) => e.name));
+  if (!page.have_more) break;
+  after = page.entries.at(-1)!.name; // a page with have_more is never empty
+}
+```
+
+Each page is one lock-free statement over its own snapshot. Names written
+between two pages may or may not appear, and a name removed in between is
+simply absent (so `have_more: true` followed by an empty page is possible).
+There are no offsets: a name that exists for the whole walk is returned exactly
+once.
+
+Ordering is byte order of the UTF-8 name (`COLLATE "C"`), identical on every
+database regardless of its locale — so `'B' < '_x' < 'a'`. It differs from
+JavaScript's default string comparison only for characters outside the Basic
+Multilingual Plane, which matters if you merge pages with locally sorted data.
+On databases whose default collation is `C` the primary-key index serves
+`after` and `prefix` directly; on other databases each page is a scan and sort
+of the store's names, which is fine for the sizes a per-service store reaches.
+No index is added — the schema stays frozen.
+
+#### `count(options?): Promise<number>`
+
+Resolves to the number of names in the store, or of those starting with
+`options.prefix` (same semantics as above). A single lock-free statement that
+scans the matching names.
+
+## Concurrency
+
+Every write to a given name — `set`, `remove`, and `update` — runs in its own
+transaction on a dedicated pooled connection and holds a per-`(namespace, name)`
+transaction-scoped advisory lock (`pg_advisory_xact_lock`) for its duration.
+Writes to the same name are therefore strictly serialized, across every process
+that shares the database, and each one observes exactly the committed result of
+the one before it. That is what makes the *previous value* contract exact and
+what lets `update` callbacks compose (including the create-from-absent case,
+which a row lock alone cannot cover).
+
+Writes to different names never block each other. Reads — `get`, `list` and
+`count` — are single lock-free statements that see the latest committed state
+and neither block writes nor are blocked by them. `removeAll` is a single bulk
+`DELETE` that does not take per-name locks; it waits for in-flight writes on
+individual rows like any other statement would.
+
+Advisory-lock keys are derived in Node (SHA-256 of the namespace and name) — no
+server-side extension is required.
+
 ## Multiple stores
 
 Each namespace is an independent store with its own table:
@@ -139,15 +233,20 @@ const settings = new PgNameValueStore(pool, { namespace: 'settings' });
 // sessions.* and settings.* never collide; removeAll() on one leaves the other intact
 ```
 
+Several instances over the same namespace (in one process or many) share the
+same data and the same per-name serialization.
+
 ## Errors
 
 | Error                 | When |
 |-----------------------|------|
-| `TypeError`           | Invalid namespace; invalid name (not a non-empty string, or > 1024 chars); a `set` value or `update` return that is not JSON-serializable. |
+| `TypeError`           | Invalid namespace; invalid name (not a non-empty string, more than 1024 bytes of UTF-8, or containing `U+0000`); a `set` value or `update` return that is not JSON-serializable; an `update` callback that is not a function; invalid `list`/`count` options (a `prefix` or `after` that is not a string within the name bounds, a `limit` that is not a positive integer, a non-boolean `values`). |
 | `SchemaMismatchError` | A table with this namespace's name already exists with a different shape. It is left untouched. Exported by the package. |
 | re-thrown value       | `update` re-throws any non-`null`/`undefined` value its callback throws. |
 
 Operational/connection failures propagate from the underlying `pg` calls.
+Argument validation errors are thrown synchronously by the constructor and as
+rejected promises by every other method, always before any I/O.
 
 ## Schema
 
@@ -163,7 +262,32 @@ CREATE TABLE {{ns}}name_value_store (
 ```
 
 The module issues only `CREATE TABLE IF NOT EXISTS` and verifies the columns of
-a pre-existing table against this shape — it never runs `ALTER` or `DROP`.
+a pre-existing table against this shape — it never runs `ALTER` or `DROP`. A
+matching table created by an operator (with any existing rows) is adopted as-is.
+`updated_at` is maintained on every write for operational inspection; it has no
+public accessor.
+
+## Development
+
+```sh
+npm install
+npm test            # embeds sql/, typechecks src + tests, runs vitest
+npm run build       # emits dist/
+npm pack --dry-run  # audit the tarball contents before publishing
+```
+
+The tests run against a real PostgreSQL. By default a throwaway cluster is
+bootstrapped with `initdb`/`pg_ctl` in a temp directory (PostgreSQL server
+binaries must be on `PATH`) and torn down afterwards. To use an existing server
+instead — for example the bundled compose file — point
+`TR_PG_NAME_VALUE_STORE_TEST_URL` at an admin connection; a scratch database
+named `tr_pg_name_value_store_test` is (re)created on it:
+
+```sh
+docker compose -f test/docker-compose.yml up -d
+TR_PG_NAME_VALUE_STORE_TEST_URL=postgres://postgres:postgres@localhost:5434/postgres npm test
+docker compose -f test/docker-compose.yml down
+```
 
 ## License
 

package/dist/index.d.ts

@@ -1,2 +1,2 @@
-export { PgNameValueStore, type PgNameValueStoreOptions, type UpdateCallback, } from './pg-name-value-store';
+export { PgNameValueStore, type PgNameValueStoreOptions, type UpdateCallback, type ListOptions, type ListEntry, type ListResult, type CountOptions, } from './pg-name-value-store';
 export { SchemaMismatchError } from './schema';

package/dist/pg-name-value-store.d.ts

@@ -21,6 +21,61 @@ export interface PgNameValueStoreOptions {
  * - return a non-serializable value → cancel (no change), a `TypeError` is thrown.
  */
 export type UpdateCallback = (current: unknown) => unknown | Promise<unknown>;
+/** Options for {@link PgNameValueStore.list}. */
+export interface ListOptions {
+    /**
+     * Only names that start with this exact string. Literal, not a pattern: `%`,
+     * `_` and `\` in the prefix match themselves. Validated like a name (at most
+     * 1024 bytes of UTF-8, no `U+0000`). Empty or omitted: no filter.
+     */
+    prefix?: string;
+    /**
+     * Keyset cursor: only names strictly after this one in byte order — normally
+     * the `name` of the last entry of the previous page. Need not be an existing
+     * name. Validated like `prefix`. Empty or omitted: start from the first name.
+     */
+    after?: string;
+    /**
+     * Page size. A positive integer; default and maximum 1000 (larger values are
+     * clamped silently — keep paging while `have_more` is true). A non-integer or
+     * a value below 1 throws `TypeError` before any I/O.
+     */
+    limit?: number;
+    /**
+     * Whether to include values (default `true`). With `false` the `value` field
+     * is omitted from every entry and the value column is not read at all, so a
+     * store of large documents can be walked cheaply.
+     */
+    values?: boolean;
+}
+/** One entry of a {@link ListResult}. */
+export interface ListEntry {
+    /** The name. */
+    name: string;
+    /**
+     * The stored value — present (possibly `null`) whenever the page was listed
+     * with `values` on; omitted entirely with `values: false`. A name with no
+     * value has no entry at all, so this is never `undefined` when present.
+     */
+    value?: unknown;
+    /** When the value was last written (millisecond precision). */
+    updated_at: Date;
+}
+/** Result of {@link PgNameValueStore.list}. */
+export interface ListResult {
+    /** The page, in ascending byte order of `name`. */
+    entries: ListEntry[];
+    /**
+     * `true` if more names followed this page in the same snapshot; continue with
+     * `after: entries.at(-1).name`.
+     */
+    have_more: boolean;
+}
+/** Options for {@link PgNameValueStore.count}. */
+export interface CountOptions {
+    /** Same semantics as {@link ListOptions.prefix}. */
+    prefix?: string;
+}
 /**
  * A persistent name → value store backed by a single PostgreSQL table.
  *
@@ -32,6 +87,16 @@ export type UpdateCallback = (current: unknown) => unknown | Promise<unknown>;
  * The constructor takes a pg `Pool`; the schema (one table per namespace) is
  * created automatically and idempotently on first use, or via an explicit
  * {@link init} call. An existing table is never altered — only verified.
+ *
+ * **Concurrency.** Every write to a given name — {@link set}, {@link remove},
+ * and {@link update} — runs in its own transaction on a dedicated pooled
+ * connection, serialized with all other writes to the same name (in any
+ * process sharing the database) by a per-(namespace, name) advisory lock. The
+ * *previous value* each of them resolves to is therefore exact: it is the value
+ * left by the immediately preceding write. Writes to different names never
+ * block each other. Reads — {@link get}, {@link list}, {@link count} — are
+ * single lock-free statements that see the latest committed state and neither
+ * block writes nor are blocked by them.
  */
 export declare class PgNameValueStore {
     #private;
@@ -54,7 +119,7 @@ export declare class PgNameValueStore {
     /**
      * Resolves to the current value of `name`, or `undefined` if `name` has no
      * value. A name explicitly set to JSON `null` resolves to `null` (distinct
-     * from `undefined`).
+     * from `undefined`). A single lock-free `SELECT`.
      */
     get(name: string): Promise<unknown>;
     /**
@@ -62,14 +127,15 @@ export declare class PgNameValueStore {
      * resolves to the **previous** value, or `undefined` if there was none.
      *
      * `value` must be JSON-serializable and must not be `undefined`; otherwise
-     * `set` throws `TypeError` and writes nothing. Performed as a single
-     * statement, so it is atomic.
+     * `set` throws `TypeError` and writes nothing. Serialized with every other
+     * write to the same name (see the class docs), so the previous value is
+     * exactly the value left by the preceding write.
      */
     set(name: string, value: unknown): Promise<unknown>;
     /**
      * Removes `name` from the store and resolves to its previous value, or
      * `undefined` if `name` had no value. Removing an absent name is a no-op that
-     * resolves to `undefined`. A single `DELETE … RETURNING`; atomic.
+     * resolves to `undefined`. Serialized with every other write to the same name.
      */
     remove(name: string): Promise<unknown>;
     /**
@@ -78,19 +144,23 @@ export declare class PgNameValueStore {
      *
      * - `cb` returns a JSON-serializable value (not `undefined`) → store it;
      * - `cb` returns `undefined` → remove `name`;
-     * - `cb` returns a non-serializable value → roll back, throw `TypeError`;
-     * - `cb` throws `null`/`undefined` → roll back (graceful cancel), no error;
-     * - `cb` throws anything else → roll back, re-throw that value unchanged.
+     * - `cb` returns a non-serializable value → no change, throw `TypeError`;
+     * - `cb` throws `null`/`undefined` → no change (graceful cancel), no error;
+     * - `cb` throws anything else → no change, re-throw that value unchanged.
      *
      * On every non-throwing outcome (store, remove, or graceful cancel) `update`
      * resolves to the value that was in the database **before** the call (or
      * `undefined` if there was none).
      *
      * The whole sequence runs in one transaction on a dedicated pooled
-     * connection. Concurrent updates of the same `name` are serialized by a
-     * per-(namespace, name) transaction-scoped advisory lock, so each callback
-     * sees the committed result of the previous update — including the
-     * create-from-absent case. An existing row is additionally taken `FOR UPDATE`.
+     * connection, serialized with every other write to the same name by the
+     * per-(namespace, name) advisory lock, so each callback sees the committed
+     * result of the previous write — including the create-from-absent case. An
+     * existing row is additionally taken `FOR UPDATE`.
+     *
+     * The callback must not write to the **same name** through this or any other
+     * store instance (it would wait on the lock the callback itself holds); writes
+     * to other names are fine.
      */
     update(name: string, cb: UpdateCallback): Promise<unknown>;
     /**
@@ -99,4 +169,25 @@ export declare class PgNameValueStore {
      * are untouched.
      */
     removeAll(): Promise<void>;
+    /**
+     * Lists one page of names (with their values unless `values: false`) in
+     * ascending **byte order** of `name` (`COLLATE "C"`, so the order is the same
+     * on every database regardless of its locale). See {@link ListOptions}.
+     *
+     * Paging is keyset paging: pass the last `name` of a page as `after` to get
+     * the next one, while `have_more` is `true`. Each page is one lock-free
+     * statement over its own snapshot, so names written between two pages may or
+     * may not appear, and a name removed in between is simply absent.
+     *
+     * The primary-key index serves this query directly on databases whose
+     * default collation is `C`; elsewhere a page is a scan and sort of the
+     * store's names, which is fine for the sizes a per-service store reaches.
+     */
+    list(options?: ListOptions): Promise<ListResult>;
+    /**
+     * Resolves to the number of names in the store, or of those starting with
+     * `prefix` (same semantics as {@link ListOptions.prefix}). A single lock-free
+     * `SELECT count(*)`, which scans the matching names.
+     */
+    count(options?: CountOptions): Promise<number>;
 }

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "tr-pg-name-value-store",
-    "version": "0.0.0",
+    "version": "2.0.0",
     "description": "Persistent name/value store on PostgreSQL. JSONB values, self-maintaining never-migrated schema, atomic read-modify-write, multiple named stores per database.",
     "keywords": [
         "name-value",
@@ -29,7 +29,8 @@
         "embed-sql": "node scripts/embed-sql.mjs",
         "prebuild": "npm run embed-sql",
         "build": "tsc -p tsconfig.json",
-        "pretest": "npm run embed-sql",
+        "typecheck": "tsc -p tsconfig.test.json",
+        "pretest": "npm run embed-sql && npm run typecheck",
         "test": "vitest run",
         "prepack": "npm run build"
     },
@@ -37,11 +38,11 @@
         "pg": ">=8"
     },
     "devDependencies": {
-        "@types/node": "^20.0.0",
-        "@types/pg": "^8.10.0",
-        "pg": "^8.11.0",
-        "typescript": "^5.4.0",
-        "vitest": "^4.1.8"
+        "@types/node": "^20.19.0",
+        "@types/pg": "^8.23.0",
+        "pg": "^8.23.0",
+        "typescript": "^5.9.0",
+        "vitest": "^4.1.0"
     },
     "repository": {
         "type": "git",