@sema-agent/core-stores-sql 7.59.0 → 7.61.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -8,12 +8,47 @@ backend is built behind this face rather than inside core. Ships its own types;
8
8
 
9
9
  ## Read this first: this release carries no SQL stores
10
10
 
11
- The first published version fixes the package's public face and how it is built, checked and released; it does not
11
+ The published versions so far fix the package's public face and how it is built, checked and released; they do not
12
12
  carry the stores yet. `createSqlStorageBackend` and `createSqlDriver` check the version pairing, read their options
13
- (refusing each wrong value by name) and then refuse — they do not return an object without its stores.
14
- `baselineDdl(dialect)` resolves to an empty list and `publishedColumns()` returns an empty list. A later release fills
15
- the face in; a removed or re-kinded name is called out as breaking in the `@sema-agent/core` release notes, the same as
16
- for core's own exports.
13
+ (refusing each wrong value by name) and then refuse — they do not return an object without its stores. Since no
14
+ driver can be made yet, the backend's last read, the driver, refuses every call (`options.driver was not made by
15
+ createSqlDriver`). `baselineDdl(dialect)` resolves to an empty list and `publishedColumns()` returns an empty list. A
16
+ later release fills the face in; a removed or re-kinded name is called out as breaking in the `@sema-agent/core`
17
+ release notes, the same as for core's own exports.
18
+
19
+ ## How a deployment wires it
20
+
21
+ One pool, one driver: make the driver once and hand it to the backend and to your own stores.
22
+
23
+ ```ts
24
+ const driver = createSqlDriver({ dialect: "pg", pool, onFailOpen: (tag, detail) => log.warn(tag, detail) });
25
+ const backend = createSqlStorageBackend({
26
+ driver,
27
+ retention: "managed",
28
+ transactionHooks: { retention: { holdLock, liveReferences, beforeDelete, audit }, session: { beforeCommit } },
29
+ });
30
+ await backend.ensureSchema(); // core's tables first
31
+ const conn = await driver.connect();
32
+ try {
33
+ await assertCoreSchemaReady(conn, driver.dialect); // the first statement of your own ensure
34
+ // …your own tables, which may reference core's
35
+ } finally {
36
+ conn.release();
37
+ }
38
+ ```
39
+
40
+ - **The driver** carries how its connections are initialized and what the engine reported (`SqlDriver.facts`, and
41
+ `SqlTxConn.facts` on each connection), and the one fail-open sink every read of the package reports through; a tag
42
+ is one word of `SQL_FAIL_OPEN_TAGS`. The backend takes only a driver `createSqlDriver` made.
43
+ - **The hooks** (`SqlTransactionHooks`) are the only way a deployment's tables take part in the backend's
44
+ transactions; each runs inside the transaction and gets its statement face only (`SqlStatementFace`).
45
+ - **The schema** of core's tables is the package's: `ensureSchema()` creates, migrates and checks them under its own
46
+ lock (`SQL_ENSURE_LOCK`), records the applied migrations in its own version table (`SQL_SCHEMA_VERSION_TABLE`) and
47
+ answers a `SqlSchemaReport`. Your own ensure runs after it, starts with `assertCoreSchemaReady`, and uses a lock of
48
+ its own.
49
+ - **File-history bytes** go to a table of the backend's, or to your object store through `fileHistory.blobs`
50
+ (`SqlBlobBytesPort`: bytes only, under keys the backend mints — whether a blob is there is the backend's index).
51
+ - **The sweeps** (`backend.sweep({ nowMs })`) are one call your scheduler makes; the receipt counts each leg.
17
52
 
18
53
  The planned consumer is a deployment that already owns SQL stores: `@sema-agent/server` re-homing them behind this face
19
54
  is the first use in view (planned, not shipped). It would build the backend from its own pool and hand the result to
@@ -43,10 +78,13 @@ those would be going around the neutral seam. Each name below is a value or a ty
43
78
 
44
79
  | Group | Names |
45
80
  | --- | --- |
46
- | Backend | `createSqlStorageBackend` `SqlStorageBackend` `SqlStorageBackendOptions` `RetentionCascadeHooks` |
47
- | Driver | `createSqlDriver` `SqlDriver` `SqlTxConn` `PgPoolLike` `MysqlPoolLike` |
81
+ | Backend | `createSqlStorageBackend` `SqlStorageBackend` `SqlStorageBackendOptions` `SqlSweepLeg` `SqlSweepReceipt` |
82
+ | Hooks | `SqlTransactionHooks` `SqlStatementFace` `SqlRetentionOp` |
83
+ | Driver | `createSqlDriver` `SqlDriver` `SqlTxConn` `SqlEngineFacts` `SqlWallSource` `PgPoolLike` `MysqlPoolLike` |
84
+ | Fail-open | `SQL_FAIL_OPEN_TAGS` `SqlFailOpenTag` `SqlFailOpenSink` |
48
85
  | Words | `SqlDialect` `SqlEngine` |
49
- | Schema | `baselineDdl` `publishedColumns` `PublishedColumn` |
86
+ | Schema | `assertCoreSchemaReady` `SQL_ENSURE_LOCK` `SQL_SCHEMA_VERSION_TABLE` `SqlSchemaReport` `baselineDdl` `publishedColumns` `PublishedColumn` |
87
+ | File history | `SqlBlobBytesPort` `SqlBlobKey` |
50
88
 
51
89
  `SqlDialect` is which SQL text is written (`pg` or `mysql`; the caller says it). `SqlEngine` is which engine answered
52
90
  (`pg`, `innodb` or `tidb`; probed, never configured). `baselineDdl` is the statements a new database is sent, recorded
package/dist/backend.d.ts CHANGED
@@ -2,18 +2,31 @@
2
2
  * The SQL storage backend: core's engine storage seats over one SQL database.
3
3
  *
4
4
  * It is an `EngineStorageBackend` — each member wires into the `RunnerDeps` seat of the same name — that always carries
5
- * the backend-level managed-retention executor, and can create its own tables. Releasing it is the contract's
6
- * `dispose()`; there is no second release verb.
5
+ * the backend-level managed-retention executor, creates and migrates its own tables, and runs the age and orphan
6
+ * sweeps over them. Releasing it is the contract's `dispose()`; there is no second release verb.
7
7
  *
8
- * In this version the package carries no SQL stores, so the factory reads its options (each wrong value refused by
9
- * name) and then refuses: a backend whose seats are missing is not returned.
8
+ * In this version the package carries no SQL stores and no driver, so the factory reads its options (each wrong value
9
+ * refused by name, the driver last) and then refuses: no driver can be made yet, so the driver read refuses every
10
+ * call, and a backend whose seats are missing is never returned.
10
11
  */
11
12
  import type { EngineStorageBackend, ManagedRetentionCapability } from "@sema-agent/core";
12
13
  import { type SqlStorageBackendOptions } from "./options.js";
14
+ import type { SqlSchemaReport } from "./schema.js";
15
+ /** One leg of {@link SqlStorageBackend.sweep}: roster rows past their age, workflow journals past their retention,
16
+ * file-history bytes no row references past the grace window, tool results past their age. */
17
+ export type SqlSweepLeg = "roster-aged" | "workflow-journal-expired" | "file-history-orphan-blobs" | "tool-results-aged";
18
+ /** What one sweep removed, per leg. */
19
+ export type SqlSweepReceipt = Readonly<Record<SqlSweepLeg, number>>;
13
20
  export interface SqlStorageBackend extends EngineStorageBackend {
14
21
  /** The managed-retention executor over this backend's tables (one transaction per call, across every seat). */
15
22
  readonly retention: ManagedRetentionCapability;
16
- /** Create the tables this backend owns, if absent. Creates; never alters or drops. */
17
- ensureSchema(): Promise<void>;
23
+ /** Create the tables and indexes this backend owns, if absent, and run its registered migrations not yet applied —
24
+ * under the package's ensure lock — then check the column sets; answers what it did. Never drops. */
25
+ ensureSchema(): Promise<SqlSchemaReport>;
26
+ /** The age and orphan sweeps over this backend's tables, one call for every leg; the deployment's scheduler calls it.
27
+ * A leg whose bound the deployment did not set removes nothing. */
28
+ sweep(input: {
29
+ readonly nowMs: number;
30
+ }): Promise<SqlSweepReceipt>;
18
31
  }
19
32
  export declare function createSqlStorageBackend(opts: SqlStorageBackendOptions): SqlStorageBackend;
@@ -0,0 +1,34 @@
1
+ declare const sqlBlobKeyBrand: unique symbol;
2
+ /** An object key the backend minted for one publication of one blob. Opaque: a port stores and returns it as is. */
3
+ export type SqlBlobKey = string & {
4
+ readonly [sqlBlobKeyBrand]: true;
5
+ };
6
+ /** The byte I/O behind the file-history store (an object store, for one). Bytes only. */
7
+ export interface SqlBlobBytesPort {
8
+ /** Store `bytes` under `key`. Idempotent; a rejection is a failed write. */
9
+ put(key: SqlBlobKey, bytes: Uint8Array): Promise<void>;
10
+ /** The bytes under `key`, or `undefined`. The backend checks them against the hash and treats a mismatch as absent. */
11
+ get(key: SqlBlobKey): Promise<Uint8Array | undefined>;
12
+ /** Remove the bytes under each key. Idempotent; a key with nothing under it is a success. */
13
+ delete(keys: readonly SqlBlobKey[]): Promise<void>;
14
+ /** The keys stored, in order, after `after`: lets the backend find objects no row names. Absent ⇒ such objects are
15
+ * visible to an operator only. */
16
+ list?(after: SqlBlobKey | undefined, limit: number): Promise<{
17
+ readonly keys: readonly SqlBlobKey[];
18
+ readonly next?: SqlBlobKey;
19
+ }>;
20
+ }
21
+ /** A byte port, read: an object whose `put` / `get` / `delete` are functions and whose `list`, when present, is one. */
22
+ export declare function readBlobBytesPort(entry: string, where: string, value: unknown): SqlBlobBytesPort;
23
+ /** The state of one object key's index row; `"none"` = no row. */
24
+ export type SqlBlobRowState = "none" | "pending" | "live" | "deleting";
25
+ /** What moves a row: a publication reserving a fresh key, its put succeeding, the sweep claiming the row, the sweep's
26
+ * delete succeeding. */
27
+ export type SqlBlobEvent = "reserve" | "publish" | "claim" | "remove";
28
+ /** The only moves a row makes, each a compare-and-set from the state on its left. */
29
+ export declare const BLOB_TRANSITIONS: readonly (readonly [SqlBlobRowState, SqlBlobEvent, SqlBlobRowState])[];
30
+ /** Where `event` moves a row in `from`, or `undefined` when it does not (the compare-and-set misses). */
31
+ export declare function blobTransition(from: SqlBlobRowState, event: SqlBlobEvent): SqlBlobRowState | undefined;
32
+ /** Whether a row in `state` makes its hash present: only a `live` row does. */
33
+ export declare function blobPresent(state: SqlBlobRowState): boolean;
34
+ export {};
@@ -0,0 +1,26 @@
1
+ import { refusal } from "./refusal.js";
2
+ const PORT_MEMBERS = Object.freeze(["put", "get", "delete"]);
3
+ export function readBlobBytesPort(entry, where, value) {
4
+ if (typeof value !== "object" || value === null)
5
+ throw refusal(entry, `${where} must be an object with put / get / delete (got ${value === null ? "null" : typeof value})`);
6
+ const port = value;
7
+ const missing = PORT_MEMBERS.filter((m) => typeof port[m] !== "function");
8
+ if (missing.length > 0)
9
+ throw refusal(entry, `${where} has no ${missing.map((m) => `${m}()`).join(" / ")} — a byte port puts, gets and deletes`);
10
+ if (port["list"] !== undefined && typeof port["list"] !== "function")
11
+ throw refusal(entry, `${where}.list must be a function when given (got ${port["list"] === null ? "null" : typeof port["list"]})`);
12
+ return value;
13
+ }
14
+ export const BLOB_TRANSITIONS = Object.freeze([
15
+ Object.freeze(["none", "reserve", "pending"]),
16
+ Object.freeze(["pending", "publish", "live"]),
17
+ Object.freeze(["pending", "claim", "deleting"]),
18
+ Object.freeze(["live", "claim", "deleting"]),
19
+ Object.freeze(["deleting", "remove", "none"]),
20
+ ]);
21
+ export function blobTransition(from, event) {
22
+ return BLOB_TRANSITIONS.find(([state, move]) => state === from && move === event)?.[2];
23
+ }
24
+ export function blobPresent(state) {
25
+ return state === "live";
26
+ }
package/dist/driver.d.ts CHANGED
@@ -1,8 +1,35 @@
1
+ import { type SqlFailOpenSink } from "./fail-open.js";
1
2
  declare const SQL_DIALECTS: readonly ["pg", "mysql"];
2
3
  /** The SQL text family the stores write in: `"pg"` (PostgreSQL) or `"mysql"` (the MySQL protocol family, TiDB included). */
3
4
  export type SqlDialect = (typeof SQL_DIALECTS)[number];
4
5
  /** The engine a driver found behind the pool: PostgreSQL, InnoDB (MySQL / MariaDB) or TiDB. Probed, never configured. */
5
6
  export type SqlEngine = "pg" | "innodb" | "tidb";
7
+ /** Where a wall's number came from: `"reported"` = the engine answered it; `"assumed"` = the engine's documented
8
+ * default stood in because the read could not be taken (a fail-open read, reported through the driver's sink). */
9
+ export type SqlWallSource = "reported" | "assumed";
10
+ /**
11
+ * What the driver found about the engine when it initialized a connection. Read, never configured: the caller states
12
+ * the dialect, the driver probes everything here. A `null` wall is a wall this engine does not have.
13
+ */
14
+ export interface SqlEngineFacts {
15
+ /** Which engine answered. */
16
+ readonly engine: SqlEngine;
17
+ /** The engine's version string, as it reported it. */
18
+ readonly version: string;
19
+ /** The transaction isolation level, read back after the driver set it. */
20
+ readonly isolation: string;
21
+ /** TiDB's transaction mode, read back after the driver set it; `null` on an engine that has no such mode. */
22
+ readonly txnMode: "pessimistic" | null;
23
+ /** The largest row the engine stores, in bytes; `null` = this engine has no single-row wall. */
24
+ readonly maxRowBytes: number | null;
25
+ /** The largest statement the connection sends, in bytes (a session value); `null` = no such wall. */
26
+ readonly maxPacketBytes: number | null;
27
+ /** Where each wall's number came from. */
28
+ readonly wallSources: {
29
+ readonly maxRowBytes: SqlWallSource;
30
+ readonly maxPacketBytes: SqlWallSource;
31
+ };
32
+ }
6
33
  interface PgResultLike {
7
34
  readonly rows: unknown[];
8
35
  readonly rowCount: number | null;
@@ -45,13 +72,22 @@ export interface SqlTxConn {
45
72
  commit(): Promise<void>;
46
73
  rollback(): Promise<void>;
47
74
  release(): void;
75
+ /** What this connection's initialization read; fixed for the connection's life. The final word on a wall is read
76
+ * here, on the connection that is about to write. */
77
+ readonly facts: SqlEngineFacts;
48
78
  }
49
79
  /** The neutral execution seam: the dialect the statements are written in, pooled statements, and transactions. */
50
80
  export interface SqlDriver {
51
81
  readonly dialect: SqlDialect;
82
+ /** What the latest connection initialization read; `null` before any connection was lent, or when the latest
83
+ * initialization was refused. An admission check with no connection in hand reads this one; the same wall judgment
84
+ * runs again on {@link SqlTxConn.facts} of the connection that writes — one judgment, read twice, not two readings. */
85
+ readonly facts: SqlEngineFacts | null;
52
86
  query(sql: string, params?: unknown[]): Promise<SqlResult>;
53
87
  connect(): Promise<SqlTxConn>;
54
88
  }
89
+ /** The one reading of a driver handed to the backend: present, and one {@link createSqlDriver} made. */
90
+ export declare function readDriver(entry: string, value: unknown): SqlDriver;
55
91
  /** The one reading of a dialect word. Any other value — `"tidb"` included — is refused by name. */
56
92
  export declare function readDialect(entry: string, value: unknown): SqlDialect;
57
93
  /** The pool, read against the face its dialect needs: a pool missing a member the driver calls is refused by name. */
@@ -61,11 +97,14 @@ export declare function refuseUnknownKeys(entry: string, where: string, value: R
61
97
  /** Read a factory's options argument as a record, or refuse. */
62
98
  export declare function readRecord(entry: string, where: string, value: unknown): Record<string, unknown>;
63
99
  /**
64
- * The SQL driver over one pool. In this version the package carries no driver: the options are read (version, dialect,
65
- * pool — each refused by name when wrong) and then the call refuses, rather than return a driver that cannot run.
100
+ * The SQL driver over one pool: `dialect` is which SQL text the stores write, `pool` is the caller's pool of that
101
+ * dialect (borrowed, never ended), and `onFailOpen` is told once per fail-open read the package takes — the driver's
102
+ * and every store's. In this version the package carries no driver: the options are read (version, dialect, pool,
103
+ * sink — each refused by name when wrong) and then the call refuses, rather than return a driver that cannot run.
66
104
  */
67
105
  export declare function createSqlDriver(opts: {
68
106
  dialect: SqlDialect;
69
107
  pool: PgPoolLike | MysqlPoolLike;
108
+ onFailOpen?: SqlFailOpenSink;
70
109
  }): SqlDriver;
71
110
  export {};
package/dist/driver.js CHANGED
@@ -1,6 +1,17 @@
1
1
  import { assertCoreVersion } from "./version-check.js";
2
2
  import { refusal } from "./refusal.js";
3
+ import { readFailOpenSink } from "./fail-open.js";
3
4
  const SQL_DIALECTS = Object.freeze(["pg", "mysql"]);
5
+ const MADE_DRIVERS = new WeakSet();
6
+ export function readDriver(entry, value) {
7
+ if (value === undefined)
8
+ throw refusal(entry, "driver is required — the driver createSqlDriver made over your pool (one pool, one driver: hand the same one to your own stores)");
9
+ if (typeof value !== "object" || value === null)
10
+ throw refusal(entry, `driver must be the object createSqlDriver returned (got ${value === null ? "null" : typeof value})`);
11
+ if (!MADE_DRIVERS.has(value))
12
+ throw refusal(entry, "options.driver was not made by createSqlDriver — an object of the same shape cannot vouch for how its connections were initialized");
13
+ return value;
14
+ }
4
15
  const isObject = (value) => typeof value === "object" && value !== null && !Array.isArray(value);
5
16
  export function readDialect(entry, value) {
6
17
  if (typeof value === "string" && SQL_DIALECTS.includes(value))
@@ -27,11 +38,13 @@ export function readRecord(entry, where, value) {
27
38
  throw refusal(entry, `${where} must be an object (got ${value === null ? "null" : Array.isArray(value) ? "an array" : typeof value})`);
28
39
  return value;
29
40
  }
41
+ const DRIVER_OPTION_KEYS = Object.freeze(["dialect", "pool", "onFailOpen"]);
30
42
  export function createSqlDriver(opts) {
31
43
  const entry = "createSqlDriver";
32
44
  assertCoreVersion(entry);
33
45
  const record = readRecord(entry, "options", opts);
34
- refuseUnknownKeys(entry, "options", record, ["dialect", "pool"]);
46
+ refuseUnknownKeys(entry, "options", record, DRIVER_OPTION_KEYS);
35
47
  readPool(entry, readDialect(entry, record["dialect"]), record["pool"]);
48
+ readFailOpenSink(entry, record["onFailOpen"]);
36
49
  throw refusal(entry, "this version of the package carries no SQL driver — the options are valid, but there is no driver to return");
37
50
  }
@@ -0,0 +1,8 @@
1
+ /** Every tag the fail-open sink can be told, in one frozen list. */
2
+ export declare const SQL_FAIL_OPEN_TAGS: readonly ["core-stores-sql.driver.packet-wall-log-unwritten", "core-stores-sql.driver.tidb-row-cap-unread", "core-stores-sql.driver.packet-cap-unread"];
3
+ /** A fail-open read the package took: one word of {@link SQL_FAIL_OPEN_TAGS}. */
4
+ export type SqlFailOpenTag = (typeof SQL_FAIL_OPEN_TAGS)[number];
5
+ /** The sink told once per fail-open read: `tag` names the read, `detail` says what stood in for it. */
6
+ export type SqlFailOpenSink = (tag: SqlFailOpenTag, detail: string) => void;
7
+ /** The sink option, read: absent, or a function. */
8
+ export declare function readFailOpenSink(entry: string, value: unknown): SqlFailOpenSink | undefined;
@@ -0,0 +1,13 @@
1
+ import { refusal } from "./refusal.js";
2
+ export const SQL_FAIL_OPEN_TAGS = Object.freeze([
3
+ "core-stores-sql.driver.packet-wall-log-unwritten",
4
+ "core-stores-sql.driver.tidb-row-cap-unread",
5
+ "core-stores-sql.driver.packet-cap-unread",
6
+ ]);
7
+ export function readFailOpenSink(entry, value) {
8
+ if (value === undefined)
9
+ return undefined;
10
+ if (typeof value !== "function")
11
+ throw refusal(entry, `onFailOpen must be a function when given (got ${value === null ? "null" : typeof value})`);
12
+ return value;
13
+ }
package/dist/index.d.ts CHANGED
@@ -1,11 +1,15 @@
1
1
  /**
2
2
  * `@sema-agent/core-stores-sql` — core's storage seats over PostgreSQL and the MySQL protocol family.
3
3
  *
4
- * The public face is neutral factories and neutral types only: no store class, no dialect class, no statement list,
5
- * no ensure leg is exported. A caller hands in a pool and a dialect word and gets core's seats back.
4
+ * The public face is neutral factories, neutral types and the closed sets a deployment reads: no store class, no
5
+ * dialect class, no statement list, no ensure leg is exported. A caller makes one driver over its pool and gets
6
+ * core's seats back from a backend over that driver.
6
7
  */
7
- export { createSqlStorageBackend, type SqlStorageBackend } from "./backend.js";
8
- export type { RetentionCascadeHooks, SqlStorageBackendOptions } from "./options.js";
9
- export { createSqlDriver, type MysqlPoolLike, type PgPoolLike, type SqlDialect, type SqlDriver, type SqlEngine, type SqlTxConn } from "./driver.js";
8
+ export { createSqlStorageBackend, type SqlStorageBackend, type SqlSweepLeg, type SqlSweepReceipt } from "./backend.js";
9
+ export type { SqlRetentionOp, SqlStatementFace, SqlStorageBackendOptions, SqlTransactionHooks } from "./options.js";
10
+ export { createSqlDriver, type MysqlPoolLike, type PgPoolLike, type SqlDialect, type SqlDriver, type SqlEngine, type SqlEngineFacts, type SqlTxConn, type SqlWallSource, } from "./driver.js";
11
+ export { SQL_FAIL_OPEN_TAGS, type SqlFailOpenSink, type SqlFailOpenTag } from "./fail-open.js";
12
+ export type { SqlBlobBytesPort, SqlBlobKey } from "./blob-index.js";
13
+ export { assertCoreSchemaReady, SQL_ENSURE_LOCK, SQL_SCHEMA_VERSION_TABLE, type SqlSchemaReport } from "./schema.js";
10
14
  export { baselineDdl } from "./ddl-recorder.js";
11
15
  export { publishedColumns, type PublishedColumn } from "./published-columns.js";
package/dist/index.js CHANGED
@@ -1,4 +1,6 @@
1
1
  export { createSqlStorageBackend } from "./backend.js";
2
- export { createSqlDriver } from "./driver.js";
2
+ export { createSqlDriver, } from "./driver.js";
3
+ export { SQL_FAIL_OPEN_TAGS } from "./fail-open.js";
4
+ export { assertCoreSchemaReady, SQL_ENSURE_LOCK, SQL_SCHEMA_VERSION_TABLE } from "./schema.js";
3
5
  export { baselineDdl } from "./ddl-recorder.js";
4
6
  export { publishedColumns } from "./published-columns.js";
package/dist/options.d.ts CHANGED
@@ -1,41 +1,91 @@
1
1
  /**
2
2
  * The options of the SQL storage backend, and their one reading.
3
3
  *
4
- * Every value is read loudly: a wrong dialect word, a pool of the other dialect, a retention word core does not
5
- * know, a hook that is not a function, or a key this version has no option for is refused by name — never read as
6
- * "absent" and replaced by a default.
4
+ * Every value is read loudly: a retention word core does not know, a hook that is not a function, a key this version
5
+ * has no option for, or a driver {@link import("./driver.js").createSqlDriver} did not make is refused by name —
6
+ * never read as "absent" and replaced by a default. The driver is read last, so every other wrong value is named
7
+ * before it.
8
+ *
9
+ * A deployment takes part in the backend's transactions through one closed table of hooks
10
+ * ({@link SqlTransactionHooks}); each hook runs inside the backend's transaction, on the connection that transaction
11
+ * holds, and gets its statement face only — the transaction is the backend's to commit or roll back. A hook sends
12
+ * statements and nothing else: a side effect outside the database (a directory, an object) is recorded as a row of
13
+ * the deployment's own in the same transaction and carried out by the deployment after the commit, so a rollback
14
+ * takes it back.
7
15
  */
8
- import { type RetentionDeclaration } from "@sema-agent/core";
9
- import { type MysqlPoolLike, type PgPoolLike, type SqlDialect, type SqlTxConn } from "./driver.js";
16
+ import { type ManagedRetentionCapability, type RetentionDeclaration, type RetentionReceipt } from "@sema-agent/core";
17
+ import { type SqlDriver, type SqlTxConn } from "./driver.js";
18
+ import { type SqlBlobBytesPort } from "./blob-index.js";
19
+ /** The statements a hook may send: the `query` of the connection the backend's transaction holds. */
20
+ export type SqlStatementFace = Pick<SqlTxConn, "query">;
21
+ /** The destructive retention calls a hook is told about — the executor's members that delete. */
22
+ export type SqlRetentionOp = Exclude<keyof ManagedRetentionCapability, "listRetentionDomains">;
10
23
  /**
11
- * How a deployment's own tables take part in a managed-retention sweep. Both hooks run inside the backend's deletion
12
- * transaction, on the connection that transaction holds, and get its statement face only — the transaction is the
13
- * backend's to commit or roll back.
24
+ * Where a deployment's tables take part in the backend's transactions. Closed: each group and each hook is named
25
+ * here, and any other key is refused.
26
+ *
27
+ * A destructive retention transaction runs, in order: `holdLock` → the backend locks its candidates (`FOR UPDATE`,
28
+ * each candidate's session row included) → `session.beforeCommit` for each candidate session
29
+ * (`deleteExpiredSessions` only) → `liveReferences` over the candidates left → `beforeDelete` over the final set (the
30
+ * candidates minus the refused minus the referenced) → the backend deletes its rows and writes its tombstones →
31
+ * `audit` → commit. A session no hook lets go is in no later hook's input.
14
32
  */
15
- export interface RetentionCascadeHooks {
16
- /** Called after the backend has locked its candidate sessions (`FOR UPDATE`): return the ids among them that a
17
- * deployment row still references. Those are skipped this round. */
18
- liveReferences?(txn: Pick<SqlTxConn, "query">, candidates: readonly string[]): Promise<ReadonlySet<string>>;
19
- /** Called before the backend deletes its own rows for the final set (the locked candidates minus the skipped):
20
- * the deployment deletes its rows for the same sessions. Each row carries the session's incarnation. */
21
- beforeDelete?(txn: Pick<SqlTxConn, "query">, rows: readonly {
22
- readonly sessionId: string;
23
- readonly incarnation: string | undefined;
24
- }[]): Promise<void>;
33
+ export interface SqlTransactionHooks {
34
+ readonly retention?: {
35
+ /** The first statement of every destructive retention transaction: the deployment locks and reads its own hold
36
+ * record for the domain and keeps the lock until the commit. `"held"` ⇒ the backend deletes nothing in that
37
+ * domain this call and counts its candidates as skipped. */
38
+ holdLock?(txn: SqlStatementFace, call: {
39
+ readonly domain: string;
40
+ readonly op: SqlRetentionOp;
41
+ }): Promise<"proceed" | "held">;
42
+ /** Called after the backend has locked its candidate sessions: return the ids among them that a deployment row
43
+ * still references. Those are skipped this round. */
44
+ liveReferences?(txn: SqlStatementFace, candidates: readonly string[]): Promise<ReadonlySet<string>>;
45
+ /** Called before the backend deletes its own rows for the final set: the deployment deletes its rows for the same
46
+ * sessions. Each row carries the session's incarnation. */
47
+ beforeDelete?(txn: SqlStatementFace, rows: readonly {
48
+ readonly sessionId: string;
49
+ readonly incarnation: string | undefined;
50
+ }[]): Promise<void>;
51
+ /** The last statement before the commit: the deployment writes its own audit row in the same transaction — the
52
+ * round, the policy generation and the fencing token are the deployment's to know. `receipt` is the call's three
53
+ * counts. */
54
+ audit?(txn: SqlStatementFace, call: {
55
+ readonly domain: string;
56
+ readonly op: SqlRetentionOp;
57
+ }, receipt: RetentionReceipt): Promise<void>;
58
+ };
59
+ readonly session?: {
60
+ /**
61
+ * Called before a write of the backend's that changes a session tree's root — an import's commit, an owner's
62
+ * delete, a retention delete — after the root's session row is locked and before anything is written.
63
+ * `{ refuse }` ⇒ nothing is written: an import or a delete rejects with an error carrying the deployment's
64
+ * sentence (not the session `conflict` code — a conflict is retried, a refusal is not); a retention delete counts
65
+ * the session as skipped and goes on with the others.
66
+ */
67
+ beforeCommit?(txn: SqlStatementFace, write: {
68
+ readonly op: "import" | "delete" | "retention-delete";
69
+ readonly rootSessionId: string;
70
+ }): Promise<"proceed" | {
71
+ readonly refuse: string;
72
+ }>;
73
+ };
25
74
  }
26
75
  /** The options of {@link import("./backend.js").createSqlStorageBackend}. */
27
76
  export interface SqlStorageBackendOptions {
28
- /** Which SQL text the stores write: `"pg"` or `"mysql"`. */
29
- dialect: SqlDialect;
30
- /** The pool of that dialect. The backend borrows from it and never ends it; the caller owns the pool. */
31
- pool: PgPoolLike | MysqlPoolLike;
77
+ /** The driver {@link import("./driver.js").createSqlDriver} made over the deployment's pool — the same one the
78
+ * deployment's own stores use. The backend borrows its connections and never ends the pool. */
79
+ driver: SqlDriver;
32
80
  /** The stores' managed-retention declaration — the word core reads (`RetentionDeclaration`). Required: the
33
81
  * deployment states it rather than inherit a default. */
34
82
  retention: RetentionDeclaration;
35
- /** The deployment's tables in a retention sweep. */
36
- retentionCascade?: RetentionCascadeHooks;
37
- /** Told once per fail-open read the stores take (`tag` names the read, `detail` says what stood in). */
38
- onFailOpen?: (tag: string, detail: string) => void;
83
+ /** The deployment's tables in the backend's transactions. */
84
+ transactionHooks?: SqlTransactionHooks;
85
+ /** The file-history store's byte storage: absent ⇒ the bytes are kept in a table of the backend's. */
86
+ fileHistory?: {
87
+ blobs?: SqlBlobBytesPort;
88
+ };
39
89
  }
40
90
  /** Read the backend's options, or refuse naming the first wrong value. */
41
91
  export declare function readBackendOptions(entry: string, value: unknown): SqlStorageBackendOptions;
package/dist/options.js CHANGED
@@ -1,8 +1,13 @@
1
1
  import { readRetentionDeclaration } from "@sema-agent/core";
2
- import { readDialect, readPool, readRecord, refuseUnknownKeys } from "./driver.js";
2
+ import { readDriver, readRecord, refuseUnknownKeys } from "./driver.js";
3
+ import { readBlobBytesPort } from "./blob-index.js";
3
4
  import { refusal } from "./refusal.js";
4
- const OPTION_KEYS = Object.freeze(["dialect", "pool", "retention", "retentionCascade", "onFailOpen"]);
5
- const CASCADE_KEYS = Object.freeze(["liveReferences", "beforeDelete"]);
5
+ const OPTION_KEYS = Object.freeze(["driver", "retention", "transactionHooks", "fileHistory"]);
6
+ const HOOK_GROUPS = Object.freeze({
7
+ retention: Object.freeze(["holdLock", "liveReferences", "beforeDelete", "audit"]),
8
+ session: Object.freeze(["beforeCommit"]),
9
+ });
10
+ const FILE_HISTORY_KEYS = Object.freeze(["blobs"]);
6
11
  function readFunction(entry, where, value) {
7
12
  if (value !== undefined && typeof value !== "function")
8
13
  throw refusal(entry, `${where} must be a function when given (got ${value === null ? "null" : typeof value})`);
@@ -17,25 +22,36 @@ function readRetention(entry, value) {
17
22
  throw refusal(entry, `retention: ${error instanceof Error ? error.message : String(error)}`, error);
18
23
  }
19
24
  }
25
+ function readTransactionHooks(entry, value) {
26
+ const groups = readRecord(entry, "transactionHooks", value);
27
+ refuseUnknownKeys(entry, "transactionHooks", groups, Object.keys(HOOK_GROUPS));
28
+ for (const [group, hooks] of Object.entries(HOOK_GROUPS)) {
29
+ if (groups[group] === undefined)
30
+ continue;
31
+ const where = `transactionHooks.${group}`;
32
+ const record = readRecord(entry, where, groups[group]);
33
+ refuseUnknownKeys(entry, where, record, hooks);
34
+ for (const hook of hooks)
35
+ readFunction(entry, `${where}.${hook}`, record[hook]);
36
+ }
37
+ return groups;
38
+ }
39
+ function readFileHistory(entry, value) {
40
+ const record = readRecord(entry, "fileHistory", value);
41
+ refuseUnknownKeys(entry, "fileHistory", record, FILE_HISTORY_KEYS);
42
+ return record["blobs"] === undefined ? {} : { blobs: readBlobBytesPort(entry, "fileHistory.blobs", record["blobs"]) };
43
+ }
20
44
  export function readBackendOptions(entry, value) {
21
45
  const record = readRecord(entry, "options", value);
22
46
  refuseUnknownKeys(entry, "options", record, OPTION_KEYS);
23
- const dialect = readDialect(entry, record["dialect"]);
24
- const pool = readPool(entry, dialect, record["pool"]);
25
47
  const retention = readRetention(entry, record["retention"]);
26
- const cascade = record["retentionCascade"];
27
- if (cascade !== undefined) {
28
- const hooks = readRecord(entry, "retentionCascade", cascade);
29
- refuseUnknownKeys(entry, "retentionCascade", hooks, CASCADE_KEYS);
30
- for (const key of CASCADE_KEYS)
31
- readFunction(entry, `retentionCascade.${key}`, hooks[key]);
32
- }
33
- readFunction(entry, "onFailOpen", record["onFailOpen"]);
48
+ const transactionHooks = record["transactionHooks"] === undefined ? undefined : readTransactionHooks(entry, record["transactionHooks"]);
49
+ const fileHistory = record["fileHistory"] === undefined ? undefined : readFileHistory(entry, record["fileHistory"]);
50
+ const driver = readDriver(entry, record["driver"]);
34
51
  return {
35
- dialect,
36
- pool,
52
+ driver,
37
53
  retention,
38
- ...(cascade !== undefined ? { retentionCascade: cascade } : {}),
39
- ...(record["onFailOpen"] !== undefined ? { onFailOpen: record["onFailOpen"] } : {}),
54
+ ...(transactionHooks !== undefined ? { transactionHooks } : {}),
55
+ ...(fileHistory !== undefined ? { fileHistory } : {}),
40
56
  };
41
57
  }
@@ -0,0 +1,59 @@
1
+ /**
2
+ * Who creates and changes core's tables, and how a deployment checks they are there before it touches its own.
3
+ *
4
+ * Core's tables have one owner — this package. The backend's `ensureSchema()` creates the missing tables and indexes,
5
+ * runs the registered migrations not yet applied (each in its own transaction, its version row written in the same
6
+ * transaction), reconciles the column sets (a database that skipped a version, or is missing a column, is refused by
7
+ * name), and answers what it did ({@link SqlSchemaReport}). All of it runs under the package's own ensure lock
8
+ * ({@link SQL_ENSURE_LOCK}: taken by a non-blocking try with jittered back-off, never a blocking wait — a session
9
+ * parked in a blocking lock holds a snapshot that a concurrent index build waits on), and the applied migrations are
10
+ * rows of the package's own version table ({@link SQL_SCHEMA_VERSION_TABLE}).
11
+ *
12
+ * A deployment's own ensure runs AFTER the backend's: its tables reference core's, so core's must be there first. The
13
+ * order is the deployment's call order — `await backend.ensureSchema()`, then its own — and the first statement of its
14
+ * own ensure is {@link assertCoreSchemaReady}, which refuses loudly when core's ensure has not run, instead of letting
15
+ * a reference to a missing table be created, ignored, or fail later on a user's request.
16
+ *
17
+ * The migration registry ships empty: this version carries no tables, and the first stores arrive with their tables
18
+ * created whole.
19
+ */
20
+ import { type SqlDialect } from "./driver.js";
21
+ import type { SqlStatementFace } from "./options.js";
22
+ /** What one `ensureSchema()` did. */
23
+ export interface SqlSchemaReport {
24
+ /** The tables it created (a table already there is not listed). */
25
+ readonly tablesCreated: readonly string[];
26
+ /** The indexes it created. */
27
+ readonly indexesCreated: readonly string[];
28
+ /** The registered migrations it applied, in the order it applied them. */
29
+ readonly migrationsApplied: readonly string[];
30
+ }
31
+ /** The package's version table: one row per applied migration. */
32
+ export declare const SQL_SCHEMA_VERSION_TABLE = "core_schema_migrations";
33
+ /** The package's ensure lock: the PostgreSQL advisory-lock key (one 64-bit key) and the MySQL named lock. A
34
+ * deployment's own ensure lock must not be either. */
35
+ export declare const SQL_ENSURE_LOCK: {
36
+ readonly pg: number;
37
+ readonly mysql: "sema_core_ensure_schema";
38
+ };
39
+ /** One registered migration: its id (its identity in the version table), the version that introduced it, and the
40
+ * statements it runs, inside the transaction that also writes its version row. */
41
+ export interface SqlMigration {
42
+ readonly id: string;
43
+ readonly introducedIn: string;
44
+ readonly run: (txn: SqlStatementFace) => Promise<void>;
45
+ }
46
+ /** The package's registered migrations, in the order they run. Empty in this version. */
47
+ export declare const SQL_MIGRATIONS: readonly SqlMigration[];
48
+ /** A migration registry, read: unique non-empty ids, each introduced in an `X.Y.Z` version, never earlier than the one
49
+ * before it. */
50
+ export declare function readMigrations(entry: string, rows: readonly SqlMigration[]): readonly SqlMigration[];
51
+ /** The check behind {@link assertCoreSchemaReady}, over a given registry. */
52
+ export declare function checkCoreSchema(entry: string, face: SqlStatementFace, dialect: SqlDialect, registry: readonly SqlMigration[]): Promise<void>;
53
+ /**
54
+ * The first statement of a deployment's own ensure: refuses unless core's ensure has run on this database — its
55
+ * version table is there and every migration this version registers is applied. Reads only that table, writes
56
+ * nothing. `face` is the statement face of a connection the deployment's ensure holds (the driver's, since the
57
+ * deployment's stores share it); `dialect` is the driver's.
58
+ */
59
+ export declare function assertCoreSchemaReady(face: SqlStatementFace, dialect: SqlDialect): Promise<void>;
package/dist/schema.js ADDED
@@ -0,0 +1,76 @@
1
+ import { readDialect } from "./driver.js";
2
+ import { refusal } from "./refusal.js";
3
+ export const SQL_SCHEMA_VERSION_TABLE = "core_schema_migrations";
4
+ export const SQL_ENSURE_LOCK = Object.freeze({ pg: 1_397_836_867, mysql: "sema_core_ensure_schema" });
5
+ export const SQL_MIGRATIONS = Object.freeze([]);
6
+ const VERSION_RE = /^\d+\.\d+\.\d+$/;
7
+ export function readMigrations(entry, rows) {
8
+ const seen = new Set();
9
+ let previous;
10
+ for (const row of rows) {
11
+ if (typeof row.id !== "string" || row.id.trim() === "")
12
+ throw refusal(entry, `a registered migration has no id (${JSON.stringify(row.id)})`);
13
+ if (seen.has(row.id))
14
+ throw refusal(entry, `the migration "${row.id}" is registered twice — its id is its identity in ${SQL_SCHEMA_VERSION_TABLE}`);
15
+ seen.add(row.id);
16
+ if (typeof row.introducedIn !== "string" || !VERSION_RE.test(row.introducedIn))
17
+ throw refusal(entry, `the migration "${row.id}" names no X.Y.Z version it was introduced in (${JSON.stringify(row.introducedIn)})`);
18
+ if (typeof row.run !== "function")
19
+ throw refusal(entry, `the migration "${row.id}" has no run()`);
20
+ const version = row.introducedIn.split(".").map(Number);
21
+ if (previous !== undefined && compareVersions(version, previous) < 0)
22
+ throw refusal(entry, `the migration "${row.id}" (${row.introducedIn}) is registered after a later version's — migrations run in the order they were introduced`);
23
+ previous = version;
24
+ }
25
+ return rows;
26
+ }
27
+ function compareVersions(a, b) {
28
+ for (let i = 0; i < 3; i += 1)
29
+ if (a[i] !== b[i])
30
+ return a[i] - b[i];
31
+ return 0;
32
+ }
33
+ const VERSION_TABLE_PROBE = Object.freeze({
34
+ pg: "SELECT to_regclass($1) IS NOT NULL AS present",
35
+ mysql: "SELECT COUNT(*) AS present FROM information_schema.tables WHERE table_schema = DATABASE() AND table_name = ?",
36
+ });
37
+ function isYes(value) {
38
+ if (value === true)
39
+ return true;
40
+ if (typeof value === "number")
41
+ return value > 0;
42
+ if (typeof value === "bigint")
43
+ return value > 0n;
44
+ return typeof value === "string" && /^[1-9]\d*$/.test(value);
45
+ }
46
+ const ORDER = "run the backend's ensureSchema() first and the deployment's own ensure after it — a deployment table that references a core table needs that table to be there";
47
+ export async function checkCoreSchema(entry, face, dialect, registry) {
48
+ const which = readDialect(entry, dialect);
49
+ if (typeof face !== "object" || face === null || typeof face.query !== "function")
50
+ throw refusal(entry, "face must be a connection's statement face (an object with query())");
51
+ const migrations = readMigrations(entry, registry);
52
+ let present;
53
+ try {
54
+ present = (await face.query(VERSION_TABLE_PROBE[which], [SQL_SCHEMA_VERSION_TABLE])).rows[0]?.["present"];
55
+ }
56
+ catch (error) {
57
+ throw refusal(entry, `could not ask whether core's version table ${SQL_SCHEMA_VERSION_TABLE} is there (${error instanceof Error ? error.message : String(error)}) — a check that cannot be made does not pass`, error);
58
+ }
59
+ if (!isYes(present))
60
+ throw refusal(entry, `core's tables are not there: its version table ${SQL_SCHEMA_VERSION_TABLE} is missing (the probe answered ${JSON.stringify(typeof present === "bigint" ? String(present) : present)}) — ${ORDER}`);
61
+ if (migrations.length === 0)
62
+ return;
63
+ let applied;
64
+ try {
65
+ applied = new Set((await face.query(`SELECT id FROM ${SQL_SCHEMA_VERSION_TABLE}`)).rows.map((row) => String(row["id"])));
66
+ }
67
+ catch (error) {
68
+ throw refusal(entry, `could not read core's version table ${SQL_SCHEMA_VERSION_TABLE} (${error instanceof Error ? error.message : String(error)}) — a check that cannot be made does not pass`, error);
69
+ }
70
+ const missing = migrations.filter((m) => !applied.has(m.id)).map((m) => m.id);
71
+ if (missing.length > 0)
72
+ throw refusal(entry, `core's tables are not those of this version: the migration(s) ${missing.join(", ")} have not run — ${ORDER}`);
73
+ }
74
+ export async function assertCoreSchemaReady(face, dialect) {
75
+ await checkCoreSchema("assertCoreSchemaReady", face, dialect, SQL_MIGRATIONS);
76
+ }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@sema-agent/core-stores-sql",
3
- "version": "7.59.0",
3
+ "version": "7.61.0",
4
4
  "private": false,
5
5
  "description": "Core's storage seats over PostgreSQL and the MySQL protocol family (MySQL, MariaDB, TiDB): neutral factories over a caller's pool. This version carries no SQL stores yet: the public face only.",
6
6
  "type": "module",
@@ -26,7 +26,7 @@
26
26
  "node": ">=20.18.1"
27
27
  },
28
28
  "peerDependencies": {
29
- "@sema-agent/core": "7.59.0"
29
+ "@sema-agent/core": "7.61.0"
30
30
  },
31
31
  "publishConfig": {
32
32
  "access": "public"
@@ -34,5 +34,5 @@
34
34
  "scripts": {
35
35
  "build": "tsc -p tsconfig.json && tsc -p tsconfig.json --emitDeclarationOnly --removeComments false"
36
36
  },
37
- "gitHead": "66b1845c9644930fa7f43fd7c388316c3b060ee1"
37
+ "gitHead": "039e5ecf552494dca0785926ba9a11b578b76fc6"
38
38
  }