@jskit-ai/database-runtime 0.1.261 → 0.1.263
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/package.json +4 -5
- package/src/server/database.js +7 -7
- package/src/server/databaseSetup.js +3 -3
- package/src/server/knexMigrationConfig.js +7 -2
- package/src/server/testDatabases.js +52 -3
- package/src/server/types.js +1 -2
- package/src/shared/databaseConnection.js +20 -1
- package/src/shared/duplicateEntry.js +10 -1
- package/src/shared/index.js +1 -9
- package/src/shared/repositoryOptions.js +41 -119
- package/test/databaseConnection.postgres.test.js +66 -0
- package/test/databaseConnection.test.js +26 -1
- package/test/duplicateEntry.test.js +16 -0
- package/test/entrypoints.boundary.test.js +18 -2
- package/test/knexMigrationConfig.test.js +39 -8
- package/test/providerRuntime.test.js +2 -2
- package/test/repositoryOptions.test.js +81 -76
- package/test/testDatabases.test.js +37 -0
- package/src/shared/transactionManager.js +0 -31
- package/src/shared/visibility.js +0 -73
- package/test/runtimeCore.test.js +0 -51
- package/test/visibility.test.js +0 -93
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## Unreleased
|
|
4
|
+
|
|
5
|
+
- `@jskit-ai/database-runtime-mysql` and `@jskit-ai/database-runtime-postgres` take `@jskit-ai/database-runtime` as a required peer at the same version, where it was a dependency, as `@jskit-ai/resource-runtime` and `@jskit-ai/crud-core` do: the application declares it beside its driver (`npm install @jskit-ai/database-runtime @jskit-ai/database-runtime-mysql`) and holds its one copy. npm installs a required peer as it installed the dependency.
|
|
6
|
+
- **Breaking (data-access redesign).** `./shared/visibility` is removed (`applyVisibility`, `applyVisibilityOwners`, `DEFAULT_VISIBILITY_COLUMNS`, also re-exported from `./shared`): a table resource declares its `ownership`, and every handle statement is scoped by it. `./shared` and `./shared/repositoryOptions` no longer export `applyForUpdate`, `createWithTransaction`, `joinOrOpenTransaction`, `lockScopedRecordId`, `resolveQueryOptions`, `resolveRepoClient`, `resolveTransaction` and `runInTransaction`: a transaction is the table handle's `withTransaction(options, work)`, a row lock its `lock` or `lockMany`, and a call's client and transaction ride in the `options` it passes on. The Before | Now table in the existing-application migration guide maps each one.
|
|
7
|
+
- `./server/testDatabases` exports `requireTestDatabase(environment)`, which answers `TEST_DB_NAME` and throws `TestDatabaseMissingError` without it, so a data test fails instead of skipping, and `tableNames(knex)`, the tables of the test database.
|
|
8
|
+
- `retryLostInsertRace(options, run)` (from `./shared`) runs a transaction once more when its insert lost a race for a unique key: `run`, usually `() => repository.withTransaction(options, work)`, failed with a duplicate whose transaction rolled back in full, outside a caller's transaction (`isLostDuplicateInsertRace`). The database reports the duplicate only once the winner has committed, so the second run's lookup by the key finds the winner's row; a second duplicate is final, and inside a caller's transaction (`options.trx`) the error reaches the caller without a rerun. Apps wrote this retry by hand around each idempotent write.
|
|
9
|
+
- The migration configuration lists the application's `migrations/constraints/` only when the directory exists, as it already did for a package's: an application without one failed `db:migrate` and `jskit test --databases` with ENOENT (scandir `migrations/constraints`).
|
|
10
|
+
- `retryDeadlocks` (and so `withTransaction` and every handle write outside a transaction) also runs again a transaction that MariaDB failed as "Record has changed since last read" (errno 1020, `ER_CHECKREAD`): with `innodb_snapshot_isolation` on, the second of two writers whose filters matched one row fails so when it locks the row the first changed. json-rest-api has rolled the transaction back, so the new run reads a fresh snapshot, and a `patchWhere` or `deleteWhere` then leaves out the row that no longer matches. `isDeadlockError` recognises it.
|
|
11
|
+
- **Breaking.** `runtime.database` is `{ driver, knex }`: its `transactionManager`, with `inTransaction(work, { trx })`, and the `RuntimeTransactionManager` type are removed, so a transaction has one way in, the table handle's `withTransaction(options, work)`, which joins the caller's transaction and retries a deadlocked outermost one. Nothing in JSKIT called it. Code that opened a transaction with it opens it on a table's handle (or its owner's API) instead.
|
|
12
|
+
- A PostgreSQL session runs in UTC whatever the server's default time zone, as a MySQL connection does with `timezone: "Z"`: the runtime's Knex and the migration configuration (`createKnexMigrationConfig`) set `TimeZone` to UTC in the pool's `afterCreate`, and the test databases' own connections keep it. PostgreSQL reads a date-time written without an offset, as the `datetime-utc` storage writes it, in the session's zone, so on a server set to another zone a `timestamptz` value moved by that zone's offset.
|
|
13
|
+
- The test databases' `tableNames()` and the migration ledger read accept every name Knex gives PostgreSQL's client (`pg`, `postgres`, `postgresql`); `postgres` used to be read as MySQL.
|
|
14
|
+
- `retryDeadlocks` runs a transaction again only when the database chose it as a deadlock victim, never after a lock wait timed out (MySQL's 1205 and PostgreSQL's `55P03`): MySQL rolls back only the timed-out statement, so a rerun could repeat the transaction's earlier work. Behaviour is unchanged; tests now pin it.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@jskit-ai/database-runtime",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.263",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"scripts": {
|
|
6
6
|
"test": "jskit test"
|
|
@@ -17,8 +17,7 @@
|
|
|
17
17
|
"./shared/databaseClient": "./src/shared/databaseClient.js",
|
|
18
18
|
"./shared/dialect": "./src/shared/dialect.js",
|
|
19
19
|
"./shared/duplicateEntry": "./src/shared/duplicateEntry.js",
|
|
20
|
-
"./shared/repositoryOptions": "./src/shared/repositoryOptions.js"
|
|
21
|
-
"./shared/visibility": "./src/shared/visibility.js"
|
|
20
|
+
"./shared/repositoryOptions": "./src/shared/repositoryOptions.js"
|
|
22
21
|
},
|
|
23
22
|
"dependencies": {
|
|
24
23
|
"knex": "^3.1.0"
|
|
@@ -72,7 +71,7 @@
|
|
|
72
71
|
},
|
|
73
72
|
{
|
|
74
73
|
"subpath": "./shared",
|
|
75
|
-
"summary": "Exports shared
|
|
74
|
+
"summary": "Exports shared database helpers (deadlock retries, record ids, JSON columns, duplicate-entry detection and date/time serialization)."
|
|
76
75
|
},
|
|
77
76
|
{
|
|
78
77
|
"subpath": "./server/types",
|
|
@@ -83,6 +82,6 @@
|
|
|
83
82
|
}
|
|
84
83
|
},
|
|
85
84
|
"peerDependencies": {
|
|
86
|
-
"@jskit-ai/kernel": "0.2.
|
|
85
|
+
"@jskit-ai/kernel": "0.2.14"
|
|
87
86
|
}
|
|
88
87
|
}
|
package/src/server/database.js
CHANGED
|
@@ -1,7 +1,6 @@
|
|
|
1
1
|
import diagnosticsChannel from "node:diagnostics_channel";
|
|
2
2
|
import { normalizeDatabaseClient, toKnexClientId } from "../shared/databaseClient.js";
|
|
3
|
-
import { resolveKnexConnectionFromEnvironment } from "../shared/databaseConnection.js";
|
|
4
|
-
import { createTransactionManager } from "../shared/transactionManager.js";
|
|
3
|
+
import { resolveKnexConnectionFromEnvironment, resolveKnexPool } from "../shared/databaseConnection.js";
|
|
5
4
|
import { loadKnexFactory } from "./knexFactory.js";
|
|
6
5
|
|
|
7
6
|
// The diagnostics channel that carries each statement the shared Knex sends (Knex's `query` event: its `sql`,
|
|
@@ -18,7 +17,8 @@ function resolveDriverDialectId(driver) {
|
|
|
18
17
|
}
|
|
19
18
|
|
|
20
19
|
/**
|
|
21
|
-
* The database connection every feature shares: the installed driver
|
|
20
|
+
* The database connection every feature shares: the installed driver and its Knex client. Table work opens its
|
|
21
|
+
* transactions through its handle's withTransaction.
|
|
22
22
|
* @param {object} dependencies
|
|
23
23
|
* @param {import("./types.js").RuntimeDatabaseDriver} dependencies.driver
|
|
24
24
|
* @param {import("@jskit-ai/kernel/server/types").RuntimeEnv} dependencies.env
|
|
@@ -30,6 +30,7 @@ function createDatabase({ driver, env }) {
|
|
|
30
30
|
throw new Error(`DB_CLIENT="${configuredClient}" does not match installed database driver "${dialectId}".`);
|
|
31
31
|
}
|
|
32
32
|
|
|
33
|
+
const pool = resolveKnexPool(dialectId);
|
|
33
34
|
/** @type {import("./types.js").RuntimeKnex} */
|
|
34
35
|
const knex = loadKnexFactory()({
|
|
35
36
|
client: toKnexClientId(dialectId),
|
|
@@ -37,16 +38,15 @@ function createDatabase({ driver, env }) {
|
|
|
37
38
|
client: dialectId,
|
|
38
39
|
defaultPort: dialectId === "pg" ? 5432 : 3306,
|
|
39
40
|
context: "database runtime"
|
|
40
|
-
})
|
|
41
|
+
}),
|
|
42
|
+
...(pool ? { pool } : {})
|
|
41
43
|
});
|
|
42
44
|
const statements = diagnosticsChannel.channel(RUNTIME_QUERY_CHANNEL);
|
|
43
45
|
knex.on("query", (query) => statements.publish(query));
|
|
44
|
-
const transactionManager = createTransactionManager({ knex });
|
|
45
46
|
|
|
46
47
|
return Object.freeze({
|
|
47
48
|
driver,
|
|
48
|
-
knex
|
|
49
|
-
transactionManager
|
|
49
|
+
knex
|
|
50
50
|
});
|
|
51
51
|
}
|
|
52
52
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import path from "node:path";
|
|
2
2
|
import { createKnexMigrationConfigFromApp } from "./knexMigrationConfig.js";
|
|
3
3
|
import { loadKnexFactory } from "./knexFactory.js";
|
|
4
|
-
import {
|
|
4
|
+
import { retryDeadlocks } from "../shared/repositoryOptions.js";
|
|
5
5
|
|
|
6
6
|
function normalizeMigrationResult(result) {
|
|
7
7
|
const [batchValue, migrationValues] = Array.isArray(result) ? result : [];
|
|
@@ -44,13 +44,13 @@ async function runDatabaseSetup({ knex, seed, appRoot = process.cwd(), environme
|
|
|
44
44
|
});
|
|
45
45
|
}
|
|
46
46
|
|
|
47
|
-
await
|
|
47
|
+
await retryDeadlocks(() => knex.transaction(async (transaction) => {
|
|
48
48
|
await seed(Object.freeze({
|
|
49
49
|
knex: transaction,
|
|
50
50
|
appRoot: path.resolve(appRoot),
|
|
51
51
|
environment
|
|
52
52
|
}));
|
|
53
|
-
});
|
|
53
|
+
}));
|
|
54
54
|
|
|
55
55
|
return Object.freeze({
|
|
56
56
|
...migrationResult,
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { existsSync } from "node:fs";
|
|
1
2
|
import path from "node:path";
|
|
2
3
|
import { access } from "node:fs/promises";
|
|
3
4
|
import { discoverInstalledPackages } from "@jskit-ai/kernel/server/support";
|
|
@@ -5,7 +6,7 @@ import {
|
|
|
5
6
|
normalizeDatabaseClient,
|
|
6
7
|
toKnexClientId
|
|
7
8
|
} from "../shared/databaseClient.js";
|
|
8
|
-
import { resolveKnexConnectionFromEnvironment } from "../shared/databaseConnection.js";
|
|
9
|
+
import { resolveKnexConnectionFromEnvironment, resolveKnexPool } from "../shared/databaseConnection.js";
|
|
9
10
|
|
|
10
11
|
/**
|
|
11
12
|
* The Knex configuration for migrating the database the application's environment names.
|
|
@@ -32,6 +33,9 @@ function createKnexMigrationConfig({
|
|
|
32
33
|
}
|
|
33
34
|
const dialectId = normalizeDatabaseClient(client);
|
|
34
35
|
const resolvedMigrationsDirectory = path.resolve(appRoot, migrationsDirectory);
|
|
36
|
+
// An application needs no constraints/ until it has a constraint; Knex refuses a missing directory.
|
|
37
|
+
const applicationConstraintsDirectory = path.join(resolvedMigrationsDirectory, "constraints");
|
|
38
|
+
const pool = resolveKnexPool(dialectId);
|
|
35
39
|
|
|
36
40
|
return {
|
|
37
41
|
client: toKnexClientId(dialectId),
|
|
@@ -40,6 +44,7 @@ function createKnexMigrationConfig({
|
|
|
40
44
|
defaultPort: dialectId === "pg" ? 5432 : 3306,
|
|
41
45
|
context: "knex migrations"
|
|
42
46
|
}),
|
|
47
|
+
...(pool ? { pool } : {}),
|
|
43
48
|
migrations: {
|
|
44
49
|
// Packages create the tables application migrations build on; package constraints
|
|
45
50
|
// can then reference application tables such as users.
|
|
@@ -47,7 +52,7 @@ function createKnexMigrationConfig({
|
|
|
47
52
|
...packageMigrationDirectories.map((directory) => path.resolve(directory)),
|
|
48
53
|
resolvedMigrationsDirectory,
|
|
49
54
|
...packageConstraintDirectories.map((directory) => path.resolve(directory)),
|
|
50
|
-
|
|
55
|
+
...(existsSync(applicationConstraintsDirectory) ? [applicationConstraintsDirectory] : [])
|
|
51
56
|
],
|
|
52
57
|
extension: "cjs",
|
|
53
58
|
sortDirsSeparately: true
|
|
@@ -19,7 +19,9 @@
|
|
|
19
19
|
// PROD_DB_NAME are never touched. Two runs at once need different database names, as before.
|
|
20
20
|
//
|
|
21
21
|
// A database test that guards a path's cost counts the statements the path sends with countQueries(), or, for a route
|
|
22
|
-
// of a server the test booted, countRuntimeQueries()
|
|
22
|
+
// of a server the test booted, countRuntimeQueries(); expectFlatQueries() in @jskit-ai/resource-runtime/test compares
|
|
23
|
+
// those counts at a few rows and at many. A data test never skips: requireTestDatabase() fails it when TEST_DB_NAME is
|
|
24
|
+
// not set.
|
|
23
25
|
import { createHash } from "node:crypto";
|
|
24
26
|
import diagnosticsChannel from "node:diagnostics_channel";
|
|
25
27
|
import { existsSync, mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
|
@@ -27,6 +29,7 @@ import os from "node:os";
|
|
|
27
29
|
import path from "node:path";
|
|
28
30
|
import { fileURLToPath } from "node:url";
|
|
29
31
|
import { parseDatabaseUrl } from "../shared/databaseConnection.js";
|
|
32
|
+
import { detectDialectFromClient } from "../shared/dialect.js";
|
|
30
33
|
import { RUNTIME_QUERY_CHANNEL } from "./database.js";
|
|
31
34
|
import { loadKnexFactory } from "./knexFactory.js";
|
|
32
35
|
|
|
@@ -127,7 +130,8 @@ function knexOn(knexConfig, database, connectionOptions = {}) {
|
|
|
127
130
|
return loadKnexFactory()({
|
|
128
131
|
...knexConfig,
|
|
129
132
|
connection: { ...knexConfig.connection, ...connectionOptions, database },
|
|
130
|
-
pool
|
|
133
|
+
// The configuration's own pool settings stay, such as a PostgreSQL session's UTC time zone.
|
|
134
|
+
pool: { ...knexConfig.pool, min: 0, max: 1 }
|
|
131
135
|
});
|
|
132
136
|
}
|
|
133
137
|
|
|
@@ -178,7 +182,7 @@ async function testDatabaseSnapshotKey(knexConfig, { appRoot = process.cwd(), sc
|
|
|
178
182
|
async function ledgerNames(knex, ledger, tables) {
|
|
179
183
|
if (!tables.includes(ledger)) return [];
|
|
180
184
|
const result = await knex.raw("SELECT name FROM ?? ORDER BY id", [ledger]);
|
|
181
|
-
return (knex
|
|
185
|
+
return (detectDialectFromClient(knex) === "postgres" ? result.rows : result[0]).map(({ name }) => name);
|
|
182
186
|
}
|
|
183
187
|
|
|
184
188
|
const mysql = {
|
|
@@ -571,14 +575,59 @@ async function countRuntimeQueries(work) {
|
|
|
571
575
|
}, work);
|
|
572
576
|
}
|
|
573
577
|
|
|
578
|
+
/** A data test that has no test database: it fails rather than skip. */
|
|
579
|
+
class TestDatabaseMissingError extends Error {
|
|
580
|
+
constructor() {
|
|
581
|
+
super("TEST_DB_NAME is not set; data tests never skip.");
|
|
582
|
+
this.name = "TestDatabaseMissingError";
|
|
583
|
+
}
|
|
584
|
+
}
|
|
585
|
+
|
|
586
|
+
/**
|
|
587
|
+
* The test database a data test runs on. A data test never skips: without TEST_DB_NAME it throws
|
|
588
|
+
* TestDatabaseMissingError, so a run without a database fails instead of passing what it never ran.
|
|
589
|
+
* @param {import("@jskit-ai/kernel/server/types").RuntimeEnv} environment The test's environment, such as `process.env`.
|
|
590
|
+
* @returns {string}
|
|
591
|
+
*/
|
|
592
|
+
function requireTestDatabase(environment) {
|
|
593
|
+
const name = String(environment.TEST_DB_NAME || "").trim();
|
|
594
|
+
if (!name) {
|
|
595
|
+
throw new TestDatabaseMissingError();
|
|
596
|
+
}
|
|
597
|
+
return name;
|
|
598
|
+
}
|
|
599
|
+
|
|
600
|
+
/**
|
|
601
|
+
* The tables of the database `knex` is connected to, sorted: MySQL's and MariaDB's base tables, PostgreSQL's in the
|
|
602
|
+
* current schema, SQLite's own tables (its internal ones aside).
|
|
603
|
+
* @param {import("knex").Knex | import("knex").Knex.Transaction} knex
|
|
604
|
+
* @returns {Promise<string[]>}
|
|
605
|
+
*/
|
|
606
|
+
async function tableNames(knex) {
|
|
607
|
+
const client = String(knex.client.config.client);
|
|
608
|
+
if (client.includes("sqlite")) {
|
|
609
|
+
const rows = await knex.raw("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'");
|
|
610
|
+
return rows.map(({ name }) => name).sort();
|
|
611
|
+
}
|
|
612
|
+
// Knex names PostgreSQL "pg", "postgres" or "postgresql".
|
|
613
|
+
if (detectDialectFromClient(knex) === "postgres") {
|
|
614
|
+
return postgres.tables(knex);
|
|
615
|
+
}
|
|
616
|
+
const [[{ name: database }]] = await knex.raw("SELECT DATABASE() AS name");
|
|
617
|
+
return mysql.tables(knex, database);
|
|
618
|
+
}
|
|
619
|
+
|
|
574
620
|
export {
|
|
575
621
|
RUN_SETTINGS_FILE,
|
|
622
|
+
TestDatabaseMissingError,
|
|
576
623
|
countQueries,
|
|
577
624
|
countRuntimeQueries,
|
|
578
625
|
matchesSnapshot,
|
|
579
626
|
prepareTestDatabases,
|
|
580
627
|
requireDisposableDatabase,
|
|
628
|
+
requireTestDatabase,
|
|
581
629
|
restoreTestDatabase,
|
|
630
|
+
tableNames,
|
|
582
631
|
testDatabaseNames,
|
|
583
632
|
testDatabaseRun,
|
|
584
633
|
testDatabaseSnapshotKey
|
package/src/server/types.js
CHANGED
|
@@ -2,8 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
/** @typedef {ReturnType<typeof import("./database.js").createDatabase>} RuntimeDatabase */
|
|
4
4
|
/** @typedef {import("knex").Knex} RuntimeKnex A Knex client: the one `runtime.database` shares, or one an application passes in. */
|
|
5
|
-
/** @typedef {
|
|
6
|
-
/** @typedef {import("knex").Knex.Transaction} RuntimeKnexTransaction A Knex transaction: the `trx` a repository or service call runs in, as the transaction manager hands it to its work. */
|
|
5
|
+
/** @typedef {import("knex").Knex.Transaction} RuntimeKnexTransaction A Knex transaction: the `trx` a repository or service call runs in. */
|
|
7
6
|
/** @typedef {import("knex").Knex.QueryBuilder} RuntimeKnexQueryBuilder A Knex query that a hook or helper narrows. */
|
|
8
7
|
|
|
9
8
|
/**
|
|
@@ -141,9 +141,28 @@ function resolveKnexConnectionFromEnvironment(
|
|
|
141
141
|
return connection;
|
|
142
142
|
}
|
|
143
143
|
|
|
144
|
+
// Runs on each new PostgreSQL connection before the pool hands it out.
|
|
145
|
+
function setPostgresSessionToUtc(connection, done) {
|
|
146
|
+
connection.query("SET TIME ZONE 'UTC'", (error) => done(error, connection));
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
/**
|
|
150
|
+
* The Knex pool settings of a dialect: a PostgreSQL session runs in UTC whatever
|
|
151
|
+
* the server's default time zone, as a MySQL connection does with timezone "Z".
|
|
152
|
+
* PostgreSQL reads a date-time written without an offset (as the "datetime-utc"
|
|
153
|
+
* storage writes it) in the session's time zone, and formats timestamptz values
|
|
154
|
+
* and date functions in it, so a non-UTC session would shift stored instants.
|
|
155
|
+
* @param {string} client The dialect id: "mysql2" or "pg".
|
|
156
|
+
* @returns {{ afterCreate: (connection: any, done: (error: unknown, connection: any) => void) => void } | undefined}
|
|
157
|
+
*/
|
|
158
|
+
function resolveKnexPool(client) {
|
|
159
|
+
return normalizeDatabaseClient(client) === "pg" ? { afterCreate: setPostgresSessionToUtc } : undefined;
|
|
160
|
+
}
|
|
161
|
+
|
|
144
162
|
export {
|
|
145
163
|
parseDatabaseUrl,
|
|
146
164
|
resolveDatabaseClientFromEnvironment,
|
|
147
165
|
resolveDatabaseConnectionFromEnvironment,
|
|
148
|
-
resolveKnexConnectionFromEnvironment
|
|
166
|
+
resolveKnexConnectionFromEnvironment,
|
|
167
|
+
resolveKnexPool
|
|
149
168
|
};
|
|
@@ -15,6 +15,13 @@ function isPostgresDuplicateEntryError(error) {
|
|
|
15
15
|
return code === "23505";
|
|
16
16
|
}
|
|
17
17
|
|
|
18
|
+
// better-sqlite3 names the constraint a duplicate broke: a unique index or the primary key.
|
|
19
|
+
const SQLITE_DUPLICATE_CODES = Object.freeze(new Set(["SQLITE_CONSTRAINT_UNIQUE", "SQLITE_CONSTRAINT_PRIMARYKEY"]));
|
|
20
|
+
|
|
21
|
+
function isSqliteDuplicateEntryError(error) {
|
|
22
|
+
return SQLITE_DUPLICATE_CODES.has(String(error?.code || "").trim().toUpperCase());
|
|
23
|
+
}
|
|
24
|
+
|
|
18
25
|
function findDuplicateEntryError(error, { dialect = "", client = null } = {}) {
|
|
19
26
|
const resolvedDialect =
|
|
20
27
|
normalizeDialect(dialect) || (client ? normalizeDialect(detectDialectFromClient(client)) : "");
|
|
@@ -28,8 +35,10 @@ function findDuplicateEntryError(error, { dialect = "", client = null } = {}) {
|
|
|
28
35
|
matches = isPostgresDuplicateEntryError(error);
|
|
29
36
|
} else if (resolvedDialect === "mysql") {
|
|
30
37
|
matches = isMysqlDuplicateEntryError(error);
|
|
38
|
+
} else if (resolvedDialect.includes("sqlite")) {
|
|
39
|
+
matches = isSqliteDuplicateEntryError(error);
|
|
31
40
|
} else {
|
|
32
|
-
matches = isMysqlDuplicateEntryError(error) || isPostgresDuplicateEntryError(error);
|
|
41
|
+
matches = isMysqlDuplicateEntryError(error) || isPostgresDuplicateEntryError(error) || isSqliteDuplicateEntryError(error);
|
|
33
42
|
}
|
|
34
43
|
|
|
35
44
|
if (matches) {
|
package/src/shared/index.js
CHANGED
|
@@ -10,12 +10,7 @@ export {
|
|
|
10
10
|
export { normalizeDialect, detectDialectFromClient } from "./dialect.js";
|
|
11
11
|
export { normalizeDatabaseClient, toKnexClientId } from "./databaseClient.js";
|
|
12
12
|
export { findDuplicateEntryError, isDuplicateEntryError, isLostDuplicateInsertRace } from "./duplicateEntry.js";
|
|
13
|
-
export { DEFAULT_VISIBILITY_COLUMNS, applyVisibility, applyVisibilityOwners } from "./visibility.js";
|
|
14
13
|
export {
|
|
15
|
-
resolveTransaction,
|
|
16
|
-
resolveQueryOptions,
|
|
17
|
-
resolveRepoClient,
|
|
18
|
-
applyForUpdate,
|
|
19
14
|
parseJsonObject,
|
|
20
15
|
stringifyJsonObject,
|
|
21
16
|
normalizeNullableString,
|
|
@@ -25,8 +20,5 @@ export {
|
|
|
25
20
|
toDbJson,
|
|
26
21
|
isDeadlockError,
|
|
27
22
|
retryDeadlocks,
|
|
28
|
-
|
|
29
|
-
createWithTransaction,
|
|
30
|
-
joinOrOpenTransaction,
|
|
31
|
-
lockScopedRecordId
|
|
23
|
+
retryLostInsertRace
|
|
32
24
|
} from "./repositoryOptions.js";
|
|
@@ -1,43 +1,5 @@
|
|
|
1
1
|
import { normalizeCanonicalRecordIdText } from "@jskit-ai/kernel/shared/support/normalize";
|
|
2
|
-
|
|
3
|
-
/**
|
|
4
|
-
* The transaction a repository or service call runs in: its options' `trx`,
|
|
5
|
-
* or null for a call outside one (a call may pass no options).
|
|
6
|
-
* @param {{ trx?: import("../server/types.js").RuntimeKnexTransaction | null } | null} [options]
|
|
7
|
-
* @returns {import("../server/types.js").RuntimeKnexTransaction | null}
|
|
8
|
-
*/
|
|
9
|
-
function resolveTransaction(options) {
|
|
10
|
-
if (!options || typeof options !== "object") {
|
|
11
|
-
return null;
|
|
12
|
-
}
|
|
13
|
-
return options.trx || null;
|
|
14
|
-
}
|
|
15
|
-
|
|
16
|
-
/**
|
|
17
|
-
* A query's transaction (see resolveTransaction) and whether it locks the rows it reads.
|
|
18
|
-
* @param {{ trx?: import("../server/types.js").RuntimeKnexTransaction | null, forUpdate?: boolean } | null} [options]
|
|
19
|
-
*/
|
|
20
|
-
function resolveQueryOptions(options) {
|
|
21
|
-
return {
|
|
22
|
-
trx: resolveTransaction(options),
|
|
23
|
-
forUpdate: options?.forUpdate === true
|
|
24
|
-
};
|
|
25
|
-
}
|
|
26
|
-
|
|
27
|
-
function resolveRepoClient(dbClient, options = {}) {
|
|
28
|
-
const { trx } = resolveQueryOptions(options);
|
|
29
|
-
return trx || dbClient;
|
|
30
|
-
}
|
|
31
|
-
|
|
32
|
-
function applyForUpdate(query, options = {}) {
|
|
33
|
-
const { forUpdate } = resolveQueryOptions(options);
|
|
34
|
-
// readable-check: allow method-probe - platform object: a Knex query builder that cannot lock rows is used as is
|
|
35
|
-
if (forUpdate && typeof query?.forUpdate === "function") {
|
|
36
|
-
return query.forUpdate();
|
|
37
|
-
}
|
|
38
|
-
|
|
39
|
-
return query;
|
|
40
|
-
}
|
|
2
|
+
import { isLostDuplicateInsertRace } from "./duplicateEntry.js";
|
|
41
3
|
|
|
42
4
|
function parseJsonObject(value, fallback = {}) {
|
|
43
5
|
const source = String(value || "").trim();
|
|
@@ -164,8 +126,14 @@ function toDbJson(value) {
|
|
|
164
126
|
|
|
165
127
|
/**
|
|
166
128
|
* True when the error reports a transaction the database aborted, and rolled
|
|
167
|
-
* back in full, as a deadlock victim
|
|
168
|
-
*
|
|
129
|
+
* back in full, as a deadlock victim, a SQLite write that found the database
|
|
130
|
+
* locked by another connection (SQLITE_BUSY), or, with MariaDB's
|
|
131
|
+
* innodb_snapshot_isolation on, a locking read of a row another transaction
|
|
132
|
+
* changed after this one's snapshot ("Record has changed since last read",
|
|
133
|
+
* errno 1020, ER_CHECKREAD); after the last two json-rest-api rolls its
|
|
134
|
+
* transaction back, so a new run starts from a fresh snapshot. A wrapper that
|
|
135
|
+
* reports its transaction committed (json-rest-api's transactionOutcome)
|
|
136
|
+
* never is one.
|
|
169
137
|
* @param {unknown} error
|
|
170
138
|
*/
|
|
171
139
|
function isDeadlockError(error) {
|
|
@@ -176,7 +144,8 @@ function isDeadlockError(error) {
|
|
|
176
144
|
return false;
|
|
177
145
|
}
|
|
178
146
|
const code = String(error.code || "").toUpperCase();
|
|
179
|
-
if (code === "ER_LOCK_DEADLOCK" || code === "40P01" ||
|
|
147
|
+
if (code === "ER_LOCK_DEADLOCK" || code === "ER_CHECKREAD" || code === "40P01" || code === "SQLITE_BUSY" ||
|
|
148
|
+
Number(error.errno) === 1213 || Number(error.errno) === 1020) {
|
|
180
149
|
return true;
|
|
181
150
|
}
|
|
182
151
|
error = error.cause;
|
|
@@ -187,6 +156,8 @@ function isDeadlockError(error) {
|
|
|
187
156
|
/**
|
|
188
157
|
* Runs a whole transaction again when the database chose it as a deadlock
|
|
189
158
|
* victim, so `run` must be safe to repeat: no external effects inside it.
|
|
159
|
+
* A lock wait that timed out is never run again: MySQL rolls back only the
|
|
160
|
+
* timed-out statement, so the transaction's earlier work could repeat.
|
|
190
161
|
* @template T
|
|
191
162
|
* @param {() => Promise<T>} run
|
|
192
163
|
* @param {{ retries?: number }} [options]
|
|
@@ -205,88 +176,42 @@ async function retryDeadlocks(run, { retries = 3 } = {}) {
|
|
|
205
176
|
}
|
|
206
177
|
}
|
|
207
178
|
|
|
208
|
-
async function runInTransaction(knex, callback, options = {}) {
|
|
209
|
-
if (typeof callback !== "function") {
|
|
210
|
-
throw new TypeError("runInTransaction requires callback.");
|
|
211
|
-
}
|
|
212
|
-
if (!knex || typeof knex !== "function") {
|
|
213
|
-
throw new TypeError("runInTransaction requires knex client.");
|
|
214
|
-
}
|
|
215
|
-
|
|
216
|
-
// readable-check: allow method-probe - platform object: a Knex client without transaction() runs the callback on itself
|
|
217
|
-
if (typeof knex.transaction !== "function") {
|
|
218
|
-
return callback(knex);
|
|
219
|
-
}
|
|
220
|
-
// Inside a transaction this opens a savepoint; a deadlock aborts the
|
|
221
|
-
// outermost transaction, and only that one retries.
|
|
222
|
-
if (knex.isTransaction) {
|
|
223
|
-
return knex.transaction(callback);
|
|
224
|
-
}
|
|
225
|
-
return retryDeadlocks(() => knex.transaction(callback), options);
|
|
226
|
-
}
|
|
227
|
-
|
|
228
|
-
function createWithTransaction(knex) {
|
|
229
|
-
return function withTransaction(work) {
|
|
230
|
-
return runInTransaction(knex, work);
|
|
231
|
-
};
|
|
232
|
-
}
|
|
233
|
-
|
|
234
179
|
/**
|
|
235
|
-
*
|
|
236
|
-
*
|
|
237
|
-
*
|
|
180
|
+
* Runs `run` once more when its insert lost a race for a unique key: another
|
|
181
|
+
* request inserted the same key first (an idempotency or request key, a
|
|
182
|
+
* revision number, a computed id), and `run`'s own transaction rolled back in
|
|
183
|
+
* full (isLostDuplicateInsertRace). The database reports the duplicate only
|
|
184
|
+
* once the winner has committed, so the second run reads a snapshot that holds
|
|
185
|
+
* the winner's row: its lookup by the key finds it, and every other check sees
|
|
186
|
+
* the current state. A second duplicate is final. `run` is a transaction
|
|
187
|
+
* opened with these `options`, usually `() => repository.withTransaction(options,
|
|
188
|
+
* work)`, and must be safe to repeat, as for a deadlock retry. Inside a
|
|
189
|
+
* caller's transaction (`options.trx`) the failed insert has failed that
|
|
190
|
+
* transaction, so `run` never runs again there: the error reaches the caller,
|
|
191
|
+
* whose own retry, if it has one, runs the whole of it again.
|
|
238
192
|
* @template T
|
|
239
|
-
* @param {{ trx?:
|
|
240
|
-
* @param {(
|
|
241
|
-
* The repository's withTransaction: opens a transaction and runs work in it.
|
|
242
|
-
* @param {(trx: import("../server/types.js").RuntimeKnexTransaction) => Promise<T>} work
|
|
193
|
+
* @param {{ trx?: unknown }} options The options `run` opens its transaction with.
|
|
194
|
+
* @param {() => Promise<T>} run
|
|
243
195
|
* @returns {Promise<T>}
|
|
244
196
|
*/
|
|
245
|
-
function
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
if (!/^[A-Za-z_][A-Za-z0-9_]*$/u.test(identifier)) {
|
|
252
|
-
throw new TypeError(`${label} must be a safe SQL identifier.`);
|
|
253
|
-
}
|
|
254
|
-
return identifier;
|
|
255
|
-
}
|
|
256
|
-
|
|
257
|
-
async function lockScopedRecordId({
|
|
258
|
-
trx,
|
|
259
|
-
tableName,
|
|
260
|
-
recordId,
|
|
261
|
-
scopeId,
|
|
262
|
-
idColumn = "id",
|
|
263
|
-
scopeColumn = "workspace_id"
|
|
264
|
-
} = {}) {
|
|
265
|
-
if (typeof trx !== "function") {
|
|
266
|
-
throw new TypeError("lockScopedRecordId requires a transaction.");
|
|
197
|
+
async function retryLostInsertRace(options, run, ...rest) {
|
|
198
|
+
if (!options || typeof options !== "object" || typeof run !== "function" || rest.length > 0) {
|
|
199
|
+
throw new TypeError(
|
|
200
|
+
"retryLostInsertRace() takes (options, run): the options run opens its transaction with, then " +
|
|
201
|
+
"run, such as () => repository.withTransaction(options, work)."
|
|
202
|
+
);
|
|
267
203
|
}
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
return
|
|
204
|
+
try {
|
|
205
|
+
return await run();
|
|
206
|
+
} catch (error) {
|
|
207
|
+
if (!isLostDuplicateInsertRace(error, options)) {
|
|
208
|
+
throw error;
|
|
209
|
+
}
|
|
210
|
+
return run();
|
|
275
211
|
}
|
|
276
|
-
const row = await trx(table)
|
|
277
|
-
.select(recordKey)
|
|
278
|
-
.where(recordKey, normalizedRecordId)
|
|
279
|
-
.andWhere(scopeKey, normalizedScopeId)
|
|
280
|
-
.forUpdate()
|
|
281
|
-
.first();
|
|
282
|
-
return normalizeDbRecordId(row?.[recordKey]);
|
|
283
212
|
}
|
|
284
213
|
|
|
285
214
|
export {
|
|
286
|
-
resolveTransaction,
|
|
287
|
-
resolveQueryOptions,
|
|
288
|
-
resolveRepoClient,
|
|
289
|
-
applyForUpdate,
|
|
290
215
|
parseJsonObject,
|
|
291
216
|
stringifyJsonObject,
|
|
292
217
|
normalizeNullableString,
|
|
@@ -296,8 +221,5 @@ export {
|
|
|
296
221
|
toDbJson,
|
|
297
222
|
isDeadlockError,
|
|
298
223
|
retryDeadlocks,
|
|
299
|
-
|
|
300
|
-
createWithTransaction,
|
|
301
|
-
joinOrOpenTransaction,
|
|
302
|
-
lockScopedRecordId
|
|
224
|
+
retryLostInsertRace
|
|
303
225
|
};
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
import assert from "node:assert/strict";
|
|
2
|
+
import test from "node:test";
|
|
3
|
+
import knexFactory from "knex";
|
|
4
|
+
import { createDatabase } from "../src/server/database.js";
|
|
5
|
+
import { resolveKnexConnectionFromEnvironment } from "../src/shared/databaseConnection.js";
|
|
6
|
+
import { toDatabaseDateTimeUtc } from "../src/shared/dateUtils.js";
|
|
7
|
+
|
|
8
|
+
// Runs beside the PostgreSQL test database: run with DB_NAME set to TEST_DB_NAME (and the usual DB_HOST, DB_PORT,
|
|
9
|
+
// DB_USER and DB_PASSWORD), and DB_CLIENT unset or pg.
|
|
10
|
+
const TEST_DB_NAME = process.env.TEST_DB_NAME?.trim();
|
|
11
|
+
const ON_TEST_DATABASE = Boolean(TEST_DB_NAME) && process.env.DB_NAME === TEST_DB_NAME;
|
|
12
|
+
const POSTGRES_CLIENT = !process.env.DB_CLIENT || process.env.DB_CLIENT === "pg";
|
|
13
|
+
const SKIP = ON_TEST_DATABASE && POSTGRES_CLIENT ? false : "not on a PostgreSQL test database (DB_NAME=$TEST_DB_NAME)";
|
|
14
|
+
|
|
15
|
+
// The default time zone of the server in this test. node-postgres sends PGOPTIONS as each new connection's startup
|
|
16
|
+
// options, so every session starts in it, as on a server configured for that zone.
|
|
17
|
+
const SERVER_TIME_ZONE = "Australia/Perth";
|
|
18
|
+
|
|
19
|
+
/**
|
|
20
|
+
* Writes one date-time as the "datetime-utc" storage writes it (UTC, without an offset) into a timestamptz column,
|
|
21
|
+
* and reads it back, on one connection.
|
|
22
|
+
* @param {import("knex").Knex} knex
|
|
23
|
+
* @param {Date} instant
|
|
24
|
+
*/
|
|
25
|
+
function roundTrip(knex, instant) {
|
|
26
|
+
return knex.transaction(async (trx) => {
|
|
27
|
+
await trx.raw("CREATE TEMPORARY TABLE stamped (at timestamptz NOT NULL) ON COMMIT DROP");
|
|
28
|
+
await trx.raw("INSERT INTO stamped (at) VALUES (?)", [toDatabaseDateTimeUtc(instant)]);
|
|
29
|
+
const { rows: [{ TimeZone: session }] } = await trx.raw("SHOW TIME ZONE");
|
|
30
|
+
const { rows: [{ at, text }] } = await trx.raw("SELECT at, at::text AS text FROM stamped");
|
|
31
|
+
return { session, at, text };
|
|
32
|
+
});
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
test("the runtime's PostgreSQL sessions run in UTC on a server that is not, so a timestamptz keeps the instant written", {
|
|
36
|
+
skip: SKIP
|
|
37
|
+
}, async (context) => {
|
|
38
|
+
const previousOptions = process.env.PGOPTIONS;
|
|
39
|
+
process.env.PGOPTIONS = `-c TimeZone=${SERVER_TIME_ZONE}`;
|
|
40
|
+
context.after(() => {
|
|
41
|
+
if (previousOptions === undefined) {
|
|
42
|
+
delete process.env.PGOPTIONS;
|
|
43
|
+
} else {
|
|
44
|
+
process.env.PGOPTIONS = previousOptions;
|
|
45
|
+
}
|
|
46
|
+
});
|
|
47
|
+
const env = { ...process.env, DB_CLIENT: "pg" };
|
|
48
|
+
const serverDefault = knexFactory({
|
|
49
|
+
client: "pg",
|
|
50
|
+
connection: resolveKnexConnectionFromEnvironment(env, { client: "pg", defaultPort: 5432 }),
|
|
51
|
+
pool: { min: 0, max: 1 }
|
|
52
|
+
});
|
|
53
|
+
const { knex } = createDatabase({ driver: { DIALECT_ID: "pg" }, env });
|
|
54
|
+
context.after(() => Promise.all([serverDefault.destroy(), knex.destroy()]));
|
|
55
|
+
const instant = new Date("2026-11-08T04:05:06.789Z");
|
|
56
|
+
|
|
57
|
+
// Without the runtime's session setting, the server reads the UTC text in its own zone: eight hours early.
|
|
58
|
+
const shifted = await roundTrip(serverDefault, instant);
|
|
59
|
+
assert.equal(shifted.session, SERVER_TIME_ZONE);
|
|
60
|
+
assert.equal(instant.getTime() - shifted.at.getTime(), 8 * 60 * 60 * 1000);
|
|
61
|
+
|
|
62
|
+
const kept = await roundTrip(knex, instant);
|
|
63
|
+
assert.equal(kept.session, "UTC");
|
|
64
|
+
assert.equal(kept.at.toISOString(), instant.toISOString());
|
|
65
|
+
assert.equal(kept.text, "2026-11-08 04:05:06.789+00");
|
|
66
|
+
});
|