@jskit-ai/database-runtime 0.1.262 → 0.1.264
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/package.json +6 -7
- package/src/server/database.js +7 -7
- package/src/server/databaseSetup.js +3 -3
- package/src/server/foreignKeyMigrations.js +64 -1
- package/src/server/knexMigrationConfig.js +7 -2
- package/src/server/testDatabases.js +109 -7
- package/src/server/types.js +1 -4
- package/src/shared/databaseConnection.js +20 -1
- package/src/shared/duplicateEntry.js +10 -1
- package/src/shared/index.js +1 -9
- package/src/shared/repositoryOptions.js +41 -119
- package/test/databaseConnection.postgres.test.js +66 -0
- package/test/databaseConnection.test.js +26 -1
- package/test/duplicateEntry.test.js +16 -0
- package/test/entrypoints.boundary.test.js +18 -2
- package/test/foreignKeyMigrations.postgres.test.js +22 -1
- package/test/foreignKeyMigrations.test.js +25 -1
- package/test/knexMigrationConfig.test.js +39 -8
- package/test/providerRuntime.test.js +2 -2
- package/test/repositoryOptions.test.js +81 -76
- package/test/testDatabases.test.js +37 -0
- package/src/shared/transactionManager.js +0 -31
- package/src/shared/visibility.js +0 -73
- package/test/runtimeCore.test.js +0 -51
- package/test/visibility.test.js +0 -93
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## Unreleased
|
|
4
|
+
|
|
5
|
+
- `./server/testDatabases` exports `createPackageTestKnex(t, { environment })`, the Knex a package's own data test runs on when it creates its tables in the test: an in-memory SQLite database, or on a `DB_CLIENT=pg` run a schema of the test's own in `TEST_DB_NAME` (on DB_HOST, DB_PORT, DB_USER and DB_PASSWORD), in UTC, with a pool of up to eight connections for concurrent transactions, dropped with everything in it after the test. It refuses `NODE_ENV=production`, and a pg run without `TEST_DB_NAME` fails (`TestDatabaseMissingError`).
|
|
6
|
+
- `listForeignKeys(knex)` (`./server/foreignKeyMigrations`) answers every foreign key of the connected database's tables in one statement, each with its table, its columns in order and the table and columns it references, on MySQL, MariaDB, PostgreSQL (the current schema) and SQLite (whose keys have no name). `testDatabaseRun` takes `checkDatabase(knex)`, run once on the prepared `TEST_DB_NAME` before any test file: its answer joins the summary and a throw fails the run.
|
|
7
|
+
- `@jskit-ai/database-runtime-mysql` and `@jskit-ai/database-runtime-postgres` take `@jskit-ai/database-runtime` as a required peer at the same version, where it was a dependency, as `@jskit-ai/resource-runtime` and `@jskit-ai/crud-core` do: the application declares it beside its driver (`npm install @jskit-ai/database-runtime @jskit-ai/database-runtime-mysql`) and holds its one copy. npm installs a required peer as it installed the dependency.
|
|
8
|
+
- **Breaking (data-access redesign).** `./shared/visibility` is removed (`applyVisibility`, `applyVisibilityOwners`, `DEFAULT_VISIBILITY_COLUMNS`, also re-exported from `./shared`): a table resource declares its `ownership`, and every handle statement is scoped by it. `./shared` and `./shared/repositoryOptions` no longer export `applyForUpdate`, `createWithTransaction`, `joinOrOpenTransaction`, `lockScopedRecordId`, `resolveQueryOptions`, `resolveRepoClient`, `resolveTransaction` and `runInTransaction`: a transaction is the table handle's `withTransaction(options, work)`, a row lock its `lock` or `lockMany`, and a call's client and transaction ride in the `options` it passes on. The Before | Now table in the existing-application migration guide maps each one.
|
|
9
|
+
- `./server/testDatabases` exports `requireTestDatabase(environment)`, which answers `TEST_DB_NAME` and throws `TestDatabaseMissingError` without it, so a data test fails instead of skipping, and `tableNames(knex)`, the tables of the test database.
|
|
10
|
+
- `retryLostInsertRace(options, run)` (from `./shared`) runs a transaction once more when its insert lost a race for a unique key: `run`, usually `() => repository.withTransaction(options, work)`, failed with a duplicate whose transaction rolled back in full, outside a caller's transaction (`isLostDuplicateInsertRace`). The database reports the duplicate only once the winner has committed, so the second run's lookup by the key finds the winner's row; a second duplicate is final, and inside a caller's transaction (`options.trx`) the error reaches the caller without a rerun. Apps wrote this retry by hand around each idempotent write.
|
|
11
|
+
- The migration configuration lists the application's `migrations/constraints/` only when the directory exists, as it already did for a package's: an application without one failed `db:migrate` and `jskit test --databases` with ENOENT (scandir `migrations/constraints`).
|
|
12
|
+
- `retryDeadlocks` (and so `withTransaction` and every handle write outside a transaction) also runs again a transaction that MariaDB failed as "Record has changed since last read" (errno 1020, `ER_CHECKREAD`): with `innodb_snapshot_isolation` on, the second of two writers whose filters matched one row fails so when it locks the row the first changed. json-rest-api has rolled the transaction back, so the new run reads a fresh snapshot, and a `patchWhere` or `deleteWhere` then leaves out the row that no longer matches. `isDeadlockError` recognises it.
|
|
13
|
+
- **Breaking.** `runtime.database` is `{ driver, knex }`: its `transactionManager`, with `inTransaction(work, { trx })`, and the `RuntimeTransactionManager` type are removed, so a transaction has one way in, the table handle's `withTransaction(options, work)`, which joins the caller's transaction and retries a deadlocked outermost one. Nothing in JSKIT called it. Code that opened a transaction with it opens it on a table's handle (or its owner's API) instead.
|
|
14
|
+
- A PostgreSQL session runs in UTC whatever the server's default time zone, as a MySQL connection does with `timezone: "Z"`: the runtime's Knex and the migration configuration (`createKnexMigrationConfig`) set `TimeZone` to UTC in the pool's `afterCreate`, and the test databases' own connections keep it. PostgreSQL reads a date-time written without an offset, as the `datetime-utc` storage writes it, in the session's zone, so on a server set to another zone a `timestamptz` value moved by that zone's offset.
|
|
15
|
+
- The test databases' `tableNames()` and the migration ledger read accept every name Knex gives PostgreSQL's client (`pg`, `postgres`, `postgresql`); `postgres` used to be read as MySQL.
|
|
16
|
+
- `retryDeadlocks` runs a transaction again only when the database chose it as a deadlock victim, never after a lock wait timed out (MySQL's 1205 and PostgreSQL's `55P03`): MySQL rolls back only the timed-out statement, so a rerun could repeat the transaction's earlier work. Behaviour is unchanged; tests now pin it.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@jskit-ai/database-runtime",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.264",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"scripts": {
|
|
6
6
|
"test": "jskit test"
|
|
@@ -17,8 +17,7 @@
|
|
|
17
17
|
"./shared/databaseClient": "./src/shared/databaseClient.js",
|
|
18
18
|
"./shared/dialect": "./src/shared/dialect.js",
|
|
19
19
|
"./shared/duplicateEntry": "./src/shared/duplicateEntry.js",
|
|
20
|
-
"./shared/repositoryOptions": "./src/shared/repositoryOptions.js"
|
|
21
|
-
"./shared/visibility": "./src/shared/visibility.js"
|
|
20
|
+
"./shared/repositoryOptions": "./src/shared/repositoryOptions.js"
|
|
22
21
|
},
|
|
23
22
|
"dependencies": {
|
|
24
23
|
"knex": "^3.1.0"
|
|
@@ -64,15 +63,15 @@
|
|
|
64
63
|
},
|
|
65
64
|
{
|
|
66
65
|
"subpath": "./server/foreignKeyMigrations",
|
|
67
|
-
"summary": "Adds a single-column foreign key only when it is missing, refusing rows without a parent, for constraints migrations on fresh and already-migrated databases."
|
|
66
|
+
"summary": "Adds a single-column foreign key only when it is missing, refusing rows without a parent, for constraints migrations on fresh and already-migrated databases, and lists every foreign key the connected database holds."
|
|
68
67
|
},
|
|
69
68
|
{
|
|
70
69
|
"subpath": "./server/testDatabases",
|
|
71
|
-
"summary": "Gives each test file or browser worker running at once its own disposable copy of TEST_DB_NAME or BROWSER_TEST_DB_NAME, restored from a schema snapshot keyed by the migrations' names and contents (MySQL, MariaDB and PostgreSQL),
|
|
70
|
+
"summary": "Gives each test file or browser worker running at once its own disposable copy of TEST_DB_NAME or BROWSER_TEST_DB_NAME, restored from a schema snapshot keyed by the migrations' names and contents (MySQL, MariaDB and PostgreSQL), counts the statements a piece of work sends, for query-count guards, and gives a package's own data test its Knex: SQLite in memory, or a PostgreSQL schema of its own on a DB_CLIENT=pg run (createPackageTestKnex)."
|
|
72
71
|
},
|
|
73
72
|
{
|
|
74
73
|
"subpath": "./shared",
|
|
75
|
-
"summary": "Exports shared
|
|
74
|
+
"summary": "Exports shared database helpers (deadlock retries, record ids, JSON columns, duplicate-entry detection and date/time serialization)."
|
|
76
75
|
},
|
|
77
76
|
{
|
|
78
77
|
"subpath": "./server/types",
|
|
@@ -83,6 +82,6 @@
|
|
|
83
82
|
}
|
|
84
83
|
},
|
|
85
84
|
"peerDependencies": {
|
|
86
|
-
"@jskit-ai/kernel": "0.2.
|
|
85
|
+
"@jskit-ai/kernel": "0.2.15"
|
|
87
86
|
}
|
|
88
87
|
}
|
package/src/server/database.js
CHANGED
|
@@ -1,7 +1,6 @@
|
|
|
1
1
|
import diagnosticsChannel from "node:diagnostics_channel";
|
|
2
2
|
import { normalizeDatabaseClient, toKnexClientId } from "../shared/databaseClient.js";
|
|
3
|
-
import { resolveKnexConnectionFromEnvironment } from "../shared/databaseConnection.js";
|
|
4
|
-
import { createTransactionManager } from "../shared/transactionManager.js";
|
|
3
|
+
import { resolveKnexConnectionFromEnvironment, resolveKnexPool } from "../shared/databaseConnection.js";
|
|
5
4
|
import { loadKnexFactory } from "./knexFactory.js";
|
|
6
5
|
|
|
7
6
|
// The diagnostics channel that carries each statement the shared Knex sends (Knex's `query` event: its `sql`,
|
|
@@ -18,7 +17,8 @@ function resolveDriverDialectId(driver) {
|
|
|
18
17
|
}
|
|
19
18
|
|
|
20
19
|
/**
|
|
21
|
-
* The database connection every feature shares: the installed driver
|
|
20
|
+
* The database connection every feature shares: the installed driver and its Knex client. Table work opens its
|
|
21
|
+
* transactions through its handle's withTransaction.
|
|
22
22
|
* @param {object} dependencies
|
|
23
23
|
* @param {import("./types.js").RuntimeDatabaseDriver} dependencies.driver
|
|
24
24
|
* @param {import("@jskit-ai/kernel/server/types").RuntimeEnv} dependencies.env
|
|
@@ -30,6 +30,7 @@ function createDatabase({ driver, env }) {
|
|
|
30
30
|
throw new Error(`DB_CLIENT="${configuredClient}" does not match installed database driver "${dialectId}".`);
|
|
31
31
|
}
|
|
32
32
|
|
|
33
|
+
const pool = resolveKnexPool(dialectId);
|
|
33
34
|
/** @type {import("./types.js").RuntimeKnex} */
|
|
34
35
|
const knex = loadKnexFactory()({
|
|
35
36
|
client: toKnexClientId(dialectId),
|
|
@@ -37,16 +38,15 @@ function createDatabase({ driver, env }) {
|
|
|
37
38
|
client: dialectId,
|
|
38
39
|
defaultPort: dialectId === "pg" ? 5432 : 3306,
|
|
39
40
|
context: "database runtime"
|
|
40
|
-
})
|
|
41
|
+
}),
|
|
42
|
+
...(pool ? { pool } : {})
|
|
41
43
|
});
|
|
42
44
|
const statements = diagnosticsChannel.channel(RUNTIME_QUERY_CHANNEL);
|
|
43
45
|
knex.on("query", (query) => statements.publish(query));
|
|
44
|
-
const transactionManager = createTransactionManager({ knex });
|
|
45
46
|
|
|
46
47
|
return Object.freeze({
|
|
47
48
|
driver,
|
|
48
|
-
knex
|
|
49
|
-
transactionManager
|
|
49
|
+
knex
|
|
50
50
|
});
|
|
51
51
|
}
|
|
52
52
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import path from "node:path";
|
|
2
2
|
import { createKnexMigrationConfigFromApp } from "./knexMigrationConfig.js";
|
|
3
3
|
import { loadKnexFactory } from "./knexFactory.js";
|
|
4
|
-
import {
|
|
4
|
+
import { retryDeadlocks } from "../shared/repositoryOptions.js";
|
|
5
5
|
|
|
6
6
|
function normalizeMigrationResult(result) {
|
|
7
7
|
const [batchValue, migrationValues] = Array.isArray(result) ? result : [];
|
|
@@ -44,13 +44,13 @@ async function runDatabaseSetup({ knex, seed, appRoot = process.cwd(), environme
|
|
|
44
44
|
});
|
|
45
45
|
}
|
|
46
46
|
|
|
47
|
-
await
|
|
47
|
+
await retryDeadlocks(() => knex.transaction(async (transaction) => {
|
|
48
48
|
await seed(Object.freeze({
|
|
49
49
|
knex: transaction,
|
|
50
50
|
appRoot: path.resolve(appRoot),
|
|
51
51
|
environment
|
|
52
52
|
}));
|
|
53
|
-
});
|
|
53
|
+
}));
|
|
54
54
|
|
|
55
55
|
return Object.freeze({
|
|
56
56
|
...migrationResult,
|
|
@@ -119,4 +119,67 @@ async function dropForeignKeyIfPresent(knex, key) {
|
|
|
119
119
|
return true;
|
|
120
120
|
}
|
|
121
121
|
|
|
122
|
-
|
|
122
|
+
// Every foreign key of the connected database's tables, one row per column: the key's name, its table and column,
|
|
123
|
+
// the referenced table and column, and the column's place in the key. Keys to another schema are left out.
|
|
124
|
+
const FOREIGN_KEY_COLUMNS = Object.freeze({
|
|
125
|
+
mysql: `SELECT CONSTRAINT_NAME AS name, TABLE_NAME AS tableName, COLUMN_NAME AS columnName,
|
|
126
|
+
REFERENCED_TABLE_NAME AS referencedTableName, REFERENCED_COLUMN_NAME AS referencedColumnName
|
|
127
|
+
FROM information_schema.KEY_COLUMN_USAGE
|
|
128
|
+
WHERE TABLE_SCHEMA = DATABASE() AND REFERENCED_TABLE_SCHEMA = DATABASE()
|
|
129
|
+
ORDER BY TABLE_NAME, CONSTRAINT_NAME, ORDINAL_POSITION`,
|
|
130
|
+
postgres: `SELECT c.conname AS name, t.relname AS "tableName", a.attname AS "columnName",
|
|
131
|
+
r.relname AS "referencedTableName", p.attname AS "referencedColumnName"
|
|
132
|
+
FROM pg_constraint c
|
|
133
|
+
JOIN pg_class t ON t.oid = c.conrelid
|
|
134
|
+
JOIN pg_class r ON r.oid = c.confrelid
|
|
135
|
+
CROSS JOIN LATERAL unnest(c.conkey, c.confkey) WITH ORDINALITY AS k(attnum, refnum, position)
|
|
136
|
+
JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = k.attnum
|
|
137
|
+
JOIN pg_attribute p ON p.attrelid = c.confrelid AND p.attnum = k.refnum
|
|
138
|
+
WHERE c.contype = 'f' AND t.relnamespace = to_regnamespace(current_schema()) AND r.relnamespace = t.relnamespace
|
|
139
|
+
ORDER BY t.relname, c.conname, k.position`,
|
|
140
|
+
// SQLite keys have no name: each is told apart by its table and id.
|
|
141
|
+
sqlite: `SELECT m.name || '#' || k.id AS name, m.name AS tableName, k."from" AS columnName,
|
|
142
|
+
k."table" AS referencedTableName, k."to" AS referencedColumnName
|
|
143
|
+
FROM sqlite_master m JOIN pragma_foreign_key_list(m.name) k
|
|
144
|
+
WHERE m.type = 'table'
|
|
145
|
+
ORDER BY m.name, k.id, k.seq`
|
|
146
|
+
});
|
|
147
|
+
|
|
148
|
+
// The rows of a raw statement, as each driver answers them.
|
|
149
|
+
const RAW_ROWS = Object.freeze({
|
|
150
|
+
mysql: ([rows]) => rows,
|
|
151
|
+
postgres: ({ rows }) => rows,
|
|
152
|
+
sqlite: (rows) => rows
|
|
153
|
+
});
|
|
154
|
+
|
|
155
|
+
/**
|
|
156
|
+
* The foreign keys the connected database holds, each with its table, its columns in order, and the table and
|
|
157
|
+
* columns it references. A SQLite key's name is "", since SQLite keeps none.
|
|
158
|
+
* @param {import("./types.js").RuntimeKnex} knex
|
|
159
|
+
* @returns {Promise<Array<{ name: string, tableName: string, columns: string[], referencedTableName: string, referencedColumns: string[] }>>}
|
|
160
|
+
*/
|
|
161
|
+
async function listForeignKeys(knex) {
|
|
162
|
+
const detected = detectDialectFromClient(knex);
|
|
163
|
+
const dialect = detected.includes("sqlite") ? "sqlite" : detected;
|
|
164
|
+
if (!Object.hasOwn(FOREIGN_KEY_COLUMNS, dialect)) {
|
|
165
|
+
throw new Error(`Listing foreign keys does not support the ${detected || "unknown"} dialect.`);
|
|
166
|
+
}
|
|
167
|
+
const keys = new Map();
|
|
168
|
+
for (const row of RAW_ROWS[dialect](await knex.raw(FOREIGN_KEY_COLUMNS[dialect]))) {
|
|
169
|
+
const id = `${row.tableName}\u0000${row.name}`;
|
|
170
|
+
if (!keys.has(id)) {
|
|
171
|
+
keys.set(id, {
|
|
172
|
+
name: dialect === "sqlite" ? "" : row.name,
|
|
173
|
+
tableName: row.tableName,
|
|
174
|
+
columns: [],
|
|
175
|
+
referencedTableName: row.referencedTableName,
|
|
176
|
+
referencedColumns: []
|
|
177
|
+
});
|
|
178
|
+
}
|
|
179
|
+
keys.get(id).columns.push(row.columnName);
|
|
180
|
+
keys.get(id).referencedColumns.push(row.referencedColumnName);
|
|
181
|
+
}
|
|
182
|
+
return [...keys.values()];
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
export { addMissingForeignKey, dropForeignKeyIfPresent, hasForeignKey, listForeignKeys };
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { existsSync } from "node:fs";
|
|
1
2
|
import path from "node:path";
|
|
2
3
|
import { access } from "node:fs/promises";
|
|
3
4
|
import { discoverInstalledPackages } from "@jskit-ai/kernel/server/support";
|
|
@@ -5,7 +6,7 @@ import {
|
|
|
5
6
|
normalizeDatabaseClient,
|
|
6
7
|
toKnexClientId
|
|
7
8
|
} from "../shared/databaseClient.js";
|
|
8
|
-
import { resolveKnexConnectionFromEnvironment } from "../shared/databaseConnection.js";
|
|
9
|
+
import { resolveKnexConnectionFromEnvironment, resolveKnexPool } from "../shared/databaseConnection.js";
|
|
9
10
|
|
|
10
11
|
/**
|
|
11
12
|
* The Knex configuration for migrating the database the application's environment names.
|
|
@@ -32,6 +33,9 @@ function createKnexMigrationConfig({
|
|
|
32
33
|
}
|
|
33
34
|
const dialectId = normalizeDatabaseClient(client);
|
|
34
35
|
const resolvedMigrationsDirectory = path.resolve(appRoot, migrationsDirectory);
|
|
36
|
+
// An application needs no constraints/ until it has a constraint; Knex refuses a missing directory.
|
|
37
|
+
const applicationConstraintsDirectory = path.join(resolvedMigrationsDirectory, "constraints");
|
|
38
|
+
const pool = resolveKnexPool(dialectId);
|
|
35
39
|
|
|
36
40
|
return {
|
|
37
41
|
client: toKnexClientId(dialectId),
|
|
@@ -40,6 +44,7 @@ function createKnexMigrationConfig({
|
|
|
40
44
|
defaultPort: dialectId === "pg" ? 5432 : 3306,
|
|
41
45
|
context: "knex migrations"
|
|
42
46
|
}),
|
|
47
|
+
...(pool ? { pool } : {}),
|
|
43
48
|
migrations: {
|
|
44
49
|
// Packages create the tables application migrations build on; package constraints
|
|
45
50
|
// can then reference application tables such as users.
|
|
@@ -47,7 +52,7 @@ function createKnexMigrationConfig({
|
|
|
47
52
|
...packageMigrationDirectories.map((directory) => path.resolve(directory)),
|
|
48
53
|
resolvedMigrationsDirectory,
|
|
49
54
|
...packageConstraintDirectories.map((directory) => path.resolve(directory)),
|
|
50
|
-
|
|
55
|
+
...(existsSync(applicationConstraintsDirectory) ? [applicationConstraintsDirectory] : [])
|
|
51
56
|
],
|
|
52
57
|
extension: "cjs",
|
|
53
58
|
sortDirsSeparately: true
|
|
@@ -19,14 +19,17 @@
|
|
|
19
19
|
// PROD_DB_NAME are never touched. Two runs at once need different database names, as before.
|
|
20
20
|
//
|
|
21
21
|
// A database test that guards a path's cost counts the statements the path sends with countQueries(), or, for a route
|
|
22
|
-
// of a server the test booted, countRuntimeQueries()
|
|
23
|
-
|
|
22
|
+
// of a server the test booted, countRuntimeQueries(); expectFlatQueries() in @jskit-ai/resource-runtime/test compares
|
|
23
|
+
// those counts at a few rows and at many. A data test never skips: requireTestDatabase() fails it when TEST_DB_NAME is
|
|
24
|
+
// not set.
|
|
25
|
+
import { createHash, randomBytes } from "node:crypto";
|
|
24
26
|
import diagnosticsChannel from "node:diagnostics_channel";
|
|
25
27
|
import { existsSync, mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
|
26
28
|
import os from "node:os";
|
|
27
29
|
import path from "node:path";
|
|
28
30
|
import { fileURLToPath } from "node:url";
|
|
29
|
-
import { parseDatabaseUrl } from "../shared/databaseConnection.js";
|
|
31
|
+
import { parseDatabaseUrl, resolveKnexConnectionFromEnvironment, resolveKnexPool } from "../shared/databaseConnection.js";
|
|
32
|
+
import { detectDialectFromClient } from "../shared/dialect.js";
|
|
30
33
|
import { RUNTIME_QUERY_CHANNEL } from "./database.js";
|
|
31
34
|
import { loadKnexFactory } from "./knexFactory.js";
|
|
32
35
|
|
|
@@ -127,7 +130,8 @@ function knexOn(knexConfig, database, connectionOptions = {}) {
|
|
|
127
130
|
return loadKnexFactory()({
|
|
128
131
|
...knexConfig,
|
|
129
132
|
connection: { ...knexConfig.connection, ...connectionOptions, database },
|
|
130
|
-
pool
|
|
133
|
+
// The configuration's own pool settings stay, such as a PostgreSQL session's UTC time zone.
|
|
134
|
+
pool: { ...knexConfig.pool, min: 0, max: 1 }
|
|
131
135
|
});
|
|
132
136
|
}
|
|
133
137
|
|
|
@@ -178,7 +182,7 @@ async function testDatabaseSnapshotKey(knexConfig, { appRoot = process.cwd(), sc
|
|
|
178
182
|
async function ledgerNames(knex, ledger, tables) {
|
|
179
183
|
if (!tables.includes(ledger)) return [];
|
|
180
184
|
const result = await knex.raw("SELECT name FROM ?? ORDER BY id", [ledger]);
|
|
181
|
-
return (knex
|
|
185
|
+
return (detectDialectFromClient(knex) === "postgres" ? result.rows : result[0]).map(({ name }) => name);
|
|
182
186
|
}
|
|
183
187
|
|
|
184
188
|
const mysql = {
|
|
@@ -478,8 +482,10 @@ function refusedPrivilege(error) {
|
|
|
478
482
|
* @param {number} options.count How many test files run at once: at most the run's test budget.
|
|
479
483
|
* @param {TestDatabaseSchema} [options.schema]
|
|
480
484
|
* @param {string} [options.appRoot]
|
|
485
|
+
* @param {(knex: import("knex").Knex) => Promise<string>} [options.checkDatabase] Runs once on the prepared TEST_DB_NAME,
|
|
486
|
+
* before any test file: what it answers joins the summary, and a throw fails the run.
|
|
481
487
|
*/
|
|
482
|
-
async function testDatabaseRun({ knexConfig, environment, count, schema, appRoot = process.cwd() }) {
|
|
488
|
+
async function testDatabaseRun({ knexConfig, environment, count, schema, appRoot = process.cwd(), checkDatabase }) {
|
|
483
489
|
const oneAtATime = (reason) => ({
|
|
484
490
|
concurrency: 1,
|
|
485
491
|
env: {},
|
|
@@ -499,6 +505,7 @@ async function testDatabaseRun({ knexConfig, environment, count, schema, appRoot
|
|
|
499
505
|
if (!refusedPrivilege(error)) throw error;
|
|
500
506
|
return oneAtATime(`the database user may not prepare ${testDatabase} (${error.code})`);
|
|
501
507
|
}
|
|
508
|
+
const checked = checkDatabase ? await withKnex(knexConfig, prepared.databases[0], checkDatabase) : "";
|
|
502
509
|
// mkdtemp makes the directory for this user alone; the settings file holds the connection password.
|
|
503
510
|
const directory = mkdtempSync(path.join(os.tmpdir(), "jskit-test-databases-"));
|
|
504
511
|
const { client, connection, migrations } = knexConfig;
|
|
@@ -512,7 +519,7 @@ async function testDatabaseRun({ knexConfig, environment, count, schema, appRoot
|
|
|
512
519
|
concurrency: prepared.databases.length,
|
|
513
520
|
env: { JSKIT_TEST_DATABASES: directory },
|
|
514
521
|
preload: TEST_DATABASE_FILE,
|
|
515
|
-
summary: prepared.summary
|
|
522
|
+
summary: `${prepared.summary}${checked ? `; ${checked}` : ""}`,
|
|
516
523
|
cleanup: () => rmSync(directory, { recursive: true, force: true })
|
|
517
524
|
};
|
|
518
525
|
}
|
|
@@ -571,14 +578,109 @@ async function countRuntimeQueries(work) {
|
|
|
571
578
|
}, work);
|
|
572
579
|
}
|
|
573
580
|
|
|
581
|
+
/** A data test that has no test database: it fails rather than skip. */
|
|
582
|
+
class TestDatabaseMissingError extends Error {
|
|
583
|
+
constructor() {
|
|
584
|
+
super("TEST_DB_NAME is not set; data tests never skip.");
|
|
585
|
+
this.name = "TestDatabaseMissingError";
|
|
586
|
+
}
|
|
587
|
+
}
|
|
588
|
+
|
|
589
|
+
/**
|
|
590
|
+
* The test database a data test runs on. A data test never skips: without TEST_DB_NAME it throws
|
|
591
|
+
* TestDatabaseMissingError, so a run without a database fails instead of passing what it never ran.
|
|
592
|
+
* @param {import("@jskit-ai/kernel/server/types").RuntimeEnv} environment The test's environment, such as `process.env`.
|
|
593
|
+
* @returns {string}
|
|
594
|
+
*/
|
|
595
|
+
function requireTestDatabase(environment) {
|
|
596
|
+
const name = String(environment.TEST_DB_NAME || "").trim();
|
|
597
|
+
if (!name) {
|
|
598
|
+
throw new TestDatabaseMissingError();
|
|
599
|
+
}
|
|
600
|
+
return name;
|
|
601
|
+
}
|
|
602
|
+
|
|
603
|
+
// A package test's PostgreSQL connections: enough for the concurrent transactions a lock or race test opens at once.
|
|
604
|
+
const PACKAGE_TEST_POOL = Object.freeze({ min: 0, max: 8 });
|
|
605
|
+
|
|
606
|
+
/**
|
|
607
|
+
* The Knex a package's own data test runs on, for a test that creates its own tables: an in-memory SQLite database,
|
|
608
|
+
* or, on a run whose DB_CLIENT is pg, a schema of the test's own in TEST_DB_NAME on the server DB_HOST, DB_PORT,
|
|
609
|
+
* DB_USER and DB_PASSWORD name, its sessions in UTC as an application's are. So one test file proves its data paths on
|
|
610
|
+
* both: the PostgreSQL run of the same files is the proof of the row locks, the many-row writes and the upserts there.
|
|
611
|
+
* The Knex is destroyed after the test, and the schema dropped with everything in it. A pg run without TEST_DB_NAME
|
|
612
|
+
* fails (TestDatabaseMissingError), as every data test does; `knex.client.config.client` tells a test which it got.
|
|
613
|
+
* @param {{ after(cleanup: () => unknown): unknown }} context The test's context (node:test's `t`).
|
|
614
|
+
* @param {{ environment: import("@jskit-ai/kernel/server/types").RuntimeEnv }} options The test's environment, such as
|
|
615
|
+
* `process.env`.
|
|
616
|
+
* @returns {Promise<import("knex").Knex>}
|
|
617
|
+
*/
|
|
618
|
+
async function createPackageTestKnex(context, { environment }) {
|
|
619
|
+
const knexFactory = loadKnexFactory();
|
|
620
|
+
if (String(environment.DB_CLIENT || "").trim() !== "pg") {
|
|
621
|
+
const knex = knexFactory({ client: "better-sqlite3", connection: { filename: ":memory:" }, useNullAsDefault: true });
|
|
622
|
+
context.after(() => knex.destroy());
|
|
623
|
+
return knex;
|
|
624
|
+
}
|
|
625
|
+
if (environment.NODE_ENV === "production") {
|
|
626
|
+
throw new Error("Package data tests never run with NODE_ENV=production.");
|
|
627
|
+
}
|
|
628
|
+
const connection = {
|
|
629
|
+
...resolveKnexConnectionFromEnvironment(environment, { client: "pg", defaultPort: 5432 }),
|
|
630
|
+
database: requireTestDatabase(environment)
|
|
631
|
+
};
|
|
632
|
+
const schema = `jskit_test_${process.pid}_${randomBytes(6).toString("hex")}`;
|
|
633
|
+
const admin = knexFactory({ client: "pg", connection, pool: { min: 0, max: 1 } });
|
|
634
|
+
try {
|
|
635
|
+
await admin.raw("CREATE SCHEMA ??", [schema]);
|
|
636
|
+
} catch (error) {
|
|
637
|
+
await admin.destroy();
|
|
638
|
+
throw error;
|
|
639
|
+
}
|
|
640
|
+
const knex = knexFactory({ client: "pg", connection, searchPath: [schema], pool: { ...resolveKnexPool("pg"), ...PACKAGE_TEST_POOL } });
|
|
641
|
+
context.after(async () => {
|
|
642
|
+
try {
|
|
643
|
+
await knex.destroy();
|
|
644
|
+
await admin.raw("DROP SCHEMA IF EXISTS ?? CASCADE", [schema]);
|
|
645
|
+
} finally {
|
|
646
|
+
await admin.destroy();
|
|
647
|
+
}
|
|
648
|
+
});
|
|
649
|
+
return knex;
|
|
650
|
+
}
|
|
651
|
+
|
|
652
|
+
/**
|
|
653
|
+
* The tables of the database `knex` is connected to, sorted: MySQL's and MariaDB's base tables, PostgreSQL's in the
|
|
654
|
+
* current schema, SQLite's own tables (its internal ones aside).
|
|
655
|
+
* @param {import("knex").Knex | import("knex").Knex.Transaction} knex
|
|
656
|
+
* @returns {Promise<string[]>}
|
|
657
|
+
*/
|
|
658
|
+
async function tableNames(knex) {
|
|
659
|
+
const client = String(knex.client.config.client);
|
|
660
|
+
if (client.includes("sqlite")) {
|
|
661
|
+
const rows = await knex.raw("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'");
|
|
662
|
+
return rows.map(({ name }) => name).sort();
|
|
663
|
+
}
|
|
664
|
+
// Knex names PostgreSQL "pg", "postgres" or "postgresql".
|
|
665
|
+
if (detectDialectFromClient(knex) === "postgres") {
|
|
666
|
+
return postgres.tables(knex);
|
|
667
|
+
}
|
|
668
|
+
const [[{ name: database }]] = await knex.raw("SELECT DATABASE() AS name");
|
|
669
|
+
return mysql.tables(knex, database);
|
|
670
|
+
}
|
|
671
|
+
|
|
574
672
|
export {
|
|
575
673
|
RUN_SETTINGS_FILE,
|
|
674
|
+
TestDatabaseMissingError,
|
|
576
675
|
countQueries,
|
|
676
|
+
createPackageTestKnex,
|
|
577
677
|
countRuntimeQueries,
|
|
578
678
|
matchesSnapshot,
|
|
579
679
|
prepareTestDatabases,
|
|
580
680
|
requireDisposableDatabase,
|
|
681
|
+
requireTestDatabase,
|
|
581
682
|
restoreTestDatabase,
|
|
683
|
+
tableNames,
|
|
582
684
|
testDatabaseNames,
|
|
583
685
|
testDatabaseRun,
|
|
584
686
|
testDatabaseSnapshotKey
|
package/src/server/types.js
CHANGED
|
@@ -2,8 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
/** @typedef {ReturnType<typeof import("./database.js").createDatabase>} RuntimeDatabase */
|
|
4
4
|
/** @typedef {import("knex").Knex} RuntimeKnex A Knex client: the one `runtime.database` shares, or one an application passes in. */
|
|
5
|
-
/** @typedef {
|
|
6
|
-
/** @typedef {import("knex").Knex.Transaction} RuntimeKnexTransaction A Knex transaction: the `trx` a repository or service call runs in, as the transaction manager hands it to its work. */
|
|
5
|
+
/** @typedef {import("knex").Knex.Transaction} RuntimeKnexTransaction A Knex transaction: the `trx` a repository or service call runs in. */
|
|
7
6
|
/** @typedef {import("knex").Knex.QueryBuilder} RuntimeKnexQueryBuilder A Knex query that a hook or helper narrows. */
|
|
8
7
|
|
|
9
8
|
/**
|
|
@@ -13,8 +12,6 @@
|
|
|
13
12
|
* contract; `runtime.database` reads its dialect from it.
|
|
14
13
|
* @typedef {object} RuntimeDatabaseDriver
|
|
15
14
|
* @property {string} DIALECT_ID The dialect id: "mysql2" or "pg".
|
|
16
|
-
* @property {(knex: RuntimeKnex, options?: { tableName?: string, idColumn?: string }) => Promise<object>} [introspectCrudTableSnapshot]
|
|
17
|
-
* Reads a table's columns for CRUD generation; the MySQL driver has it.
|
|
18
15
|
*/
|
|
19
16
|
|
|
20
17
|
export {};
|
|
@@ -141,9 +141,28 @@ function resolveKnexConnectionFromEnvironment(
|
|
|
141
141
|
return connection;
|
|
142
142
|
}
|
|
143
143
|
|
|
144
|
+
// Runs on each new PostgreSQL connection before the pool hands it out.
|
|
145
|
+
function setPostgresSessionToUtc(connection, done) {
|
|
146
|
+
connection.query("SET TIME ZONE 'UTC'", (error) => done(error, connection));
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
/**
|
|
150
|
+
* The Knex pool settings of a dialect: a PostgreSQL session runs in UTC whatever
|
|
151
|
+
* the server's default time zone, as a MySQL connection does with timezone "Z".
|
|
152
|
+
* PostgreSQL reads a date-time written without an offset (as the "datetime-utc"
|
|
153
|
+
* storage writes it) in the session's time zone, and formats timestamptz values
|
|
154
|
+
* and date functions in it, so a non-UTC session would shift stored instants.
|
|
155
|
+
* @param {string} client The dialect id: "mysql2" or "pg".
|
|
156
|
+
* @returns {{ afterCreate: (connection: any, done: (error: unknown, connection: any) => void) => void } | undefined}
|
|
157
|
+
*/
|
|
158
|
+
function resolveKnexPool(client) {
|
|
159
|
+
return normalizeDatabaseClient(client) === "pg" ? { afterCreate: setPostgresSessionToUtc } : undefined;
|
|
160
|
+
}
|
|
161
|
+
|
|
144
162
|
export {
|
|
145
163
|
parseDatabaseUrl,
|
|
146
164
|
resolveDatabaseClientFromEnvironment,
|
|
147
165
|
resolveDatabaseConnectionFromEnvironment,
|
|
148
|
-
resolveKnexConnectionFromEnvironment
|
|
166
|
+
resolveKnexConnectionFromEnvironment,
|
|
167
|
+
resolveKnexPool
|
|
149
168
|
};
|
|
@@ -15,6 +15,13 @@ function isPostgresDuplicateEntryError(error) {
|
|
|
15
15
|
return code === "23505";
|
|
16
16
|
}
|
|
17
17
|
|
|
18
|
+
// better-sqlite3 names the constraint a duplicate broke: a unique index or the primary key.
|
|
19
|
+
const SQLITE_DUPLICATE_CODES = Object.freeze(new Set(["SQLITE_CONSTRAINT_UNIQUE", "SQLITE_CONSTRAINT_PRIMARYKEY"]));
|
|
20
|
+
|
|
21
|
+
function isSqliteDuplicateEntryError(error) {
|
|
22
|
+
return SQLITE_DUPLICATE_CODES.has(String(error?.code || "").trim().toUpperCase());
|
|
23
|
+
}
|
|
24
|
+
|
|
18
25
|
function findDuplicateEntryError(error, { dialect = "", client = null } = {}) {
|
|
19
26
|
const resolvedDialect =
|
|
20
27
|
normalizeDialect(dialect) || (client ? normalizeDialect(detectDialectFromClient(client)) : "");
|
|
@@ -28,8 +35,10 @@ function findDuplicateEntryError(error, { dialect = "", client = null } = {}) {
|
|
|
28
35
|
matches = isPostgresDuplicateEntryError(error);
|
|
29
36
|
} else if (resolvedDialect === "mysql") {
|
|
30
37
|
matches = isMysqlDuplicateEntryError(error);
|
|
38
|
+
} else if (resolvedDialect.includes("sqlite")) {
|
|
39
|
+
matches = isSqliteDuplicateEntryError(error);
|
|
31
40
|
} else {
|
|
32
|
-
matches = isMysqlDuplicateEntryError(error) || isPostgresDuplicateEntryError(error);
|
|
41
|
+
matches = isMysqlDuplicateEntryError(error) || isPostgresDuplicateEntryError(error) || isSqliteDuplicateEntryError(error);
|
|
33
42
|
}
|
|
34
43
|
|
|
35
44
|
if (matches) {
|
package/src/shared/index.js
CHANGED
|
@@ -10,12 +10,7 @@ export {
|
|
|
10
10
|
export { normalizeDialect, detectDialectFromClient } from "./dialect.js";
|
|
11
11
|
export { normalizeDatabaseClient, toKnexClientId } from "./databaseClient.js";
|
|
12
12
|
export { findDuplicateEntryError, isDuplicateEntryError, isLostDuplicateInsertRace } from "./duplicateEntry.js";
|
|
13
|
-
export { DEFAULT_VISIBILITY_COLUMNS, applyVisibility, applyVisibilityOwners } from "./visibility.js";
|
|
14
13
|
export {
|
|
15
|
-
resolveTransaction,
|
|
16
|
-
resolveQueryOptions,
|
|
17
|
-
resolveRepoClient,
|
|
18
|
-
applyForUpdate,
|
|
19
14
|
parseJsonObject,
|
|
20
15
|
stringifyJsonObject,
|
|
21
16
|
normalizeNullableString,
|
|
@@ -25,8 +20,5 @@ export {
|
|
|
25
20
|
toDbJson,
|
|
26
21
|
isDeadlockError,
|
|
27
22
|
retryDeadlocks,
|
|
28
|
-
|
|
29
|
-
createWithTransaction,
|
|
30
|
-
joinOrOpenTransaction,
|
|
31
|
-
lockScopedRecordId
|
|
23
|
+
retryLostInsertRace
|
|
32
24
|
} from "./repositoryOptions.js";
|