@eltjs/elt-sqlserver 0.10.0 → 0.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -10,9 +10,9 @@ The words in these documents have one meaning each. See the glossary in [CONTEXT
|
|
|
10
10
|
|
|
11
11
|
| Now (first release) | Later | Never |
|
|
12
12
|
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
13
|
-
| `SqlServerDestination`: the four destination sync modes, evolve, reload, reader views, extended properties, file chunks | Native `VECTOR` over TDS, when tedious has it ([tedious #1841](https://github.com/tediousjs/tedious/pull/1841)) | A search API
|
|
13
|
+
| `SqlServerDestination`: the four destination sync modes, evolve, reload, reader views, extended properties, file chunks | Native `VECTOR` over TDS, when tedious has it ([tedious #1841](https://github.com/tediousjs/tedious/pull/1841)) | A search API beyond the `_nearest` function of a vector index: any other search is the host's SQL (`@zukhruf/vectors/sqlserver`) |
|
|
14
14
|
| Numbered commits, positions, history, readings, retention, a clear that forgets | `float16` vector columns (`HalfVector`), for a model of more than 1,998 dimensions, such as `text-embedding-3-large` (3,072) | System-versioned temporal tables as the history ([ADR 0001](./docs/adr/0001-the-sql-server-destination-takes-the-declarations-of-elt-postgresql.md)) |
|
|
15
|
-
| `View` and `ComputedDataset`, with `vector(n)` rule columns as `VECTOR(n)` and declared indexes: nonclustered, and DiskANN
|
|
15
|
+
| `View` and `ComputedDataset`, with `vector(n)` rule columns as `VECTOR(n)` and declared indexes: nonclustered, and DiskANN with its search `marts.<name>_nearest` | A bulk load through Azure Blob Storage and `BULK INSERT`, as the bulk mode of Airbyte, when a load is too slow through JSON batches | An ORM, or a query builder over the reader's SQL |
|
|
16
16
|
| `SqlServerCheckpointStore`, `SqlServerSyncHistory` (with `derived_status`, `catalog` and named readers), `readPassStatus` | Microsoft Entra ID logins (managed identity), which `mssql` supports through `authentication` | An `mssql://` URL |
|
|
17
17
|
| The prerequisites in other packages: the `vector` rule column of elt, `dimensions` on `Embedding`, and the extraction of the rules that the destinations share ([ADR 0012](./docs/adr/0012-the-rules-that-the-destinations-share-move-into-elt-before-the-third-destination.md)) | SQL Server 2022 and older, which have no `VECTOR` and no `JSON` type | A migration of stored data between versions (elt AGENTS.md: stored data is disposable) |
|
|
18
18
|
|
|
@@ -240,23 +240,20 @@ new ComputedDataset({
|
|
|
240
240
|
- The vector index is `CREATE VECTOR INDEX … ON [_warehouse].[raw_staff_passage_vectors] ([vector]) WITH (METRIC = 'cosine', TYPE = 'DiskANN')`, which each refresh makes when the table does not have it, also after a changed declaration makes the table again (see "Declare a computed dataset" below). An index made on an empty table starts as an exact index, and Azure SQL builds its graph after the rows come (`@zukhruf/vectors`, "SQL Server: what the probes found").
|
|
241
241
|
- SQL Server 2025 in Docker makes a vector index only with `PREVIEW_FEATURES` on and only on an `int` clustered key, so it refuses the index of a computed dataset (error 42217); the tests check the statement there and nothing more ([ADR 0013, probe](./docs/adr/0013-a-computed-dataset-declares-its-indexes-and-the-destination-reconciles-them.md#probe-2026-11-21-vector-indexes-in-sql-server-2025-in-docker)).
|
|
242
242
|
- Azure SQL Database makes the index with no preview feature, on the `bigint` `_elt_row` of the raw table, and the `MERGE` of each commit works on the indexed table ([ADR 0013, Azure probe](./docs/adr/0013-a-computed-dataset-declares-its-indexes-and-the-destination-reconciles-them.md#probe-2026-11-21-vector-indexes-on-azure-sql-database)).
|
|
243
|
-
-
|
|
243
|
+
- Each table with a vector index has its search in `marts`, an inline table-valued function named after the reader view of the table with `_nearest`: `marts.staff_passage_vectors_nearest(@query VECTOR(1536), @k INT)`. It gives the `k` nearest rows of the raw table, each with its key and its `distance`, and, for the parts table of a `TableRule`, `marts.<name>_parts_nearest` gives the key, `seq` and `distance` of each part. Its body is `SELECT TOP (@k) WITH APPROXIMATE … FROM VECTOR_SEARCH(TABLE = [_warehouse].[raw_<name>] …) ORDER BY distance` with the metric of the index, because `VECTOR_SEARCH` uses the index only on a base table, and readers read only `marts` ([ADR 0014](./docs/adr/0014-each-vector-index-has-a-search-function-in-marts.md)). A row with no vector, such as a failed one, is not in the result. The join to other rows of `marts` can be in the same query, and the plan keeps the index:
|
|
244
244
|
|
|
245
245
|
```sql
|
|
246
|
-
DECLARE @qv VECTOR(1536) = CAST(@q AS VECTOR(1536));
|
|
247
|
-
WITH nearest AS (
|
|
248
|
-
SELECT TOP (10) WITH APPROXIMATE v.id, r.distance
|
|
249
|
-
FROM VECTOR_SEARCH(TABLE = _warehouse.raw_staff_passage_vectors AS v, COLUMN = vector, SIMILAR_TO = @qv, METRIC = 'cosine') AS r
|
|
250
|
-
WHERE v.status = 'computed'
|
|
251
|
-
ORDER BY r.distance)
|
|
252
246
|
SELECT p.id, p.text, n.distance
|
|
253
|
-
FROM
|
|
247
|
+
FROM marts.staff_passage_vectors_nearest(CAST(@q AS VECTOR(1536)), 10) AS n
|
|
248
|
+
JOIN marts.staff_passages AS p ON p.id = n.id
|
|
254
249
|
ORDER BY n.distance;
|
|
255
250
|
```
|
|
256
251
|
|
|
257
|
-
-
|
|
258
|
-
- The raw table
|
|
259
|
-
-
|
|
252
|
+
- Each refresh, after the indexes, makes the function, makes it again with `CREATE OR ALTER` when its stored definition differs, such as after a change of the metric or of the dimensions, and drops it when the table has no vector index. A table has at most one vector index, and its key cannot be named `distance`; the constructor refuses both with a `TypeError`. A copy whose reader name is the name of the function fails the pipeline before anything is made.
|
|
253
|
+
- The function and the raw table have one owner, so a reader that may read the function reads it with no permission on `_warehouse`: `GRANT SELECT ON SCHEMA::marts` covers each function of `marts`. The `readers` of `SqlServerSyncHistory` get `SELECT` on the numbering of `_warehouse` only.
|
|
254
|
+
- The function searches the rows of now: the raw table, also inside a reading as of an older position. A search as of a position reads the reader view `marts.staff_passage_vectors` in a reading, with `VECTOR_DISTANCE`, which is exact and uses no index. `VECTOR_SEARCH` through the reader view, a `UNION ALL` of the raw table and its history, also runs as an exact scan.
|
|
255
|
+
- The optimizer of Azure SQL chooses between the index and an exact scan by cost: a small table, or an index whose graph is not built yet, is searched exactly.
|
|
256
|
+
- SQL Server 2025 in Docker parses no `TOP … WITH APPROXIMATE` (error 102), and refuses the vector index before the function, so the function exists only on Azure SQL; the tests check its refusals and its drop there.
|
|
260
257
|
- See the recipe [Chunk documents, embed each chunk and search them with a vector index](./docs/recipes/search-chunks-with-a-vector-index.md).
|
|
261
258
|
|
|
262
259
|
## File bytes in the database
|
|
@@ -465,6 +462,7 @@ Each decision above that depends on the behavior of the server has a probe, whos
|
|
|
465
462
|
- Done: How SQL Server resolves a bare table name inside a view in the schema `marts`, when the user that runs the query has another default schema. The schema of the view, then `dbo`, never the default schema of the user; the user with another default schema ran in Docker only ([ADR 0006, probe](./docs/adr/0006-a-change-of-a-table-refreshes-the-views-over-it.md#probe-2026-11-20-a-bare-table-name-inside-a-view)).
|
|
466
463
|
- Done: Whether a point-in-time restore of Azure SQL changes `recovery_fork_guid` in `sys.database_recovery_status`, so a restored database can start a new epoch. First in Docker (Ezz, 2026-11-19): on SQL Server 2025 each restore changes it and the numbers repeat after the restore point; Azure SQL Database has no `sys.database_recovery_status` ([ADR 0002, probe](./docs/adr/0002-a-commit-number-is-the-rowversion-of-its-commit-row.md#probe-2026-11-20-recovery_fork_guid-after-a-restore)). A point-in-time restore on Azure SQL Database (a scratch server in Ezz's own subscription, Ezz, 2026-11-20) changes `database_id`, `create_date` and `physical_database_name` of `sys.databases`, which a plain database user can read, and the numbers repeat; the rename to the original name keeps `database_id` and `physical_database_name`, and a change of service objective also changes `physical_database_name`. On Azure SQL Database the epoch keeps all three and starts a new epoch when any one changes, because a missed restore gives wrong rows and a needless new epoch only a full read ([ADR 0002, probe](./docs/adr/0002-a-commit-number-is-the-rowversion-of-its-commit-row.md#probe-2026-11-20-an-azure-sql-database-point-in-time-restore)).
|
|
467
464
|
- Done: A vector index of a computed dataset on Azure SQL Database (a scratch server in Ezz's own subscription, 2026-11-21), through a real pipeline: the DiskANN index on the `bigint` `_elt_row`, the `MERGE` and its `OUTPUT deleted.*` on the indexed table, the names in `sys.indexes`, a change of the metric on a filled table with no recompute, and a reading as of a position. Each holds. `VECTOR_SEARCH` through `marts.<name>` runs with no index, so the search reads the raw table ([ADR 0013, Azure probe](./docs/adr/0013-a-computed-dataset-declares-its-indexes-and-the-destination-reconciles-them.md#probe-2026-11-21-vector-indexes-on-azure-sql-database)).
|
|
465
|
+
- Done: The search function of a vector index on Azure SQL Database (a scratch server in Ezz's own subscription, 2026-11-21): an inline table-valued function with `SELECT TOP (@k) WITH APPROXIMATE` over `VECTOR_SEARCH` on the raw table keeps the `Vector Index Seek`, also when the caller joins a reader view, and a user with `SELECT` on `marts` only calls it and may not read `_warehouse`. A real pipeline then made the function with its index, made it again for a new metric, and dropped it with the index ([ADR 0014, Azure probe](./docs/adr/0014-each-vector-index-has-a-search-function-in-marts.md#probe-2026-11-21-a-search-function-on-azure-sql-database)).
|
|
468
466
|
|
|
469
467
|
## Recipes
|
|
470
468
|
|
package/dist/computed-dataset.js
CHANGED
|
@@ -104,6 +104,8 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
104
104
|
indexes: new TableIndexes({
|
|
105
105
|
dataset: name,
|
|
106
106
|
table,
|
|
107
|
+
reader: name,
|
|
108
|
+
keys: [compute.key.name],
|
|
107
109
|
columns: declared,
|
|
108
110
|
indexes,
|
|
109
111
|
}),
|
|
@@ -143,6 +145,7 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
143
145
|
this.qualifiedName,
|
|
144
146
|
`${warehouse}.${rawTable(this.name)}`,
|
|
145
147
|
...('parts' in this.#compute ? this.#compute.parts.names : []),
|
|
148
|
+
...this.#tables.flatMap(({ indexes }) => indexes.names),
|
|
146
149
|
])
|
|
147
150
|
if (relations.has(identifiers.key(relation)))
|
|
148
151
|
throw new TypeError(`Computed dataset ${this.name} needs ${relation}, which a copy of the pipeline makes`);
|
package/dist/sql-server-as-of.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { parsePosition,
|
|
1
|
+
import { parsePosition, readingNumber } from '@eltjs/elt';
|
|
2
2
|
import { connect } from './sql-server-connection.js';
|
|
3
3
|
import { committedAt, numberingIn, requireRowVersions, settledMark, sqlServerDatabase, } from './sql-server-numbering.js';
|
|
4
4
|
import { now } from './sql-server-reader-view.js';
|
|
@@ -23,26 +23,21 @@ export async function readAsOf(database, position) {
|
|
|
23
23
|
if (settled === null)
|
|
24
24
|
return { expired: 'No load has committed yet' };
|
|
25
25
|
await connection.query('SET TRANSACTION ISOLATION LEVEL SNAPSHOT; BEGIN TRANSACTION');
|
|
26
|
-
const
|
|
27
|
-
if (
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
const number = asked?.number ?? numbered?.ends.at(-1);
|
|
33
|
-
const commit = number === undefined ? undefined : await committedAt(connection, number);
|
|
34
|
-
if (numbered === null || number === undefined || commit === undefined)
|
|
26
|
+
const read = readingNumber(await numberingIn(connection, settled), asked, sqlServerDatabase, 'the database');
|
|
27
|
+
if ('expired' in read)
|
|
28
|
+
return read;
|
|
29
|
+
const { epoch, number } = read;
|
|
30
|
+
const commit = await committedAt(connection, number);
|
|
31
|
+
if (commit === undefined)
|
|
35
32
|
return {
|
|
36
|
-
expired:
|
|
37
|
-
? 'The database has no position yet'
|
|
38
|
-
: `Position ${position} names no commit of the database`,
|
|
33
|
+
expired: `Position ${epoch}:${number} names no commit of the database`,
|
|
39
34
|
};
|
|
40
35
|
await connection.query(`DECLARE @position bigint = CAST(@number AS bigint);
|
|
41
36
|
EXEC sp_set_session_context @key = N'elt.position', @value = @position, @read_only = 1;`, { number: number < commit.newest ? String(number) : now });
|
|
42
37
|
const owned = stack.move();
|
|
43
38
|
return {
|
|
44
39
|
sql: connection,
|
|
45
|
-
position: `${
|
|
40
|
+
position: `${epoch}:${number}`,
|
|
46
41
|
at: Temporal.Instant.fromEpochMilliseconds(commit.at.getTime()),
|
|
47
42
|
[Symbol.asyncDispose]: () => owned.disposeAsync(),
|
|
48
43
|
};
|
|
@@ -10,9 +10,12 @@ export type SqlServerIndex = {
|
|
|
10
10
|
};
|
|
11
11
|
export declare class TableIndexes {
|
|
12
12
|
#private;
|
|
13
|
-
|
|
13
|
+
readonly names: readonly string[];
|
|
14
|
+
constructor({ dataset, table, reader, keys, columns, indexes, }: {
|
|
14
15
|
dataset: string;
|
|
15
16
|
table: string;
|
|
17
|
+
reader: string;
|
|
18
|
+
keys: readonly string[];
|
|
16
19
|
columns: readonly Declared[];
|
|
17
20
|
indexes: readonly SqlServerIndex[];
|
|
18
21
|
});
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { hash } from 'node:crypto';
|
|
2
2
|
import { identifiers, quote } from './sql-server-identifiers.js';
|
|
3
|
+
import { inMarts, marts } from './sql-server-schemas.js';
|
|
3
4
|
// The start of the name of each index that the destination manages. A
|
|
4
5
|
// person's own index has another name, so the destination never drops it.
|
|
5
6
|
const managed = 'ix_elt_';
|
|
@@ -7,15 +8,31 @@ const managed = 'ix_elt_';
|
|
|
7
8
|
// that the destination derives from its declaration, so a changed
|
|
8
9
|
// declaration is another index. reconcile brings the table to them in place,
|
|
9
10
|
// with no make of the table again and no recompute.
|
|
11
|
+
// A vector index has its search, the inline table-valued function
|
|
12
|
+
// marts.<reader>_nearest(@query, @k), named after the reader view of the
|
|
13
|
+
// table: VECTOR_SEARCH uses the index only on a base table, and readers read
|
|
14
|
+
// only marts. It gives the keys and the distance of the k nearest rows of the
|
|
15
|
+
// table as it is now. It and the table have one owner, so a reader that may
|
|
16
|
+
// read the function needs nothing of _warehouse (ADR 0014).
|
|
10
17
|
export class TableIndexes {
|
|
18
|
+
// The relations of marts that the table's search takes, schema-qualified:
|
|
19
|
+
// none when the table has no vector index.
|
|
20
|
+
names;
|
|
11
21
|
#table;
|
|
12
22
|
// The statement that makes each declared index, by its name.
|
|
13
23
|
#declared;
|
|
24
|
+
// The function that searches the vector index, in marts.
|
|
25
|
+
#search;
|
|
26
|
+
// The definition of the search, from FUNCTION on, as SQL Server keeps it;
|
|
27
|
+
// undefined when the table has no vector index.
|
|
28
|
+
#nearest;
|
|
14
29
|
// Refuses an index that the table cannot have, before anything is made: a
|
|
15
30
|
// column that the table does not have, a vector index on a column that is
|
|
16
|
-
// not a vector,
|
|
17
|
-
//
|
|
18
|
-
|
|
31
|
+
// not a vector, an index key that SQL Server refuses (error 1919 for a
|
|
32
|
+
// (max) type, 1978 for a vector), and a search that cannot have its name
|
|
33
|
+
// or its columns: a second vector index, and a key named distance. keys
|
|
34
|
+
// are the columns that name a row of the table, which the search gives.
|
|
35
|
+
constructor({ dataset, table, reader, keys, columns, indexes, }) {
|
|
19
36
|
const column = (name) => {
|
|
20
37
|
const found = columns.find((declared) => identifiers.key(declared.name) === identifiers.key(name));
|
|
21
38
|
if (found === undefined)
|
|
@@ -46,13 +63,31 @@ export class TableIndexes {
|
|
|
46
63
|
`CREATE NONCLUSTERED INDEX ${quote(indexName)} ON ${table} (${keys.map(quote).join(', ')})`,
|
|
47
64
|
];
|
|
48
65
|
}));
|
|
66
|
+
const name = identifiers.relation(`${reader}_nearest`);
|
|
67
|
+
this.#search = inMarts(name);
|
|
68
|
+
const vectors = indexes.filter((index) => index.type === 'vector');
|
|
69
|
+
const [vector] = vectors;
|
|
70
|
+
if (vectors.length > 1)
|
|
71
|
+
throw new TypeError(`Computed dataset ${dataset} declares ${vectors.length} vector indexes on ${table}, and ${this.#search} searches one`);
|
|
72
|
+
const distance = keys.find((key) => identifiers.key(key) === 'distance');
|
|
73
|
+
if (vector !== undefined && distance !== undefined)
|
|
74
|
+
throw new TypeError(`Computed dataset ${dataset} cannot search ${table} through ${this.#search}: its column ${distance} has the name of the distance that the search gives`);
|
|
75
|
+
this.names = Object.freeze(vector === undefined ? [] : [`${marts}.${name}`]);
|
|
76
|
+
if (vector === undefined)
|
|
77
|
+
this.#nearest = undefined;
|
|
78
|
+
else {
|
|
79
|
+
const searched = column(vector.column);
|
|
80
|
+
this.#nearest = `FUNCTION ${this.#search}(@query ${searched.type}, @k int) RETURNS TABLE AS RETURN SELECT TOP (@k) WITH APPROXIMATE ${keys.map((key) => `t.${quote(key)}`).join(', ')}, r.[distance] FROM VECTOR_SEARCH(TABLE = ${table} AS t, COLUMN = ${quote(searched.name)}, SIMILAR_TO = @query, METRIC = '${vector.metric}') AS r ORDER BY r.[distance]`;
|
|
81
|
+
}
|
|
49
82
|
Object.freeze(this);
|
|
50
83
|
}
|
|
51
84
|
// Drops each managed index of the table that is no longer declared, then
|
|
52
|
-
// makes each declared index that the table does not have
|
|
53
|
-
//
|
|
54
|
-
//
|
|
55
|
-
//
|
|
85
|
+
// makes each declared index that the table does not have, then makes the
|
|
86
|
+
// search of the vector index, or makes it again when its stored
|
|
87
|
+
// definition differs, or drops it when the table has no vector index.
|
|
88
|
+
// Each statement runs outside a transaction, since SQL Server makes a
|
|
89
|
+
// vector index only there (error 574); a refresh that stops between two of
|
|
90
|
+
// them makes the rest the next time.
|
|
56
91
|
async reconcile(connection) {
|
|
57
92
|
const stored = new Set((await connection.query(`SELECT [name] FROM sys.indexes WHERE [object_id] = OBJECT_ID(@table) AND [name] LIKE N'${managed.replaceAll('_', '[_]')}%'`, { table: this.#table })).map(({ name }) => String(name)));
|
|
58
93
|
for (const name of stored)
|
|
@@ -61,8 +96,26 @@ export class TableIndexes {
|
|
|
61
96
|
for (const [name, statement] of this.#declared)
|
|
62
97
|
if (!stored.has(name))
|
|
63
98
|
await connection.query(statement);
|
|
99
|
+
const search = await searchDefinition(connection, this.#search);
|
|
100
|
+
if (this.#nearest === undefined) {
|
|
101
|
+
if (search !== undefined)
|
|
102
|
+
await connection.query(`DROP FUNCTION ${this.#search}`);
|
|
103
|
+
}
|
|
104
|
+
else if (search !== this.#nearest)
|
|
105
|
+
await connection.query(`CREATE OR ALTER ${this.#nearest}`);
|
|
64
106
|
}
|
|
65
107
|
}
|
|
108
|
+
// The definition of the inline table-valued function name, from FUNCTION on,
|
|
109
|
+
// or undefined when the database has none of that name. SQL Server keeps the
|
|
110
|
+
// text of the statement that made the function, with CREATE OR ALTER kept
|
|
111
|
+
// as CREATE (Azure probe of ADR 0014), so the words before FUNCTION are
|
|
112
|
+
// skipped.
|
|
113
|
+
async function searchDefinition(connection, name) {
|
|
114
|
+
const [stored] = await connection.query("SELECT OBJECT_DEFINITION(OBJECT_ID(@name, N'IF')) AS [definition]", { name });
|
|
115
|
+
return typeof stored?.definition === 'string'
|
|
116
|
+
? stored.definition.replace(/^CREATE\s+(?:OR\s+ALTER\s+)?/i, '')
|
|
117
|
+
: undefined;
|
|
118
|
+
}
|
|
66
119
|
// The name of an index: the managed prefix and the first 16 hex digits of the
|
|
67
120
|
// SHA-256 of its declaration.
|
|
68
121
|
const derivedName = (declaration) => `${managed}${hash('sha256', JSON.stringify(declaration)).slice(0, 16)}`;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@eltjs/elt-sqlserver",
|
|
3
|
-
"version": "0.10.
|
|
3
|
+
"version": "0.10.1",
|
|
4
4
|
"description": "The SQL Server destination of elt: each stream is a typed table in _warehouse, with a reader view in marts that the host names, numbered commits that readers read as of a position, and a checkpoint store and a sync history in the same database.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"repository": {
|
|
@@ -38,8 +38,8 @@
|
|
|
38
38
|
"access": "public"
|
|
39
39
|
},
|
|
40
40
|
"dependencies": {
|
|
41
|
-
"@eltjs/elt": "^0.10.
|
|
42
|
-
"@eltjs/sdk-microsoft-sql-server": "^0.10.
|
|
41
|
+
"@eltjs/elt": "^0.10.1",
|
|
42
|
+
"@eltjs/sdk-microsoft-sql-server": "^0.10.1",
|
|
43
43
|
"@zukhruf/async": "^0.8.51",
|
|
44
44
|
"@zukhruf/mutex": "^0.8.31",
|
|
45
45
|
"@zukhruf/vectors": "^0.8.51"
|