@eltjs/elt-sqlserver 0.10.0 → 0.10.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -10,9 +10,9 @@ The words in these documents have one meaning each. See the glossary in [CONTEXT
10
10
 
11
11
  | Now (first release) | Later | Never |
12
12
  | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
13
- | `SqlServerDestination`: the four destination sync modes, evolve, reload, reader views, extended properties, file chunks | Native `VECTOR` over TDS, when tedious has it ([tedious #1841](https://github.com/tediousjs/tedious/pull/1841)) | A search API in the destination: search is the host's SQL (`@zukhruf/vectors/sqlserver`) |
13
+ | `SqlServerDestination`: the four destination sync modes, evolve, reload, reader views, extended properties, file chunks | Native `VECTOR` over TDS, when tedious has it ([tedious #1841](https://github.com/tediousjs/tedious/pull/1841)) | A search API beyond the `_nearest` function of a vector index: any other search is the host's SQL (`@zukhruf/vectors/sqlserver`) |
14
14
  | Numbered commits, positions, history, readings, retention, a clear that forgets | `float16` vector columns (`HalfVector`), for a model of more than 1,998 dimensions, such as `text-embedding-3-large` (3,072) | System-versioned temporal tables as the history ([ADR 0001](./docs/adr/0001-the-sql-server-destination-takes-the-declarations-of-elt-postgresql.md)) |
15
- | `View` and `ComputedDataset`, with `vector(n)` rule columns as `VECTOR(n)` and declared indexes: nonclustered, and DiskANN for `VECTOR_SEARCH` | A bulk load through Azure Blob Storage and `BULK INSERT`, as the bulk mode of Airbyte, when a load is too slow through JSON batches | An ORM, or a query builder over the reader's SQL |
15
+ | `View` and `ComputedDataset`, with `vector(n)` rule columns as `VECTOR(n)` and declared indexes: nonclustered, and DiskANN with its search `marts.<name>_nearest` | A bulk load through Azure Blob Storage and `BULK INSERT`, as the bulk mode of Airbyte, when a load is too slow through JSON batches | An ORM, or a query builder over the reader's SQL |
16
16
  | `SqlServerCheckpointStore`, `SqlServerSyncHistory` (with `derived_status`, `catalog` and named readers), `readPassStatus` | Microsoft Entra ID logins (managed identity), which `mssql` supports through `authentication` | An `mssql://` URL |
17
17
  | The prerequisites in other packages: the `vector` rule column of elt, `dimensions` on `Embedding`, and the extraction of the rules that the destinations share ([ADR 0012](./docs/adr/0012-the-rules-that-the-destinations-share-move-into-elt-before-the-third-destination.md)) | SQL Server 2022 and older, which have no `VECTOR` and no `JSON` type | A migration of stored data between versions (elt AGENTS.md: stored data is disposable) |
18
18
 
@@ -240,23 +240,20 @@ new ComputedDataset({
240
240
  - The vector index is `CREATE VECTOR INDEX … ON [_warehouse].[raw_staff_passage_vectors] ([vector]) WITH (METRIC = 'cosine', TYPE = 'DiskANN')`, which each refresh makes when the table does not have it, also after a changed declaration makes the table again (see "Declare a computed dataset" below). An index made on an empty table starts as an exact index, and Azure SQL builds its graph after the rows come (`@zukhruf/vectors`, "SQL Server: what the probes found").
241
241
  - SQL Server 2025 in Docker makes a vector index only with `PREVIEW_FEATURES` on and only on an `int` clustered key, so it refuses the index of a computed dataset (error 42217); the tests check the statement there and nothing more ([ADR 0013, probe](./docs/adr/0013-a-computed-dataset-declares-its-indexes-and-the-destination-reconciles-them.md#probe-2026-11-21-vector-indexes-in-sql-server-2025-in-docker)).
242
242
  - Azure SQL Database makes the index with no preview feature, on the `bigint` `_elt_row` of the raw table, and the `MERGE` of each commit works on the indexed table ([ADR 0013, Azure probe](./docs/adr/0013-a-computed-dataset-declares-its-indexes-and-the-destination-reconciles-them.md#probe-2026-11-21-vector-indexes-on-azure-sql-database)).
243
- - The search is the host's own SQL, with every feature of Azure SQL. `VECTOR_SEARCH` uses the index only on the raw table, and only when no join is in the same query block. Thus the nearest rows come from `_warehouse.raw_<name>` in a common table expression, and a join comes after it:
243
+ - Each table with a vector index has its search in `marts`, an inline table-valued function named after the reader view of the table with `_nearest`: `marts.staff_passage_vectors_nearest(@query VECTOR(1536), @k INT)`. It gives the `k` nearest rows of the raw table, each with its key and its `distance`, and, for the parts table of a `TableRule`, `marts.<name>_parts_nearest` gives the key, `seq` and `distance` of each part. Its body is `SELECT TOP (@k) WITH APPROXIMATE … FROM VECTOR_SEARCH(TABLE = [_warehouse].[raw_<name>] …) ORDER BY distance` with the metric of the index, because `VECTOR_SEARCH` uses the index only on a base table, and readers read only `marts` ([ADR 0014](./docs/adr/0014-each-vector-index-has-a-search-function-in-marts.md)). A row with no vector, such as a failed one, is not in the result. The join to other rows of `marts` can be in the same query, and the plan keeps the index:
244
244
 
245
245
  ```sql
246
- DECLARE @qv VECTOR(1536) = CAST(@q AS VECTOR(1536));
247
- WITH nearest AS (
248
- SELECT TOP (10) WITH APPROXIMATE v.id, r.distance
249
- FROM VECTOR_SEARCH(TABLE = _warehouse.raw_staff_passage_vectors AS v, COLUMN = vector, SIMILAR_TO = @qv, METRIC = 'cosine') AS r
250
- WHERE v.status = 'computed'
251
- ORDER BY r.distance)
252
246
  SELECT p.id, p.text, n.distance
253
- FROM nearest AS n JOIN marts.staff_passages AS p ON p.id = n.id
247
+ FROM marts.staff_passage_vectors_nearest(CAST(@q AS VECTOR(1536)), 10) AS n
248
+ JOIN marts.staff_passages AS p ON p.id = n.id
254
249
  ORDER BY n.distance;
255
250
  ```
256
251
 
257
- - `VECTOR_SEARCH` through the reader view `marts.staff_passage_vectors`, a `UNION ALL` of the raw table and its history, gives the same rows with an exact scan, and no index.
258
- - The raw table holds the rows of now. A search as of a position reads the reader view in a reading, with `VECTOR_DISTANCE` or `VECTOR_SEARCH`, and no index.
259
- - A reader that searches needs `SELECT` on the raw table. The `readers` of `SqlServerSyncHistory` get `SELECT` on the numbering of `_warehouse` only.
252
+ - Each refresh, after the indexes, makes the function, makes it again with `CREATE OR ALTER` when its stored definition differs, such as after a change of the metric or of the dimensions, and drops it when the table has no vector index. A table has at most one vector index, and its key cannot be named `distance`; the constructor refuses both with a `TypeError`. A copy whose reader name is the name of the function fails the pipeline before anything is made.
253
+ - The function and the raw table have one owner, so a reader that may read the function reads it with no permission on `_warehouse`: `GRANT SELECT ON SCHEMA::marts` covers each function of `marts`. The `readers` of `SqlServerSyncHistory` get `SELECT` on the numbering of `_warehouse` only.
254
+ - The function searches the rows of now: the raw table, also inside a reading as of an older position. A search as of a position reads the reader view `marts.staff_passage_vectors` in a reading, with `VECTOR_DISTANCE`, which is exact and uses no index. `VECTOR_SEARCH` through the reader view, a `UNION ALL` of the raw table and its history, also runs as an exact scan.
255
+ - The optimizer of Azure SQL chooses between the index and an exact scan by cost: a small table, or an index whose graph is not built yet, is searched exactly.
256
+ - SQL Server 2025 in Docker parses no `TOP … WITH APPROXIMATE` (error 102), and refuses the vector index before the function, so the function exists only on Azure SQL; the tests check its refusals and its drop there.
260
257
  - See the recipe [Chunk documents, embed each chunk and search them with a vector index](./docs/recipes/search-chunks-with-a-vector-index.md).
261
258
 
262
259
  ## File bytes in the database
@@ -465,6 +462,7 @@ Each decision above that depends on the behavior of the server has a probe, whos
465
462
  - Done: How SQL Server resolves a bare table name inside a view in the schema `marts`, when the user that runs the query has another default schema. The schema of the view, then `dbo`, never the default schema of the user; the user with another default schema ran in Docker only ([ADR 0006, probe](./docs/adr/0006-a-change-of-a-table-refreshes-the-views-over-it.md#probe-2026-11-20-a-bare-table-name-inside-a-view)).
466
463
  - Done: Whether a point-in-time restore of Azure SQL changes `recovery_fork_guid` in `sys.database_recovery_status`, so a restored database can start a new epoch. First in Docker (Ezz, 2026-11-19): on SQL Server 2025 each restore changes it and the numbers repeat after the restore point; Azure SQL Database has no `sys.database_recovery_status` ([ADR 0002, probe](./docs/adr/0002-a-commit-number-is-the-rowversion-of-its-commit-row.md#probe-2026-11-20-recovery_fork_guid-after-a-restore)). A point-in-time restore on Azure SQL Database (a scratch server in Ezz's own subscription, Ezz, 2026-11-20) changes `database_id`, `create_date` and `physical_database_name` of `sys.databases`, which a plain database user can read, and the numbers repeat; the rename to the original name keeps `database_id` and `physical_database_name`, and a change of service objective also changes `physical_database_name`. On Azure SQL Database the epoch keeps all three and starts a new epoch when any one changes, because a missed restore gives wrong rows and a needless new epoch only a full read ([ADR 0002, probe](./docs/adr/0002-a-commit-number-is-the-rowversion-of-its-commit-row.md#probe-2026-11-20-an-azure-sql-database-point-in-time-restore)).
467
464
  - Done: A vector index of a computed dataset on Azure SQL Database (a scratch server in Ezz's own subscription, 2026-11-21), through a real pipeline: the DiskANN index on the `bigint` `_elt_row`, the `MERGE` and its `OUTPUT deleted.*` on the indexed table, the names in `sys.indexes`, a change of the metric on a filled table with no recompute, and a reading as of a position. Each holds. `VECTOR_SEARCH` through `marts.<name>` runs with no index, so the search reads the raw table ([ADR 0013, Azure probe](./docs/adr/0013-a-computed-dataset-declares-its-indexes-and-the-destination-reconciles-them.md#probe-2026-11-21-vector-indexes-on-azure-sql-database)).
465
+ - Done: The search function of a vector index on Azure SQL Database (a scratch server in Ezz's own subscription, 2026-11-21): an inline table-valued function with `SELECT TOP (@k) WITH APPROXIMATE` over `VECTOR_SEARCH` on the raw table keeps the `Vector Index Seek`, also when the caller joins a reader view, and a user with `SELECT` on `marts` only calls it and may not read `_warehouse`. A real pipeline then made the function with its index, made it again for a new metric, and dropped it with the index ([ADR 0014, Azure probe](./docs/adr/0014-each-vector-index-has-a-search-function-in-marts.md#probe-2026-11-21-a-search-function-on-azure-sql-database)).
468
466
 
469
467
  ## Recipes
470
468
 
@@ -104,6 +104,8 @@ export class ComputedDataset extends DerivedDataset {
104
104
  indexes: new TableIndexes({
105
105
  dataset: name,
106
106
  table,
107
+ reader: name,
108
+ keys: [compute.key.name],
107
109
  columns: declared,
108
110
  indexes,
109
111
  }),
@@ -143,6 +145,7 @@ export class ComputedDataset extends DerivedDataset {
143
145
  this.qualifiedName,
144
146
  `${warehouse}.${rawTable(this.name)}`,
145
147
  ...('parts' in this.#compute ? this.#compute.parts.names : []),
148
+ ...this.#tables.flatMap(({ indexes }) => indexes.names),
146
149
  ])
147
150
  if (relations.has(identifiers.key(relation)))
148
151
  throw new TypeError(`Computed dataset ${this.name} needs ${relation}, which a copy of the pipeline makes`);
@@ -1,4 +1,4 @@
1
- import { parsePosition, readingExpiry } from '@eltjs/elt';
1
+ import { parsePosition, readingNumber } from '@eltjs/elt';
2
2
  import { connect } from './sql-server-connection.js';
3
3
  import { committedAt, numberingIn, requireRowVersions, settledMark, sqlServerDatabase, } from './sql-server-numbering.js';
4
4
  import { now } from './sql-server-reader-view.js';
@@ -23,26 +23,21 @@ export async function readAsOf(database, position) {
23
23
  if (settled === null)
24
24
  return { expired: 'No load has committed yet' };
25
25
  await connection.query('SET TRANSACTION ISOLATION LEVEL SNAPSHOT; BEGIN TRANSACTION');
26
- const numbered = await numberingIn(connection, settled);
27
- if (asked !== undefined) {
28
- const expired = readingExpiry(numbered, asked, sqlServerDatabase, 'the database');
29
- if (expired !== undefined)
30
- return { expired };
31
- }
32
- const number = asked?.number ?? numbered?.ends.at(-1);
33
- const commit = number === undefined ? undefined : await committedAt(connection, number);
34
- if (numbered === null || number === undefined || commit === undefined)
26
+ const read = readingNumber(await numberingIn(connection, settled), asked, sqlServerDatabase, 'the database');
27
+ if ('expired' in read)
28
+ return read;
29
+ const { epoch, number } = read;
30
+ const commit = await committedAt(connection, number);
31
+ if (commit === undefined)
35
32
  return {
36
- expired: position === undefined
37
- ? 'The database has no position yet'
38
- : `Position ${position} names no commit of the database`,
33
+ expired: `Position ${epoch}:${number} names no commit of the database`,
39
34
  };
40
35
  await connection.query(`DECLARE @position bigint = CAST(@number AS bigint);
41
36
  EXEC sp_set_session_context @key = N'elt.position', @value = @position, @read_only = 1;`, { number: number < commit.newest ? String(number) : now });
42
37
  const owned = stack.move();
43
38
  return {
44
39
  sql: connection,
45
- position: `${numbered.epoch}:${number}`,
40
+ position: `${epoch}:${number}`,
46
41
  at: Temporal.Instant.fromEpochMilliseconds(commit.at.getTime()),
47
42
  [Symbol.asyncDispose]: () => owned.disposeAsync(),
48
43
  };
@@ -10,9 +10,12 @@ export type SqlServerIndex = {
10
10
  };
11
11
  export declare class TableIndexes {
12
12
  #private;
13
- constructor({ dataset, table, columns, indexes, }: {
13
+ readonly names: readonly string[];
14
+ constructor({ dataset, table, reader, keys, columns, indexes, }: {
14
15
  dataset: string;
15
16
  table: string;
17
+ reader: string;
18
+ keys: readonly string[];
16
19
  columns: readonly Declared[];
17
20
  indexes: readonly SqlServerIndex[];
18
21
  });
@@ -1,5 +1,6 @@
1
1
  import { hash } from 'node:crypto';
2
2
  import { identifiers, quote } from './sql-server-identifiers.js';
3
+ import { inMarts, marts } from './sql-server-schemas.js';
3
4
  // The start of the name of each index that the destination manages. A
4
5
  // person's own index has another name, so the destination never drops it.
5
6
  const managed = 'ix_elt_';
@@ -7,15 +8,31 @@ const managed = 'ix_elt_';
7
8
  // that the destination derives from its declaration, so a changed
8
9
  // declaration is another index. reconcile brings the table to them in place,
9
10
  // with no make of the table again and no recompute.
11
+ // A vector index has its search, the inline table-valued function
12
+ // marts.<reader>_nearest(@query, @k), named after the reader view of the
13
+ // table: VECTOR_SEARCH uses the index only on a base table, and readers read
14
+ // only marts. It gives the keys and the distance of the k nearest rows of the
15
+ // table as it is now. It and the table have one owner, so a reader that may
16
+ // read the function needs nothing of _warehouse (ADR 0014).
10
17
  export class TableIndexes {
18
+ // The relations of marts that the table's search takes, schema-qualified:
19
+ // none when the table has no vector index.
20
+ names;
11
21
  #table;
12
22
  // The statement that makes each declared index, by its name.
13
23
  #declared;
24
+ // The function that searches the vector index, in marts.
25
+ #search;
26
+ // The definition of the search, from FUNCTION on, as SQL Server keeps it;
27
+ // undefined when the table has no vector index.
28
+ #nearest;
14
29
  // Refuses an index that the table cannot have, before anything is made: a
15
30
  // column that the table does not have, a vector index on a column that is
16
- // not a vector, and an index key that SQL Server refuses (error 1919 for
17
- // a (max) type, 1978 for a vector).
18
- constructor({ dataset, table, columns, indexes, }) {
31
+ // not a vector, an index key that SQL Server refuses (error 1919 for a
32
+ // (max) type, 1978 for a vector), and a search that cannot have its name
33
+ // or its columns: a second vector index, and a key named distance. keys
34
+ // are the columns that name a row of the table, which the search gives.
35
+ constructor({ dataset, table, reader, keys, columns, indexes, }) {
19
36
  const column = (name) => {
20
37
  const found = columns.find((declared) => identifiers.key(declared.name) === identifiers.key(name));
21
38
  if (found === undefined)
@@ -46,13 +63,31 @@ export class TableIndexes {
46
63
  `CREATE NONCLUSTERED INDEX ${quote(indexName)} ON ${table} (${keys.map(quote).join(', ')})`,
47
64
  ];
48
65
  }));
66
+ const name = identifiers.relation(`${reader}_nearest`);
67
+ this.#search = inMarts(name);
68
+ const vectors = indexes.filter((index) => index.type === 'vector');
69
+ const [vector] = vectors;
70
+ if (vectors.length > 1)
71
+ throw new TypeError(`Computed dataset ${dataset} declares ${vectors.length} vector indexes on ${table}, and ${this.#search} searches one`);
72
+ const distance = keys.find((key) => identifiers.key(key) === 'distance');
73
+ if (vector !== undefined && distance !== undefined)
74
+ throw new TypeError(`Computed dataset ${dataset} cannot search ${table} through ${this.#search}: its column ${distance} has the name of the distance that the search gives`);
75
+ this.names = Object.freeze(vector === undefined ? [] : [`${marts}.${name}`]);
76
+ if (vector === undefined)
77
+ this.#nearest = undefined;
78
+ else {
79
+ const searched = column(vector.column);
80
+ this.#nearest = `FUNCTION ${this.#search}(@query ${searched.type}, @k int) RETURNS TABLE AS RETURN SELECT TOP (@k) WITH APPROXIMATE ${keys.map((key) => `t.${quote(key)}`).join(', ')}, r.[distance] FROM VECTOR_SEARCH(TABLE = ${table} AS t, COLUMN = ${quote(searched.name)}, SIMILAR_TO = @query, METRIC = '${vector.metric}') AS r ORDER BY r.[distance]`;
81
+ }
49
82
  Object.freeze(this);
50
83
  }
51
84
  // Drops each managed index of the table that is no longer declared, then
52
- // makes each declared index that the table does not have. Each statement
53
- // runs outside a transaction, since SQL Server makes a vector index only
54
- // there (error 574); a refresh that stops between two of them makes the
55
- // rest the next time.
85
+ // makes each declared index that the table does not have, then makes the
86
+ // search of the vector index, or makes it again when its stored
87
+ // definition differs, or drops it when the table has no vector index.
88
+ // Each statement runs outside a transaction, since SQL Server makes a
89
+ // vector index only there (error 574); a refresh that stops between two of
90
+ // them makes the rest the next time.
56
91
  async reconcile(connection) {
57
92
  const stored = new Set((await connection.query(`SELECT [name] FROM sys.indexes WHERE [object_id] = OBJECT_ID(@table) AND [name] LIKE N'${managed.replaceAll('_', '[_]')}%'`, { table: this.#table })).map(({ name }) => String(name)));
58
93
  for (const name of stored)
@@ -61,8 +96,26 @@ export class TableIndexes {
61
96
  for (const [name, statement] of this.#declared)
62
97
  if (!stored.has(name))
63
98
  await connection.query(statement);
99
+ const search = await searchDefinition(connection, this.#search);
100
+ if (this.#nearest === undefined) {
101
+ if (search !== undefined)
102
+ await connection.query(`DROP FUNCTION ${this.#search}`);
103
+ }
104
+ else if (search !== this.#nearest)
105
+ await connection.query(`CREATE OR ALTER ${this.#nearest}`);
64
106
  }
65
107
  }
108
+ // The definition of the inline table-valued function name, from FUNCTION on,
109
+ // or undefined when the database has none of that name. SQL Server keeps the
110
+ // text of the statement that made the function, with CREATE OR ALTER kept
111
+ // as CREATE (Azure probe of ADR 0014), so the words before FUNCTION are
112
+ // skipped.
113
+ async function searchDefinition(connection, name) {
114
+ const [stored] = await connection.query("SELECT OBJECT_DEFINITION(OBJECT_ID(@name, N'IF')) AS [definition]", { name });
115
+ return typeof stored?.definition === 'string'
116
+ ? stored.definition.replace(/^CREATE\s+(?:OR\s+ALTER\s+)?/i, '')
117
+ : undefined;
118
+ }
66
119
  // The name of an index: the managed prefix and the first 16 hex digits of the
67
120
  // SHA-256 of its declaration.
68
121
  const derivedName = (declaration) => `${managed}${hash('sha256', JSON.stringify(declaration)).slice(0, 16)}`;
@@ -62,6 +62,8 @@ export class SqlServerPartsTable {
62
62
  this.indexes = new TableIndexes({
63
63
  dataset,
64
64
  table: this.table,
65
+ reader: view.name,
66
+ keys: [rule.key.name, 'seq'],
65
67
  columns: this.declared,
66
68
  indexes,
67
69
  });
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@eltjs/elt-sqlserver",
3
- "version": "0.10.0",
3
+ "version": "0.10.1",
4
4
  "description": "The SQL Server destination of elt: each stream is a typed table in _warehouse, with a reader view in marts that the host names, numbered commits that readers read as of a position, and a checkpoint store and a sync history in the same database.",
5
5
  "license": "Apache-2.0",
6
6
  "repository": {
@@ -38,8 +38,8 @@
38
38
  "access": "public"
39
39
  },
40
40
  "dependencies": {
41
- "@eltjs/elt": "^0.10.0",
42
- "@eltjs/sdk-microsoft-sql-server": "^0.10.0",
41
+ "@eltjs/elt": "^0.10.1",
42
+ "@eltjs/sdk-microsoft-sql-server": "^0.10.1",
43
43
  "@zukhruf/async": "^0.8.51",
44
44
  "@zukhruf/mutex": "^0.8.31",
45
45
  "@zukhruf/vectors": "^0.8.51"