@memberjunction/ai-vectors-sqlserver 0.0.1 → 5.39.0

package/README.md

@@ -1,45 +1,88 @@
 # @memberjunction/ai-vectors-sqlserver
 
-## ⚠️ IMPORTANT NOTICE ⚠️
-
-**This package is created solely for the purpose of setting up OIDC (OpenID Connect) trusted publishing with npm.**
-
-This is **NOT** a functional package and contains **NO** code or functionality beyond the OIDC setup configuration.
-
-## Purpose
-
-This package exists to:
-1. Configure OIDC trusted publishing for the package name `@memberjunction/ai-vectors-sqlserver`
-2. Enable secure, token-less publishing from CI/CD workflows
-3. Establish provenance for packages published under this name
-
-## What is OIDC Trusted Publishing?
-
-OIDC trusted publishing allows package maintainers to publish packages directly from their CI/CD workflows without needing to manage npm access tokens. Instead, it uses OpenID Connect to establish trust between the CI/CD provider (like GitHub Actions) and npm.
-
-## Setup Instructions
-
-To properly configure OIDC trusted publishing for this package:
-
-1. Go to [npmjs.com](https://www.npmjs.com/) and navigate to your package settings
-2. Configure the trusted publisher (e.g., GitHub Actions)
-3. Specify the repository and workflow that should be allowed to publish
-4. Use the configured workflow to publish your actual package
-
-## DO NOT USE THIS PACKAGE
-
-This package is a placeholder for OIDC configuration only. It:
-- Contains no executable code
-- Provides no functionality
-- Should not be installed as a dependency
-- Exists only for administrative purposes
-
-## More Information
-
-For more details about npm's trusted publishing feature, see:
-- [npm Trusted Publishing Documentation](https://docs.npmjs.com/generating-provenance-statements)
-- [GitHub Actions OIDC Documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect)
-
----
-
-**Maintained for OIDC setup purposes only**
+A **colocated** MemberJunction vector-database provider backed by **SQL Server 2025 native vectors**
+(the `VECTOR(N)` type and the `VECTOR_SEARCH` DiskANN table-valued function). It stores and queries
+embeddings inside the application's own SQL Server database — borrowing the active data provider's
+connection rather than opening a separate pool — so vectors live alongside the entity rows they
+describe and can be searched with the database's native ANN index.
+
+Registered with the MJ class factory as **`SQLServerVectorDatabase`** (`@RegisterClass(VectorDBBase, 'SQLServerVectorDatabase')`).
+
+> **Status:** validated end-to-end against a live **SQL Server 2025 RTM** container (17.0.4050)
+> in `sibling` storage mode — index create, upsert, exact query, metadata (`JSON_VALUE`) filtering,
+> and result mapping all confirmed working. The `entityColumn` mode and the approximate
+> (`VECTOR_SEARCH`) path on Azure SQL Database remain unexercised; treat those as untested.
+
+## How it works
+
+This is a [colocated](../../Database/README.md) provider: it implements no connection logic of its
+own. Instead it receives the active relational connection through MemberJunction's
+`IColocatedVectorHost` adapter (implemented by `SQLServerDataProvider`) and runs all DDL/DML through
+it. When a transaction is open on that connection, vector writes participate in it.
+
+### Query path
+
+The provider prefers the DiskANN-aware **`VECTOR_SEARCH`** TVF (`SELECT TOP (N) WITH APPROXIMATE`),
+which engages the vector index and is dramatically faster than a brute-force scan at scale. But that
+surface is **not present in every SQL Server 2025 build** — verified live, **boxed SQL Server 2025 RTM
+(17.0.4050) does not have `VECTOR_SEARCH`/`WITH APPROXIMATE` or `CREATE VECTOR INDEX … DiskANN`**; only
+the `VECTOR(n)` type and the exact `VECTOR_DISTANCE` function ship there. (The approximate surface is
+currently Azure SQL Database, and likely a later boxed CU.)
+
+So the provider **detects support lazily and falls back**: it attempts the `VECTOR_SEARCH` query once;
+if the server rejects it, it caches that fact process-wide and routes all subsequent queries through an
+exact **`VECTOR_DISTANCE`** scan. Where the TVF *is* available, filtered queries additionally dispatch
+on cardinality — below a configurable threshold (default 50,000 matching rows) they use the exact path
+anyway, because DiskANN's graph walk fails to converge when a filter's row cluster is disjoint from the
+query vector's neighborhood. Net: correct results everywhere, accelerated where the index exists.
+
+### Storage modes
+
+Configured per index via `MJVectorIndex.ProviderConfig` (which flows to `CreateIndexParams.additionalParams`):
+
+| Mode | Where vectors live | Filters resolve against | Use it for |
+|---|---|---|---|
+| `sibling` (default) | an MJ-managed sibling table (`id`/`embedding`/`metadata`/`content`) | the JSON `metadata` column | generic, entity-agnostic, multi-model indexes |
+| `entityColumn` | an existing `VECTOR` column on an entity table | **live entity columns** | adopting embeddings already stored on a table — no re-vectorization |
+
+#### `entityColumn` configuration
+
+```json
+{
+  "storageMode": "entityColumn",
+  "sourceTable": "Recommendation.Content",
+  "vectorColumn": "Embedding",
+  "keyColumn": "ID",
+  "entityName": "Content",
+  "selectColumns": ["Title", "URL", "Source", "ContentType", "Date"],
+  "iterativeFilterThreshold": 50000
+}
+```
+
+The provider projects the listed columns into each result's metadata and synthesizes the MJ
+`RecordID`/`Entity` fields so the `SearchEngine` renders these results identically to sibling/external
+indexes. `entityColumn` upsert/delete operate by a single scalar key (`keyColumn`); composite-PK
+entities are not supported in this mode.
+
+## Enabling
+
+1. Create an `MJ: Vector Databases` row with `ClassKey = 'SQLServerVectorDatabase'` (no `DefaultURL`
+   or `CredentialID` — it uses the application's connection).
+2. Create `MJ: Vector Indexes` rows pointing at it, with the `ProviderConfig` above for `entityColumn`
+   mode (or none for `sibling` mode).
+3. Search flows through `@memberjunction/search-engine`'s `VectorSearchProvider`; sync flows through
+   `@memberjunction/ai-vector-sync`'s `EntityVectorSyncer`.
+
+## Limitations
+
+- Requires **SQL Server 2025** (major version ≥ 17); the provider fails loud on older servers.
+- **Approximate / DiskANN search** depends on the `VECTOR_SEARCH` TVF, which is absent on boxed
+  SQL Server 2025 RTM (the provider transparently falls back to exact `VECTOR_DISTANCE` — correct,
+  but O(rows) per query, so large corpora on the boxed product will be slow until a CU adds the TVF).
+- **Hybrid keyword search** (full-text `CONTAINS` + vector) is not yet implemented — queries are
+  vector-only. (The pgvector colocated provider does RRF hybrid today.)
+- `entityColumn` mode supports **single-column keys** only.
+
+## License
+
+ISC

package/dist/index.d.ts

@@ -0,0 +1,3 @@
+export * from './models/SQLServerVectorDatabase.js';
+export * from './models/sqlserverColocatedSQL.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,kCAAkC,CAAC;AACjD,cAAc,gCAAgC,CAAC"}

package/dist/index.js

@@ -0,0 +1,3 @@
+export * from './models/SQLServerVectorDatabase.js';
+export * from './models/sqlserverColocatedSQL.js';
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,kCAAkC,CAAC;AACjD,cAAc,gCAAgC,CAAC"}

package/dist/models/SQLServerVectorDatabase.d.ts

@@ -0,0 +1,109 @@
+import { BaseRequestParams, BaseResponse, ColocatedQueryOptions, ColocatedQueryResult, CreateIndexParams, EditIndexParams, IndexList, ListVectorIDsParams, ListVectorIDsResult, QueryOptions, SharedIndexFilterOptions, UpdateOptions, VectorDBBase, VectorRecord } from '@memberjunction/ai-vectordb';
+import { UserInfo } from '@memberjunction/core';
+/**
+ * Colocated MemberJunction vector provider backed by **SQL Server 2025 native vectors**
+ * (`VECTOR(N)` + the `VECTOR_SEARCH` DiskANN table-valued function), stored in the application's
+ * own database. Borrows the active data provider's connection via {@link IColocatedVectorHost}.
+ *
+ * **Two storage modes** (per index, via `MJVectorIndex.ProviderConfig` → `CreateIndexParams.additionalParams`):
+ *  - `sibling` (default): MJ creates and owns a sibling table (`id`/`embedding`/`metadata`/`content`).
+ *    Generic, entity-agnostic, multi-model. Filters resolve against the JSON `metadata` column.
+ *  - `entityColumn`: vectors live on an existing entity table's `VECTOR` column. Filters resolve
+ *    against **live entity columns**, and results project real entity fields. This is the migration
+ *    target for systems that already store embeddings on their own tables (e.g. a `Content.Embedding`
+ *    column) — point MJ at the column instead of re-vectorizing.
+ *
+ * **Query path** prefers the DiskANN-aware `VECTOR_SEARCH` TVF (never `ORDER BY VECTOR_DISTANCE`,
+ * which doesn't engage the index). But that surface isn't in every 2025 build — verified live, boxed
+ * SQL Server 2025 RTM lacks it — so the provider lazily detects (per host) whether `VECTOR_SEARCH`
+ * works and **falls back to an exact `VECTOR_DISTANCE` scan** when it doesn't. Where the TVF exists,
+ * filtered queries dispatch on cardinality: small filtered sets use the exact path (DiskANN doesn't
+ * converge when the filter cluster is disjoint from the query neighborhood), large sets use DiskANN.
+ * See {@link sqlserverColocatedSQL} for the production-verified SQL shapes and the reasoning behind each.
+ *
+ * Validated end-to-end against a live SQL Server 2025 RTM container in sibling and entityColumn modes
+ * (exact fallback path). The approximate path on Azure SQL Database remains untested.
+ *
+ * Registered with the MJ class factory as `'SQLServerVectorDatabase'`.
+ */
+export declare class SQLServerVectorDatabase extends VectorDBBase {
+    constructor(apiKey: string);
+    get SupportsColocatedQuery(): boolean;
+    get SupportsHybridSearch(): boolean;
+    private get Host();
+    private get Schema();
+    private Qualify;
+    private Run;
+    private AssertVectorCapable;
+    private EnsureInfrastructure;
+    private GetResolvedIndex;
+    /** Validate & narrow the untyped `CreateIndexParams.additionalParams` bag into a typed config,
+     *  throwing on malformed shapes rather than asserting (`as`) a shape that may not hold. */
+    private NormalizeIndexConfig;
+    private ParseSelectColumns;
+    /** Resolve the schema-qualified table string for an entityColumn source spec, validating each
+     *  identifier segment (accepts `[schema].[table]`, `schema.table`, or bare `table`). */
+    private QualifyEntityTable;
+    ListIndexes(): Promise<IndexList>;
+    GetIndex(params: BaseRequestParams): Promise<BaseResponse>;
+    CreateIndex(params: CreateIndexParams): Promise<BaseResponse>;
+    private CreateSiblingIndex;
+    private RegisterEntityColumnIndex;
+    /** Attempt to create a DiskANN vector index; tolerate failure (preview surface / pre-existing). */
+    private TryCreateVectorIndex;
+    private UpsertRegistry;
+    DeleteIndex(params: BaseRequestParams): Promise<BaseResponse>;
+    EditIndex(_params: EditIndexParams): Promise<BaseResponse>;
+    /**
+     * Whether a given host server supports the `VECTOR_SEARCH` DiskANN TVF
+     * (`SELECT TOP (N) WITH APPROXIMATE`). Verified live: boxed **SQL Server 2025 RTM does NOT** —
+     * only `VECTOR_DISTANCE` (exact) ships there; the approximate/DiskANN surface is currently Azure
+     * SQL Database (and likely a later boxed CU). Detected lazily on first query.
+     *
+     * Keyed **per host**, not process-wide: one MJ process can hold connections to multiple SQL
+     * Servers (parallel client-to-many-servers, or mixed 2022/2025 backends), and capability differs
+     * per server — a process-global flag would let one server's probe poison another's path.
+     */
+    private static ApproximateSearchSupportedByHost;
+    ColocatedQuery(params: ColocatedQueryOptions, _contextUser?: UserInfo): Promise<ColocatedQueryResult>;
+    /**
+     * Execute the vector search, choosing the exact vs. approximate path and transparently falling
+     * back to exact `VECTOR_DISTANCE` when this server lacks the `VECTOR_SEARCH` TVF.
+     *
+     * Path selection:
+     *  - If the approximate surface is known-absent → always exact.
+     *  - Else for filtered queries, count the filter cardinality and use exact below the threshold
+     *    (DiskANN doesn't converge when the filter cluster is disjoint from the query neighborhood).
+     *  - Else attempt approximate; if it errors with a "feature not supported" signature, cache that
+     *    fact and retry exact.
+     */
+    private RunVectorQuery;
+    /** Best-effort heuristic: does this error indicate the VECTOR_SEARCH/DiskANN surface is absent on
+     *  this server? Matches on message text (brittle across CUs/locales), but only gates a fallback to
+     *  an equally-correct exact query, so a false negative just rethrows and a false positive is slow,
+     *  not wrong. */
+    private IsApproximateUnsupportedError;
+    /** Map a query result row to a ColocatedMatch, synthesizing metadata per storage mode. */
+    private RowToMatch;
+    private ToMetadataValue;
+    QueryIndex(params: QueryOptions, contextUser?: UserInfo): Promise<BaseResponse>;
+    CreateRecord(record: VectorRecord, indexName?: string): Promise<BaseResponse>;
+    CreateRecords(records: VectorRecord[], indexName?: string): Promise<BaseResponse>;
+    private UpsertRecords;
+    /** Extract the scalar key value from a vector record id (CompositeKey URL `Field|Value`). */
+    private ExtractKeyValue;
+    UpdateRecord(record: UpdateOptions): Promise<BaseResponse>;
+    UpdateRecords(records: UpdateOptions): Promise<BaseResponse>;
+    DeleteRecord(record: VectorRecord, indexName?: string): Promise<BaseResponse>;
+    DeleteRecords(records: VectorRecord[], indexName?: string): Promise<BaseResponse>;
+    private DeleteByKeys;
+    DeleteAllRecords(indexName: string, _namespace?: string): Promise<BaseResponse>;
+    GetRecord(params: BaseRequestParams): Promise<BaseResponse>;
+    GetRecords(params: BaseRequestParams): Promise<BaseResponse>;
+    ListVectorIDs(params: ListVectorIDsParams): Promise<ListVectorIDsResult>;
+    BuildMetadataFilter(options: SharedIndexFilterOptions): object | undefined;
+    private ParseMetadata;
+    private Success;
+    private Failure;
+}
+//# sourceMappingURL=SQLServerVectorDatabase.d.ts.map

package/dist/models/SQLServerVectorDatabase.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"SQLServerVectorDatabase.d.ts","sourceRoot":"","sources":["../../src/models/SQLServerVectorDatabase.ts"],"names":[],"mappings":"AACA,OAAO,EACH,iBAAiB,EACjB,YAAY,EAEZ,qBAAqB,EACrB,oBAAoB,EACpB,iBAAiB,EACjB,eAAe,EAGf,SAAS,EAET,mBAAmB,EACnB,mBAAmB,EACnB,YAAY,EAGZ,wBAAwB,EACxB,aAAa,EAEb,YAAY,EAEZ,YAAY,EACf,MAAM,6BAA6B,CAAC;AACrC,OAAO,EAAuB,QAAQ,EAAE,MAAM,sBAAsB,CAAC;AAwDrE;;;;;;;;;;;;;;;;;;;;;;;;;GAyBG;AACH,qBACa,uBAAwB,SAAQ,YAAY;gBACzC,MAAM,EAAE,MAAM;IAI1B,IAAoB,sBAAsB,IAAI,OAAO,CAEpD;IAGD,IAAoB,oBAAoB,IAAI,OAAO,CAElD;IAMD,OAAO,KAAK,IAAI,GAOf;IAED,OAAO,KAAK,MAAM,GAEjB;IAED,OAAO,CAAC,OAAO;IAIf,OAAO,CAAC,GAAG;YAQG,mBAAmB;YAYnB,oBAAoB;YAwBpB,gBAAgB;IAqC9B;+FAC2F;IAC3F,OAAO,CAAC,oBAAoB;IAuB5B,OAAO,CAAC,kBAAkB;IAe1B;4FACwF;IACxF,OAAO,CAAC,kBAAkB;IAab,WAAW,IAAI,OAAO,CAAC,SAAS,CAAC;IAmBjC,QAAQ,CAAC,MAAM,EAAE,iBAAiB,GAAG,OAAO,CAAC,YAAY,CAAC;IAoB1D,WAAW,CAAC,MAAM,EAAE,iBAAiB,GAAG,OAAO,CAAC,YAAY,CAAC;YAmB5D,kBAAkB;YAsBlB,yBAAyB;IAoBvC,mGAAmG;YACrF,oBAAoB;YAWpB,cAAc;IAgBf,WAAW,CAAC,MAAM,EAAE,iBAAiB,GAAG,OAAO,CAAC,YAAY,CAAC;IAgB7D,SAAS,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,YAAY,CAAC;IAQvE;;;;;;;;;OASG;IACH,OAAO,CAAC,MAAM,CAAC,gCAAgC,CAAgD;IAEzE,cAAc,CAAC,MAAM,EAAE,qBAAqB,EAAE,YAAY,CAAC,EAAE,QAAQ,GAAG,OAAO,CAAC,oBAAoB,CAAC;IAuB3H;;;;;;;;;;OAUG;YACW,cAAc;IAoC5B;;;qBAGiB;IACjB,OAAO,CAAC,6BAA6B;IAKrC,0FAA0F;IAC1F,OAAO,CAAC,UAAU;IAmClB,OAAO,CAAC,eAAe;IAUV,UAAU,CAAC,MAAM,EAAE,YAAY,EAAE,WAAW,CAAC,EAAE,QAAQ,GAAG,OAAO,CAAC,YAAY,CAAC;IAoC/E,YAAY,CAAC,MAAM,EAAE,YAAY,EAAE,SAAS,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,YAAY,CAAC;IAO7E,aAAa,CAAC,OAAO,EAAE,YAAY,EAAE,EAAE,SAAS,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,YAAY,CAAC;YAOhF,aAAa;IAoC3B,6FAA6F;IAC7F,OAAO,CAAC,eAAe;IAKV,YAAY,CAAC,MAAM,EAAE,aAAa,GAAG,OAAO,CAAC,YAAY,CAAC;IAW1D,aAAa,CAAC,OAAO,EAAE,aAAa,GAAG,OAAO,CAAC,YAAY,CAAC;IAQ5D,YAAY,CAAC,MAAM,EAAE,YAAY,EAAE,SAAS,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,YAAY,CAAC;IAO7E,aAAa,CAAC,OAAO,EAAE,YAAY,EAAE,EAAE,SAAS,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,YAAY,CAAC;YAOhF,YAAY;IAwBb,gBAAgB,CAAC,SAAS,EAAE,MAAM,EAAE,UAAU,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,YAAY,CAAC;IAmB/E,SAAS,CAAC,MAAM,EAAE,iBAAiB,GAAG,OAAO,CAAC,YAAY,CAAC;IA0B3D,UAAU,CAAC,MAAM,EAAE,iBAAiB,GAAG,OAAO,CAAC,YAAY,CAAC;IA6B5D,aAAa,CAAC,MAAM,EAAE,mBAAmB,GAAG,OAAO,CAAC,mBAAmB,CAAC;IAuBrE,mBAAmB,CAAC,OAAO,EAAE,wBAAwB,GAAG,MAAM,GAAG,SAAS;IAS1F,OAAO,CAAC,aAAa;IAcrB,OAAO,CAAC,OAAO;IAIf,OAAO,CAAC,OAAO;CAGlB"}