semanticdb-v2 2.0.0-alpha.5 → 2.0.0-alpha.7

package/README.md

@@ -1,9 +1,23 @@
 # SemanticDB
 
-SemanticDB translates its own LogicForm into SQL and executes it on a single core compute layer.
+SemanticDB translates its own LogicForm into SQL and executes it on one or more core compute layers.
 Callers provide schemas, schema mappings, operators, and database connection information.
 SemanticDB creates the matching provider and owns its connection lifecycle.
 
+Install SemanticDB together with only the database driver used by the application. Database
+drivers are optional peer dependencies and are loaded on the first execution for that connection;
+Redis cache support is included by the main package.
+
+```bash
+npm install semanticdb-v2 mysql2
+```
+
+Use `@clickhouse/client`, `@databricks/sql`, `dmdb`, `mysql2` (MySQL, Doris, StarRocks, and GBase),
+`mssql`, `pg`, `snowflake-sdk`, or `oracledb` for the corresponding connection kind. MaxCompute
+uses a caller-supplied parameterized execution client. If a configured driver is missing, execution
+fails with an installation command for that driver. SemanticDB caches its internally created
+providers across executions; call `await Logicform.close()` during application shutdown.
+
 ## Public API
 
 ```ts
@@ -50,6 +64,11 @@ const response = await Logicform.execute(
 );
 ```
 
+A successful response always includes `total`, the number of result rows before global
+`limit`/`page` pagination. For grouped LogicForms this is the number of groups after `having` and
+`limitBy`. Paginated physical queries include a separate `sqls` entry with `type: 'total'` for the
+count query; unpaginated and executor-merged results derive the value without another SQL query.
+
 Every executable LogicForm, including LogicForm objects used as subqueries, must declare `version`. This
 SDK accepts semantic versions greater than or equal to `2.0`; missing, malformed, or older versions
 fail with `UNSUPPORTED_LOGICFORM_VERSION` before Schema resolution or database access. A relation
@@ -100,12 +119,18 @@ Plugins may own Schema metadata, legacy conversion, `entity_id` selection, and O
 remaining scoped to one execution. See [Register an external plugin](docs/register-plugin.md).
 
 Parent-child Schemas may declare `parentProperty`. `$recursive` filters a property to a root node and
-all descendants; object properties infer the tree Schema from `ref`. MySQL, Doris, StarRocks, and
-PostgreSQL compile this as a parameterized recursive CTE.
+all descendants; object properties infer the tree Schema from `ref`. MySQL, Doris, StarRocks,
+PostgreSQL, ClickHouse 24.4+, and Databricks compile this as a parameterized recursive CTE.
 
 Top-level projections may set `schema` to read metrics from another registered Schema. SemanticDB
 splits those projections into independent provider queries, aligns shared query/groupby dimensions,
 and merges rows by the group keys before applying `having`, `sort`, `limitBy`, and pagination.
+When the selected Schema mappings use separate physical connections, SemanticDB executes every
+single-connection fragment with its native provider and registers the aggregated fragment results
+in an in-process DuckDB coordinator. DuckDB applies the final full-outer group merge, cross-fragment
+conditions, ordering, totals, and pagination. It is embedded in the Node.js process and does not
+require a service or listening port. `federation.maxMaterializedRows` limits the total number of
+fragment rows admitted to the coordinator and defaults to 100,000.
 
 The progress stream emits `started`, `validated`, `normalized`, `planned`, `lineage`,
 `compiled`, `executing`, `hydrating`, `result`, and `completed`. Every event includes a cumulative
@@ -118,7 +143,9 @@ SQL string for diagnostics:
 
 ```ts
 interface ExecutedSql {
-  type: 'semantic-resolution' | 'computation';
+  type: 'semantic-resolution' | 'computation' | 'total';
+  connectionId?: string;
+  provider?: CoreComputeProvider['kind'];
   explain: string;
   logicform: LogicformType;
   sql: string;
@@ -131,11 +158,123 @@ interface ExecutedSql {
 time-watermark resolution and object entity population. `computation` identifies SQL that calculates
 the requested LogicForm result. `sqls` includes every physical SQL involved in the execution; an
 entry remains visible when its result is served by the query cache.
+Coordinator SQL uses the reserved connection id `__semanticdb_coordinator__` and provider
+`duckdb`; other entries identify the physical connection that executed them. Successful results
+also expose `providers`, the physical connection/provider pairs that contributed rows. The legacy
+singular `provider` field is `duckdb` for a federated result.
 
 Providers execute only parameterized `sql + parameters`; SemanticDB never executes `displaySql`.
 Because `displaySql` contains literal values and may expose sensitive data, use it only in trusted
 debugging and observability channels.
 
+### Native SQL and table discovery
+
+`DBConnector` provides trusted physical database operations independently of LogicForm execution:
+
+```ts
+import { DBConnector, Logicform } from 'semanticdb-v2';
+import type { DatabaseConnection } from 'semanticdb-v2';
+
+const connection = {
+  id: 'analytics',
+  kind: 'mysql',
+  host: 'localhost',
+  user: 'analytics',
+  password: process.env.MYSQL_PASSWORD ?? '',
+  database: 'analytics',
+} satisfies DatabaseConnection;
+
+const tables = await DBConnector.getTableList({ connection });
+// [{ name: 'orders', database: 'analytics', type: 'table' }, ...]
+
+const description = await DBConnector.describeTable({ connection, table: 'orders' });
+// {
+//   columns: [{ name: 'id', type: 'int', comment: 'Order ID', primaryKey: true, exampleValue: 42 }, ...],
+//   tableInfo: { name: 'orders', database: 'analytics', type: 'table', comment: 'Orders' },
+// }
+
+const { rows } = await DBConnector.runSQL({
+  connection,
+  sql: 'SELECT * FROM orders WHERE status = ?',
+  parameters: ['paid'],
+});
+
+const updated = await DBConnector.runSQL({
+  connection,
+  sql: 'UPDATE orders SET status = ? WHERE id = ?',
+  parameters: ['shipped', 42],
+});
+// { rows: [], affectedRows: 1 }
+
+// The exact same object belongs in Logicform.execute's connections array:
+// await Logicform.execute(logicform, { connections: [connection], schemas, schemaMappings });
+
+await Logicform.close(); // DBConnector.close() is the same shutdown operation.
+```
+
+Both APIs share providers and connection pools by connection ID and `providerRegistry`. An ID must
+retain the same configuration until shutdown. Providers and pools are initialized lazily. Either
+`close()` releases **all** SDK-managed connections, including those used by the other API; it does
+not close just the last connector connection.
+
+`runSQL` returns `{ rows: DataRow[], affectedRows?: number }`. Commands without a result set return
+an empty array; affected-row counts are included only when the driver reports them. SQL is passed
+through without automatic LIMIT, dialect translation, Schema mapping, hydration, generated `_id`,
+or query caching. It does **not** enforce `ExecuteOptions.permissions`: use trusted callers and
+appropriately restricted database accounts. Supply values through `parameters` using the native
+provider placeholders: `?` for MySQL-compatible databases, Databricks, Dameng, Snowflake and MaxCompute;
+`$1` for PostgreSQL; `@p0` for SQL Server; `:1` for Oracle; typed `{p0:String}`-style
+placeholders for ClickHouse. ClickHouse uses JSONEachRow output; do not override it with another
+FORMAT. Each call is independent, with Oracle/Dameng command execution automatically committed;
+cross-call transactions and multi-statement/multi-result execution are not supported.
+
+`getTableList` returns `{ name, database?, schema?, type: 'table' | 'view' }[]` for accessible tables
+and views, retaining namespaces to distinguish same-name tables. Optional `database` and `schema`
+override only this metadata lookup, never the cached connection configuration:
+
+- MySQL, Doris, StarRocks, GBase and ClickHouse: `database`; `schema` is rejected.
+- PostgreSQL, Oracle and Dameng: `schema`; cross-database overrides are rejected. Defaults to the
+  current schema (or the configured Dameng schema).
+- SQL Server and Snowflake: `database` and `schema`. SQL Server lists all accessible schemas in
+  the database by default; Snowflake defaults to the current schema.
+- Databricks: `database` means catalog, and `schema` means schema. Defaults come from the connection's
+  `catalog` and `schema` (`connection.database` remains the existing legacy alias for schema).
+- MaxCompute: the existing injected client handles `SHOW TABLES`, returning rows with `table_name`.
+  It may implement `getTableList(request)` for project/schema overrides and view metadata, and
+  `runSQL(request)` for affected-row counts. Otherwise commands must return `[]` through `execute`.
+
+`describeTable({ connection, table, database?, schema?, includeSample? })` inspects one physical
+table or view. `table` is a literal **unqualified** name; provide its namespace separately, using
+the same scope rules as `getTableList`. A dot inside `table` is part of the identifier, not a separator.
+Do not pass a SQL expression or the legacy semantic Schema/`from` object. SQL Server descriptions
+default to the connection's current default schema; pass `schema` explicitly when describing a
+table returned from another schema. Column order follows the database's definition.
+
+The result has `columns: { name, type, comment, primaryKey?, exampleValue? }[]` and
+`tableInfo: { name, database?, schema?, type, comment }`. These are physical names/native types,
+not semantic Properties; no `_id` is added. Comments default to an empty string. Primary-key
+membership includes every column of composite keys; unique-only indexes are not primary keys.
+ClickHouse reports its storage primary key, which does not imply relational uniqueness. If the
+adapter cannot determine primary-key membership (Databricks/Snowflake), `primaryKey` is omitted
+rather than guessed. Example values preserve driver types, including `null`.
+
+Like the legacy API, descriptions read one example row by default. Set `includeSample: false`
+for metadata-only inspection, avoiding a data read. Empty tables and failed sample queries still
+return metadata without example values; metadata failures, tables not found in the selected namespace, and
+cancellation reject. All SQL uses bound metadata values and quoted identifiers, bypassing
+LogicForm permissions and caching just like `runSQL`.
+
+MaxCompute clients may implement `describeTable(request)` to provide project/schema-aware
+metadata. Otherwise the fallback runs `DESC` and expects structured rows with `column_name`,
+`column_type`, optional `column_comment` and `is_primary_key`. Legacy UDBC ASCII `Info` reports
+must be normalized by the client; unparsed reports are rejected rather than producing empty columns.
+
+All operations accept `signal` and the same optional `providerRegistry` as `Logicform.execute`.
+Custom providers opt in through `runSQL(request)`, `getTableList(request)` and `describeTable(request)`; missing capabilities
+fail explicitly without changing the existing provider `execute` contract. Unsupported discovery
+scopes are rejected rather than silently ignored. Cancellation does not guarantee rollback of
+commands that have already reached the database.
+
 ### Permissions
 
 Pass table, row, and column permissions together through `ExecuteOptions.permissions`:
@@ -200,10 +339,18 @@ names shared by both sides unless an exceptional custom operator explicitly supp
 See [Register an operator](docs/register-operator.md) for the component contract.
 
 The root entry point accepts a `connections` array of `DatabaseConnection` items. Every item has a unique `id`; set `kind` to `mysql`,
-`doris`, `starrocks`, `clickhouse`, `snowflake`, `oracle`, or `postgresql`, then add that driver's
+`doris`, `starrocks`, `gbase`, `maxcompute`, `clickhouse`, `databricks`, `dameng`, `snowflake`,
+`sqlserver`, `oracle`, or `postgresql`, then add that driver's
 native connection fields directly on the item. Each Schema Mapping selects an item through
 `connectionId`. SemanticDB lazily creates and reuses the matching internal provider. Call
 `await Logicform.close()` during application shutdown to close every connection pool and session.
+One LogicForm may reference mappings on multiple connections. This is one logical execution, not
+one cross-database SQL statement: physical fragments run independently and their aggregate results
+are combined by the embedded coordinator. A relation join whose two sides themselves live on
+different connections is not yet materialized automatically. When the related Schema has replicas
+on multiple connections, the Planner selects the mapping colocated with the main mapping and emits
+the `LEFT JOIN` on that connection. It fails explicitly only when no target mapping exists on the
+main connection.
 
 MySQL-compatible providers default to 30 connections, at most 10 idle connections, a 60-second
 idle timeout, and TCP keepalive. Native `mysql2` pool options on the connection item override these
@@ -239,6 +386,66 @@ pagination, date, aggregate, string, regex, parameter, and display-SQL behavior
 MySQL-compatible SQL helper. Database-specific differences should be implemented by overriding the
 corresponding internal dialect capability.
 
+### GBase connection
+
+GBase 8a uses its MySQL-compatible protocol through `mysql2`. It defaults to port `5258`; all
+ordinary `mysql2` connection and pool fields remain available:
+
+```ts
+const gbase = {
+  id: 'gbase-main',
+  kind: 'gbase',
+  host: process.env.GBASE_HOST,
+  user: process.env.GBASE_USER,
+  password: process.env.GBASE_PASSWORD,
+  database: process.env.GBASE_DATABASE,
+} as const;
+```
+
+### MaxCompute connection
+
+MaxCompute requires a caller-owned client that preserves the Provider's parameterized execution
+contract. The client receives SQL and parameters separately and owns the actual MaxCompute/JDBC
+transport:
+
+```ts
+const maxcompute = {
+  id: 'maxcompute-main',
+  kind: 'maxcompute',
+  client: {
+    execute: ({ sql, parameters, signal }) => maxComputeGateway.execute({
+      sql, parameters, signal,
+    }),
+    close: () => maxComputeGateway.close(),
+  },
+} as const;
+```
+
+The legacy UDBC `Statement` endpoint accepts only a complete SQL string, so SemanticDB does not
+use it or interpolate values locally. A gateway implementation must bind `parameters` before this
+connection can execute queries.
+
+### SQL Server connection
+
+SQL Server uses `mssql`, bracket-quoted identifiers, and named `@p0`, `@p1` parameters:
+
+```ts
+const sqlserver = {
+  id: 'sqlserver-main',
+  kind: 'sqlserver',
+  host: process.env.SQLSERVER_HOST,
+  port: 1433,
+  user: process.env.SQLSERVER_USER,
+  password: process.env.SQLSERVER_PASSWORD,
+  database: process.env.SQLSERVER_DATABASE,
+  options: { encrypt: true },
+} as const;
+```
+
+Pagination uses `OFFSET ... FETCH`; SemanticDB supplies a neutral top-level ordering when the
+LogicForm has no sort. JSON array relations use `OPENJSON`. SQL Server has no native regular
+expression predicate, so `$regex` is rejected explicitly.
+
 ### Snowflake connection
 
 Snowflake connection fields are passed to the official `snowflake-sdk`:
@@ -261,6 +468,52 @@ await Logicform.execute(logicform, {
 });
 ```
 
+### Databricks connection
+
+Databricks uses `@databricks/sql` and supports a personal access token or OAuth M2M. `catalog` and
+`schema` select the initial SQL Warehouse session namespace (`database` is accepted as a legacy
+alias for `schema`):
+
+```ts
+const databricks = {
+  id: 'databricks-main',
+  kind: 'databricks',
+  host: process.env.DATABRICKS_SERVER_HOSTNAME,
+  path: process.env.DATABRICKS_HTTP_PATH,
+  authType: 'databricks-oauth',
+  oauthClientId: process.env.DATABRICKS_CLIENT_ID,
+  oauthClientSecret: process.env.DATABRICKS_CLIENT_SECRET,
+  catalog: 'main',
+  schema: 'analytics',
+} as const;
+```
+
+Use `authType: 'access-token'` with `token` for PAT authentication. Values are sent through the
+driver's ordinal parameter binding and are not interpolated into executable SQL. `authType` is
+required: PAT and OAuth fields are mutually exclusive, and authentication is never inferred from
+the presence of credentials.
+
+### Dameng connection
+
+Dameng uses the official `dmdb` pool and the `dameng` connection kind:
+
+```ts
+const dameng = {
+  id: 'dameng-main',
+  kind: 'dameng',
+  host: process.env.DAMENG_HOST,
+  port: 5236,
+  user: process.env.DAMENG_USER,
+  password: process.env.DAMENG_PASSWORD,
+  schema: process.env.DAMENG_SCHEMA,
+} as const;
+```
+
+The provider uses positional binds and object-format result rows. Recursive hierarchy queries and
+JSON-array relations are rejected until their behavior can be verified against supported DM server
+versions; scalar queries, grouping, date operations, regex, windows, joins, and pagination compile
+with the DM/Oracle-style dialect.
+
 ### Oracle connection
 
 Oracle uses the official `oracledb` driver in Thin mode and does not require Oracle Client
@@ -331,9 +584,20 @@ comments, or parameter placeholders. Property UDFs do not support the legacy `fu
 
 All compiled provider queries, including the internal time-watermark query, can share one
 caller-owned query cache. Redis is published from `semanticdb-v2/cache/redis`; cache identity
-includes namespace, provider, SQL, and parameters. Physical expiration belongs to each Schema
+includes namespace, connection id, provider, SQL, and parameters. Physical expiration belongs to each Schema
 mapping's `freshness` policy. See [Query cache and Redis](docs/query-cache.md).
 
+Schema mappings can also expose same-connection entity feature tables as lazy physical
+enrichments. A mapped feature property adds a `LEFT JOIN` only when referenced, and a point-in-time
+feature can be assigned to the containing valid-time entity row. See
+[Schema Mapping feature enrichments](docs/schema-mapping-enrichments.md).
+
+For aggregate sources, set `granularity` on mapped date or hierarchy Properties.
+Other available dimensions are inferred from their Property mappings, so there is no
+separate Schema Mapping `dimensions` field. The planner selects a source that covers
+the requested properties and physical time or hierarchy grain. See
+[Schema Mapping physical granularity](docs/schema-mapping-granularity.md).
+
 When Redis is shared with other applications, pass an application-specific key prefix:
 
 ```ts
@@ -455,7 +719,8 @@ npm run debug
 
 ## Current scope
 
-- ClickHouse, MySQL, Oracle, and Snowflake core compute providers
+- ClickHouse, Databricks, Dameng, Doris, GBase, MaxCompute, MySQL, Oracle, PostgreSQL, Snowflake,
+  SQL Server, and StarRocks core compute providers
 - Projections, conditional aggregates, and `$sum`, `$count`, `$countDistinct`, `$avg`, `$min`, `$max`
 - Structured arithmetic, share-of-total, period comparison, period-boundary, date-bucket,
   cumulative period, trusted SQL expression, string, case, and window operators

package/dist/cache/redis-query-cache.esm.js

@@ -1 +1 @@
-import{createClient as b}from"redis";import{createHash as g}from"node:crypto";const a=e=>e===void 0?{type:"undefined"}:typeof e=="bigint"?{type:"bigint",value:e.toString()}:typeof e=="number"&&!Number.isFinite(e)?{type:"number",value:String(e)}:e instanceof Date?{type:"date",value:e.toISOString()}:Array.isArray(e)?e.map(a):e&&typeof e=="object"?Object.fromEntries(Object.entries(e).sort(([t],[r])=>t.localeCompare(r)).map(([t,r])=>[t,a(r)])):e,d=e=>g("sha256").update(JSON.stringify(a(e))).digest("hex"),A=e=>d({namespace:e.namespace,provider:e.provider,sql:e.sql,parameters:e.parameters}),S=(e,t)=>d({namespace:e,source:t}),o=e=>e===void 0?{type:"undefined"}:typeof e=="bigint"?{type:"bigint",value:e.toString()}:typeof e=="number"&&!Number.isFinite(e)?{type:"number",value:String(e)}:e instanceof Date?{type:"date",value:e.toISOString()}:Array.isArray(e)?{type:"array",value:e.map(o)}:e&&typeof e=="object"?{type:"object",value:Object.entries(e).map(([t,r])=>[t,o(r)])}:e,y=e=>{if(e===null||typeof e!="object")return e;switch(e.type){case"undefined":return;case"bigint":return BigInt(e.value);case"number":return Number(e.value);case"date":return new Date(e.value);case"array":return e.value.map(y);case"object":return Object.fromEntries(e.value.map(([t,r])=>[t,y(r)]))}},w=e=>JSON.stringify(o(e)),x=e=>{const t=y(JSON.parse(e));if(!(!Array.isArray(t)||t.some(r=>!r||typeof r!="object"||Array.isArray(r))))return t},l=Number.MAX_SAFE_INTEGER;class p{#e;#t;#r;constructor(t,r={}){this.#e=t,this.#t=r.keyPrefix??"semanticdb:cache",this.#r=r.ownsClient??!1}#n(t){return`${this.#t}:query:${A(t)}`}#i(t,r){return`${this.#t}:source:${S(t,r)}`}async get(t){const r=await this.#e.get(this.#n(t));if(r!==null)return x(r)}async set(t,r,i){const n=this.#n(t),f=w(r),c=i.expiresAt?.getTime(),m=c??l,s=this.#e.multi();c===void 0?s.set(n,f):s.set(n,f,{PXAT:c});for(const h of new Set(i.sources)){const u=this.#i(t.namespace,h);s.zRemRangeByScore(u,0,Date.now()),s.zAdd(u,{score:m,value:n})}await s.exec()}async invalidateSource(t,r){const i=this.#i(t,r),n=await this.#e.zRange(i,0,-1);n.length>0&&await this.#e.del(n),await this.#e.del(i)}async close(){this.#r&&await this.#e.close()}}const j=async(e={})=>{const t=b(e.url?{url:e.url}:{});return t.on("error",()=>{}),await t.connect(),new p(t,{...e.keyPrefix?{keyPrefix:e.keyPrefix}:{},ownsClient:!0})};export{p as RedisQueryCache,j as createRedisQueryCache};
+import{createClient as b}from"redis";import{createHash as g}from"node:crypto";const o=e=>e===void 0?{type:"undefined"}:typeof e=="bigint"?{type:"bigint",value:e.toString()}:typeof e=="number"&&!Number.isFinite(e)?{type:"number",value:String(e)}:e instanceof Date?{type:"date",value:e.toISOString()}:Array.isArray(e)?e.map(o):e&&typeof e=="object"?Object.fromEntries(Object.entries(e).sort(([t],[r])=>t.localeCompare(r)).map(([t,r])=>[t,o(r)])):e,d=e=>g("sha256").update(JSON.stringify(o(e))).digest("hex"),A=e=>d({namespace:e.namespace,provider:e.provider,connectionId:e.connectionId,sql:e.sql,parameters:e.parameters}),S=(e,t)=>d({namespace:e,source:t}),a=e=>e===void 0?{type:"undefined"}:typeof e=="bigint"?{type:"bigint",value:e.toString()}:typeof e=="number"&&!Number.isFinite(e)?{type:"number",value:String(e)}:e instanceof Date?{type:"date",value:e.toISOString()}:Array.isArray(e)?{type:"array",value:e.map(a)}:e&&typeof e=="object"?{type:"object",value:Object.entries(e).map(([t,r])=>[t,a(r)])}:e,f=e=>{if(e===null||typeof e!="object")return e;switch(e.type){case"undefined":return;case"bigint":return BigInt(e.value);case"number":return Number(e.value);case"date":return new Date(e.value);case"array":return e.value.map(f);case"object":return Object.fromEntries(e.value.map(([t,r])=>[t,f(r)]))}},w=e=>JSON.stringify(a(e)),x=e=>{const t=f(JSON.parse(e));if(!(!Array.isArray(t)||t.some(r=>!r||typeof r!="object"||Array.isArray(r))))return t},l=Number.MAX_SAFE_INTEGER;class p{#e;#t;#r;constructor(t,r={}){this.#e=t,this.#t=r.keyPrefix??"semanticdb:cache",this.#r=r.ownsClient??!1}#n(t){return`${this.#t}:query:${A(t)}`}#i(t,r){return`${this.#t}:source:${S(t,r)}`}async get(t){const r=await this.#e.get(this.#n(t));if(r!==null)return x(r)}async set(t,r,i){const n=this.#n(t),y=w(r),c=i.expiresAt?.getTime(),m=c??l,s=this.#e.multi();c===void 0?s.set(n,y):s.set(n,y,{PXAT:c});for(const h of new Set(i.sources)){const u=this.#i(t.namespace,h);s.zRemRangeByScore(u,0,Date.now()),s.zAdd(u,{score:m,value:n})}await s.exec()}async invalidateSource(t,r){const i=this.#i(t,r),n=await this.#e.zRange(i,0,-1);n.length>0&&await this.#e.del(n),await this.#e.del(i)}async close(){this.#r&&await this.#e.close()}}const j=async(e={})=>{const t=b(e.url?{url:e.url}:{});return t.on("error",()=>{}),await t.connect(),new p(t,{...e.keyPrefix?{keyPrefix:e.keyPrefix}:{},ownsClient:!0})};export{p as RedisQueryCache,j as createRedisQueryCache};

package/dist/cache/redis-query-cache.js

@@ -1 +1 @@
-"use strict";var b=require("redis"),g=require("node:crypto");const a=e=>e===void 0?{type:"undefined"}:typeof e=="bigint"?{type:"bigint",value:e.toString()}:typeof e=="number"&&!Number.isFinite(e)?{type:"number",value:String(e)}:e instanceof Date?{type:"date",value:e.toISOString()}:Array.isArray(e)?e.map(a):e&&typeof e=="object"?Object.fromEntries(Object.entries(e).sort(([t],[r])=>t.localeCompare(r)).map(([t,r])=>[t,a(r)])):e,d=e=>g.createHash("sha256").update(JSON.stringify(a(e))).digest("hex"),A=e=>d({namespace:e.namespace,provider:e.provider,sql:e.sql,parameters:e.parameters}),S=(e,t)=>d({namespace:e,source:t}),o=e=>e===void 0?{type:"undefined"}:typeof e=="bigint"?{type:"bigint",value:e.toString()}:typeof e=="number"&&!Number.isFinite(e)?{type:"number",value:String(e)}:e instanceof Date?{type:"date",value:e.toISOString()}:Array.isArray(e)?{type:"array",value:e.map(o)}:e&&typeof e=="object"?{type:"object",value:Object.entries(e).map(([t,r])=>[t,o(r)])}:e,y=e=>{if(e===null||typeof e!="object")return e;switch(e.type){case"undefined":return;case"bigint":return BigInt(e.value);case"number":return Number(e.value);case"date":return new Date(e.value);case"array":return e.value.map(y);case"object":return Object.fromEntries(e.value.map(([t,r])=>[t,y(r)]))}},w=e=>JSON.stringify(o(e)),x=e=>{const t=y(JSON.parse(e));if(!(!Array.isArray(t)||t.some(r=>!r||typeof r!="object"||Array.isArray(r))))return t},l=Number.MAX_SAFE_INTEGER;class p{#e;#t;#r;constructor(t,r={}){this.#e=t,this.#t=r.keyPrefix??"semanticdb:cache",this.#r=r.ownsClient??!1}#n(t){return`${this.#t}:query:${A(t)}`}#i(t,r){return`${this.#t}:source:${S(t,r)}`}async get(t){const r=await this.#e.get(this.#n(t));if(r!==null)return x(r)}async set(t,r,i){const n=this.#n(t),u=w(r),c=i.expiresAt?.getTime(),h=c??l,s=this.#e.multi();c===void 0?s.set(n,u):s.set(n,u,{PXAT:c});for(const m of new Set(i.sources)){const f=this.#i(t.namespace,m);s.zRemRangeByScore(f,0,Date.now()),s.zAdd(f,{score:h,value:n})}await s.exec()}async invalidateSource(t,r){const i=this.#i(t,r),n=await this.#e.zRange(i,0,-1);n.length>0&&await this.#e.del(n),await this.#e.del(i)}async close(){this.#r&&await this.#e.close()}}const C=async(e={})=>{const t=b.createClient(e.url?{url:e.url}:{});return t.on("error",()=>{}),await t.connect(),new p(t,{...e.keyPrefix?{keyPrefix:e.keyPrefix}:{},ownsClient:!0})};exports.RedisQueryCache=p,exports.createRedisQueryCache=C;
+"use strict";var b=require("redis"),g=require("node:crypto");const a=e=>e===void 0?{type:"undefined"}:typeof e=="bigint"?{type:"bigint",value:e.toString()}:typeof e=="number"&&!Number.isFinite(e)?{type:"number",value:String(e)}:e instanceof Date?{type:"date",value:e.toISOString()}:Array.isArray(e)?e.map(a):e&&typeof e=="object"?Object.fromEntries(Object.entries(e).sort(([t],[r])=>t.localeCompare(r)).map(([t,r])=>[t,a(r)])):e,d=e=>g.createHash("sha256").update(JSON.stringify(a(e))).digest("hex"),A=e=>d({namespace:e.namespace,provider:e.provider,connectionId:e.connectionId,sql:e.sql,parameters:e.parameters}),S=(e,t)=>d({namespace:e,source:t}),o=e=>e===void 0?{type:"undefined"}:typeof e=="bigint"?{type:"bigint",value:e.toString()}:typeof e=="number"&&!Number.isFinite(e)?{type:"number",value:String(e)}:e instanceof Date?{type:"date",value:e.toISOString()}:Array.isArray(e)?{type:"array",value:e.map(o)}:e&&typeof e=="object"?{type:"object",value:Object.entries(e).map(([t,r])=>[t,o(r)])}:e,u=e=>{if(e===null||typeof e!="object")return e;switch(e.type){case"undefined":return;case"bigint":return BigInt(e.value);case"number":return Number(e.value);case"date":return new Date(e.value);case"array":return e.value.map(u);case"object":return Object.fromEntries(e.value.map(([t,r])=>[t,u(r)]))}},w=e=>JSON.stringify(o(e)),x=e=>{const t=u(JSON.parse(e));if(!(!Array.isArray(t)||t.some(r=>!r||typeof r!="object"||Array.isArray(r))))return t},l=Number.MAX_SAFE_INTEGER;class p{#e;#t;#r;constructor(t,r={}){this.#e=t,this.#t=r.keyPrefix??"semanticdb:cache",this.#r=r.ownsClient??!1}#n(t){return`${this.#t}:query:${A(t)}`}#i(t,r){return`${this.#t}:source:${S(t,r)}`}async get(t){const r=await this.#e.get(this.#n(t));if(r!==null)return x(r)}async set(t,r,i){const n=this.#n(t),y=w(r),c=i.expiresAt?.getTime(),h=c??l,s=this.#e.multi();c===void 0?s.set(n,y):s.set(n,y,{PXAT:c});for(const m of new Set(i.sources)){const f=this.#i(t.namespace,m);s.zRemRangeByScore(f,0,Date.now()),s.zAdd(f,{score:h,value:n})}await s.exec()}async invalidateSource(t,r){const i=this.#i(t,r),n=await this.#e.zRange(i,0,-1);n.length>0&&await this.#e.del(n),await this.#e.del(i)}async close(){this.#r&&await this.#e.close()}}const C=async(e={})=>{const t=b.createClient(e.url?{url:e.url}:{});return t.on("error",()=>{}),await t.connect(),new p(t,{...e.keyPrefix?{keyPrefix:e.keyPrefix}:{},ownsClient:!0})};exports.RedisQueryCache=p,exports.createRedisQueryCache=C;

package/dist/cache/types.d.ts

@@ -2,6 +2,8 @@ import type { CoreComputeProvider, DataRow, SqlParameters } from '../physical/in
 export interface QueryCacheIdentity {
     readonly namespace: string;
     readonly provider: CoreComputeProvider['kind'];
+    /** Distinguishes separate databases using the same provider dialect. */
+    readonly connectionId?: string;
     readonly sql: string;
     readonly parameters: SqlParameters;
 }

package/dist/executor/connection-registry.d.ts

@@ -1,12 +1 @@
-import type { CoreComputeProvider, DatabaseConnection } from '../physical/index';
-type ProviderFactory = (connection: DatabaseConnection) => CoreComputeProvider;
-export declare class ConnectionRegistry {
-    #private;
-    private readonly create;
-    constructor(create?: ProviderFactory);
-    get(connection: DatabaseConnection): CoreComputeProvider;
-    close(): Promise<void>;
-}
-export declare const getConnectionProvider: (connection: DatabaseConnection) => CoreComputeProvider;
-export declare const closeConnections: () => Promise<void>;
-export {};
+export { ConnectionRegistry, getConnectionProvider, closeConnections } from '../physical/index';

package/dist/executor/execute-query.d.ts

@@ -1,13 +1,18 @@
 import type { CompiledQuery } from '../compiler/index';
 import type { DataRow } from '../physical/index';
 import type { LogicalPlan } from '../planner/index';
+import type { SchemaMappingDefinition, SchemaMappingRegistry } from '../schema-mappings/index';
 import type { InternalExecuteOptions } from './types';
 export interface PhysicalQuerySource {
     readonly schemaId: string;
     readonly source: string;
+    readonly connectionId?: string;
+    /** Base mapping source that owns an enrichment source. */
+    readonly mappingSource?: string;
 }
+export declare const resolvePhysicalSourceMapping: (source: PhysicalQuerySource, mappings: SchemaMappingRegistry) => SchemaMappingDefinition | undefined;
 export declare const collectPlanSources: (plan: LogicalPlan) => readonly PhysicalQuerySource[];
 export declare const mergePhysicalSources: (groups: readonly (readonly PhysicalQuerySource[])[]) => readonly PhysicalQuerySource[];
-export declare const executeCompiledQuery: (compiled: CompiledQuery, sources: readonly PhysicalQuerySource[], options: Pick<InternalExecuteOptions, "cache" | "provider" | "schemaMappings" | "signal">) => Promise<DataRow[]>;
+export declare const executeCompiledQuery: (compiled: CompiledQuery, sources: readonly PhysicalQuerySource[], options: Pick<InternalExecuteOptions, "cache" | "provider" | "schemaMappings" | "signal" | "connectionId">) => Promise<DataRow[]>;
 /** Apply final SQL shaping after all plans, views, and derived sources are compiled. */
 export declare const shortenCompiledQuery: (compiled: CompiledQuery) => CompiledQuery;

package/dist/executor/federated-execution.d.ts

@@ -0,0 +1,37 @@
+import type { CompiledQuery } from '../compiler/index';
+import type { LogicformType } from '../logicform-types';
+import type { OperatorRegistry } from '../operators/index';
+import type { CoreComputeProvider, DataRow } from '../physical/index';
+import type { LogicalPlan } from '../planner/index';
+import type { SchemaMappingRegistry } from '../schema-mappings/index';
+import type { InternalExecuteOptions } from './types';
+export declare const coordinatorConnectionId = "__semanticdb_coordinator__";
+export interface FederatedCompiledQuery {
+    readonly compiled: CompiledQuery;
+    readonly connectionId: string;
+    readonly provider: CoreComputeProvider;
+    readonly type: 'computation' | 'total';
+    readonly logicform: LogicformType;
+}
+interface FederatedExecutionOptions {
+    readonly logicform: LogicformType;
+    readonly operators: OperatorRegistry;
+    readonly schemaMappings: SchemaMappingRegistry;
+    readonly providerForConnection: (connectionId: string) => CoreComputeProvider;
+    readonly cache?: InternalExecuteOptions['cache'];
+    readonly signal?: AbortSignal;
+    readonly maxMaterializedRows: number;
+    readonly onCompiled: (query: FederatedCompiledQuery) => void;
+}
+export interface FederatedExecutionResult {
+    readonly rows: DataRow[];
+    readonly total: number;
+    readonly connections: readonly {
+        readonly connectionId: string;
+        readonly provider: CoreComputeProvider['kind'];
+    }[];
+}
+export declare const planConnectionIds: (plan: LogicalPlan, schemaMappings: SchemaMappingRegistry) => readonly string[];
+export declare const isFederatedComposition: (plan: LogicalPlan, schemaMappings: SchemaMappingRegistry) => boolean;
+export declare const executeFederatedComposition: (plan: LogicalPlan, options: FederatedExecutionOptions) => Promise<FederatedExecutionResult>;
+export {};

package/dist/executor/fetch-entities.d.ts

@@ -21,9 +21,11 @@ interface FetchEntityOptions {
     readonly resolveSchema: (schemaId: string) => ResolvedSemanticSchema;
     readonly permissions?: Permissions;
     readonly provider: CoreComputeProvider;
+    readonly providerForConnection?: (connectionId: string) => CoreComputeProvider;
+    readonly preferredConnectionId?: string;
     readonly cache?: QueryCacheExecutionOptions;
     readonly signal?: AbortSignal;
-    readonly onCompiled?: (compiled: CompiledQuery, logicform: LogicformType) => void;
+    readonly onCompiled?: (compiled: CompiledQuery, logicform: LogicformType, provider: CoreComputeProvider, connectionId?: string) => void;
 }
 export declare const fetchEntities: (request: EntityFetchRequest, options: FetchEntityOptions) => Promise<DataRow[]>;
 export {};

package/dist/executor/result-hydrator/types.d.ts

@@ -1,6 +1,7 @@
 import type { ResultProperty } from '../../operators/index';
 import type { DataRow } from '../../physical/index';
 import type { ResolvedSemanticProperty, ResolvedSemanticSchema, SemanticTypeRegistry } from '../../semantic-types/index';
+import type { SemanticDBPlugin } from '../../plugins/index';
 export type HydrationProperty = Pick<ResultProperty, 'isArray' | 'name' | 'primal_type' | 'ref' | 'type'> | ResolvedSemanticProperty;
 export interface EntityFetchRequest {
     readonly schema: ResolvedSemanticSchema;
@@ -11,4 +12,5 @@ export interface ResultHydratorOptions {
     readonly semanticTypes: SemanticTypeRegistry;
     readonly resolveSchema: (schemaId: string) => ResolvedSemanticSchema;
     readonly fetchEntities: (request: EntityFetchRequest) => Promise<readonly DataRow[]>;
+    readonly plugins?: readonly SemanticDBPlugin[];
 }

package/dist/executor/total.d.ts

@@ -0,0 +1,11 @@
+import type { CompiledQuery } from '../compiler/index';
+import type { CoreComputeProvider, DataRow } from '../physical/index';
+import type { LogicalPlan } from '../planner/index';
+import type { LogicformType } from '../logicform-types';
+/** Remove global result paging while retaining HAVING and per-group limitBy semantics. */
+export declare const withoutGlobalPagination: (plan: LogicalPlan) => LogicalPlan;
+/** Describe the total SQL with the same unpaged global result as its plan. */
+export declare const withoutGlobalPaginationLogicform: (logicform: LogicformType) => LogicformType;
+/** Count the rows produced by a complete unpaginated logical result. */
+export declare const compileTotalQuery: (unpaged: CompiledQuery, provider: CoreComputeProvider) => CompiledQuery;
+export declare const readTotal: (rows: readonly DataRow[]) => number;

package/dist/executor/types.d.ts

@@ -1,8 +1,8 @@
 import type { OperatorDefinition, OperatorRegistry } from '../operators/index';
-import type { CoreComputeProvider, DataRow, SqlParameters } from '../physical/index';
+import type { CoreComputeProvider, DataRow, ProviderRegistry, SqlParameters } from '../physical/index';
 import type { DatabaseConnection } from '../physical/index';
 import type { LogicalPlan, QueryLineage } from '../planner/index';
-import type { SchemaMappingDefinition, SchemaMappingRegistry } from '../schema-mappings/index';
+import type { SchemaMappingDefinition, SchemaMappingRegistry, SchemaMappingSelection } from '../schema-mappings/index';
 import type { ResolvedSemanticSchema, SemanticSchemaDraft, SemanticTypeRegistry, TimeWatermarkPolicy } from '../semantic-types/index';
 import type { LogicformType, PublicLogicformType } from '../logicform-types';
 import type { QueryCacheExecutionOptions } from '../cache/index';
@@ -36,6 +36,7 @@ export type ExecuteProgressEvent = {
 } | {
     readonly stage: 'executing';
     readonly provider: CoreComputeProvider['kind'];
+    readonly connectionId?: string;
     readonly partialResult: PartialExecuteResult;
 } | {
     readonly stage: 'hydrating';
@@ -56,8 +57,9 @@ export type ExecuteProgressEvent = {
 export type ProgressCallback = (event: ExecuteProgressEvent) => void | Promise<void>;
 interface ExecuteBaseOptions {
     readonly schemas: Readonly<Record<string, SemanticSchemaDraft>>;
+    /** Caller-owned Registry; plugin SemanticTypes are copied into the execution Registry. */
     readonly semanticTypes?: SemanticTypeRegistry;
-    /** Caller-defined operators supplied as plain definition objects. */
+    /** Caller-defined Operators; explicit definitions override same-name built-ins or plugin Operators. */
     readonly operators?: OperatorDefinition | readonly OperatorDefinition[];
     /** Caller-owned capability plugins appended to the built-in plugin set for this execution. */
     readonly plugins?: readonly SemanticDBPlugin[];
@@ -73,30 +75,54 @@ interface ExecuteBaseOptions {
     readonly signal?: AbortSignal;
     /** Default time watermark policy; Schema and Property policies override it. */
     readonly timeWatermark?: TimeWatermarkPolicy;
+    /** Resource limits for in-process combination of separate physical connections. */
+    readonly federation?: {
+        /** Defaults to 100,000 rows across all materialized aggregate branches. */
+        readonly maxMaterializedRows?: number;
+    };
 }
 export interface ExecuteOptions extends ExecuteBaseOptions {
     /** Physical mappings supplied as JSON definitions. */
     readonly schemaMappings: readonly SchemaMappingDefinition[];
     /** Database connections addressable by SchemaMappingDefinition.connectionId. */
     readonly connections: readonly DatabaseConnection[];
+    /** Provider factories used for built-in or application-defined connection kinds. */
+    readonly providerRegistry?: ProviderRegistry;
 }
 /** Internal execution seam used by executor tests after a provider has been resolved. */
 export interface InternalExecuteOptions extends Omit<ExecuteBaseOptions, 'operators'> {
     readonly operators?: OperatorRegistry;
+    /** The public execution boundary has already merged built-in, plugin, and explicit Operators. */
+    readonly operatorsResolved?: boolean;
+    /** The public execution boundary has already merged plugin SemanticTypes. */
+    readonly semanticTypesResolved?: boolean;
     readonly localeRegistry?: LocaleRegistry;
     readonly provider: CoreComputeProvider;
+    /** Resolve providers for additional physical connections in a federated execution. */
+    readonly providerForConnection?: (connectionId: string) => CoreComputeProvider;
     readonly schemaMappings: SchemaMappingRegistry;
     /** Selected connection for public execution; omitted only by internal provider tests. */
     readonly connectionId?: string;
 }
 export interface ExecuteSuccessResult {
     readonly result: DataRow[];
+    /** Result rows or grouped rows before global limit/page pagination. */
+    readonly total: number;
     /** Schema describing the computed result rows. */
     readonly schema: ResolvedSemanticSchema;
     readonly normedLogicform: PublicLogicformType;
     readonly sqls: readonly ExecutedSql[];
     readonly lineage: QueryLineage;
     readonly provider: CoreComputeProvider['kind'];
+    /** Physical connections that contributed rows to this result. */
+    readonly providers: readonly {
+        readonly connectionId: string;
+        readonly provider: CoreComputeProvider['kind'];
+    }[];
+    /** Physical mapping decisions, including why every non-selected candidate lost. */
+    readonly schemaMappingSelections: readonly SchemaMappingSelection[];
+    /** Plugin-owned materialization for a custom LogicForm output type. */
+    readonly output?: unknown;
 }
 export interface ExecuteFailureResult {
     readonly errorCode: '100';
@@ -109,10 +135,13 @@ export interface ExecuteFailureResult {
     readonly lineage?: QueryLineage;
 }
 export type ExecuteResult = ExecuteSuccessResult | ExecuteFailureResult;
-export type ExecutedSqlType = 'computation' | 'semantic-resolution';
+export type ExecutedSqlType = 'computation' | 'semantic-resolution' | 'total';
 export interface ExecutedSql {
     /** Why this SQL participates in the LogicForm execution. */
     readonly type: ExecutedSqlType;
+    /** Physical connection that executed this SQL, or the in-process coordinator id. */
+    readonly connectionId?: string;
+    readonly provider?: CoreComputeProvider['kind'];
     /** Human-readable, localized explanation of why this SQL is executed. */
     readonly explain: string;
     /** Normalized LogicForm represented by this SQL statement. */