semanticdb-v2 2.0.0-alpha.1 → 2.0.0-alpha.10

package/README.md

@@ -1,23 +1,37 @@
 # SemanticDB
 
-SemanticDB translates its own LogicForm into SQL and executes it on a single core compute layer.
+SemanticDB translates its own LogicForm into SQL and executes it on one or more core compute layers.
 Callers provide schemas, schema mappings, operators, and database connection information.
 SemanticDB creates the matching provider and owns its connection lifecycle.
 
+Install SemanticDB together with only the database driver used by the application. Database
+drivers are optional peer dependencies and are loaded on the first execution for that connection;
+Redis cache support is included by the main package.
+
+```bash
+npm install semanticdb-v2 mysql2
+```
+
+Use `@clickhouse/client`, `@databricks/sql`, `dmdb`, `mysql2` (MySQL, Doris, StarRocks, and GBase),
+`mssql`, `pg`, `snowflake-sdk`, or `oracledb` for the corresponding connection kind. MaxCompute
+uses a caller-supplied parameterized execution client. If a configured driver is missing, execution
+fails with an installation command for that driver. SemanticDB caches its internally created
+providers across executions; call `await Logicform.close()` during application shutdown.
+
 ## Public API
 
 ```ts
 import { Logicform } from 'semanticdb-v2';
-import { SchemaMappingRegistry } from 'semanticdb-v2';
 
-const schemaMappings = new SchemaMappingRegistry([
+const schemaMappings = [
   {
     schemaId: 'sales',
     connectionId: 'analytics',
     source: 'analytics.sales_detail',
+    mappings: [],
     freshness: { mode: 'ttl', ttlSeconds: 300 },
   },
-]);
+];
 
 const response = await Logicform.execute(
   {
@@ -34,8 +48,9 @@ const response = await Logicform.execute(
     connections: [{
       id: 'analytics',
       kind: 'clickhouse',
-      url: 'http://localhost:8123',
-      username: 'default',
+      host: 'localhost',
+      port: 8123,
+      user: 'default',
       password: process.env.CLICKHOUSE_PASSWORD,
       database: 'analytics',
     }],
@@ -49,27 +64,43 @@ const response = await Logicform.execute(
 );
 ```
 
-Every executable LogicForm, including LogicForms used as subqueries, must declare `version`. This
-SDK accepts semantic versions greater than or equal to `2.0`; missing, malformed, or older versions
-fail with `UNSUPPORTED_LOGICFORM_VERSION` before Schema resolution or database access. A relation
-query shaped as `{ schema, query }` is part of its parent LogicForm and inherits the parent's version.
+A successful response always includes `total`, the number of result rows before global
+`limit`/`page` pagination. For grouped LogicForms this is the number of groups after `having` and
+`limitBy`. Paginated physical queries include a separate `sqls` entry with `type: 'total'` for the
+count query; unpaginated and executor-merged results derive the value without another SQL query.
+
+The v2 LogicForm contract requires `version` on executable LogicForms, including LogicForm objects
+used as subqueries, and accepts semantic versions greater than or equal to `2.0`. At the public
+`Logicform.execute()` boundary, versionless and 1.x legacy input is converted to 2.0 before v2
+normalization; malformed and other unsupported versions fail with `UNSUPPORTED_LOGICFORM_VERSION`.
+Direct use of the v2 normalizer requires a supported version. A relation query shaped as
+`{ schema, query }` is part of its parent LogicForm and inherits the parent's version.
 
 PredItems without an explicit `name` receive a human-readable name during normalization. Generated
 names default to Chinese; pass `locale: 'en-US'` in `ExecuteOptions` for the built-in English
-names. Every operator owns its localized default names and receives the requested locale directly:
+names. SemanticDB ships `zh-CN` and `en-US` locale files backed by i18next. Applications can add or
+override locale files through `ExecuteOptions.locales`:
 
 ```ts
-operators.register({
-  name: '$weightedAverage',
-  defaultName: ({ operandName, locale }) => locale.toLowerCase().startsWith('en')
-    ? `Weighted average of ${operandName}`
-    : `${operandName}加权平均值`,
-  kind: 'aggregate',
-  toSQL: ({ column }) => `weighted_avg(${column})`,
+await Logicform.execute(logicform, {
+  locale: 'ja-JP',
+  locales: {
+    locale: 'ja-JP',
+    messages: {
+      'operator.$sum': '{operandName}の合計',
+      'component.numerator': '{outputName} 分子',
+      'component.denominator': '{outputName} 分母',
+    },
+  },
+  // schemas, schemaMappings, connections, ...
 });
 ```
 
-The same locale applies to `preds`, `groupby`, nested expressions, custom-function components,
+Locale lookup uses the exact locale first, then the matching language, and finally `zh-CN`.
+Multiple locale files can be supplied as an array. A custom operator still owns its fallback name
+through `defaultName()` and can call the context's `t()` helper when it defines translation keys.
+
+The same locale applies to `preds`, `groupby`, nested expressions, custom-function subqueries,
 and entity-population queries. A PredItem's normalized `query` is included in its
 generated name, for example `门店为上海的销售额合计`. Operator-specific `args` also participate
 through each operator's `defaultName()`; period operators distinguish同比、环比 and other offsets.
@@ -78,7 +109,8 @@ An explicitly authored `name` is preserved unchanged.
 `entity_id` identifies one entity by `hierarchy.property` on hierarchy Schemas and by the unique
 `type: 'ID'` property on other Schemas. Normalization adds that identity filter unless the query
 already contains one, in which case the explicit query wins. The legacy value `placeholder` is
-preserved as entity context but never becomes a SQL filter. This also applies to relation queries
+preserved as entity context but never becomes a SQL filter. String IDs and safe-integer IDs are
+accepted and retain their original type. This also applies to relation queries
 shaped as `{ schema, query, entity_id }`. `representation` is presentation metadata for frontend
 consumers and does not change normalization, planning, or execution. See
 [Hierarchy plugin](docs/hierarchy-plugin.md) for Schema configuration, `$hierarchyLevel`, legacy
@@ -89,12 +121,18 @@ Plugins may own Schema metadata, legacy conversion, `entity_id` selection, and O
 remaining scoped to one execution. See [Register an external plugin](docs/register-plugin.md).
 
 Parent-child Schemas may declare `parentProperty`. `$recursive` filters a property to a root node and
-all descendants; object properties infer the tree Schema from `ref`. MySQL, Doris, StarRocks, and
-PostgreSQL compile this as a parameterized recursive CTE.
+all descendants; object properties infer the tree Schema from `ref`. MySQL, Doris, StarRocks,
+PostgreSQL, ClickHouse 24.4+, and Databricks compile this as a parameterized recursive CTE.
 
 Top-level projections may set `schema` to read metrics from another registered Schema. SemanticDB
 splits those projections into independent provider queries, aligns shared query/groupby dimensions,
 and merges rows by the group keys before applying `having`, `sort`, `limitBy`, and pagination.
+When the selected Schema mappings use separate physical connections, SemanticDB executes every
+single-connection fragment with its native provider and registers the aggregated fragment results
+in an in-process DuckDB coordinator. DuckDB applies the final full-outer group merge, cross-fragment
+conditions, ordering, totals, and pagination. It is embedded in the Node.js process and does not
+require a service or listening port. `federation.maxMaterializedRows` limits the total number of
+fragment rows admitted to the coordinator and defaults to 100,000.
 
 The progress stream emits `started`, `validated`, `normalized`, `planned`, `lineage`,
 `compiled`, `executing`, `hydrating`, `result`, and `completed`. Every event includes a cumulative
@@ -107,7 +145,11 @@ SQL string for diagnostics:
 
 ```ts
 interface ExecutedSql {
-  type: 'semantic-resolution' | 'computation';
+  type: 'semantic-resolution' | 'computation' | 'total';
+  connectionId?: string;
+  provider?: CoreComputeProvider['kind'];
+  explain: string;
+  logicform: LogicformType;
   sql: string;
   parameters: readonly unknown[];
   displaySql: string;
@@ -115,14 +157,127 @@ interface ExecutedSql {
 ```
 
 `semantic-resolution` identifies supporting SQL used to resolve semantic input or output, such as
-latest-data-date lookup and object entity population. `computation` identifies SQL that calculates
+time-watermark resolution and object entity population. `computation` identifies SQL that calculates
 the requested LogicForm result. `sqls` includes every physical SQL involved in the execution; an
 entry remains visible when its result is served by the query cache.
+Coordinator SQL uses the reserved connection id `__semanticdb_coordinator__` and provider
+`duckdb`; other entries identify the physical connection that executed them. Successful results
+also expose `providers`, the physical connection/provider pairs that contributed rows. The legacy
+singular `provider` field is `duckdb` for a federated result.
 
 Providers execute only parameterized `sql + parameters`; SemanticDB never executes `displaySql`.
 Because `displaySql` contains literal values and may expose sensitive data, use it only in trusted
 debugging and observability channels.
 
+### Native SQL and table discovery
+
+`DBConnector` provides trusted physical database operations independently of LogicForm execution:
+
+```ts
+import { DBConnector, Logicform } from 'semanticdb-v2';
+import type { DatabaseConnection } from 'semanticdb-v2';
+
+const connection = {
+  id: 'analytics',
+  kind: 'mysql',
+  host: 'localhost',
+  user: 'analytics',
+  password: process.env.MYSQL_PASSWORD ?? '',
+  database: 'analytics',
+} satisfies DatabaseConnection;
+
+const tables = await DBConnector.getTableList({ connection });
+// [{ name: 'orders', database: 'analytics', type: 'table' }, ...]
+
+const description = await DBConnector.describeTable({ connection, table: 'orders' });
+// {
+//   columns: [{ name: 'id', type: 'int', comment: 'Order ID', primaryKey: true, exampleValue: 42 }, ...],
+//   tableInfo: { name: 'orders', database: 'analytics', type: 'table', comment: 'Orders' },
+// }
+
+const { rows } = await DBConnector.runSQL({
+  connection,
+  sql: 'SELECT * FROM orders WHERE status = ?',
+  parameters: ['paid'],
+});
+
+const updated = await DBConnector.runSQL({
+  connection,
+  sql: 'UPDATE orders SET status = ? WHERE id = ?',
+  parameters: ['shipped', 42],
+});
+// { rows: [], affectedRows: 1 }
+
+// The exact same object belongs in Logicform.execute's connections array:
+// await Logicform.execute(logicform, { connections: [connection], schemas, schemaMappings });
+
+await Logicform.close(); // DBConnector.close() is the same shutdown operation.
+```
+
+Both APIs share providers and connection pools by connection ID and `providerRegistry`. When a
+connection's configuration changes under the same ID, the next operation creates a new provider
+and closes the previous one. Providers and pools are initialized lazily. Either
+`close()` releases **all** SDK-managed connections, including those used by the other API; it does
+not close just the last connector connection.
+
+`runSQL` returns `{ rows: DataRow[], affectedRows?: number }`. Commands without a result set return
+an empty array; affected-row counts are included only when the driver reports them. SQL is passed
+through without automatic LIMIT, dialect translation, Schema mapping, hydration, generated `_id`,
+or query caching. It does **not** enforce `ExecuteOptions.permissions`: use trusted callers and
+appropriately restricted database accounts. Supply values through `parameters` using the native
+provider placeholders: `?` for MySQL-compatible databases, Databricks, Dameng, Snowflake and MaxCompute;
+`$1` for PostgreSQL; `@p0` for SQL Server; `:1` for Oracle; typed `{p0:String}`-style
+placeholders for ClickHouse. ClickHouse uses JSONEachRow output; do not override it with another
+FORMAT. Each call is independent, with Oracle/Dameng command execution automatically committed;
+cross-call transactions and multi-statement/multi-result execution are not supported.
+
+`getTableList` returns `{ name, database?, schema?, type: 'table' | 'view' }[]` for accessible tables
+and views, retaining namespaces to distinguish same-name tables. Optional `database` and `schema`
+override only this metadata lookup, never the cached connection configuration:
+
+- MySQL, Doris, StarRocks, GBase and ClickHouse: `database`; `schema` is rejected.
+- PostgreSQL, Oracle and Dameng: `schema`; cross-database overrides are rejected. Defaults to the
+  current schema (or the configured Dameng schema).
+- SQL Server and Snowflake: `database` and `schema`. SQL Server lists all accessible schemas in
+  the database by default; Snowflake defaults to the current schema.
+- Databricks: `database` means catalog, and `schema` means schema. Defaults come from the connection's
+  `catalog` and `schema` (`connection.database` remains the existing legacy alias for schema).
+- MaxCompute: the existing injected client handles `SHOW TABLES`, returning rows with `table_name`.
+  It may implement `getTableList(request)` for project/schema overrides and view metadata, and
+  `runSQL(request)` for affected-row counts. Otherwise commands must return `[]` through `execute`.
+
+`describeTable({ connection, table, database?, schema?, includeSample? })` inspects one physical
+table or view. `table` is a literal **unqualified** name; provide its namespace separately, using
+the same scope rules as `getTableList`. A dot inside `table` is part of the identifier, not a separator.
+Do not pass a SQL expression or the legacy semantic Schema/`from` object. SQL Server descriptions
+default to the connection's current default schema; pass `schema` explicitly when describing a
+table returned from another schema. Column order follows the database's definition.
+
+The result has `columns: { name, type, comment, primaryKey?, exampleValue? }[]` and
+`tableInfo: { name, database?, schema?, type, comment }`. These are physical names/native types,
+not semantic Properties; no `_id` is added. Comments default to an empty string. Primary-key
+membership includes every column of composite keys; unique-only indexes are not primary keys.
+ClickHouse reports its storage primary key, which does not imply relational uniqueness. If the
+adapter cannot determine primary-key membership (Databricks/Snowflake), `primaryKey` is omitted
+rather than guessed. Example values preserve driver types, including `null`.
+
+Like the legacy API, descriptions read one example row by default. Set `includeSample: false`
+for metadata-only inspection, avoiding a data read. Empty tables and failed sample queries still
+return metadata without example values; metadata failures, tables not found in the selected namespace, and
+cancellation reject. All SQL uses bound metadata values and quoted identifiers, bypassing
+LogicForm permissions and caching just like `runSQL`.
+
+MaxCompute clients may implement `describeTable(request)` to provide project/schema-aware
+metadata. Otherwise the fallback runs `DESC` and expects structured rows with `column_name`,
+`column_type`, optional `column_comment` and `is_primary_key`. Legacy UDBC ASCII `Info` reports
+must be normalized by the client; unparsed reports are rejected rather than producing empty columns.
+
+All operations accept `signal` and the same optional `providerRegistry` as `Logicform.execute`.
+Custom providers opt in through `runSQL(request)`, `getTableList(request)` and `describeTable(request)`; missing capabilities
+fail explicitly without changing the existing provider `execute` contract. Unsupported discovery
+scopes are rejected rather than silently ignored. Cancellation does not guarantee rollback of
+commands that have already reached the database.
+
 ### Permissions
 
 Pass table, row, and column permissions together through `ExecuteOptions.permissions`:
@@ -165,26 +320,44 @@ await Logicform.execute(logicform, {
 
 When `permissions` is omitted, execution remains unrestricted. When it is present, its Schema keys
 are the table allowlist. `rowPermissions` is always combined with the caller query using `AND`,
-including internal latest-data-date queries, related-table joins, and object population queries.
+including internal time-watermark queries, related-table joins, and object population queries.
 `columnPermissions.whitelist` accepts property names or IDs and restricts caller projections,
 filters, grouping, operator inputs, and relation paths. A row policy may still use a non-whitelisted
 column as an internal filter. If an ID is not whitelisted, SemanticDB generates the executor-owned
 result-row `_id` instead of exposing the physical ID as that identity.
 
-### Custom composite metrics
+### Component operators
+
+Composite metrics are regular `OperatorDefinition` values with `kind: 'component'`. Their
+`createComponent` hook can return query components merged through a CTE or JavaScript. Register
+them in the same `OperatorRegistry` as scalar and aggregate operators. Operator names are free-form;
+the `$` prefix is a convention used by built-in operators, not a requirement.
 
-Pass legacy-compatible `genComponents`, `mergeComponents`, and `genMergeSQL` definitions through
-`ExecuteOptions.customFunctions`. JavaScript merge supports async callbacks; SQL merge compiles
-components into CTEs. `Logicform.genComponents()` expands the same definitions into an indicator
-tree without accessing a database. Cross-Schema components independently pass table, row, and
-column permissions. See [Custom composite metrics](docs/custom-functions.md) for the complete
-contract, merge-mode selection, and trusted function-string behavior.
+Every composite component is exposed in result rows by default. Its `name` is its final result-column
+name and may contain Chinese characters or spaces. Built-ins derive child names from the parent output,
+for example `总销售额占比分子` and `总销售额占比分母`. Operators do not return `outputName` or an
+exposure flag; the executor owns the parent output binding. Components are joined on groupby output
+names shared by both sides unless an exceptional custom operator explicitly supplies `joinKeys`.
+
+See [Register an operator](docs/register-operator.md) for the component contract.
 
 The root entry point accepts a `connections` array of `DatabaseConnection` items. Every item has a unique `id`; set `kind` to `mysql`,
-`doris`, `starrocks`, `clickhouse`, `snowflake`, `oracle`, or `postgresql`, then add that driver's
+`doris`, `starrocks`, `gbase`, `maxcompute`, `clickhouse`, `databricks`, `dameng`, `snowflake`,
+`sqlserver`, `oracle`, or `postgresql`, then add that driver's
 native connection fields directly on the item. Each Schema Mapping selects an item through
 `connectionId`. SemanticDB lazily creates and reuses the matching internal provider. Call
 `await Logicform.close()` during application shutdown to close every connection pool and session.
+One LogicForm may reference mappings on multiple connections. This is one logical execution, not
+one cross-database SQL statement: physical fragments run independently and their aggregate results
+are combined by the embedded coordinator. A relation join whose two sides themselves live on
+different connections is not yet materialized automatically. When the related Schema has replicas
+on multiple connections, the Planner selects the mapping colocated with the main mapping and emits
+the `LEFT JOIN` on that connection. It fails explicitly only when no target mapping exists on the
+main connection.
+
+MySQL-compatible providers default to 30 connections, at most 10 idle connections, a 60-second
+idle timeout, and TCP keepalive. Native `mysql2` pool options on the connection item override these
+defaults.
 
 ### PostgreSQL connection
 
@@ -216,6 +389,66 @@ pagination, date, aggregate, string, regex, parameter, and display-SQL behavior
 MySQL-compatible SQL helper. Database-specific differences should be implemented by overriding the
 corresponding internal dialect capability.
 
+### GBase connection
+
+GBase 8a uses its MySQL-compatible protocol through `mysql2`. It defaults to port `5258`; all
+ordinary `mysql2` connection and pool fields remain available:
+
+```ts
+const gbase = {
+  id: 'gbase-main',
+  kind: 'gbase',
+  host: process.env.GBASE_HOST,
+  user: process.env.GBASE_USER,
+  password: process.env.GBASE_PASSWORD,
+  database: process.env.GBASE_DATABASE,
+} as const;
+```
+
+### MaxCompute connection
+
+MaxCompute requires a caller-supplied client that preserves the Provider's parameterized execution
+contract. The client receives SQL and parameters separately and owns the actual MaxCompute/JDBC
+transport. SemanticDB calls the client's `close()` when its provider is closed:
+
+```ts
+const maxcompute = {
+  id: 'maxcompute-main',
+  kind: 'maxcompute',
+  client: {
+    execute: ({ sql, parameters, signal }) => maxComputeGateway.execute({
+      sql, parameters, signal,
+    }),
+    close: () => maxComputeGateway.close(),
+  },
+} as const;
+```
+
+The legacy UDBC `Statement` endpoint accepts only a complete SQL string, so SemanticDB does not
+use it or interpolate values locally. A gateway implementation must bind `parameters` before this
+connection can execute queries.
+
+### SQL Server connection
+
+SQL Server uses `mssql`, bracket-quoted identifiers, and named `@p0`, `@p1` parameters:
+
+```ts
+const sqlserver = {
+  id: 'sqlserver-main',
+  kind: 'sqlserver',
+  host: process.env.SQLSERVER_HOST,
+  port: 1433,
+  user: process.env.SQLSERVER_USER,
+  password: process.env.SQLSERVER_PASSWORD,
+  database: process.env.SQLSERVER_DATABASE,
+  options: { encrypt: true },
+} as const;
+```
+
+Pagination uses `OFFSET ... FETCH`; SemanticDB supplies a neutral top-level ordering when the
+LogicForm has no sort. JSON array relations use `OPENJSON`. SQL Server has no native regular
+expression predicate, so `$regex` is rejected explicitly.
+
 ### Snowflake connection
 
 Snowflake connection fields are passed to the official `snowflake-sdk`:
@@ -238,6 +471,52 @@ await Logicform.execute(logicform, {
 });
 ```
 
+### Databricks connection
+
+Databricks uses `@databricks/sql` and supports a personal access token or OAuth M2M. `catalog` and
+`schema` select the initial SQL Warehouse session namespace (`database` is accepted as a legacy
+alias for `schema`):
+
+```ts
+const databricks = {
+  id: 'databricks-main',
+  kind: 'databricks',
+  host: process.env.DATABRICKS_SERVER_HOSTNAME,
+  path: process.env.DATABRICKS_HTTP_PATH,
+  authType: 'databricks-oauth',
+  oauthClientId: process.env.DATABRICKS_CLIENT_ID,
+  oauthClientSecret: process.env.DATABRICKS_CLIENT_SECRET,
+  catalog: 'main',
+  schema: 'analytics',
+} as const;
+```
+
+Use `authType: 'access-token'` with `token` for PAT authentication. Values are sent through the
+driver's ordinal parameter binding and are not interpolated into executable SQL. `authType` is
+required: PAT and OAuth fields are mutually exclusive, and authentication is never inferred from
+the presence of credentials.
+
+### Dameng connection
+
+Dameng uses the official `dmdb` pool and the `dameng` connection kind:
+
+```ts
+const dameng = {
+  id: 'dameng-main',
+  kind: 'dameng',
+  host: process.env.DAMENG_HOST,
+  port: 5236,
+  user: process.env.DAMENG_USER,
+  password: process.env.DAMENG_PASSWORD,
+  schema: process.env.DAMENG_SCHEMA,
+} as const;
+```
+
+The provider uses positional binds and object-format result rows. Recursive hierarchy queries and
+JSON-array relations are rejected until their behavior can be verified against supported DM server
+versions; scalar queries, grouping, date operations, regex, windows, joins, and pagination compile
+with the DM/Oracle-style dialect.
+
 ### Oracle connection
 
 Oracle uses the official `oracledb` driver in Thin mode and does not require Oracle Client
@@ -262,7 +541,8 @@ await Logicform.execute(logicform, {
 `Schema.properties[].type` is an extensible semantic type. SemanticDB supplies the built-in
 types and derives each property's `primal_type` while resolving the Schema. A supplied
 `primal_type` must match the registered type. `isArray` describes the value container and
-defaults to `false`; for example, tags are represented as `type: 'category', isArray: true`.
+defaults to `false`. The built-in `tag` type is a string array and sets `isArray: true`
+automatically; `type: 'category', isArray: true` remains available for other categorical arrays.
 
 ```ts
 import { createSemanticTypeRegistry, Logicform } from 'semanticdb-v2';
@@ -306,19 +586,38 @@ comments, or parameter placeholders. Property UDFs do not support the legacy `fu
 
 ### Query cache
 
-All compiled provider queries, including the internal latest-data-date query, can share one
-caller-owned query cache. Redis is published from `semanticdb-v2/cache/redis`; cache identity
-includes namespace, provider, SQL, and parameters. Physical expiration belongs to each Schema
-mapping's `freshness` policy. See [Query cache and Redis](docs/query-cache.md).
+All compiled provider queries, including the internal time-watermark query, can share one
+internally managed Redis query cache. Pass its JSON connection configuration through `cache`;
+cache identity includes namespace, connection id, provider, SQL, and parameters. Physical
+expiration belongs to each Schema mapping's `freshness` policy. See
+[Query cache and Redis](docs/query-cache.md).
 
-When Redis is shared with other applications, pass an application-specific key prefix:
+Schema mappings can also expose same-connection entity feature tables as lazy physical
+enrichments. A mapped feature property adds a `LEFT JOIN` only when referenced, and a point-in-time
+feature can be assigned to the containing valid-time entity row. See
+[Schema Mapping feature enrichments](docs/schema-mapping-enrichments.md).
 
-```ts
-import { createRedisQueryCache } from 'semanticdb-v2/cache/redis';
+For aggregate sources, set `granularity` on mapped date or hierarchy Properties.
+Other available dimensions are inferred from their Property mappings, so there is no
+separate Schema Mapping `dimensions` field. The planner selects a source that covers
+the requested properties and physical time or hierarchy grain. See
+[Schema Mapping physical granularity](docs/schema-mapping-granularity.md).
+
+When Redis is shared with other applications, pass an application-specific key prefix in the
+execution JSON. SemanticDB reuses the connection by `cache.id`:
 
-const cache = await createRedisQueryCache({
-  url: process.env.REDIS_URL,
-  keyPrefix: process.env.REDIS_KEY_PREFIX,
+```ts
+await Logicform.execute(logicform, {
+  schemas,
+  schemaMappings,
+  connections,
+  cache: {
+    id: 'redis-main',
+    kind: 'redis',
+    url: process.env.REDIS_URL ?? 'redis://localhost:6379/0',
+    keyPrefix: process.env.REDIS_KEY_PREFIX,
+    namespace: 'tenant-a:mysql-sales',
+  },
 });
 ```
 
@@ -432,7 +731,8 @@ npm run debug
 
 ## Current scope
 
-- ClickHouse, MySQL, Oracle, and Snowflake core compute providers
+- ClickHouse, Databricks, Dameng, Doris, GBase, MaxCompute, MySQL, Oracle, PostgreSQL, Snowflake,
+  SQL Server, and StarRocks core compute providers
 - Projections, conditional aggregates, and `$sum`, `$count`, `$countDistinct`, `$avg`, `$min`, `$max`
 - Structured arithmetic, share-of-total, period comparison, period-boundary, date-bucket,
   cumulative period, trusted SQL expression, string, case, and window operators
@@ -442,7 +742,7 @@ npm run debug
 - `$and` and `$or`, HAVING, sorting, row-count and percentage pagination, and per-group `limitBy`
 - Query-level and output-column lineage
 - Multi-level scalar and array object relation paths, array groupby, and batched entity population
-- Nested `from` LogicForms compiled as parameterized derived tables
+- Nested `from` LogicForm objects compiled as parameterized derived tables
 
 Exports and metadata persistence are intentionally deferred. Directly selected array-object
 properties are populated without expanding the root result rows.

package/dist/cache/index.d.ts

@@ -1,3 +1,4 @@
 export { nextDailyExpiration, resolveQueryExpiration, } from './expiry';
 export { queryCacheIdentityHash, sourceCacheIndexHash, } from './identity';
-export type { QueryCache, QueryCacheExecutionOptions, QueryCacheIdentity, QueryCacheWriteOptions, } from './types';
+export type { InvalidateQueryCacheSourceOptions, QueryCache, QueryCacheConfiguration, QueryCacheExecutionOptions, QueryCacheIdentity, QueryCacheWriteOptions, } from './types';
+export { QueryCacheManager } from './manager';

package/dist/cache/manager.d.ts

@@ -0,0 +1,5 @@
+import type { InvalidateQueryCacheSourceOptions } from './types';
+/** Operations for internally managed query-cache connections. */
+export declare const QueryCacheManager: Readonly<{
+    invalidateSource: ({ cache, source }: InvalidateQueryCacheSourceOptions) => Promise<void>;
+}>;

package/dist/cache/registry.d.ts

@@ -0,0 +1,14 @@
+import type { QueryCache, QueryCacheConfiguration, QueryCacheExecutionOptions } from './types';
+type QueryCacheConnection = Omit<QueryCacheConfiguration, 'namespace'>;
+type QueryCacheFactory = (connection: QueryCacheConnection) => Promise<QueryCache>;
+/** Internally owns and reuses cache connections by stable configuration id. */
+export declare class QueryCacheRegistry {
+    #private;
+    private readonly create;
+    constructor(create?: QueryCacheFactory);
+    get(configuration: QueryCacheConfiguration): Promise<QueryCacheExecutionOptions>;
+    close(): Promise<void>;
+}
+export declare const resolveQueryCache: (configuration: QueryCacheConfiguration) => Promise<QueryCacheExecutionOptions>;
+export declare const closeQueryCaches: () => Promise<void>;
+export {};

package/dist/cache/types.d.ts

@@ -2,6 +2,8 @@ import type { CoreComputeProvider, DataRow, SqlParameters } from '../physical/in
 export interface QueryCacheIdentity {
     readonly namespace: string;
     readonly provider: CoreComputeProvider['kind'];
+    /** Distinguishes separate databases using the same provider dialect. */
+    readonly connectionId?: string;
     readonly sql: string;
     readonly parameters: SqlParameters;
 }
@@ -25,3 +27,18 @@ export interface QueryCacheExecutionOptions {
     /** Isolates tenants and physical data sources that can produce identical SQL. */
     readonly namespace: string;
 }
+/** Redis connection and cache namespace supplied as JSON at the public execution boundary. */
+export interface QueryCacheConfiguration {
+    /** Stable identity used to reuse one internally managed Redis connection. */
+    readonly id: string;
+    readonly kind: 'redis';
+    readonly url: string;
+    readonly keyPrefix?: string;
+    /** Isolates tenants and physical data sources that can produce identical SQL. */
+    readonly namespace: string;
+}
+export interface InvalidateQueryCacheSourceOptions {
+    readonly cache: QueryCacheConfiguration;
+    /** Physical Schema Mapping source to invalidate. */
+    readonly source: string;
+}

package/dist/close.d.ts

@@ -0,0 +1 @@
+export declare const closeSemanticDB: () => Promise<void>;

package/dist/compatibility/index.d.ts

@@ -1,2 +1,2 @@
-export { assertNoLegacyLogicformFields, normalizeLegacyLogicform, splitLegacySchemas, } from './logicform';
-export { normalizeSchemas } from '../executor/compatibility';
+export { assertNoLegacyLogicformFields, normalizeLegacyLogicform, } from './logicform';
+export { isLegacyEmptyMapping, normalizeLegacySchemaMappings, normalizeSchemas, } from '../executor/compatibility';

package/dist/compatibility/logicform.d.ts

@@ -1,13 +1,7 @@
 import type { LogicformType } from '../logicform-types';
 type SchemaRegistry = Readonly<Record<string, unknown>>;
 /** Converts a legacy LogicForm tree into the v2 wire shape without mutating input. */
-export declare const normalizeLegacyLogicform: (logicform: unknown) => LogicformType;
-/**
- * Converts the legacy `pred.schema` fan-out into v2 composition. The planner
- * intentionally rejects cross-schema PredItems, so this has to happen before
- * the v2 normalizer sees the tree.
- */
-export declare const splitLegacySchemas: (logicform: LogicformType, schemas?: SchemaRegistry) => LogicformType;
+export declare const normalizeLegacyLogicform: (logicform: unknown, schemas?: SchemaRegistry) => LogicformType;
 /** Rejects legacy wire fields when a caller explicitly declares a v2 LogicForm. */
 export declare const assertNoLegacyLogicformFields: (logicform: unknown) => void;
 export {};

package/dist/compiler/compile-cte.d.ts

@@ -1,4 +1,4 @@
-import type { OperatorCteCalculation, OperatorGroupKeyShift } from '../operators/index';
+import type { OperatorComponentCalculation, OperatorGroupKeyShift } from '../operators/index';
 import type { CoreComputeProvider } from '../physical/index';
 import type { LogicalPlan } from '../planner/index';
 import type { CompiledQuery } from './types';
@@ -9,4 +9,19 @@ export interface PreparedCteComponent {
     readonly groupKeyShifts?: readonly OperatorGroupKeyShift[];
     readonly joinKeys?: readonly string[];
 }
-export declare const compileCteCalculation: (calculation: OperatorCteCalculation, components: readonly PreparedCteComponent[], outputPlan: LogicalPlan, provider: CoreComputeProvider, operators: OperatorRegistry) => CompiledQuery;
+/** A compiled top-level execution unit that can be joined into one SQL query. */
+export interface PreparedCteQuery {
+    readonly plan: LogicalPlan;
+    readonly compiled: CompiledQuery;
+    /** Result projection names exposed by this execution unit (excluding groups). */
+    readonly outputs: readonly string[];
+}
+/**
+ * Combine independent SQL execution units into one statement.
+ *
+ * Component operators are compiled independently because their input ranges can
+ * differ (for example, current and previous periods). Keeping each unit as a
+ * CTE preserves those ranges while allowing the provider to execute one query.
+ */
+export declare const compileCteQueries: (units: readonly PreparedCteQuery[], outputPlan: LogicalPlan, provider: CoreComputeProvider, operators?: OperatorRegistry) => CompiledQuery;
+export declare const compileCteCalculation: (calculation: OperatorComponentCalculation, components: readonly PreparedCteComponent[], outputPlan: LogicalPlan, outputName: string, provider: CoreComputeProvider, operators: OperatorRegistry) => CompiledQuery;

package/dist/compiler/index.d.ts

@@ -1,4 +1,5 @@
 export { compilePlan } from './compile-plan';
-export { compileCteCalculation } from './compile-cte';
+export { compileCteCalculation, compileCteQueries } from './compile-cte';
+export type { PreparedCteQuery } from './compile-cte';
 export { shortenSQL } from './shorten-sql';
 export type { CompiledQuery } from './types';