@mastra/duckdb 1.3.0-alpha.0 → 1.3.0

package/CHANGELOG.md

@@ -1,5 +1,65 @@
 # @mastra/duckdb
 
+## 1.3.0
+
+### Minor Changes
+
+- - **Added** `listBranches` and `getSpans` implementations. ([#16154](https://github.com/mastra-ai/mastra/pull/16154))
+  - Historical span data is queryable immediately; no migration required.
+
+- Added `count_distinct` aggregation and server-side TopK to the metrics storage API so dashboards built on high-cardinality fields (like `threadId` or `resourceId`) stay fast and bounded. ([#16137](https://github.com/mastra-ai/mastra/pull/16137))
+
+  **New aggregation**
+
+  `getMetricAggregate`, `getMetricBreakdown`, and `getMetricTimeSeries` accept `aggregation: 'count_distinct'` with a `distinctColumn`. Backends pick the most efficient native implementation — `uniq` on ClickHouse, `approx_count_distinct` on DuckDB.
+
+  `distinctColumn` is restricted to a low/medium-cardinality categorical allowlist (`entityType`, `entityName`, `parentEntityType`, `parentEntityName`, `rootEntityType`, `rootEntityName`, `name`, `provider`, `model`, `environment`, `executionSource`, `serviceName`). ID columns are not allowed — distinct counts over near-unique values converge to the row count and are rarely useful.
+
+  ```ts
+  await store.getMetricAggregate({
+    name: ['mastra_llm_tokens_total'],
+    aggregation: 'count_distinct',
+    distinctColumn: 'model',
+    filters: { timestamp: { start, end } },
+  });
+  ```
+
+  **Server-side TopK**
+
+  `getMetricBreakdown` accepts `limit` and `orderDirection`, so breakdowns never return the full cardinality of a column from the database. Ordering is always by the aggregated `value`; `orderDirection` flips between top-N (`DESC`, default) and bottom-N (`ASC`).
+
+  ```ts
+  await store.getMetricBreakdown({
+    name: ['mastra_agent_duration_ms'],
+    aggregation: 'sum',
+    groupBy: ['threadId'],
+    limit: 20,
+    orderDirection: 'DESC',
+  });
+  ```
+
+### Patch Changes
+
+- Improved performance of `listTraces` and `listBranches` on DuckDB. The Traces and Branches lists in the observability UI now load noticeably faster, especially on large span tables, because filtering and pagination happen up front and the store only assembles full span data for the rows on the page being viewed. ([#16165](https://github.com/mastra-ai/mastra/pull/16165))
+
+  No API or behavior changes — return shapes and filter semantics are unchanged, and no migration is required.
+
+- Added direct score lookup support to observability storage so score records can be fetched by `scoreId` without scanning paginated score lists, including DuckDB and ClickHouse vNext observability stores. ([#16162](https://github.com/mastra-ai/mastra/pull/16162))
+
+- Updated dependencies [[`6dcd65f`](https://github.com/mastra-ai/mastra/commit/6dcd65f2a34069e6dc43ba35f1d11119b9b40bef), [`86c0298`](https://github.com/mastra-ai/mastra/commit/86c0298e647306423c842f9d5ac827bd616bd13d), [`c05c9a1`](https://github.com/mastra-ai/mastra/commit/c05c9a13230988cef6d438a62f37760f31927bc7), [`ca28c23`](https://github.com/mastra-ai/mastra/commit/ca28c232a2f18801a6cf20fe053479237b4d4fb0), [`e24aacb`](https://github.com/mastra-ai/mastra/commit/e24aacba07bd66f5d95b636dc24016fca26b52cf), [`7679a63`](https://github.com/mastra-ai/mastra/commit/7679a634eae8e8ca459fd87538fdf72b4389b07f), [`7fce309`](https://github.com/mastra-ai/mastra/commit/7fce30912b14170bfc41f0ac736cca0f39fe0cd4), [`1d64a76`](https://github.com/mastra-ai/mastra/commit/1d64a765861a0772ea187bab76e5ed37bf82d042), [`1c2dda8`](https://github.com/mastra-ai/mastra/commit/1c2dda805fbfccc0abf55d4cb20cc34402dc3f0c), [`c721164`](https://github.com/mastra-ai/mastra/commit/c7211643f7ac861f83b19a3757cc921487fc9d75), [`1b55954`](https://github.com/mastra-ai/mastra/commit/1b559541c1e08a10e49d01ffc51a634dfc37a286), [`7997c2e`](https://github.com/mastra-ai/mastra/commit/7997c2e55ddd121562a4098cd8d2b89c68433bf1), [`5adc55e`](https://github.com/mastra-ai/mastra/commit/5adc55e63407be8ee977914957d68bcc2a075ceb), [`7679a63`](https://github.com/mastra-ai/mastra/commit/7679a634eae8e8ca459fd87538fdf72b4389b07f), [`a0d9b6d`](https://github.com/mastra-ai/mastra/commit/a0d9b6d6b810aeaa9e177a0dcc99a4402e609634), [`e97ccb9`](https://github.com/mastra-ai/mastra/commit/e97ccb900f8b7a390ce82c9f8eb8d6eb2c5e3777), [`c5daf48`](https://github.com/mastra-ai/mastra/commit/c5daf48556e98c46ae06caf00f92c249912007e9), [`70017d7`](https://github.com/mastra-ai/mastra/commit/70017d72ab741b5d7040e2a15c251a317782e39e), [`cd96779`](https://github.com/mastra-ai/mastra/commit/cd9677937f113b2856dc8b9f3d4bdabcee58bb2e), [`b0c7022`](https://github.com/mastra-ai/mastra/commit/b0c70224f80dad7c0cdbfb22cbff22e0f75c064f), [`e4942bc`](https://github.com/mastra-ai/mastra/commit/e4942bc7fdc903572f7d84f26d5e15f9d39c763d)]:
+  - @mastra/core@1.32.0
+
+## 1.3.0-alpha.1
+
+### Patch Changes
+
+- Improved performance of `listTraces` and `listBranches` on DuckDB. The Traces and Branches lists in the observability UI now load noticeably faster, especially on large span tables, because filtering and pagination happen up front and the store only assembles full span data for the rows on the page being viewed. ([#16165](https://github.com/mastra-ai/mastra/pull/16165))
+
+  No API or behavior changes — return shapes and filter semantics are unchanged, and no migration is required.
+
+- Updated dependencies [[`ca28c23`](https://github.com/mastra-ai/mastra/commit/ca28c232a2f18801a6cf20fe053479237b4d4fb0)]:
+  - @mastra/core@1.32.0-alpha.3
+
 ## 1.3.0-alpha.0
 
 ### Minor Changes

package/dist/docs/SKILL.md

@@ -3,7 +3,7 @@ name: mastra-duckdb
 description: Documentation for @mastra/duckdb. Use when working with @mastra/duckdb APIs, configuration, or implementation.
 metadata:
   package: "@mastra/duckdb"
-  version: "1.3.0-alpha.0"
+  version: "1.3.0"
 ---
 
 ## When to use

package/dist/docs/assets/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.3.0-alpha.0",
+  "version": "1.3.0",
   "package": "@mastra/duckdb",
   "exports": {
     "DuckDBConnection": {

package/dist/index.cjs

@@ -583,7 +583,7 @@ var ObservabilityStorageDuckDB = class extends storage.ObservabilityStorage {
       return null;
     }
     if (!this.loadPromise) {
-      this.loadPromise = import('./observability-YJW7WSXX.cjs').then(({ ObservabilityStorageDuckDB: ObservabilityStorageDuckDB2 }) => {
+      this.loadPromise = import('./observability-2PJ44OVC.cjs').then(({ ObservabilityStorageDuckDB: ObservabilityStorageDuckDB2 }) => {
         const delegate = new ObservabilityStorageDuckDB2({ db: this.db });
         this.delegate = delegate;
         return delegate;

package/dist/index.js

@@ -582,7 +582,7 @@ var ObservabilityStorageDuckDB = class extends ObservabilityStorage {
       return null;
     }
     if (!this.loadPromise) {
-      this.loadPromise = import('./observability-2R7ZOHEZ.js').then(({ ObservabilityStorageDuckDB: ObservabilityStorageDuckDB2 }) => {
+      this.loadPromise = import('./observability-7LL4LLXY.js').then(({ ObservabilityStorageDuckDB: ObservabilityStorageDuckDB2 }) => {
         const delegate = new ObservabilityStorageDuckDB2({ db: this.db });
         this.delegate = delegate;
         return delegate;

package/dist/{observability-YJW7WSXX.cjs → observability-2PJ44OVC.cjs}

@@ -2436,11 +2436,91 @@ function rowToSpanRecord(row) {
     updatedAt: null
   };
 }
-function buildHasChildErrorClause(hasChildError) {
+function buildHasChildErrorClause(hasChildError, rootAlias) {
   if (hasChildError === void 0) return "";
-  const base = `SELECT 1 FROM reconstructed_spans c WHERE c.traceId = root_spans.traceId AND c.spanId != root_spans.spanId AND c.error IS NOT NULL`;
+  const base = `SELECT 1 FROM span_events c WHERE c.traceId = ${rootAlias}.traceId AND c.spanId != ${rootAlias}.spanId AND c.error IS NOT NULL`;
   return hasChildError ? `EXISTS (${base})` : `NOT EXISTS (${base})`;
 }
+var PREFILTER_KEYS = /* @__PURE__ */ new Set([
+  "traceId",
+  "spanId",
+  "parentSpanId",
+  "name",
+  "spanType",
+  "source",
+  "entityType",
+  "entityId",
+  "entityName",
+  "entityVersionId",
+  "experimentId",
+  "userId",
+  "organizationId",
+  "resourceId",
+  "runId",
+  "sessionId",
+  "threadId",
+  "requestId",
+  "environment",
+  "serviceName"
+]);
+var SAFE_PREFILTER_ORDER_FIELDS = /* @__PURE__ */ new Set(["startedAt"]);
+function intersectTimestampRange(existing, incoming) {
+  if (!existing) return { ...incoming };
+  const merged = { ...existing };
+  if (incoming.start !== void 0) {
+    if (merged.start === void 0 || incoming.start.getTime() > merged.start.getTime()) {
+      merged.start = incoming.start;
+      merged.startExclusive = incoming.startExclusive;
+    } else if (incoming.start.getTime() === merged.start.getTime()) {
+      merged.startExclusive = (merged.startExclusive ?? false) || (incoming.startExclusive ?? false);
+    }
+  }
+  if (incoming.end !== void 0) {
+    if (merged.end === void 0 || incoming.end.getTime() < merged.end.getTime()) {
+      merged.end = incoming.end;
+      merged.endExclusive = incoming.endExclusive;
+    } else if (incoming.end.getTime() === merged.end.getTime()) {
+      merged.endExclusive = (merged.endExclusive ?? false) || (incoming.endExclusive ?? false);
+    }
+  }
+  return merged;
+}
+function partitionAnchorFilters(filters) {
+  const prefilter = {};
+  const postAgg = {};
+  let hasChildError;
+  for (const [key, value] of Object.entries(filters)) {
+    if (value === void 0 || value === null) continue;
+    if (key === "hasChildError") {
+      if (typeof value === "boolean") hasChildError = value;
+      continue;
+    }
+    if (key === "startedAt") {
+      prefilter.timestamp = intersectTimestampRange(
+        prefilter.timestamp,
+        value
+      );
+      continue;
+    }
+    if (key === "endedAt") {
+      postAgg.endedAt = value;
+      const dateRange = value;
+      if (dateRange?.end) {
+        prefilter.timestamp = intersectTimestampRange(prefilter.timestamp, {
+          end: dateRange.end,
+          endExclusive: dateRange.endExclusive
+        });
+      }
+      continue;
+    }
+    if (PREFILTER_KEYS.has(key)) {
+      prefilter[key] = value;
+      continue;
+    }
+    postAgg[key] = value;
+  }
+  return { prefilter, postAgg, hasChildError };
+}
 function toValuesTuple(row) {
   return [
     v(row.eventType),
@@ -2619,44 +2699,82 @@ async function listTraces(db, args) {
   const page = Number(args.pagination?.page ?? 0);
   const perPage = Number(args.pagination?.perPage ?? 10);
   const orderBy = { field: args.orderBy?.field ?? "startedAt", direction: args.orderBy?.direction ?? "DESC" };
-  const { clause: filterClause, params: filterParams } = buildWhereClause(filters);
-  const orderByClause = buildOrderByClause(orderBy);
-  const { clause: paginationClause, params: paginationParams } = buildPaginationClause({ page, perPage });
-  const filterParts = [];
-  if (filterClause) filterParts.push(filterClause.replace(/^WHERE\s+/i, ""));
-  const hasChildError = typeof filters.hasChildError === "boolean" ? filters.hasChildError : void 0;
-  const childErrorClause = buildHasChildErrorClause(hasChildError);
-  if (childErrorClause) filterParts.push(childErrorClause);
-  const combinedFilterClause = filterParts.length > 0 ? `WHERE ${filterParts.join(" AND ")}` : "";
-  const cteSql = `
-    WITH reconstructed_spans AS (
+  const { prefilter, postAgg, hasChildError } = partitionAnchorFilters(filters);
+  const { clause: prefilterClause, params: prefilterParams } = buildWhereClause(prefilter);
+  const prefilterParts = [`eventType = 'start'`, `parentSpanId IS NULL`];
+  if (prefilterClause) prefilterParts.push(prefilterClause.replace(/^WHERE\s+/i, ""));
+  const prefilterWhere = `WHERE ${prefilterParts.join(" AND ")}`;
+  const outerAlias = "outer_root";
+  const orderDir = orderBy.direction.toUpperCase();
+  if (orderDir !== "ASC" && orderDir !== "DESC") {
+    throw new Error(`Invalid sort direction: ${orderBy.direction}`);
+  }
+  const canOrderInPrefilter = SAFE_PREFILTER_ORDER_FIELDS.has(orderBy.field);
+  const hasPostAggFilters = Object.keys(postAgg).length > 0 || hasChildError !== void 0 || !canOrderInPrefilter;
+  if (!hasPostAggFilters) {
+    const prefilterOrderBy = `ORDER BY timestamp ${orderDir}`;
+    const offset = page * perPage;
+    const countSql2 = `
+      SELECT COUNT(*) as total
+      FROM span_events AS ${outerAlias}
+      ${prefilterWhere}
+    `;
+    const countResult2 = await db.query(countSql2, prefilterParams);
+    const total2 = Number(countResult2[0]?.total ?? 0);
+    const pageSql = `
+      WITH page_roots AS (
+        SELECT traceId, spanId
+        FROM span_events AS ${outerAlias}
+        ${prefilterWhere}
+        ${prefilterOrderBy}
+        LIMIT ? OFFSET ?
+      )
       ${SPAN_RECONSTRUCT_SELECT}
+      WHERE (traceId, spanId) IN (SELECT traceId, spanId FROM page_roots)
       GROUP BY traceId, spanId
+      ${buildOrderByClause(orderBy)}
+    `;
+    const rows2 = await db.query(pageSql, [...prefilterParams, perPage, offset]);
+    const spans2 = rows2.map((row) => rowToSpanRecord(row));
+    return {
+      pagination: { total: total2, page, perPage, hasMore: (page + 1) * perPage < total2 },
+      spans: storage.toTraceSpans(spans2)
+    };
+  }
+  const { clause: postAggClause, params: postAggParams } = buildWhereClause(postAgg);
+  const postAggParts = [];
+  if (postAggClause) postAggParts.push(postAggClause.replace(/^WHERE\s+/i, ""));
+  const childErrorClause = buildHasChildErrorClause(hasChildError, "root_spans");
+  if (childErrorClause) postAggParts.push(childErrorClause);
+  const postAggWhere = postAggParts.length > 0 ? `WHERE ${postAggParts.join(" AND ")}` : "";
+  const cteSql = `
+    WITH candidate_roots AS (
+      SELECT traceId, spanId
+      FROM span_events AS ${outerAlias}
+      ${prefilterWhere}
     ),
     root_spans AS (
-      SELECT * FROM reconstructed_spans
-      WHERE parentSpanId IS NULL
+      ${SPAN_RECONSTRUCT_SELECT}
+      WHERE (traceId, spanId) IN (SELECT traceId, spanId FROM candidate_roots)
+      GROUP BY traceId, spanId
     )
   `;
+  const orderByClause = buildOrderByClause(orderBy);
+  const { clause: paginationClause, params: paginationParams } = buildPaginationClause({ page, perPage });
   const countSql = `
     ${cteSql}
-    SELECT COUNT(*) as total FROM root_spans ${combinedFilterClause}
+    SELECT COUNT(*) as total FROM root_spans ${postAggWhere}
   `;
-  const countResult = await db.query(countSql, filterParams);
+  const countResult = await db.query(countSql, [...prefilterParams, ...postAggParams]);
   const total = Number(countResult[0]?.total ?? 0);
   const dataSql = `
     ${cteSql}
-    SELECT * FROM root_spans ${combinedFilterClause} ${orderByClause} ${paginationClause}
+    SELECT * FROM root_spans ${postAggWhere} ${orderByClause} ${paginationClause}
   `;
-  const rows = await db.query(dataSql, [...filterParams, ...paginationParams]);
+  const rows = await db.query(dataSql, [...prefilterParams, ...postAggParams, ...paginationParams]);
   const spans = rows.map((row) => rowToSpanRecord(row));
   return {
-    pagination: {
-      total,
-      page,
-      perPage,
-      hasMore: (page + 1) * perPage < total
-    },
+    pagination: { total, page, perPage, hasMore: (page + 1) * perPage < total },
     spans: storage.toTraceSpans(spans)
   };
 }
@@ -2682,37 +2800,92 @@ async function listBranches(db, args) {
   const page = Number(args.pagination?.page ?? 0);
   const perPage = Number(args.pagination?.perPage ?? 10);
   const orderBy = { field: args.orderBy?.field ?? "startedAt", direction: args.orderBy?.direction ?? "DESC" };
-  const { spanType, ...passthroughFilters } = filters;
-  const { clause: filterClause, params: filterParams } = buildWhereClause(passthroughFilters);
-  const orderByClause = buildOrderByClause(orderBy);
-  const { clause: paginationClause, params: paginationParams } = buildPaginationClause({ page, perPage });
-  let prefilterClause;
-  let prefilterParams;
-  if (typeof spanType === "string") {
-    if (!storage.BRANCH_SPAN_TYPES.includes(spanType)) {
+  const userSpanType = filters.spanType;
+  if (typeof userSpanType === "string" && !storage.BRANCH_SPAN_TYPES.includes(userSpanType)) {
+    return {
+      pagination: { total: 0, page, perPage, hasMore: false },
+      branches: []
+    };
+  }
+  const { spanType: _spanType, ...rest } = filters;
+  const { prefilter, postAgg} = partitionAnchorFilters(rest);
+  const { clause: prefilterClause, params: prefilterFilterParams } = buildWhereClause(prefilter);
+  const prefilterParts = [`eventType = 'start'`];
+  let spanTypeParams;
+  if (typeof userSpanType === "string") {
+    prefilterParts.push(`spanType = ?`);
+    spanTypeParams = [userSpanType];
+  } else {
+    prefilterParts.push(`spanType IN (${BRANCH_SPAN_TYPE_PLACEHOLDERS})`);
+    spanTypeParams = [...storage.BRANCH_SPAN_TYPES];
+  }
+  if (prefilterClause) prefilterParts.push(prefilterClause.replace(/^WHERE\s+/i, ""));
+  const prefilterWhere = `WHERE ${prefilterParts.join(" AND ")}`;
+  const prefilterParams = [...spanTypeParams, ...prefilterFilterParams];
+  const outerAlias = "outer_anchor";
+  const orderDir = orderBy.direction.toUpperCase();
+  if (orderDir !== "ASC" && orderDir !== "DESC") {
+    throw new Error(`Invalid sort direction: ${orderBy.direction}`);
+  }
+  const canOrderInPrefilter = SAFE_PREFILTER_ORDER_FIELDS.has(orderBy.field);
+  const hasPostAggFilters = Object.keys(postAgg).length > 0 || !canOrderInPrefilter;
+  if (!hasPostAggFilters) {
+    const prefilterOrderBy = `ORDER BY timestamp ${orderDir}`;
+    const offset = page * perPage;
+    const countSql2 = `
+      SELECT COUNT(*) as total
+      FROM span_events AS ${outerAlias}
+      ${prefilterWhere}
+    `;
+    const countResult2 = await db.query(countSql2, prefilterParams);
+    const total2 = Number(countResult2[0]?.total ?? 0);
+    if (total2 === 0) {
       return {
         pagination: { total: 0, page, perPage, hasMore: false },
         branches: []
       };
     }
-    prefilterClause = `WHERE spanType = ?`;
-    prefilterParams = [spanType];
-  } else {
-    prefilterClause = `WHERE spanType IN (${BRANCH_SPAN_TYPE_PLACEHOLDERS})`;
-    prefilterParams = [...storage.BRANCH_SPAN_TYPES];
+    const pageSql = `
+      WITH page_anchors AS (
+        SELECT traceId, spanId
+        FROM span_events AS ${outerAlias}
+        ${prefilterWhere}
+        ${prefilterOrderBy}
+        LIMIT ? OFFSET ?
+      )
+      ${SPAN_RECONSTRUCT_SELECT}
+      WHERE (traceId, spanId) IN (SELECT traceId, spanId FROM page_anchors)
+      GROUP BY traceId, spanId
+      ${buildOrderByClause(orderBy)}
+    `;
+    const rows2 = await db.query(pageSql, [...prefilterParams, perPage, offset]);
+    const spans2 = rows2.map((row) => rowToSpanRecord(row));
+    return {
+      pagination: { total: total2, page, perPage, hasMore: (page + 1) * perPage < total2 },
+      branches: storage.toTraceSpans(spans2)
+    };
   }
+  const { clause: postAggClause, params: postAggParams } = buildWhereClause(postAgg);
+  const postAggWhere = postAggClause ? postAggClause : "";
+  const orderByClause = buildOrderByClause(orderBy);
+  const { clause: paginationClause, params: paginationParams } = buildPaginationClause({ page, perPage });
   const cteSql = `
-    WITH branch_anchors AS (
+    WITH candidate_anchors AS (
+      SELECT traceId, spanId
+      FROM span_events AS ${outerAlias}
+      ${prefilterWhere}
+    ),
+    branch_anchors AS (
       ${SPAN_RECONSTRUCT_SELECT}
-      ${prefilterClause}
+      WHERE (traceId, spanId) IN (SELECT traceId, spanId FROM candidate_anchors)
       GROUP BY traceId, spanId
     )
   `;
   const countSql = `
     ${cteSql}
-    SELECT COUNT(*) as total FROM branch_anchors ${filterClause}
+    SELECT COUNT(*) as total FROM branch_anchors ${postAggWhere}
   `;
-  const countResult = await db.query(countSql, [...prefilterParams, ...filterParams]);
+  const countResult = await db.query(countSql, [...prefilterParams, ...postAggParams]);
   const total = Number(countResult[0]?.total ?? 0);
   if (total === 0) {
     return {
@@ -2722,17 +2895,12 @@ async function listBranches(db, args) {
   }
   const dataSql = `
     ${cteSql}
-    SELECT * FROM branch_anchors ${filterClause} ${orderByClause} ${paginationClause}
+    SELECT * FROM branch_anchors ${postAggWhere} ${orderByClause} ${paginationClause}
   `;
-  const rows = await db.query(dataSql, [...prefilterParams, ...filterParams, ...paginationParams]);
+  const rows = await db.query(dataSql, [...prefilterParams, ...postAggParams, ...paginationParams]);
   const spans = rows.map((row) => rowToSpanRecord(row));
   return {
-    pagination: {
-      total,
-      page,
-      perPage,
-      hasMore: (page + 1) * perPage < total
-    },
+    pagination: { total, page, perPage, hasMore: (page + 1) * perPage < total },
     branches: storage.toTraceSpans(spans)
   };
 }
@@ -2958,5 +3126,5 @@ var ObservabilityStorageDuckDB = class extends storage.ObservabilityStorage {
 };
 
 exports.ObservabilityStorageDuckDB = ObservabilityStorageDuckDB;
-//# sourceMappingURL=observability-YJW7WSXX.cjs.map
-//# sourceMappingURL=observability-YJW7WSXX.cjs.map
+//# sourceMappingURL=observability-2PJ44OVC.cjs.map
+//# sourceMappingURL=observability-2PJ44OVC.cjs.map