@mastra/clickhouse 1.14.0 → 1.15.0-alpha.1

package/CHANGELOG.md

@@ -1,5 +1,47 @@
 # @mastra/clickhouse
 
+## 1.15.0-alpha.1
+
+### Patch Changes
+
+- Fixed resource-scoped message includes across storage adapters so included context cannot cross resource boundaries. ([#20984](https://github.com/mastra-ai/mastra/pull/20984))
+
+- Updated dependencies [[`6445eba`](https://github.com/mastra-ai/mastra/commit/6445eba6020abac681aba1cc9289f446cb400cbe), [`df31eb0`](https://github.com/mastra-ai/mastra/commit/df31eb0c7087d782a0d9346e467f9a4af4b0eef6), [`fcd0667`](https://github.com/mastra-ai/mastra/commit/fcd0667a4e378be35c9a1b1eb19cce78fbfd7282), [`bab06b1`](https://github.com/mastra-ai/mastra/commit/bab06b18923873a584bdfc71a6b4ec7fb4727fb7)]:
+  - @mastra/core@1.58.0-alpha.5
+
+## 1.15.0-alpha.0
+
+### Minor Changes
+
+- Memory list reads now surface database errors instead of silently returning empty results. ([#17910](https://github.com/mastra-ai/mastra/pull/17910))
+
+  Previously, the paginated memory reads (`listThreads`, `listMessages`, `listMessagesByResourceId`, and `listMessagesById`) caught backend failures, logged them, and returned an empty payload like `{ threads: [], total: 0, hasMore: false }`. A transient outage (locked table, dropped connection) was therefore indistinguishable from a genuinely empty result, so an agent reading conversation history during a brief failure would treat it as "no history" and could overwrite real state. These methods now re-throw the failure as a `MastraError`. Validation (USER) errors and genuinely empty results are unchanged.
+
+  **Behavior change**
+
+  Callers that previously received an empty result on a backend failure will now receive a thrown `MastraError`. If you call these read methods directly (rather than through an agent, which already surfaces errors), wrap them so a transient outage doesn't crash the caller:
+
+  ```ts
+  try {
+    const { threads } = await storage.listThreads({ resourceId });
+    // ...use threads
+  } catch (error) {
+    // a real backend failure. Decide whether to retry, surface, or degrade.
+    // An empty thread list no longer hides here; it only means "no threads".
+  }
+  ```
+
+### Patch Changes
+
+- Fixed the ClickHouse lightweight trace list: responses no longer carry the `input`, `output` and `attributes` payload blobs, and delta polling now works. ([#20677](https://github.com/mastra-ai/mastra/pull/20677))
+
+  `listTracesLight` rows now carry a short `inputPreview` in place of the full input, plus the span `metadata` and a computed `status`, so Studio's configurable trace columns keep working — in both page and delta mode. Response payloads stay small as prompts grow. There is no schema change and no migration.
+
+  Requires `@mastra/core` >= 1.57.0, which ships the shared `buildInputPreview` and `computeTraceStatus` helpers this store now imports (peer dependency bumped accordingly).
+
+- Updated dependencies [[`e7109ee`](https://github.com/mastra-ai/mastra/commit/e7109ee6f731bacc79c885906f3c7dca8d8f013a), [`772c0c8`](https://github.com/mastra-ai/mastra/commit/772c0c897cec383258de2e6178147f8014767c7b), [`578bf2e`](https://github.com/mastra-ai/mastra/commit/578bf2e6a88e9d5b8bf502204e15a95dfbb679ae), [`06b2d87`](https://github.com/mastra-ai/mastra/commit/06b2d87e63bcdd0ed59215c6789692b9b12de376), [`ac01d63`](https://github.com/mastra-ai/mastra/commit/ac01d6355974aec73fdb8781449ed12bac582094), [`a810a05`](https://github.com/mastra-ai/mastra/commit/a810a058f62ad407cfc1701e0be36ae91145d7cf), [`f8da216`](https://github.com/mastra-ai/mastra/commit/f8da21633e7eb0e31c9ce0fc30567870d19416d3), [`6104347`](https://github.com/mastra-ai/mastra/commit/61043473ba6bfd0a25156824e853e13165562e6c), [`45bfb88`](https://github.com/mastra-ai/mastra/commit/45bfb88fd52f1dd3be20e2a38905777c96499c90), [`e3b9307`](https://github.com/mastra-ai/mastra/commit/e3b9307098daefbfae2a52ae2ef51bc9fc701190), [`d6834c5`](https://github.com/mastra-ai/mastra/commit/d6834c5a7866b16734d23900163c2414ed70d791), [`c52d346`](https://github.com/mastra-ai/mastra/commit/c52d3462ec831a5d95926ecd3d3373f5928ad2e5), [`0023e79`](https://github.com/mastra-ai/mastra/commit/0023e7919431078280abd11c89d1edeae35fcc69), [`c2ad51e`](https://github.com/mastra-ai/mastra/commit/c2ad51e2467f901eecba8c9f4a45e22a50bd7c18), [`3dc97ea`](https://github.com/mastra-ai/mastra/commit/3dc97ea415fad353b48a13095fad1835933cc12a), [`3d01cd3`](https://github.com/mastra-ai/mastra/commit/3d01cd387321b6f9c5cac31d487c84bf51b19c78), [`7bf3086`](https://github.com/mastra-ai/mastra/commit/7bf308663f0115ca74ad20554ade740f06640859), [`a8dd139`](https://github.com/mastra-ai/mastra/commit/a8dd1391a9fe9a6632c25809ef236980afa9a020), [`e5786be`](https://github.com/mastra-ai/mastra/commit/e5786be02bb903073082bd9d6da880ebaacc343f), [`2093fbd`](https://github.com/mastra-ai/mastra/commit/2093fbd53bb744bae19ec89f6d73db9a66fbe8a7), [`e7a5da4`](https://github.com/mastra-ai/mastra/commit/e7a5da4ef8e4dd452d2f232961b4e682a85ffe43), [`7b4393d`](https://github.com/mastra-ai/mastra/commit/7b4393d557411fdcf07b0e30e5acaf7cc85154ae)]:
+  - @mastra/core@1.58.0-alpha.1
+
 ## 1.14.0
 
 ### Minor Changes

package/dist/docs/SKILL.md

@@ -3,7 +3,7 @@ name: mastra-clickhouse
 description: Documentation for @mastra/clickhouse. Use when working with @mastra/clickhouse APIs, configuration, or implementation.
 metadata:
   package: "@mastra/clickhouse"
-  version: "1.14.0"
+  version: "1.15.0-alpha.1"
 ---
 
 ## When to use

package/dist/docs/assets/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.14.0",
+  "version": "1.15.0-alpha.1",
   "package": "@mastra/clickhouse",
   "exports": {},
   "modules": {}

package/dist/docs/references/reference-storage-composite.md

@@ -251,6 +251,64 @@ const memoryStore = await storage.getStore('memory')
 const thread = await memoryStore?.getThreadById({ threadId: '...' })
 ```
 
+## Closing connections
+
+`close()` releases the connections of the stores a composite was built from: the `default` and `editor` stores, plus any domain that owns its own client. Each store is closed once, even when it backs several domains. When passed to the Mastra class, `close()` is called by `shutdown()`:
+
+```typescript
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { PostgresStore } from '@mastra/pg'
+import { Mastra } from '@mastra/core'
+
+const pgStore = new PostgresStore({
+  id: 'pg-storage',
+  connectionString: process.env.DATABASE_URL,
+})
+
+export const mastra = new Mastra({
+  storage: new MastraCompositeStore({ id: 'composite', default: pgStore }),
+})
+
+process.on('SIGTERM', async () => {
+  // Releases the Postgres pool, so the process can exit
+  await mastra.shutdown()
+})
+```
+
+A store you construct only to supply a domain isn't reachable through the composite. Keep a reference to it and close it yourself:
+
+```typescript
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { ClickhouseStore } from '@mastra/clickhouse'
+import { PostgresStore } from '@mastra/pg'
+import { Mastra } from '@mastra/core'
+
+const pgStore = new PostgresStore({
+  id: 'pg-storage',
+  connectionString: process.env.DATABASE_URL,
+})
+
+const clickhouseStore = new ClickhouseStore({
+  id: 'clickhouse-storage',
+  url: process.env.CLICKHOUSE_URL,
+  username: process.env.CLICKHOUSE_USERNAME,
+  password: process.env.CLICKHOUSE_PASSWORD,
+})
+
+export const mastra = new Mastra({
+  storage: new MastraCompositeStore({
+    id: 'composite',
+    default: pgStore,
+    domains: { observability: clickhouseStore.stores?.observability },
+  }),
+})
+
+process.on('SIGTERM', async () => {
+  await mastra.shutdown()
+  await clickhouseStore.close()
+})
+```
+
 ## Use cases
 
 ### Separate databases for different workloads

package/dist/index.cjs

@@ -1185,7 +1185,10 @@ var MemoryStorageClickhouse = class extends _mastra_core_storage.MemoryStorage {
 				hasMore: false
 			};
 			if (perPageForQuery === 0 && include && include.length > 0) {
-				const includeResult = await this._getIncludedMessages({ include });
+				const includeResult = await this._getIncludedMessages({
+					include,
+					resourceId
+				});
 				const list = new _mastra_core_agent.MessageList().add(includeResult, "memory");
 				return {
 					messages: this._sortMessages(list.get.all.db(), field, direction),
@@ -1252,7 +1255,10 @@ var MemoryStorageClickhouse = class extends _mastra_core_storage.MemoryStorage {
 			};
 			const messageIds = new Set(paginatedMessages.map((m) => m.id));
 			if (include && include.length > 0) {
-				const includeMessages = await this._getIncludedMessages({ include });
+				const includeMessages = await this._getIncludedMessages({
+					include,
+					resourceId
+				});
 				for (const includeMsg of includeMessages) if (!messageIds.has(includeMsg.id)) {
 					paginatedMessages.push(includeMsg);
 					messageIds.add(includeMsg.id);
@@ -1270,6 +1276,7 @@ var MemoryStorageClickhouse = class extends _mastra_core_storage.MemoryStorage {
 				hasMore: metadataFilter && perPageForResponse !== false ? offset + paginatedCount < total : perPageForResponse === false ? false : allThreadMessagesReturned ? false : offset + paginatedCount < total
 			};
 		} catch (error) {
+			if (error instanceof _mastra_core_error.MastraError && error.category === _mastra_core_error.ErrorCategory.USER) throw error;
 			const mastraError = new _mastra_core_error.MastraError({
 				id: (0, _mastra_core_storage.createStorageErrorId)("CLICKHOUSE", "LIST_MESSAGES", "FAILED"),
 				domain: _mastra_core_error.ErrorDomain.STORAGE,
@@ -1281,13 +1288,7 @@ var MemoryStorageClickhouse = class extends _mastra_core_storage.MemoryStorage {
 			}, error);
 			this.logger?.error?.(mastraError.toString());
 			this.logger?.trackException?.(mastraError);
-			return {
-				messages: [],
-				total: 0,
-				page,
-				perPage: perPageForResponse,
-				hasMore: false
-			};
+			throw mastraError;
 		}
 	}
 	_sortMessages(messages, field, direction) {
@@ -1300,15 +1301,24 @@ var MemoryStorageClickhouse = class extends _mastra_core_storage.MemoryStorage {
 			return direction === "ASC" ? String(aValue).localeCompare(String(bValue)) : String(bValue).localeCompare(String(aValue));
 		});
 	}
-	async _getIncludedMessages({ include }) {
+	/**
+	* Fetches the messages named by `include` together with their surrounding context.
+	*
+	* @param include - Message ids to pin, each with an optional before/after window.
+	* @param resourceId - When set, restricts both the pinned messages and their context
+	* to that resource so an id from another resource returns nothing.
+	*/
+	async _getIncludedMessages({ include, resourceId }) {
 		if (!include || include.length === 0) return [];
 		const targetIds = include.map((inc) => inc.id).filter(Boolean);
 		if (targetIds.length === 0) return [];
 		const { messages: targetDocs } = await this.listMessagesById({ messageIds: targetIds });
-		const targetMap = new Map(targetDocs.map((msg) => [msg.id, {
+		const scopedTargetDocs = resourceId ? targetDocs.filter((msg) => msg.resourceId === resourceId) : targetDocs;
+		const targetMap = new Map(scopedTargetDocs.map((msg) => [msg.id, {
 			threadId: msg.threadId,
 			createdAt: msg.createdAt
 		}]));
+		const resourceCondition = resourceId ? ` AND "resourceId" = {var_resource:String}` : "";
 		const unionQueries = [];
 		const params = {};
 		let paramIdx = 1;
@@ -1323,7 +1333,7 @@ var MemoryStorageClickhouse = class extends _mastra_core_storage.MemoryStorage {
         SELECT id, content, role, type, "createdAt", thread_id AS "threadId", "resourceId"
         FROM "${_mastra_core_storage.TABLE_MESSAGES}"
         WHERE thread_id = {${threadParam}:String}
-          AND createdAt <= parseDateTime64BestEffort({${createdAtParam}:String}, 3)
+          AND createdAt <= parseDateTime64BestEffort({${createdAtParam}:String}, 3)${resourceCondition}
         ORDER BY createdAt DESC, id DESC
         LIMIT {${limitParam}:Int64}
       `);
@@ -1339,7 +1349,7 @@ var MemoryStorageClickhouse = class extends _mastra_core_storage.MemoryStorage {
           SELECT id, content, role, type, "createdAt", thread_id AS "threadId", "resourceId"
           FROM "${_mastra_core_storage.TABLE_MESSAGES}"
           WHERE thread_id = {${threadParam2}:String}
-            AND createdAt > parseDateTime64BestEffort({${createdAtParam2}:String}, 3)
+            AND createdAt > parseDateTime64BestEffort({${createdAtParam2}:String}, 3)${resourceCondition}
           ORDER BY createdAt ASC, id ASC
           LIMIT {${limitParam2}:Int64}
         `);
@@ -1350,6 +1360,7 @@ var MemoryStorageClickhouse = class extends _mastra_core_storage.MemoryStorage {
 			}
 		}
 		if (unionQueries.length === 0) return [];
+		if (resourceId) params.var_resource = resourceId;
 		let finalQuery;
 		if (unionQueries.length === 1) finalQuery = unionQueries[0];
 		else finalQuery = `SELECT * FROM (${unionQueries.join(" UNION ALL ")}) ORDER BY "createdAt" ASC, id ASC`;
@@ -3873,6 +3884,7 @@ function rowToSpanRecord(row) {
 function rowToLightSpanRecord(row) {
 	const startedAt = toDate(row.startedAt);
 	const endedAt = row.isEvent ? startedAt : toDateOrNull(row.endedAt);
+	const error = parseJson(row.error) ?? void 0;
 	return {
 		traceId: row.traceId,
 		spanId: row.spanId,
@@ -3885,7 +3897,13 @@ function rowToLightSpanRecord(row) {
 		entityType: nullableEntityType(row.entityType),
 		entityId: nullableString(row.entityId),
 		entityName: nullableString(row.entityName),
-		error: parseJson(row.error) ?? void 0,
+		error,
+		status: (0, _mastra_core_storage.computeTraceStatus)({
+			error,
+			endedAt
+		}),
+		metadata: parseJson(row.metadataRaw) ?? void 0,
+		inputPreview: (0, _mastra_core_storage.buildInputPreview)(row.input ?? null),
 		createdAt: startedAt,
 		updatedAt: null
 	};
@@ -6208,93 +6226,48 @@ async function getRootSpan(client, args) {
 	return { span: rowToSpanRecord(rows[0]) };
 }
 /**
-* Shared page-mode helper used by listTracesLight.
+* Columns a trace list renders. `input` is selected only so the row mapper can
+* derive a short `inputPreview` at read time (see `rowToLightSpanRecord`); the
+* raw blob itself never leaves the store.
 */
-async function listTraceRows(client, args, selectClause, mapRows) {
-	const { filters, pagination, orderBy } = _mastra_core_storage.listTracesArgsSchema.parse(args);
-	const page = pagination?.page ?? 0;
-	const perPage = pagination?.perPage ?? 10;
-	const { conditions, params } = buildTraceFilterConditions(filters, "r");
-	if (filters?.hasChildError != null) if (filters.hasChildError) conditions.push(`EXISTS (
-        SELECT 1 FROM ${TABLE_SPAN_EVENTS} c
-        WHERE c.traceId = r.traceId
-          AND c.parentSpanId IS NOT NULL
-          AND c.error IS NOT NULL
-      )`);
-	else conditions.push(`NOT EXISTS (
-        SELECT 1 FROM ${TABLE_SPAN_EVENTS} c
-        WHERE c.traceId = r.traceId
-          AND c.parentSpanId IS NOT NULL
-          AND c.error IS NOT NULL
-      )`);
-	const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
-	const orderClause = buildTraceOrderByClause(orderBy);
-	const countRows = await (await client.query({
-		query: `
-      SELECT count() as cnt FROM (
-        SELECT dedupeKey
-        FROM ${TABLE_TRACE_ROOTS} r
-        ${whereClause}
-        ORDER BY dedupeKey
-        LIMIT 1 BY dedupeKey
-      )
-    `,
-		query_params: params,
-		format: "JSONEachRow",
-		clickhouse_settings: CH_SETTINGS
-	})).json();
-	const total = Number(countRows[0]?.cnt ?? 0);
-	if (total === 0) return {
-		pagination: {
-			total: 0,
-			page,
-			perPage,
-			hasMore: false
-		},
-		spans: []
-	};
-	const rows = await (await client.query({
-		query: `
-      SELECT ${selectClause} FROM (
-        SELECT ${selectClause}
-        FROM ${TABLE_TRACE_ROOTS} r
-        ${whereClause}
-        ORDER BY dedupeKey
-        LIMIT 1 BY dedupeKey
-      )
-      ORDER BY ${orderClause}
-      LIMIT {limit:UInt32}
-      OFFSET {offset:UInt32}
-    `,
-		query_params: {
-			...params,
-			limit: perPage,
-			offset: page * perPage
-		},
-		format: "JSONEachRow",
-		clickhouse_settings: CH_SETTINGS
-	})).json();
-	return {
-		pagination: {
-			total,
-			page,
-			perPage,
-			hasMore: (page + 1) * perPage < total
-		},
-		spans: mapRows(rows)
-	};
-}
+const LIGHT_TRACE_ROOT_FIELDS = [
+	"traceId",
+	"spanId",
+	"parentSpanId",
+	"name",
+	"spanType",
+	"isEvent",
+	"startedAt",
+	"endedAt",
+	"entityType",
+	"entityId",
+	"entityName",
+	"error",
+	"metadataRaw",
+	"input"
+];
+const FULL_PROJECTION = {
+	innerSelect: "*",
+	outerSelect: "*",
+	deltaSelect: "r.* EXCEPT(startedAt, traceId, dedupeKey)"
+};
+const LIGHT_PROJECTION = {
+	innerSelect: ["dedupeKey", ...LIGHT_TRACE_ROOT_FIELDS].join(", "),
+	outerSelect: LIGHT_TRACE_ROOT_FIELDS.join(", "),
+	deltaSelect: LIGHT_TRACE_ROOT_FIELDS.filter((field) => field !== "startedAt" && field !== "traceId").map((field) => `r.${field}`).join(", ")
+};
 /**
-* List traces with optional filtering, pagination, and ordering.
+* Shared implementation behind listTraces and listTracesLight.
 *
 * Reads from trace_roots (root spans only).
-* Uses two-stage query for ReplacingMergeTree deduplication:
+* Page mode uses a two-stage query for ReplacingMergeTree deduplication:
 *   Inner: filter + deterministic ORDER BY + LIMIT 1 BY dedupeKey
 *   Outer: final ordering + pagination
+* Delta mode joins the append-only delta table and returns only rows past the cursor.
 *
 * hasChildError is handled via EXISTS subquery against span_events.
 */
-async function listTraces(client, args, strategy) {
+async function listTraceRows(client, args, strategy, projection, mapRows) {
 	const { mode, filters, pagination, orderBy, after, limit } = _mastra_core_storage.listTracesArgsSchema.parse(args);
 	const page = pagination?.page ?? 0;
 	const perPage = pagination?.perPage ?? 10;
@@ -6324,10 +6297,11 @@ async function listTraces(client, args, strategy) {
 			},
 			deltaCursor: streamHeadCursor
 		};
-		const rows = await queryTracesAfterCursor(client, whereClause, params, limit, validateCursorId(after));
+		const afterCursor = validateCursorId(after);
+		const rows = await queryTracesAfterCursor(client, projection.deltaSelect, whereClause, params, limit, afterCursor);
 		const visibleRows = rows.slice(0, limit);
 		return {
-			spans: (0, _mastra_core_storage.toTraceSpans)(visibleRows.map(rowToSpanRecord)),
+			spans: mapRows(visibleRows),
 			delta: {
 				limit,
 				hasMore: rows.length > limit
@@ -6362,10 +6336,10 @@ async function listTraces(client, args, strategy) {
 		spans: [],
 		...deltaCursorEnabled ? { deltaCursor: currentDeltaCursor } : {}
 	};
-	const spans = (await (await client.query({
+	const rows = await (await client.query({
 		query: `
-      SELECT * FROM (
-        SELECT *
+      SELECT ${projection.outerSelect} FROM (
+        SELECT ${projection.innerSelect}
         FROM ${TABLE_TRACE_ROOTS} r
         ${whereClause}
         ORDER BY dedupeKey
@@ -6382,7 +6356,7 @@ async function listTraces(client, args, strategy) {
 		},
 		format: "JSONEachRow",
 		clickhouse_settings: CH_SETTINGS
-	})).json()).map(rowToSpanRecord);
+	})).json();
 	return {
 		pagination: {
 			total,
@@ -6390,32 +6364,27 @@ async function listTraces(client, args, strategy) {
 			perPage,
 			hasMore: (page + 1) * perPage < total
 		},
-		spans: (0, _mastra_core_storage.toTraceSpans)(spans),
+		spans: mapRows(rows),
 		...deltaCursorEnabled ? { deltaCursor: currentDeltaCursor } : {}
 	};
 }
-const LIGHT_TRACE_ROOT_COLUMNS = [
-	"traceId",
-	"spanId",
-	"parentSpanId",
-	"name",
-	"spanType",
-	"isEvent",
-	"startedAt",
-	"endedAt",
-	"entityType",
-	"entityId",
-	"entityName",
-	"error"
-].join(", ");
-async function listTracesLight(client, args) {
-	return listTraceRows(client, args, LIGHT_TRACE_ROOT_COLUMNS, (rows) => rows.map(rowToLightSpanRecord));
+/** List traces with optional filtering, pagination, and ordering. */
+async function listTraces(client, args, strategy) {
+	return listTraceRows(client, args, strategy, FULL_PROJECTION, (rows) => (0, _mastra_core_storage.toTraceSpans)(rows.map(rowToSpanRecord)));
+}
+/**
+* List traces projecting only the columns a trace list renders.
+* Skips the attributes/output blobs and reduces `input` to a short preview in
+* the mapper, so the response payload stays flat as traces grow.
+*/
+async function listTracesLight(client, args, strategy) {
+	return listTraceRows(client, args, strategy, LIGHT_PROJECTION, (rows) => rows.map(rowToLightSpanRecord));
 }
-async function queryTracesAfterCursor(client, whereClause, params, limit, cursorId) {
+async function queryTracesAfterCursor(client, deltaSelect, whereClause, params, limit, cursorId) {
 	return await (await client.query({
 		query: `
         SELECT
-          r.* EXCEPT(startedAt, traceId, dedupeKey),
+          ${deltaSelect},
           r.startedAt AS startedAt,
           r.traceId AS traceId,
           r.dedupeKey AS dedupeKey,
@@ -7392,7 +7361,7 @@ var ObservabilityStorageClickhouseVNext = class extends _mastra_core_storage.Obs
 	}
 	async listTracesLight(args) {
 		try {
-			return await listTracesLight(this.#client, args);
+			return await listTracesLight(this.#client, args, this.#deltaCursorStrategy);
 		} catch (error) {
 			if (error instanceof _mastra_core_error.MastraError) throw error;
 			throw new _mastra_core_error.MastraError({