@mastra/clickhouse 1.5.1-alpha.1 → 1.5.2-alpha.0

package/CHANGELOG.md

@@ -1,5 +1,27 @@
 # @mastra/clickhouse
 
+## 1.5.2-alpha.0
+
+### Patch Changes
+
+- Changed ClickHouse background task deletes to use lightweight `DELETE FROM` instead of `ALTER TABLE ... DELETE` mutations with `mutations_sync`. This makes deleted rows immediately invisible to subsequent reads without forcing part rewrites for each delete. ([#15902](https://github.com/mastra-ai/mastra/pull/15902))
+
+  Improved bulk background task deletion to push filtering into ClickHouse instead of fetching all matching task IDs into Node.js memory first. This avoids unnecessary network transfer and out-of-memory risk when deleting large result sets. As a safety guard, calling `deleteTasks` with no filters is now a no-op — use `dangerouslyClearAll` to wipe the table.
+
+- Updated dependencies [[`703a443`](https://github.com/mastra-ai/mastra/commit/703a44390c587d9c0b8ae94ec4edd8afb2a74044), [`808df1b`](https://github.com/mastra-ai/mastra/commit/808df1b39358b5f10b7317107e42b1fda7c87185)]:
+  - @mastra/core@1.29.1-alpha.1
+
+## 1.5.1
+
+### Patch Changes
+
+- Fixed `mastra dev` repeatedly reporting `MIGRATION REQUIRED` on ClickHouse Cloud after `mastra migrate` had already run successfully. The observability migration check now recognizes the engine-name variants that ClickHouse Cloud and replicated clusters use in place of `ReplacingMergeTree`. ([#15623](https://github.com/mastra-ai/mastra/pull/15623))
+
+- Improved ClickHouse v-next observability initialization errors to include the underlying ClickHouse message in the standard error text. This makes init failures actionable in loggers that only print `error.message`. ([#15588](https://github.com/mastra-ai/mastra/pull/15588))
+
+- Updated dependencies [[`f112db1`](https://github.com/mastra-ai/mastra/commit/f112db179557ae9b5a0f1d25dc47f928d7d61cd9), [`21d9706`](https://github.com/mastra-ai/mastra/commit/21d970604d89eee970cbf8013d26d7551aff6ea5), [`0a0aa94`](https://github.com/mastra-ai/mastra/commit/0a0aa94729592e99885af2efb90c56aaada62247), [`ed07df3`](https://github.com/mastra-ai/mastra/commit/ed07df32a9d539c8261e892fc1bade783f5b41a6), [`01a7d51`](https://github.com/mastra-ai/mastra/commit/01a7d513493d21562f677f98550f7ceb165ba78c)]:
+  - @mastra/core@1.27.0
+
 ## 1.5.1-alpha.1
 
 ### Patch Changes

package/README.md

@@ -10,7 +10,9 @@ npm install @mastra/clickhouse
 
 ## Prerequisites
 
-- Clickhouse server (version 21.12 or higher recommended)
+- Clickhouse server (version 23.3 or higher required for delete operations; earlier versions may work for read/write operations)
+  - Lightweight `DELETE FROM` requires ClickHouse 22.8+ with `allow_experimental_lightweight_delete = 1` (for 22.8–23.2), or 23.3+ where it is generally available.
+  - The `deleteTask`, `deleteTasks`, and `deleteMessages` methods use `DELETE FROM` — ensure your server supports lightweight delete before using those operations.
 - Node.js 22.13.0 or later
 
 ## Usage
@@ -128,7 +130,6 @@ The store uses different table engines for different types of data:
 
 ### Operations Not Currently Supported
 
-- `deleteMessages(messageIds)`: Message deletion is not supported in ClickHouse
 - AI Observability (traces/spans): Not currently supported
 
 ## Data Types

package/dist/docs/SKILL.md

@@ -3,7 +3,7 @@ name: mastra-clickhouse
 description: Documentation for @mastra/clickhouse. Use when working with @mastra/clickhouse APIs, configuration, or implementation.
 metadata:
   package: "@mastra/clickhouse"
-  version: "1.5.1-alpha.1"
+  version: "1.5.2-alpha.0"
 ---
 
 ## When to use
@@ -16,6 +16,7 @@ Read the individual reference documents for detailed explanations and code examp
 
 ### Reference
 
+- [Reference: ClickHouse storage](references/reference-storage-clickhouse.md) - Documentation for the ClickHouse storage implementation in Mastra, including the vNext observability domain.
 - [Reference: Composite storage](references/reference-storage-composite.md) - Documentation for combining multiple storage backends in Mastra.
 
 

package/dist/docs/assets/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.5.1-alpha.1",
+  "version": "1.5.2-alpha.0",
   "package": "@mastra/clickhouse",
   "exports": {},
   "modules": {}

package/dist/docs/references/reference-storage-clickhouse.md

@@ -0,0 +1,274 @@
+# ClickHouse storage
+
+[ClickHouse](https://clickhouse.com/) is a columnar database designed for analytical workloads. The `@mastra/clickhouse` package provides storage adapters for several Mastra storage domains and is the recommended backend for production observability.
+
+ClickHouse is most commonly used as the dedicated observability backend in a [composite storage](https://mastra.ai/reference/storage/composite) setup, with another database serving the remaining domains. It can also back the supported domains on its own through `ClickhouseStore`.
+
+## When to use ClickHouse
+
+- Production observability for traces, logs, metrics, scores, and feedback.
+- Append-heavy workloads where columnar storage and compression help keep costs down.
+- Deployment platforms with ephemeral filesystems (such as [Railway](#deploying-with-railway-and-similar-platforms), Fly.io, Render, Heroku, or container schedulers) where embedded backends like DuckDB cannot persist data.
+
+For local development, [LibSQL](https://mastra.ai/reference/storage/libsql) or `@mastra/duckdb` are usually a better fit because they need no external service.
+
+## Installation
+
+**npm**:
+
+```bash
+npm install @mastra/clickhouse@latest
+```
+
+**pnpm**:
+
+```bash
+pnpm add @mastra/clickhouse@latest
+```
+
+**Yarn**:
+
+```bash
+yarn add @mastra/clickhouse@latest
+```
+
+**Bun**:
+
+```bash
+bun add @mastra/clickhouse@latest
+```
+
+You will also need a running ClickHouse server. See [Hosting options](#hosting-options) for managed and self-hosted choices.
+
+## Usage
+
+### Observability with vNext (recommended)
+
+`ObservabilityStorageClickhouseVNext` is the current observability domain implementation. It uses an insert-only schema backed by `ReplacingMergeTree` and is optimized for the volume produced by traces, logs, metrics, scores, and feedback.
+
+Compose it with another storage adapter so observability writes do not contend with your application data:
+
+```typescript
+import { Mastra } from '@mastra/core'
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { LibSQLStore } from '@mastra/libsql'
+import { ObservabilityStorageClickhouseVNext } from '@mastra/clickhouse'
+import { Observability, DefaultExporter } from '@mastra/observability'
+
+const observabilityStore = new ObservabilityStorageClickhouseVNext({
+  url: process.env.CLICKHOUSE_URL!,
+  username: process.env.CLICKHOUSE_USERNAME!,
+  password: process.env.CLICKHOUSE_PASSWORD!,
+})
+
+export const mastra = new Mastra({
+  storage: new MastraCompositeStore({
+    id: 'composite-storage',
+    default: new LibSQLStore({
+      id: 'mastra-storage',
+      url: 'file:./mastra.db',
+    }),
+    domains: {
+      observability: observabilityStore,
+    },
+  }),
+  observability: new Observability({
+    configs: {
+      default: {
+        serviceName: 'mastra',
+        exporters: [new DefaultExporter()],
+      },
+    },
+  }),
+})
+```
+
+`DefaultExporter` automatically selects the `insert-only` strategy when ClickHouse is the observability backend, which gives the highest write throughput. See [tracing strategies](https://mastra.ai/docs/observability/tracing/exporters/default) for details.
+
+### Observability with the legacy domain
+
+`ObservabilityStorageClickhouse` is the original observability adapter and remains supported for projects that have not migrated to the vNext schema. The configuration shape is the same as the vNext class.
+
+```typescript
+import { ObservabilityStorageClickhouse } from '@mastra/clickhouse'
+
+const observabilityStore = new ObservabilityStorageClickhouse({
+  url: process.env.CLICKHOUSE_URL!,
+  username: process.env.CLICKHOUSE_USERNAME!,
+  password: process.env.CLICKHOUSE_PASSWORD!,
+})
+```
+
+New projects should use `ObservabilityStorageClickhouseVNext` instead.
+
+### Full storage with `ClickhouseStore`
+
+`ClickhouseStore` implements the `memory`, `workflows`, `scores`, and `observability` domains. Use it when you want a single backend for the supported domains. For most observability deployments, the composite setup above is preferable because it isolates observability writes from primary application data.
+
+```typescript
+import { Mastra } from '@mastra/core'
+import { ClickhouseStore } from '@mastra/clickhouse'
+
+export const mastra = new Mastra({
+  storage: new ClickhouseStore({
+    id: 'clickhouse-storage',
+    url: process.env.CLICKHOUSE_URL!,
+    username: process.env.CLICKHOUSE_USERNAME!,
+    password: process.env.CLICKHOUSE_PASSWORD!,
+  }),
+})
+```
+
+### Bring your own ClickHouse client
+
+Pass a pre-configured client when you need custom connection settings such as request timeouts, compression, or interceptors:
+
+```typescript
+import { createClient } from '@clickhouse/client'
+import { ClickhouseStore } from '@mastra/clickhouse'
+
+const client = createClient({
+  url: process.env.CLICKHOUSE_URL!,
+  username: process.env.CLICKHOUSE_USERNAME!,
+  password: process.env.CLICKHOUSE_PASSWORD!,
+  request_timeout: 60_000,
+  compression: { request: true, response: true },
+})
+
+const storage = new ClickhouseStore({ id: 'clickhouse-storage', client })
+```
+
+The same `client` form is accepted by `ObservabilityStorageClickhouse` and `ObservabilityStorageClickhouseVNext`.
+
+## Configuration
+
+### `ClickhouseStore` options
+
+**id** (`string`): Unique identifier for this storage instance.
+
+**url** (`string`): ClickHouse server URL (for example, \`https\://your-instance.clickhouse.cloud:8443\` or \`http\://localhost:8123\`). Required when not passing a pre-configured \`client\`.
+
+**username** (`string`): ClickHouse username. Required when not passing a pre-configured \`client\`.
+
+**password** (`string`): ClickHouse password. Required when not passing a pre-configured \`client\`. Can be an empty string for the default user on a local instance.
+
+**client** (`ClickHouseClient`): Pre-configured ClickHouse client from \`@clickhouse/client\`. Use this when you need custom request settings. Mutually exclusive with the credential fields above.
+
+**ttl** (`object`): Per-table TTL configuration applied at table creation time. Accepts row-level and column-level TTLs in interval units from \`NANOSECOND\` through \`YEAR\`.
+
+**disableInit** (`boolean`): When \`true\`, the store does not run table creation or migrations on first use. Call \`storage.init()\` explicitly from your deployment scripts. (Default: `false`)
+
+`ClickhouseStore` also accepts every option from `ClickHouseClientConfigOptions` (such as `database`, `request_timeout`, `compression`, `keep_alive`, and `max_open_connections`).
+
+### Observability domain options
+
+`ObservabilityStorageClickhouse` and `ObservabilityStorageClickhouseVNext` accept the same connection options as `ClickhouseStore` (`url`, `username`, `password`, or a pre-configured `client`).
+
+## Hosting options
+
+ClickHouse runs anywhere you can reach it over HTTP. Two common choices:
+
+- **[ClickHouse Cloud](https://clickhouse.com/cloud)**: Managed service with a free trial tier. Provides connection details directly compatible with `url`, `username`, and `password`.
+- **Self-hosted**: Run the official [`clickhouse/clickhouse-server`](https://hub.docker.com/r/clickhouse/clickhouse-server) container or install from the [official packages](https://clickhouse.com/docs/en/install). Suitable for VPS, dedicated hardware, or Kubernetes.
+
+For local development:
+
+```bash
+docker run -d --name mastra-clickhouse \
+  -p 8123:8123 -p 9000:9000 \
+  -e CLICKHOUSE_USER=default \
+  -e CLICKHOUSE_PASSWORD=password \
+  clickhouse/clickhouse-server
+```
+
+```typescript
+new ObservabilityStorageClickhouseVNext({
+  url: 'http://localhost:8123',
+  username: 'default',
+  password: 'password',
+})
+```
+
+## Deploying with Railway and similar platforms
+
+Platforms like [Railway](https://railway.com), [Fly.io](https://fly.io), [Render](https://render.com), and Heroku run application containers on ephemeral filesystems. Embedded observability backends such as DuckDB require a writable, persistent local file, so they either lose data on restart or fail to deploy entirely on these platforms.
+
+Use ClickHouse instead. Because ClickHouse is reached over HTTP, the same connection works from any host:
+
+```typescript
+import { Mastra } from '@mastra/core'
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { PostgresStore } from '@mastra/pg'
+import { ObservabilityStorageClickhouseVNext } from '@mastra/clickhouse'
+import { Observability, DefaultExporter } from '@mastra/observability'
+
+export const mastra = new Mastra({
+  storage: new MastraCompositeStore({
+    id: 'composite-storage',
+    default: new PostgresStore({
+      id: 'pg',
+      connectionString: process.env.DATABASE_URL!,
+    }),
+    domains: {
+      observability: new ObservabilityStorageClickhouseVNext({
+        url: process.env.CLICKHOUSE_URL!,
+        username: process.env.CLICKHOUSE_USERNAME!,
+        password: process.env.CLICKHOUSE_PASSWORD!,
+      }),
+    },
+  }),
+  observability: new Observability({
+    configs: {
+      default: {
+        serviceName: 'mastra',
+        exporters: [new DefaultExporter()],
+      },
+    },
+  }),
+})
+```
+
+Two ways to provision the database:
+
+- **Managed**: Use ClickHouse Cloud. Set `CLICKHOUSE_URL`, `CLICKHOUSE_USERNAME`, and `CLICKHOUSE_PASSWORD` as environment variables in your hosting platform.
+- **Self-hosted on Railway**: Add a ClickHouse service to your Railway project from the official Docker image, then reference it in the application service through Railway's private networking.
+
+The same approach applies to other hosts with ephemeral filesystems. For application data that should also live off-host, pair this setup with a managed PostgreSQL or LibSQL/Turso instance for the `default` storage.
+
+> **Warning:** Do not point an embedded backend like DuckDB at a path inside an ephemeral container filesystem. Data written there is lost when the container restarts, and on some platforms the path is read-only.
+
+## Initialization
+
+When passed to the `Mastra` class, `ClickhouseStore` calls `init()` automatically to create the schema and run any pending migrations. The same applies to `ObservabilityStorageClickhouseVNext` when used through `MastraCompositeStore`.
+
+If you manage storage outside of `Mastra`, call `init()` explicitly:
+
+```typescript
+import { ObservabilityStorageClickhouseVNext } from '@mastra/clickhouse'
+
+const observability = new ObservabilityStorageClickhouseVNext({
+  url: process.env.CLICKHOUSE_URL!,
+  username: process.env.CLICKHOUSE_USERNAME!,
+  password: process.env.CLICKHOUSE_PASSWORD!,
+})
+
+await observability.init()
+```
+
+In CI/CD pipelines, set `disableInit: true` on `ClickhouseStore` and run `init()` from a deployment step that uses elevated credentials. Runtime application credentials can then be limited to read and insert.
+
+## Observability
+
+ClickHouse is the recommended backend for production observability:
+
+- **Insert-only strategy**: `DefaultExporter` writes completed spans in batches without per-span updates, which is the highest-throughput strategy available.
+- **Columnar compression**: Span attributes and log payloads compress well compared to the same data in row-oriented databases.
+
+For the full strategy matrix and production guidance, see the [`DefaultExporter` reference](https://mastra.ai/docs/observability/tracing/exporters/default).
+
+## Related
+
+- [Storage overview](https://mastra.ai/reference/storage/overview)
+- [Composite storage](https://mastra.ai/reference/storage/composite)
+- [`DefaultExporter`](https://mastra.ai/docs/observability/tracing/exporters/default)
+- [Observability overview](https://mastra.ai/docs/observability/overview)

package/dist/docs/references/reference-storage-composite.md

@@ -218,12 +218,12 @@ const storage = new MastraCompositeStore({
 
 Observability data can quickly overwhelm general-purpose databases in production. A single agent interaction can generate hundreds of spans, and high-traffic applications can produce thousands of traces per day.
 
-**ClickHouse** is recommended for production observability because it's optimized for high-volume, write-heavy analytics workloads. Use composite storage to route observability to ClickHouse while keeping other data in your primary database:
+**[ClickHouse](https://mastra.ai/reference/storage/clickhouse)** is recommended for production observability because it's optimized for high-volume, write-heavy analytics workloads. Use composite storage to route observability to ClickHouse while keeping other data in your primary database:
 
 ```typescript
 import { MastraCompositeStore } from '@mastra/core/storage'
 import { MemoryPG, WorkflowsPG, ScoresPG } from '@mastra/pg'
-import { ObservabilityStorageClickhouse } from '@mastra/clickhouse'
+import { ObservabilityStorageClickhouseVNext } from '@mastra/clickhouse'
 
 const storage = new MastraCompositeStore({
   id: 'composite',
@@ -231,7 +231,7 @@ const storage = new MastraCompositeStore({
     memory: new MemoryPG({ connectionString: process.env.DATABASE_URL }),
     workflows: new WorkflowsPG({ connectionString: process.env.DATABASE_URL }),
     scores: new ScoresPG({ connectionString: process.env.DATABASE_URL }),
-    observability: new ObservabilityStorageClickhouse({
+    observability: new ObservabilityStorageClickhouseVNext({
       url: process.env.CLICKHOUSE_URL,
       username: process.env.CLICKHOUSE_USERNAME,
       password: process.env.CLICKHOUSE_PASSWORD,
@@ -240,4 +240,6 @@ const storage = new MastraCompositeStore({
 })
 ```
 
+> **Note:** `ObservabilityStorageClickhouseVNext` is the current observability domain implementation. The legacy `ObservabilityStorageClickhouse` class is also exported and remains supported for projects that have not migrated. See the [ClickHouse storage reference](https://mastra.ai/reference/storage/clickhouse) for details.
+
 > **Info:** This approach is also required when using storage providers that don't support observability (like Convex, DynamoDB, or Cloudflare). See the [DefaultExporter documentation](https://mastra.ai/docs/observability/tracing/exporters/default) for the full list of supported providers.

package/dist/index.cjs

@@ -516,16 +516,11 @@ var ClickhouseDB = class extends base.MastraBase {
   }
   async clearTable({ tableName }) {
     try {
-      await this.client.query({
-        query: `TRUNCATE TABLE ${tableName}`,
-        clickhouse_settings: {
-          // Allows to insert serialized JS Dates (such as '2023-12-06T10:54:48.000Z')
-          date_time_input_format: "best_effort",
-          date_time_output_format: "iso",
-          use_client_time_zone: 1,
-          output_format_json_quote_64bit_integers: 0
-        }
+      await this.client.command({ query: `SYSTEM STOP MERGES ${tableName}` });
+      await this.client.command({
+        query: `TRUNCATE TABLE ${tableName}`
       });
+      await this.client.command({ query: `SYSTEM START MERGES ${tableName}` });
     } catch (error$1) {
       throw new error.MastraError(
         {
@@ -787,7 +782,7 @@ var BackgroundTasksStorageClickhouse = class extends storage.BackgroundTasksStor
     const rows = await result.json();
     return rows.length > 0 ? rowToTask(rows[0]) : null;
   }
-  async listTasks(filter) {
+  buildFilterClause(filter) {
     const conditions = [];
     const params = {};
     if (filter.status) {
@@ -821,6 +816,10 @@ var BackgroundTasksStorageClickhouse = class extends storage.BackgroundTasksStor
       params.var_to_date = filter.toDate.toISOString();
     }
     const where = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
+    return { where, params };
+  }
+  async listTasks(filter) {
+    const { where, params } = this.buildFilterClause(filter);
     const countResult = await this.client.query({
       query: `SELECT count() as count FROM ${storage.TABLE_BACKGROUND_TASKS} FINAL ${where}`,
       query_params: params,
@@ -850,19 +849,18 @@ var BackgroundTasksStorageClickhouse = class extends storage.BackgroundTasksStor
     return { tasks, total };
   }
   async deleteTask(taskId) {
-    await this.client.query({
-      query: `ALTER TABLE ${storage.TABLE_BACKGROUND_TASKS} DELETE WHERE id = {var_id:String}`,
+    await this.client.command({
+      query: `DELETE FROM ${storage.TABLE_BACKGROUND_TASKS} WHERE id = {var_id:String}`,
       query_params: { var_id: taskId }
     });
   }
   async deleteTasks(filter) {
-    const { tasks } = await this.listTasks(filter);
-    for (const task of tasks) {
-      await this.client.query({
-        query: `ALTER TABLE ${storage.TABLE_BACKGROUND_TASKS} DELETE WHERE id = {var_id:String}`,
-        query_params: { var_id: task.id }
-      });
-    }
+    const { where, params } = this.buildFilterClause(filter);
+    if (!where) return;
+    await this.client.command({
+      query: `DELETE FROM ${storage.TABLE_BACKGROUND_TASKS} ${where}`,
+      query_params: params
+    });
   }
   async getRunningCount() {
     const result = await this.client.query({