@mastra/cloudflare-d1 1.0.1 → 1.0.2-alpha.0

package/CHANGELOG.md

@@ -1,5 +1,18 @@
 # @mastra/cloudflare-d1
 
+## 1.0.2-alpha.0
+
+### Patch Changes
+
+- Added resilient column handling to insert and update operations. Unknown columns in records are now silently dropped instead of causing SQL errors, ensuring forward compatibility when newer domain packages add fields that haven't been migrated yet. ([#14021](https://github.com/mastra-ai/mastra/pull/14021))
+
+  For example, calling `db.insert({ tableName, record: { id: '1', title: 'Hello', futureField: 'value' } })` will silently ignore `futureField` if it doesn't exist in the database table, rather than throwing. The same applies to `update` — unknown fields in the data payload are dropped before building the SQL statement.
+
+- Fixed slow semantic recall in the libsql, Cloudflare D1, and ClickHouse storage adapters. Recall performance no longer degrades as threads grow larger. (Fixes #11702) ([#14022](https://github.com/mastra-ai/mastra/pull/14022))
+
+- Updated dependencies [[`4f71b43`](https://github.com/mastra-ai/mastra/commit/4f71b436a4a6b8839842d8da47b57b84509af56c), [`a070277`](https://github.com/mastra-ai/mastra/commit/a07027766ce195ba74d0783116d894cbab25d44c), [`b628b91`](https://github.com/mastra-ai/mastra/commit/b628b9128b372c0f54214d902b07279f03443900), [`332c014`](https://github.com/mastra-ai/mastra/commit/332c014e076b81edf7fe45b58205882726415e90), [`6b63153`](https://github.com/mastra-ai/mastra/commit/6b63153878ea841c0f4ce632ba66bb33e57e9c1b), [`4246e34`](https://github.com/mastra-ai/mastra/commit/4246e34cec9c26636d0965942268e6d07c346671), [`b8837ee`](https://github.com/mastra-ai/mastra/commit/b8837ee77e2e84197609762bfabd8b3da326d30c), [`5d950f7`](https://github.com/mastra-ai/mastra/commit/5d950f7bf426a215a1808f0abef7de5c8336ba1c), [`28c85b1`](https://github.com/mastra-ai/mastra/commit/28c85b184fc32b40f7f160483c982da6d388ecbd), [`e9a08fb`](https://github.com/mastra-ai/mastra/commit/e9a08fbef1ada7e50e961e2f54f55e8c10b4a45c), [`631ffd8`](https://github.com/mastra-ai/mastra/commit/631ffd82fed108648b448b28e6a90e38c5f53bf5), [`aae2295`](https://github.com/mastra-ai/mastra/commit/aae2295838a2d329ad6640829e87934790ffe5b8), [`aa61f29`](https://github.com/mastra-ai/mastra/commit/aa61f29ff8095ce46a4ae16e46c4d8c79b2b685b), [`7ff3714`](https://github.com/mastra-ai/mastra/commit/7ff37148515439bb3be009a60e02c3e363299760), [`41d79a1`](https://github.com/mastra-ai/mastra/commit/41d79a14bd8cb6de1e2565fd0a04786bae2f211b), [`e673376`](https://github.com/mastra-ai/mastra/commit/e6733763ad1321aa7e5ae15096b9c2104f93b1f3), [`b2204c9`](https://github.com/mastra-ai/mastra/commit/b2204c98a42848bbfb6f0440f005dc2b6354f1cd), [`a1bf1e3`](https://github.com/mastra-ai/mastra/commit/a1bf1e385ed4c0ef6f11b56c5887442970d127f2), [`b6f647a`](https://github.com/mastra-ai/mastra/commit/b6f647ae2388e091f366581595feb957e37d5b40), [`0c57b8b`](https://github.com/mastra-ai/mastra/commit/0c57b8b0a69a97b5a4ae3f79be6c610f29f3cf7b), [`b081f27`](https://github.com/mastra-ai/mastra/commit/b081f272cf411716e1d6bd72ceac4bcee2657b19), [`0c09eac`](https://github.com/mastra-ai/mastra/commit/0c09eacb1926f64cfdc9ae5c6d63385cf8c9f72c), [`6b9b93d`](https://github.com/mastra-ai/mastra/commit/6b9b93d6f459d1ba6e36f163abf62a085ddb3d64), [`31b6067`](https://github.com/mastra-ai/mastra/commit/31b6067d0cc3ab10e1b29c36147f3b5266bc714a), [`797ac42`](https://github.com/mastra-ai/mastra/commit/797ac4276de231ad2d694d9aeca75980f6cd0419), [`0bc289e`](https://github.com/mastra-ai/mastra/commit/0bc289e2d476bf46c5b91c21969e8d0c6864691c), [`9b75a06`](https://github.com/mastra-ai/mastra/commit/9b75a06e53ebb0b950ba7c1e83a0142047185f46), [`4c3a1b1`](https://github.com/mastra-ai/mastra/commit/4c3a1b122ea083e003d71092f30f3b31680b01c0), [`85cc3b3`](https://github.com/mastra-ai/mastra/commit/85cc3b3b6f32ae4b083c26498f50d5b250ba944b), [`97ea28c`](https://github.com/mastra-ai/mastra/commit/97ea28c746e9e4147d56047bbb1c4a92417a3fec), [`d567299`](https://github.com/mastra-ai/mastra/commit/d567299cf81e02bd9d5221d4bc05967d6c224161), [`716ffe6`](https://github.com/mastra-ai/mastra/commit/716ffe68bed81f7c2690bc8581b9e140f7bf1c3d), [`8296332`](https://github.com/mastra-ai/mastra/commit/8296332de21c16e3dfc3d0b2d615720a6dc88f2f), [`4df2116`](https://github.com/mastra-ai/mastra/commit/4df211619dd922c047d396ca41cd7027c8c4c8e7), [`2219c1a`](https://github.com/mastra-ai/mastra/commit/2219c1acbd21da116da877f0036ffb985a9dd5a3), [`17c4145`](https://github.com/mastra-ai/mastra/commit/17c4145166099354545582335b5252bdfdfd908b)]:
+  - @mastra/core@1.11.0-alpha.0
+
 ## 1.0.1
 
 ### Patch Changes

package/dist/docs/SKILL.md

@@ -3,7 +3,7 @@ name: mastra-cloudflare-d1
 description: Documentation for @mastra/cloudflare-d1. Use when working with @mastra/cloudflare-d1 APIs, configuration, or implementation.
 metadata:
   package: "@mastra/cloudflare-d1"
-  version: "1.0.1"
+  version: "1.0.2-alpha.0"
 ---
 
 ## When to use

package/dist/docs/assets/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.0.1",
+  "version": "1.0.2-alpha.0",
   "package": "@mastra/cloudflare-d1",
   "exports": {},
   "modules": {}

package/dist/docs/references/reference-storage-cloudflare-d1.md

@@ -2,7 +2,7 @@
 
 The Cloudflare D1 storage implementation provides a serverless SQL database solution using Cloudflare D1, supporting relational operations and transactional consistency.
 
-> **Observability Not Supported:** Cloudflare D1 storage **does not support the observability domain**. Traces from the `DefaultExporter` cannot be persisted to D1, and Mastra Studio's observability features won't work with D1 as your only storage provider. To enable observability, use [composite storage](https://mastra.ai/reference/storage/composite) to route observability data to a supported provider like ClickHouse or PostgreSQL.
+> **Observability Not Supported:** Cloudflare D1 storage **doesn't support the observability domain**. Traces from the `DefaultExporter` can't be persisted to D1, and Mastra Studio's observability features won't work with D1 as your only storage provider. To enable observability, use [composite storage](https://mastra.ai/reference/storage/composite) to route observability data to a supported provider like ClickHouse or PostgreSQL.
 
 > **Row Size Limit:** Cloudflare D1 enforces a **1 MiB maximum row size**. This limit can be exceeded when storing messages with base64-encoded attachments such as images. See [Handling large attachments](https://mastra.ai/docs/memory/storage) for workarounds including uploading attachments to external storage.
 
@@ -85,7 +85,7 @@ export default {
 > - The **property name** (`D1Database`) must match the `binding` name in your `wrangler.toml`
 > - The **type** (`: D1Database`) is from `@cloudflare/workers-types` for TypeScript type checking
 >
-> At runtime, Cloudflare Workers provides the actual D1 database instance via `env.D1Database`. You cannot use `D1Database` directly outside of the worker's context.
+> At runtime, Cloudflare Workers provides the actual D1 database instance via `env.D1Database`. You can't use `D1Database` directly outside of the worker's context.
 
 ### Using with REST API
 
@@ -129,15 +129,15 @@ Or in `wrangler.jsonc`:
 
 ## Parameters
 
-**binding?:** (`D1Database`): Cloudflare D1 Workers binding (for Workers runtime)
+**binding** (`D1Database`): Cloudflare D1 Workers binding (for Workers runtime)
 
-**accountId?:** (`string`): Cloudflare Account ID (for REST API)
+**accountId** (`string`): Cloudflare Account ID (for REST API)
 
-**databaseId?:** (`string`): Cloudflare D1 Database ID (for REST API)
+**databaseId** (`string`): Cloudflare D1 Database ID (for REST API)
 
-**apiToken?:** (`string`): Cloudflare API Token (for REST API)
+**apiToken** (`string`): Cloudflare API Token (for REST API)
 
-**tablePrefix?:** (`string`): Optional prefix for all table names (useful for environment isolation)
+**tablePrefix** (`string`): Optional prefix for all table names (useful for environment isolation)
 
 ## Additional Notes
 
@@ -207,7 +207,7 @@ export default {
 }
 ```
 
-> **Warning:** If `init()` is not called, tables won't be created and storage operations will fail silently or throw errors.
+> **Warning:** If `init()` isn't called, tables won't be created and storage operations will fail silently or throw errors.
 
 ### Transactions & Consistency
 

package/dist/index.cjs

@@ -291,6 +291,8 @@ var D1DB = class extends base.MastraBase {
   client;
   binding;
   tablePrefix;
+  /** Cache of actual table columns: tableName -> Promise<Set<columnName>> (stores in-flight promise to coalesce concurrent calls) */
+  tableColumnsCache = /* @__PURE__ */ new Map();
   constructor(config) {
     super({
       component: "STORAGE",
@@ -399,6 +401,38 @@ var D1DB = class extends base.MastraBase {
       throw new Error("Neither binding nor client is configured");
     }
   }
+  /**
+   * Gets the set of column names that actually exist in the database table.
+   * Results are cached; the cache is invalidated when alterTable() adds new columns.
+   */
+  async getKnownColumnNames(tableName) {
+    const cached = this.tableColumnsCache.get(tableName);
+    if (cached) return cached;
+    const promise = this.getTableColumns(tableName).then((columns) => {
+      const names = new Set(columns.map((c) => c.name));
+      if (names.size === 0) {
+        this.tableColumnsCache.delete(tableName);
+      }
+      return names;
+    });
+    this.tableColumnsCache.set(tableName, promise);
+    return promise;
+  }
+  /**
+   * Filters a record to only include columns that exist in the actual database table.
+   * Unknown columns are silently dropped to ensure forward compatibility.
+   */
+  async filterRecordToKnownColumns(tableName, record) {
+    const knownColumns = await this.getKnownColumnNames(tableName);
+    if (knownColumns.size === 0) return record;
+    const filtered = {};
+    for (const [key, value] of Object.entries(record)) {
+      if (knownColumns.has(key)) {
+        filtered[key] = value;
+      }
+    }
+    return filtered;
+  }
   async getTableColumns(tableName) {
     try {
       const sql = `PRAGMA table_info(${tableName})`;
@@ -465,6 +499,7 @@ var D1DB = class extends base.MastraBase {
       const { sql, params } = query.build();
       await this.executeQuery({ sql, params });
       this.logger.debug(`Created table ${fullTableName}`);
+      this.tableColumnsCache.delete(fullTableName);
     } catch (error$1) {
       throw new error.MastraError(
         {
@@ -497,8 +532,8 @@ var D1DB = class extends base.MastraBase {
     }
   }
   async dropTable({ tableName }) {
+    const fullTableName = this.getTableName(tableName);
     try {
-      const fullTableName = this.getTableName(tableName);
       const sql = `DROP TABLE IF EXISTS ${fullTableName}`;
       await this.executeQuery({ sql });
       this.logger.debug(`Dropped table ${fullTableName}`);
@@ -512,11 +547,13 @@ var D1DB = class extends base.MastraBase {
         },
         error$1
       );
+    } finally {
+      this.tableColumnsCache.delete(fullTableName);
     }
   }
   async alterTable(args) {
+    const fullTableName = this.getTableName(args.tableName);
     try {
-      const fullTableName = this.getTableName(args.tableName);
       const existingColumns = await this.getTableColumns(fullTableName);
       const existingColumnNames = new Set(existingColumns.map((col) => col.name));
       for (const [columnName, column] of Object.entries(args.schema)) {
@@ -538,14 +575,18 @@ var D1DB = class extends base.MastraBase {
         },
         error$1
       );
+    } finally {
+      this.tableColumnsCache.delete(fullTableName);
     }
   }
   async insert({ tableName, record }) {
     try {
       const fullTableName = this.getTableName(tableName);
       const processedRecord = await this.processRecord(record);
-      const columns = Object.keys(processedRecord);
-      const values = Object.values(processedRecord);
+      const filteredRecord = await this.filterRecordToKnownColumns(fullTableName, processedRecord);
+      const columns = Object.keys(filteredRecord);
+      if (columns.length === 0) return;
+      const values = Object.values(filteredRecord);
       const query = createSqlBuilder().insert(fullTableName, columns, values);
       const { sql, params } = query.build();
       await this.executeQuery({ sql, params });
@@ -566,8 +607,12 @@ var D1DB = class extends base.MastraBase {
       if (records.length === 0) return;
       const fullTableName = this.getTableName(tableName);
       const processedRecords = await Promise.all(records.map((record) => this.processRecord(record)));
-      const columns = Object.keys(processedRecords[0] || {});
-      for (const record of processedRecords) {
+      const filteredRecords = await Promise.all(
+        processedRecords.map((r) => this.filterRecordToKnownColumns(fullTableName, r))
+      );
+      for (const record of filteredRecords) {
+        const columns = Object.keys(record);
+        if (columns.length === 0) continue;
         const values = Object.values(record);
         const query = createSqlBuilder().insert(fullTableName, columns, values);
         const { sql, params } = query.build();
@@ -643,12 +688,14 @@ var D1DB = class extends base.MastraBase {
         const batch = records.slice(i, i + batchSize);
         const recordsToInsert = batch;
         if (recordsToInsert.length > 0) {
-          const firstRecord = recordsToInsert[0];
-          const columns = Object.keys(firstRecord || {});
-          for (const record of recordsToInsert) {
+          const filteredRecords = await Promise.all(
+            recordsToInsert.map((r) => this.filterRecordToKnownColumns(fullTableName, r || {}))
+          );
+          for (const record of filteredRecords) {
+            const columns = Object.keys(record);
+            if (columns.length === 0) continue;
             const values = columns.map((col) => {
-              if (!record) return null;
-              const value = typeof col === "string" ? record[col] : null;
+              const value = record[col];
               return this.serializeValue(value);
             });
             const recordToUpsert = columns.reduce(
@@ -1134,56 +1181,79 @@ var MemoryStorageD1 = class extends storage.MemoryStorage {
       );
     }
   }
+  _sortMessages(messages, field, direction) {
+    return messages.sort((a, b) => {
+      const isDateField = field === "createdAt" || field === "updatedAt";
+      const aValue = isDateField ? new Date(a[field]).getTime() : a[field];
+      const bValue = isDateField ? new Date(b[field]).getTime() : b[field];
+      if (aValue === bValue) {
+        return a.id.localeCompare(b.id);
+      }
+      if (typeof aValue === "number" && typeof bValue === "number") {
+        return direction === "ASC" ? aValue - bValue : bValue - aValue;
+      }
+      return direction === "ASC" ? String(aValue).localeCompare(String(bValue)) : String(bValue).localeCompare(String(aValue));
+    });
+  }
   async _getIncludedMessages(include) {
     if (!include || include.length === 0) return null;
+    const tableName = this.#db.getTableName(storage.TABLE_MESSAGES);
+    const targetIds = include.map((inc) => inc.id).filter(Boolean);
+    if (targetIds.length === 0) return null;
+    const idPlaceholders = targetIds.map(() => "?").join(", ");
+    const targetResult = await this.#db.executeQuery({
+      sql: `SELECT id, thread_id, createdAt FROM ${tableName} WHERE id IN (${idPlaceholders})`,
+      params: targetIds
+    });
+    if (!Array.isArray(targetResult) || targetResult.length === 0) return null;
+    const targetMap = new Map(
+      targetResult.map((r) => [r.id, { threadId: r.thread_id, createdAt: r.createdAt }])
+    );
     const unionQueries = [];
     const params = [];
-    let paramIdx = 1;
-    const tableName = this.#db.getTableName(storage.TABLE_MESSAGES);
     for (const inc of include) {
       const { id, withPreviousMessages = 0, withNextMessages = 0 } = inc;
-      unionQueries.push(`
-                SELECT * FROM (
-                  WITH target_thread AS (
-                    SELECT thread_id FROM ${tableName} WHERE id = ?
-                  ),
-                  ordered_messages AS (
-                    SELECT
-                      *,
-                      ROW_NUMBER() OVER (ORDER BY createdAt ASC) AS row_num
-                    FROM ${tableName}
-                    WHERE thread_id = (SELECT thread_id FROM target_thread)
-                  )
-                  SELECT
-                    m.id,
-                    m.content,
-                    m.role,
-                    m.type,
-                    m.createdAt,
-                    m.thread_id AS threadId,
-                    m.resourceId
-                  FROM ordered_messages m
-                  WHERE m.id = ?
-                  OR EXISTS (
-                    SELECT 1 FROM ordered_messages target
-                    WHERE target.id = ?
-                    AND (
-                      (m.row_num <= target.row_num + ? AND m.row_num > target.row_num)
-                      OR
-                      (m.row_num >= target.row_num - ? AND m.row_num < target.row_num)
-                    )
-                  )
-                ) AS query_${paramIdx}
-            `);
-      params.push(id, id, id, withNextMessages, withPreviousMessages);
-      paramIdx++;
-    }
-    const finalQuery = unionQueries.join(" UNION ALL ") + " ORDER BY createdAt ASC";
+      const target = targetMap.get(id);
+      if (!target) continue;
+      unionQueries.push(`SELECT * FROM (
+        SELECT id, content, role, type, createdAt, thread_id AS threadId, resourceId
+        FROM ${tableName}
+        WHERE thread_id = ?
+          AND createdAt <= ?
+        ORDER BY createdAt DESC, id DESC
+        LIMIT ?
+      )`);
+      params.push(target.threadId, target.createdAt, withPreviousMessages + 1);
+      if (withNextMessages > 0) {
+        unionQueries.push(`SELECT * FROM (
+          SELECT id, content, role, type, createdAt, thread_id AS threadId, resourceId
+          FROM ${tableName}
+          WHERE thread_id = ?
+            AND createdAt > ?
+          ORDER BY createdAt ASC, id ASC
+          LIMIT ?
+        )`);
+        params.push(target.threadId, target.createdAt, withNextMessages);
+      }
+    }
+    if (unionQueries.length === 0) return null;
+    let finalQuery;
+    if (unionQueries.length === 1) {
+      finalQuery = unionQueries[0];
+    } else {
+      finalQuery = `${unionQueries.join(" UNION ALL ")} ORDER BY createdAt ASC, id ASC`;
+    }
     const messages = await this.#db.executeQuery({ sql: finalQuery, params });
     if (!Array.isArray(messages)) {
       return [];
     }
-    const processedMessages = messages.map((message) => {
+    const seen = /* @__PURE__ */ new Set();
+    const processedMessages = messages.filter((message) => {
+      const id = message.id;
+      if (seen.has(id)) return false;
+      seen.add(id);
+      return true;
+    }).map((message) => {
       const processedMsg = {};
       for (const [key, value] of Object.entries(message)) {
         if (key === `type` && value === `v2`) continue;
@@ -1283,6 +1353,23 @@ var MemoryStorageD1 = class extends storage.MemoryStorage {
         queryParams.push(endDate);
       }
       const { field, direction } = this.parseOrderBy(orderBy, "ASC");
+      if (perPage === 0 && (!include || include.length === 0)) {
+        return { messages: [], total: 0, page, perPage: perPageForResponse, hasMore: false };
+      }
+      if (perPage === 0 && include && include.length > 0) {
+        const includeResult = await this._getIncludedMessages(include);
+        if (!Array.isArray(includeResult) || includeResult.length === 0) {
+          return { messages: [], total: 0, page, perPage: perPageForResponse, hasMore: false };
+        }
+        const list2 = new agent.MessageList().add(includeResult, "memory");
+        return {
+          messages: this._sortMessages(list2.get.all.db(), field, direction),
+          total: 0,
+          page,
+          perPage: perPageForResponse,
+          hasMore: false
+        };
+      }
       query += ` ORDER BY "${field}" ${direction}`;
       if (perPage !== Number.MAX_SAFE_INTEGER) {
         query += ` LIMIT ? OFFSET ?`;
@@ -1342,19 +1429,7 @@ var MemoryStorageD1 = class extends storage.MemoryStorage {
         }
       }
       const list = new agent.MessageList().add(paginatedMessages, "memory");
-      let finalMessages = list.get.all.db();
-      finalMessages = finalMessages.sort((a, b) => {
-        const isDateField = field === "createdAt" || field === "updatedAt";
-        const aValue = isDateField ? new Date(a[field]).getTime() : a[field];
-        const bValue = isDateField ? new Date(b[field]).getTime() : b[field];
-        if (aValue === bValue) {
-          return a.id.localeCompare(b.id);
-        }
-        if (typeof aValue === "number" && typeof bValue === "number") {
-          return direction === "ASC" ? aValue - bValue : bValue - aValue;
-        }
-        return direction === "ASC" ? String(aValue).localeCompare(String(bValue)) : String(bValue).localeCompare(String(aValue));
-      });
+      const finalMessages = this._sortMessages(list.get.all.db(), field, direction);
       const returnedThreadMessageIds = new Set(finalMessages.filter((m) => m.threadId === threadId).map((m) => m.id));
       const allThreadMessagesReturned = returnedThreadMessageIds.size >= total;
       const hasMore = perPageInput === false ? false : allThreadMessagesReturned ? false : offset + paginatedCount < total;