@mastra/spanner 1.6.2-alpha.0 → 1.6.2-alpha.1

package/CHANGELOG.md

@@ -1,5 +1,47 @@
 # @mastra/spanner
 
+## 1.6.2-alpha.1
+
+### Patch Changes
+
+- Fixed versioned dataset item lookups to return the item visible in the requested dataset snapshot. ([#21979](https://github.com/mastra-ai/mastra/pull/21979))
+
+- Added HTTP endpoints for caller-driven experiments: `POST /datasets/:datasetId/experiments` now accepts `start: false` to create an experiment without spawning the runner (with an optional target and run-level `scorerIds`), and new routes `POST /datasets/:datasetId/experiments/:experimentId/items/:itemId/run`, `POST /datasets/:datasetId/experiments/:experimentId/results`, and `POST /datasets/:datasetId/experiments/:experimentId/finalize` let external orchestrators run one item server-side, submit externally computed per-item results (idempotent upsert on retries), and finalize the run with server-computed counts. ([#21888](https://github.com/mastra-ai/mastra/pull/21888))
+
+- Added `createDatasetExperiment()`, `runExperimentItem()`, `submitExperimentResult()`, and `finalizeExperiment()` methods so a caller-owned orchestrator (for example Temporal) can drive an experiment loop while Mastra either executes each item server-side or ingests externally computed results. ([#21888](https://github.com/mastra-ai/mastra/pull/21888))
+
+- Added `upsertExperimentResult()` to the experiments storage domain plus an `attempt` column on experiment results and a nullable target with an optional `scorerIds` column on experiments, enabling retry-safe result writes for caller-driven experiments (retried submissions with the same `(experimentId, itemId, attempt)` key converge on a single row). `saveScore()` now accepts an optional caller-supplied `id` and upserts on it, so retried experiment submissions replace their previous score rows (latest wins) instead of accumulating duplicates. ([#21888](https://github.com/mastra-ai/mastra/pull/21888))
+
+- Added caller-driven experiments so an external orchestrator (for example Temporal workers) can own the experiment loop while Mastra stays the system of record. ([#21888](https://github.com/mastra-ai/mastra/pull/21888))
+
+  Create an experiment with `dataset.createExperiment()` (idempotent when you pass your own id). With a target, Mastra runs each item for you: call `dataset.runExperimentItem()` per item and Mastra executes the registered agent or workflow, resolves scorers (experiment `scorers`, falling back to item `scorerIds`, then dataset `scorerIds`), and upserts the result. Without a target, run everything yourself and report per-item results with `dataset.submitExperimentResult()` (upsert semantics on `(experimentId, itemId, attempt)` so retried workers converge on a single row). Either way, close the run with `dataset.finalizeExperiment()` and Mastra computes per-item succeeded/failed/skipped counts from the persisted rows. Results go into the same storage as native runs, so Studio views, comparisons, and review summaries work unchanged.
+
+  ```typescript
+  // Caller drives the loop, Mastra runs each item
+  const { experimentId } = await dataset.createExperiment({
+    id: workflowRunId,
+    targetType: 'agent',
+    targetId: 'support-agent',
+    scorers: ['accuracy'],
+  });
+
+  await dataset.runExperimentItem({ experimentId, itemId });
+
+  // Or: caller runs everything, Mastra ingests results
+  const ingest = await dataset.createExperiment({ id: workflowRunId });
+  await dataset.submitExperimentResult({
+    experimentId: ingest.experimentId,
+    itemId,
+    output,
+    scores: [{ scorerId: 'accuracy', score: 0.92 }],
+  });
+
+  const experiment = await dataset.finalizeExperiment({ experimentId });
+  ```
+
+- Updated dependencies [[`9267e9b`](https://github.com/mastra-ai/mastra/commit/9267e9b3d9c2fcf16936050495a787054c2431ab), [`acc3471`](https://github.com/mastra-ai/mastra/commit/acc3471de5f3fde8027ee4e355af292b2bc1bc30), [`b6a771e`](https://github.com/mastra-ai/mastra/commit/b6a771ef23d203ddb348efca8065eff65def8191), [`9267e9b`](https://github.com/mastra-ai/mastra/commit/9267e9b3d9c2fcf16936050495a787054c2431ab), [`26d4016`](https://github.com/mastra-ai/mastra/commit/26d40160ff7f7d8bf95fee2039a52cbc83863533), [`9267e9b`](https://github.com/mastra-ai/mastra/commit/9267e9b3d9c2fcf16936050495a787054c2431ab), [`9267e9b`](https://github.com/mastra-ai/mastra/commit/9267e9b3d9c2fcf16936050495a787054c2431ab), [`57c5103`](https://github.com/mastra-ai/mastra/commit/57c51035a2a36e3df3c4f32f46bb789a66ed5946)]:
+  - @mastra/core@1.61.0-alpha.3
+
 ## 1.6.2-alpha.0
 
 ### Patch Changes

package/dist/docs/SKILL.md

@@ -3,7 +3,7 @@ name: mastra-spanner
 description: Documentation for @mastra/spanner. Use when working with @mastra/spanner APIs, configuration, or implementation.
 metadata:
   package: "@mastra/spanner"
-  version: "1.6.2-alpha.0"
+  version: "1.6.2-alpha.1"
 ---
 
 ## When to use

package/dist/docs/assets/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.6.2-alpha.0",
+  "version": "1.6.2-alpha.1",
   "package": "@mastra/spanner",
   "exports": {},
   "modules": {}

package/dist/index.cjs

@@ -3651,8 +3651,11 @@ var DatasetsSpanner = class DatasetsSpanner extends _mastra_core_storage.Dataset
 			if (args.datasetVersion !== void 0) {
 				sql = `SELECT * FROM ${tableName}
                WHERE ${quoteIdent("id", "column name")} = @id
-                 AND ${quoteIdent("datasetVersion", "column name")} = @datasetVersion
-                 AND ${quoteIdent("isDeleted", "column name")} = FALSE LIMIT 1`;
+                 AND ${quoteIdent("datasetVersion", "column name")} <= @datasetVersion
+                 AND (${quoteIdent("validTo", "column name")} IS NULL OR ${quoteIdent("validTo", "column name")} > @datasetVersion)
+                 AND ${quoteIdent("isDeleted", "column name")} = FALSE
+               ORDER BY ${quoteIdent("datasetVersion", "column name")} DESC
+               LIMIT 1`;
 				params.datasetVersion = args.datasetVersion;
 			} else sql = `SELECT * FROM ${tableName}
                WHERE ${quoteIdent("id", "column name")} = @id
@@ -3983,8 +3986,9 @@ function rowToExperiment(row) {
 		datasetVersion: t.datasetVersion == null ? null : Number(t.datasetVersion),
 		organizationId: t.organizationId ?? null,
 		projectId: t.projectId ?? null,
-		targetType: t.targetType,
-		targetId: String(t.targetId),
+		targetType: t.targetType ?? null,
+		targetId: t.targetId == null ? null : String(t.targetId),
+		scorerIds: t.scorerIds ?? null,
 		status: t.status,
 		totalItems: Number(t.totalItems ?? 0),
 		succeededCount: Number(t.succeededCount ?? 0),
@@ -4016,6 +4020,7 @@ function rowToExperimentResult(row) {
 		startedAt: toDate(t.startedAt),
 		completedAt: toDate(t.completedAt),
 		retryCount: Number(t.retryCount ?? 0),
+		attempt: t.attempt == null ? 0 : Number(t.attempt),
 		traceId: t.traceId ?? null,
 		status: t.status ?? null,
 		tags: t.tags ?? null,
@@ -4068,7 +4073,8 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 				"experimentSetId",
 				"comparisonId",
 				"variantId",
-				"trialIndex"
+				"trialIndex",
+				"scorerIds"
 			]
 		});
 		await this.db.alterTable({
@@ -4077,7 +4083,8 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 			ifNotExists: [
 				"comment",
 				"organizationId",
-				"projectId"
+				"projectId",
+				"attempt"
 			]
 		});
 		await this.createDefaultIndexes();
@@ -4106,9 +4113,13 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 				columns: ["experimentId", "startedAt"]
 			},
 			{
-				name: "mastra_experiment_results_exp_item_idx",
+				name: "mastra_experiment_results_exp_item_attempt_idx",
 				table: _mastra_core_storage.TABLE_EXPERIMENT_RESULTS,
-				columns: ["experimentId", "itemId"],
+				columns: [
+					"experimentId",
+					"itemId",
+					"attempt"
+				],
 				unique: true
 			},
 			{
@@ -4125,6 +4136,7 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 	}
 	async createDefaultIndexes() {
 		if (this.skipDefaultIndexes) return;
+		await this.db.dropIndex("mastra_experiment_results_exp_item_idx").catch(() => {});
 		await this.db.createIndexes(this.getDefaultIndexDefinitions());
 	}
 	async createCustomIndexes() {
@@ -4154,8 +4166,9 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 				datasetVersion: input.datasetVersion ?? null,
 				organizationId: input.organizationId ?? null,
 				projectId: input.projectId ?? null,
-				targetType: input.targetType,
-				targetId: input.targetId,
+				targetType: input.targetType ?? null,
+				targetId: input.targetId ?? null,
+				scorerIds: input.scorerIds ?? null,
 				status: "pending",
 				totalItems: input.totalItems,
 				succeededCount: 0,
@@ -4186,6 +4199,7 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 					projectId: experiment.projectId,
 					targetType: experiment.targetType,
 					targetId: experiment.targetId,
+					scorerIds: experiment.scorerIds,
 					status: experiment.status,
 					totalItems: experiment.totalItems,
 					succeededCount: 0,
@@ -4465,6 +4479,7 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 				startedAt: input.startedAt,
 				completedAt: input.completedAt,
 				retryCount: input.retryCount,
+				attempt: input.attempt ?? 0,
 				traceId: input.traceId ?? null,
 				status: input.status ?? null,
 				tags: input.tags ?? null,
@@ -4487,6 +4502,7 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 					startedAt: result.startedAt,
 					completedAt: result.completedAt,
 					retryCount: result.retryCount,
+					attempt: result.attempt,
 					traceId: result.traceId,
 					status: result.status,
 					tags: result.tags,
@@ -4507,6 +4523,116 @@ var ExperimentsSpanner = class ExperimentsSpanner extends _mastra_core_storage.E
 			}, error);
 		}
 	}
+	async upsertExperimentResult(input) {
+		try {
+			const attempt = input.attempt ?? 0;
+			const [rows] = await this.database.run({
+				sql: `SELECT ${quoteIdent("id", "column name")} FROM ${quoteIdent(_mastra_core_storage.TABLE_EXPERIMENT_RESULTS, "table name")}
+              WHERE ${quoteIdent("experimentId", "column name")} = @experimentId
+                AND ${quoteIdent("itemId", "column name")} = @itemId
+                AND COALESCE(${quoteIdent("attempt", "column name")}, 0) = @attempt`,
+				params: {
+					experimentId: input.experimentId,
+					itemId: input.itemId,
+					attempt
+				},
+				json: true
+			});
+			let existingId = rows[0]?.id;
+			if (!existingId) try {
+				return await this.addExperimentResult({
+					...input,
+					attempt
+				});
+			} catch (insertError) {
+				if (!(0, _mastra_core_storage.hasErrorCode)(insertError, /* @__PURE__ */ new Set([6]))) throw insertError;
+				const [winnerRows] = await this.database.run({
+					sql: `SELECT ${quoteIdent("id", "column name")} FROM ${quoteIdent(_mastra_core_storage.TABLE_EXPERIMENT_RESULTS, "table name")}
+                  WHERE ${quoteIdent("experimentId", "column name")} = @experimentId
+                    AND ${quoteIdent("itemId", "column name")} = @itemId
+                    AND COALESCE(${quoteIdent("attempt", "column name")}, 0) = @attempt`,
+					params: {
+						experimentId: input.experimentId,
+						itemId: input.itemId,
+						attempt
+					},
+					json: true
+				});
+				existingId = winnerRows[0]?.id;
+				if (!existingId) throw insertError;
+			}
+			await this.db.runDml({
+				sql: `UPDATE ${quoteIdent(_mastra_core_storage.TABLE_EXPERIMENT_RESULTS, "table name")} SET
+                ${quoteIdent("itemDatasetVersion", "column name")} = @itemDatasetVersion,
+                ${quoteIdent("organizationId", "column name")} = @organizationId,
+                ${quoteIdent("projectId", "column name")} = @projectId,
+                ${quoteIdent("input", "column name")} = @input,
+                ${quoteIdent("output", "column name")} = @output,
+                ${quoteIdent("groundTruth", "column name")} = @groundTruth,
+                ${quoteIdent("error", "column name")} = @error,
+                ${quoteIdent("startedAt", "column name")} = @startedAt,
+                ${quoteIdent("completedAt", "column name")} = @completedAt,
+                ${quoteIdent("retryCount", "column name")} = @retryCount,
+                ${quoteIdent("attempt", "column name")} = @attempt,
+                ${quoteIdent("traceId", "column name")} = @traceId,
+                ${quoteIdent("status", "column name")} = @status,
+                ${quoteIdent("tags", "column name")} = @tags,
+                ${quoteIdent("toolMockReport", "column name")} = @toolMockReport
+              WHERE ${quoteIdent("id", "column name")} = @id`,
+				params: {
+					id: existingId,
+					itemDatasetVersion: input.itemDatasetVersion ?? null,
+					organizationId: input.organizationId ?? null,
+					projectId: input.projectId ?? null,
+					input: JSON.stringify(input.input),
+					output: input.output == null ? null : JSON.stringify(input.output),
+					groundTruth: input.groundTruth == null ? null : JSON.stringify(input.groundTruth),
+					error: input.error == null ? null : JSON.stringify(input.error),
+					startedAt: input.startedAt,
+					completedAt: input.completedAt,
+					retryCount: input.retryCount,
+					attempt,
+					traceId: input.traceId ?? null,
+					status: input.status ?? null,
+					tags: input.tags == null ? null : JSON.stringify(input.tags),
+					toolMockReport: input.toolMockReport == null ? null : JSON.stringify(input.toolMockReport)
+				},
+				types: {
+					itemDatasetVersion: "int64",
+					organizationId: "string",
+					projectId: "string",
+					input: "json",
+					output: "json",
+					groundTruth: "json",
+					error: "json",
+					traceId: "string",
+					status: "string",
+					tags: "json",
+					toolMockReport: "json"
+				}
+			});
+			const updated = await this.getExperimentResultById({ id: existingId });
+			if (!updated) throw new _mastra_core_error.MastraError({
+				id: (0, _mastra_core_storage.createStorageErrorId)("SPANNER", "UPSERT_EXPERIMENT_RESULT", "NOT_FOUND"),
+				domain: _mastra_core_error.ErrorDomain.STORAGE,
+				category: _mastra_core_error.ErrorCategory.USER,
+				text: `Experiment result ${existingId} not found after upsert`,
+				details: { id: existingId }
+			});
+			return updated;
+		} catch (error) {
+			if (error instanceof _mastra_core_error.MastraError) throw error;
+			throw new _mastra_core_error.MastraError({
+				id: (0, _mastra_core_storage.createStorageErrorId)("SPANNER", "UPSERT_EXPERIMENT_RESULT", "FAILED"),
+				domain: _mastra_core_error.ErrorDomain.STORAGE,
+				category: _mastra_core_error.ErrorCategory.THIRD_PARTY,
+				details: {
+					experimentId: input.experimentId,
+					itemId: input.itemId
+				}
+			}, error);
+		}
+	}
 	async updateExperimentResult(input) {
 		try {
 			if (input.status === void 0 && input.tags === void 0 && input.comment === void 0) {
@@ -10800,9 +10926,10 @@ var ScoresSpanner = class ScoresSpanner extends _mastra_core_storage.ScoresStora
 			}, error);
 		}
 		try {
-			const scoreId = (0, crypto.randomUUID)();
+			const suppliedId = validatedScore.id;
+			const scoreId = suppliedId ?? (0, crypto.randomUUID)();
 			const now = /* @__PURE__ */ new Date();
-			const { scorer, preprocessStepResult, analyzeStepResult, metadata, input, output, additionalContext, requestContext, entity, ...rest } = validatedScore;
+			const { id: _suppliedId, scorer, preprocessStepResult, analyzeStepResult, metadata, input, output, additionalContext, requestContext, entity, ...rest } = validatedScore;
 			const insertedRecord = {
 				id: scoreId,
 				...rest,
@@ -10818,7 +10945,11 @@ var ScoresSpanner = class ScoresSpanner extends _mastra_core_storage.ScoresStora
 				createdAt: now,
 				updatedAt: now
 			};
-			await this.db.insert({
+			if (suppliedId) await this.db.upsert({
+				tableName: _mastra_core_storage.TABLE_SCORERS,
+				record: insertedRecord
+			});
+			else await this.db.insert({
 				tableName: _mastra_core_storage.TABLE_SCORERS,
 				record: insertedRecord
 			});