@textopt/langsmith 0.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Charlie Duong
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,73 @@
+# @textopt/langsmith
+
+LangSmith experiment reporting for [textopt](https://github.com/ctdio/textopt#readme).
+
+`createLangSmithReporter` writes a run to LangSmith as **one experiment per accepted candidate over one fixed dataset**. Selecting several of those experiments in LangSmith's comparison view renders the candidate x instance score matrix selection reads: which instances a candidate won, and which it paid for.
+
+It works with any optimizer. The reporter reads the acceptance payload every search emits — the candidate text and its per-instance row over the validation set — rather than one search's event union, and folds whatever else the optimizer put on the event (GEPA's lineage, MIPRO's menu choices, SIMBA's step) into the experiment's metadata.
+
+Install `langsmith` separately. This package matches its `Client` structurally and declares no runtime dependency on it.
+
+## Usage
+
+```ts
+import { GepaOptimizer } from "textopt/gepa";
+import { createLangSmithReporter } from "@textopt/langsmith";
+import { Client } from "langsmith";
+
+const instanceId = ({ datum }: { datum: Ticket }) => datum.id;
+
+const result = await new GepaOptimizer({ trackBestOutputs: true }).optimize({
+  seedCandidate,
+  trainingSet,
+  validationSet,
+  testSet,
+  adapter,
+  reflect,
+  maxMetricCalls: 300,
+  instanceId,
+  reporters: [
+    createLangSmithReporter({
+      client: new Client(),
+      dataset: "ticket-triage",
+      experimentPrefix: "gepa-2026-08-19",
+      validationSet,
+      testSet,
+      instanceId,
+      toInput: (datum) => ({ ticket: datum.text }),
+      toExpected: (datum) => ({ label: datum.label }),
+    }),
+  ],
+});
+```
+
+Experiments are named `<experimentPrefix>/cand-<candidateId>`, starting with `cand-0` for the seed — the baseline every later candidate is read against. Each carries the candidate's text, iteration, parents, source and aggregate score as metadata, so a score that moved sits next to the edit that moved it.
+
+| Option             | Default              | Effect                                                              |
+| ------------------ | -------------------- | ------------------------------------------------------------------- |
+| `client`           | required             | A LangSmith `Client`, or anything with the same shape.              |
+| `dataset`          | required             | Dataset holding the validation split. Created once, then reused.    |
+| `experimentPrefix` | required             | Names the experiments. Use a value unique to the run.               |
+| `validationSet`    | required             | The validation set the run was given, in the same order.            |
+| `testSet`          | none                 | Uploaded as its own dataset, swept once, for the winner only.       |
+| `testDataset`      | `<dataset>-held-out` | Names that second dataset.                                          |
+| `instanceId`       | the row's position   | Keys a dataset row across runs.                                     |
+| `toInput`          | the datum itself     | The example's `inputs`.                                             |
+| `toExpected`       | none                 | The example's `outputs`, when the dataset should carry a reference. |
+| `concurrency`      | `8`                  | Rows uploaded at once within one experiment.                        |
+
+## What is deliberately not logged
+
+**Minibatch rollouts.** They are small random subsets of the _training_ set that differ every iteration, so they cannot be compared across candidates and would bury the experiments that can. They are still traceable: with `LANGSMITH_TRACING=1`, `@textopt/langsmith`'s sibling `@textopt/langchain` tags every rollout with its iteration, phase, split and candidate id.
+
+**A held-out experiment per candidate.** GEPA scores the held-out set only once selection is over, precisely so that nothing can be chosen against it. There is one held-out experiment, for the winner, on its own dataset — and no option to change that, because the first time a candidate is picked because it looked better there, the number stops meaning anything.
+
+**Zeros for unmeasured instances.** An instance the evaluation policy skipped, or one an infrastructure failure lost, has no row. Written as a zero it reads as a regression that never happened.
+
+## Notes
+
+Uploads are queued, not awaited: `onEvent` runs on the search's hot path, and the reporter's `flush` is awaited once as the run ends, including when it ends by throwing. A LangSmith that is unreachable degrades to a warning — the run that bought the rollouts finishes either way.
+
+Dataset row ids are derived from the dataset name and `instanceId`, not assigned by the server, so a repeated or resumed run writes to the rows it already has. If the dataset already exists, its examples are left alone; a validation set that changed shape wants a new `dataset` name.
+
+Uploading is chatty: one run plus one feedback per measured instance per accepted candidate. Ten candidates over a hundred validation instances is a few thousand calls.

package/dist/index.cjs

@@ -0,0 +1,223 @@
+Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
+let textopt = require("textopt");
+//#region src/reporter.ts
+const DEFAULT_CONCURRENCY = 8;
+/**
+* A UUIDv5 namespace for this library, so a dataset row's id is a pure
+* function of the dataset it belongs to and the instance it names. Fixed
+* forever: changing it renames every row textopt has ever written.
+*/
+const TEXTOPT_NAMESPACE = "6f9619ff-8b86-d011-b42d-00c04fc964ff";
+/**
+* Reports a GEPA run to LangSmith as one experiment per accepted candidate
+* over one fixed dataset — which is what makes the candidates comparable.
+* Selecting several in LangSmith's comparison view renders the candidate x
+* instance score matrix the Pareto frontier is chosen on: which instances a
+* candidate won, and which it paid for.
+*
+* Minibatch rollouts are deliberately absent. They are small random subsets of
+* the *training* set that differ every iteration, so they cannot be compared
+* across candidates and would bury the experiments that can. They remain
+* traceable through the adapter, which tags every rollout with its iteration,
+* phase, split and candidate id.
+*
+* The held-out sweep gets exactly one experiment, for the winner, on its own
+* dataset. There is no option to log one per candidate: GEPA evaluates the
+* held-out set only after selection is over precisely so that nothing can be
+* chosen against it, and a per-candidate view hands that back the moment
+* someone reads it.
+*/
+function createLangSmithReporter(options) {
+	const { client, dataset, experimentPrefix, validationSet, testSet, testDataset = `${dataset}-held-out`, instanceId = ({ index }) => String(index), toInput = (datum) => datum, toExpected, concurrency = DEFAULT_CONCURRENCY } = options;
+	let pending = Promise.resolve();
+	let validationDatasetId;
+	let heldOutDatasetId;
+	function enqueue(work) {
+		pending = pending.then(work).catch((err) => {
+			console.warn("[textopt-langsmith] upload failed", { err });
+		});
+	}
+	async function ensureDataset(args) {
+		if (await client.hasDataset({ datasetName: args.name })) return (await client.readDataset({ datasetName: args.name })).id;
+		const created = await client.createDataset(args.name, { description: `textopt: ${String(args.rows.length)} instances` });
+		await client.createExamples(await Promise.all(args.rows.map(async (datum, index) => ({
+			id: await exampleIdFor({
+				dataset: args.name,
+				instance: instanceId({
+					datum,
+					index
+				})
+			}),
+			dataset_id: created.id,
+			inputs: toInput(datum),
+			...toExpected === void 0 ? {} : { outputs: toExpected(datum) },
+			metadata: { textopt_instance_id: instanceId({
+				datum,
+				index
+			}) }
+		}))));
+		return created.id;
+	}
+	async function writeExperiment(args) {
+		const project = await client.createProject({
+			projectName: args.projectName,
+			referenceDatasetId: args.datasetId,
+			metadata: args.metadata
+		});
+		const scored = args.rows.flatMap((datum, index) => {
+			const score = args.scores[index];
+			return score === void 0 ? [] : [{
+				datum,
+				index,
+				score
+			}];
+		});
+		await (0, textopt.mapWithConcurrency)({
+			items: scored,
+			limit: concurrency,
+			task: async ({ datum, index, score }) => {
+				const runId = crypto.randomUUID();
+				const now = Date.now();
+				await client.createRun({
+					id: runId,
+					name: args.projectName,
+					run_type: "chain",
+					inputs: toInput(datum),
+					outputs: { output: args.outputs?.[index] ?? null },
+					project_name: args.projectName,
+					reference_example_id: await exampleIdFor({
+						dataset: args.datasetName,
+						instance: instanceId({
+							datum,
+							index
+						})
+					}),
+					start_time: now,
+					end_time: now
+				});
+				await client.createFeedback(runId, "score", {
+					score,
+					sessionId: project.id
+				});
+			}
+		});
+	}
+	return {
+		onEvent: (event) => {
+			if ((0, textopt.isCandidateAccepted)(event)) enqueue(async () => {
+				validationDatasetId ??= ensureDataset({
+					name: dataset,
+					rows: validationSet
+				});
+				await writeExperiment({
+					projectName: `${experimentPrefix}/cand-${String(event.candidateId)}`,
+					datasetId: await validationDatasetId,
+					datasetName: dataset,
+					rows: validationSet,
+					scores: event.instanceScores,
+					outputs: event.outputs,
+					metadata: experimentMetadata({
+						candidate: event.candidate,
+						candidateId: event.candidateId,
+						aggregateScore: event.aggregateScore,
+						search: searchSpecific({ ...event })
+					})
+				});
+			});
+			if ((0, textopt.isRunFinished)(event) && event.testInstanceScores !== void 0 && testSet !== void 0) {
+				const heldOut = event.testInstanceScores;
+				const outputs = event.testOutputs;
+				const { bestCandidateId, testScore } = event;
+				enqueue(async () => {
+					heldOutDatasetId ??= ensureDataset({
+						name: testDataset,
+						rows: testSet
+					});
+					await writeExperiment({
+						projectName: `${experimentPrefix}/held-out-cand-${String(bestCandidateId)}`,
+						datasetId: await heldOutDatasetId,
+						datasetName: testDataset,
+						rows: testSet,
+						scores: heldOut,
+						outputs,
+						metadata: {
+							textopt_candidate_id: bestCandidateId,
+							textopt_split: "test",
+							...testScore === void 0 ? {} : { textopt_score: testScore }
+						}
+					});
+				});
+			}
+		},
+		flush: async () => {
+			await pending;
+		}
+	};
+}
+function experimentMetadata(args) {
+	const search = Object.fromEntries(Object.entries(args.search).map(([key, value]) => [`textopt_${key}`, value]));
+	return {
+		...args.candidate,
+		textopt_candidate_id: args.candidateId,
+		textopt_score: args.aggregateScore,
+		...search
+	};
+}
+/**
+* The fields the emitting optimizer added to the shared acceptance payload:
+* GEPA's lineage, MIPRO's menu choices, SIMBA's step. Read structurally rather
+* than per optimizer so a new search reports its own vocabulary without this
+* package learning about it.
+*/
+function searchSpecific(event) {
+	const shared = /* @__PURE__ */ new Set([
+		"type",
+		"candidateId",
+		"candidate",
+		"aggregateScore",
+		"instanceScores",
+		"outputs"
+	]);
+	return Object.fromEntries(Object.entries(event).filter(([key]) => !shared.has(key)));
+}
+/**
+* A dataset row's id, derived from the dataset and the instance it names
+* rather than assigned by the server.
+*
+* Server-assigned ids would give a resumed or repeated run a fresh set of
+* rows, and every experiment logged before the restart would have nothing left
+* to compare against.
+*/
+async function exampleIdFor(args) {
+	return uuidV5({
+		namespace: TEXTOPT_NAMESPACE,
+		name: `${args.dataset}:${args.instance}`
+	});
+}
+/** RFC 4122 name-based UUID, SHA-1 flavour. LangSmith ids must be UUIDs. */
+async function uuidV5(args) {
+	const namespaceBytes = uuidToBytes(args.namespace);
+	const nameBytes = new TextEncoder().encode(args.name);
+	const payload = new Uint8Array(namespaceBytes.length + nameBytes.length);
+	payload.set(namespaceBytes);
+	payload.set(nameBytes, namespaceBytes.length);
+	const bytes = new Uint8Array(await crypto.subtle.digest("SHA-1", payload)).slice(0, 16);
+	bytes[6] = bytes[6] & 15 | 80;
+	bytes[8] = bytes[8] & 63 | 128;
+	const hex = [...bytes].map((byte) => byte.toString(16).padStart(2, "0")).join("");
+	return [
+		hex.slice(0, 8),
+		hex.slice(8, 12),
+		hex.slice(12, 16),
+		hex.slice(16, 20),
+		hex.slice(20)
+	].join("-");
+}
+function uuidToBytes(uuid) {
+	const hex = uuid.replaceAll("-", "");
+	const bytes = /* @__PURE__ */ new Uint8Array(16);
+	for (let index = 0; index < bytes.length; index += 1) bytes[index] = Number.parseInt(hex.slice(index * 2, index * 2 + 2), 16);
+	return bytes;
+}
+//#endregion
+exports.createLangSmithReporter = createLangSmithReporter;

package/dist/index.d.cts

@@ -0,0 +1,107 @@
+import { OptimizerEvent, Reporter } from "textopt";
+//#region src/reporter.d.ts
+interface LangSmithDataset {
+  id: string;
+}
+interface LangSmithProject {
+  id: string;
+  /** Optional because LangSmith's own `TracerSession` declares it so. */
+  name?: string;
+}
+interface LangSmithExample {
+  id?: string;
+  dataset_id?: string;
+  inputs: Record<string, unknown>;
+  outputs?: Record<string, unknown>;
+  metadata?: Record<string, unknown>;
+}
+interface LangSmithRun {
+  id?: string;
+  name: string;
+  run_type: string;
+  inputs: Record<string, unknown>;
+  outputs?: Record<string, unknown>;
+  project_name?: string;
+  reference_example_id?: string;
+  start_time?: number;
+  end_time?: number;
+}
+/**
+ * The part of LangSmith's `Client` this reporter uses. Structural rather than
+ * imported so the package carries no runtime dependency on the SDK, and so a
+ * test can hand it a recording client instead of a network.
+ */
+interface LangSmithClientLike {
+  hasDataset(args: {
+    datasetName: string;
+  }): Promise<boolean>;
+  readDataset(args: {
+    datasetName: string;
+  }): Promise<LangSmithDataset>;
+  createDataset(name: string, options?: {
+    description?: string;
+  }): Promise<LangSmithDataset>;
+  createExamples(uploads: LangSmithExample[]): Promise<LangSmithExample[]>;
+  createProject(args: {
+    projectName: string;
+    referenceDatasetId?: string;
+    metadata?: Record<string, unknown>;
+  }): Promise<LangSmithProject>;
+  createRun(run: LangSmithRun): Promise<unknown>;
+  createFeedback(runId: string | null, key: string, options: {
+    score?: number;
+    sessionId?: string;
+  }): Promise<unknown>;
+}
+interface LangSmithReporterOptions<Datum> {
+  client: LangSmithClientLike;
+  /** Dataset holding the validation split. Created once, reused across runs. */
+  dataset: string;
+  /** Experiments are named `<experimentPrefix>/cand-<candidateId>`. */
+  experimentPrefix: string;
+  /** The validation set the run was given, in the same order. */
+  validationSet: readonly Datum[];
+  /**
+   * The held-out set the run was given. Uploaded as its own dataset, and only
+   * once the winner has been chosen — see the note on the reporter itself.
+   */
+  testSet?: readonly Datum[];
+  /** Defaults to `<dataset>-held-out`. */
+  testDataset?: string;
+  /**
+   * Names a dataset row. Defaults to its position, which is enough for a run
+   * whose validation set is stable; give a real id if the set is ever
+   * reordered or re-generated, since a row's identity is what a later run
+   * matches against when it writes to the same dataset.
+   */
+  instanceId?: (args: {
+    datum: Datum;
+    index: number;
+  }) => string;
+  toInput?: (datum: Datum) => Record<string, unknown>;
+  toExpected?: (datum: Datum) => Record<string, unknown>;
+  /** Rows uploaded at once within one experiment. Default 8. */
+  concurrency?: number;
+}
+/**
+ * Reports a GEPA run to LangSmith as one experiment per accepted candidate
+ * over one fixed dataset — which is what makes the candidates comparable.
+ * Selecting several in LangSmith's comparison view renders the candidate x
+ * instance score matrix the Pareto frontier is chosen on: which instances a
+ * candidate won, and which it paid for.
+ *
+ * Minibatch rollouts are deliberately absent. They are small random subsets of
+ * the *training* set that differ every iteration, so they cannot be compared
+ * across candidates and would bury the experiments that can. They remain
+ * traceable through the adapter, which tags every rollout with its iteration,
+ * phase, split and candidate id.
+ *
+ * The held-out sweep gets exactly one experiment, for the winner, on its own
+ * dataset. There is no option to log one per candidate: GEPA evaluates the
+ * held-out set only after selection is over precisely so that nothing can be
+ * chosen against it, and a per-candidate view hands that back the moment
+ * someone reads it.
+ */
+declare function createLangSmithReporter<Datum>(options: LangSmithReporterOptions<Datum>): Reporter<OptimizerEvent>;
+//#endregion
+export { type LangSmithClientLike, type LangSmithDataset, type LangSmithExample, type LangSmithProject, type LangSmithReporterOptions, type LangSmithRun, createLangSmithReporter };

package/dist/index.d.mts

@@ -0,0 +1,107 @@
+import { OptimizerEvent, Reporter } from "textopt";
+//#region src/reporter.d.ts
+interface LangSmithDataset {
+  id: string;
+}
+interface LangSmithProject {
+  id: string;
+  /** Optional because LangSmith's own `TracerSession` declares it so. */
+  name?: string;
+}
+interface LangSmithExample {
+  id?: string;
+  dataset_id?: string;
+  inputs: Record<string, unknown>;
+  outputs?: Record<string, unknown>;
+  metadata?: Record<string, unknown>;
+}
+interface LangSmithRun {
+  id?: string;
+  name: string;
+  run_type: string;
+  inputs: Record<string, unknown>;
+  outputs?: Record<string, unknown>;
+  project_name?: string;
+  reference_example_id?: string;
+  start_time?: number;
+  end_time?: number;
+}
+/**
+ * The part of LangSmith's `Client` this reporter uses. Structural rather than
+ * imported so the package carries no runtime dependency on the SDK, and so a
+ * test can hand it a recording client instead of a network.
+ */
+interface LangSmithClientLike {
+  hasDataset(args: {
+    datasetName: string;
+  }): Promise<boolean>;
+  readDataset(args: {
+    datasetName: string;
+  }): Promise<LangSmithDataset>;
+  createDataset(name: string, options?: {
+    description?: string;
+  }): Promise<LangSmithDataset>;
+  createExamples(uploads: LangSmithExample[]): Promise<LangSmithExample[]>;
+  createProject(args: {
+    projectName: string;
+    referenceDatasetId?: string;
+    metadata?: Record<string, unknown>;
+  }): Promise<LangSmithProject>;
+  createRun(run: LangSmithRun): Promise<unknown>;
+  createFeedback(runId: string | null, key: string, options: {
+    score?: number;
+    sessionId?: string;
+  }): Promise<unknown>;
+}
+interface LangSmithReporterOptions<Datum> {
+  client: LangSmithClientLike;
+  /** Dataset holding the validation split. Created once, reused across runs. */
+  dataset: string;
+  /** Experiments are named `<experimentPrefix>/cand-<candidateId>`. */
+  experimentPrefix: string;
+  /** The validation set the run was given, in the same order. */
+  validationSet: readonly Datum[];
+  /**
+   * The held-out set the run was given. Uploaded as its own dataset, and only
+   * once the winner has been chosen — see the note on the reporter itself.
+   */
+  testSet?: readonly Datum[];
+  /** Defaults to `<dataset>-held-out`. */
+  testDataset?: string;
+  /**
+   * Names a dataset row. Defaults to its position, which is enough for a run
+   * whose validation set is stable; give a real id if the set is ever
+   * reordered or re-generated, since a row's identity is what a later run
+   * matches against when it writes to the same dataset.
+   */
+  instanceId?: (args: {
+    datum: Datum;
+    index: number;
+  }) => string;
+  toInput?: (datum: Datum) => Record<string, unknown>;
+  toExpected?: (datum: Datum) => Record<string, unknown>;
+  /** Rows uploaded at once within one experiment. Default 8. */
+  concurrency?: number;
+}
+/**
+ * Reports a GEPA run to LangSmith as one experiment per accepted candidate
+ * over one fixed dataset — which is what makes the candidates comparable.
+ * Selecting several in LangSmith's comparison view renders the candidate x
+ * instance score matrix the Pareto frontier is chosen on: which instances a
+ * candidate won, and which it paid for.
+ *
+ * Minibatch rollouts are deliberately absent. They are small random subsets of
+ * the *training* set that differ every iteration, so they cannot be compared
+ * across candidates and would bury the experiments that can. They remain
+ * traceable through the adapter, which tags every rollout with its iteration,
+ * phase, split and candidate id.
+ *
+ * The held-out sweep gets exactly one experiment, for the winner, on its own
+ * dataset. There is no option to log one per candidate: GEPA evaluates the
+ * held-out set only after selection is over precisely so that nothing can be
+ * chosen against it, and a per-candidate view hands that back the moment
+ * someone reads it.
+ */
+declare function createLangSmithReporter<Datum>(options: LangSmithReporterOptions<Datum>): Reporter<OptimizerEvent>;
+//#endregion
+export { type LangSmithClientLike, type LangSmithDataset, type LangSmithExample, type LangSmithProject, type LangSmithReporterOptions, type LangSmithRun, createLangSmithReporter };

package/dist/index.mjs

@@ -0,0 +1,222 @@
+import { isCandidateAccepted, isRunFinished, mapWithConcurrency } from "textopt";
+//#region src/reporter.ts
+const DEFAULT_CONCURRENCY = 8;
+/**
+* A UUIDv5 namespace for this library, so a dataset row's id is a pure
+* function of the dataset it belongs to and the instance it names. Fixed
+* forever: changing it renames every row textopt has ever written.
+*/
+const TEXTOPT_NAMESPACE = "6f9619ff-8b86-d011-b42d-00c04fc964ff";
+/**
+* Reports a GEPA run to LangSmith as one experiment per accepted candidate
+* over one fixed dataset — which is what makes the candidates comparable.
+* Selecting several in LangSmith's comparison view renders the candidate x
+* instance score matrix the Pareto frontier is chosen on: which instances a
+* candidate won, and which it paid for.
+*
+* Minibatch rollouts are deliberately absent. They are small random subsets of
+* the *training* set that differ every iteration, so they cannot be compared
+* across candidates and would bury the experiments that can. They remain
+* traceable through the adapter, which tags every rollout with its iteration,
+* phase, split and candidate id.
+*
+* The held-out sweep gets exactly one experiment, for the winner, on its own
+* dataset. There is no option to log one per candidate: GEPA evaluates the
+* held-out set only after selection is over precisely so that nothing can be
+* chosen against it, and a per-candidate view hands that back the moment
+* someone reads it.
+*/
+function createLangSmithReporter(options) {
+	const { client, dataset, experimentPrefix, validationSet, testSet, testDataset = `${dataset}-held-out`, instanceId = ({ index }) => String(index), toInput = (datum) => datum, toExpected, concurrency = DEFAULT_CONCURRENCY } = options;
+	let pending = Promise.resolve();
+	let validationDatasetId;
+	let heldOutDatasetId;
+	function enqueue(work) {
+		pending = pending.then(work).catch((err) => {
+			console.warn("[textopt-langsmith] upload failed", { err });
+		});
+	}
+	async function ensureDataset(args) {
+		if (await client.hasDataset({ datasetName: args.name })) return (await client.readDataset({ datasetName: args.name })).id;
+		const created = await client.createDataset(args.name, { description: `textopt: ${String(args.rows.length)} instances` });
+		await client.createExamples(await Promise.all(args.rows.map(async (datum, index) => ({
+			id: await exampleIdFor({
+				dataset: args.name,
+				instance: instanceId({
+					datum,
+					index
+				})
+			}),
+			dataset_id: created.id,
+			inputs: toInput(datum),
+			...toExpected === void 0 ? {} : { outputs: toExpected(datum) },
+			metadata: { textopt_instance_id: instanceId({
+				datum,
+				index
+			}) }
+		}))));
+		return created.id;
+	}
+	async function writeExperiment(args) {
+		const project = await client.createProject({
+			projectName: args.projectName,
+			referenceDatasetId: args.datasetId,
+			metadata: args.metadata
+		});
+		const scored = args.rows.flatMap((datum, index) => {
+			const score = args.scores[index];
+			return score === void 0 ? [] : [{
+				datum,
+				index,
+				score
+			}];
+		});
+		await mapWithConcurrency({
+			items: scored,
+			limit: concurrency,
+			task: async ({ datum, index, score }) => {
+				const runId = crypto.randomUUID();
+				const now = Date.now();
+				await client.createRun({
+					id: runId,
+					name: args.projectName,
+					run_type: "chain",
+					inputs: toInput(datum),
+					outputs: { output: args.outputs?.[index] ?? null },
+					project_name: args.projectName,
+					reference_example_id: await exampleIdFor({
+						dataset: args.datasetName,
+						instance: instanceId({
+							datum,
+							index
+						})
+					}),
+					start_time: now,
+					end_time: now
+				});
+				await client.createFeedback(runId, "score", {
+					score,
+					sessionId: project.id
+				});
+			}
+		});
+	}
+	return {
+		onEvent: (event) => {
+			if (isCandidateAccepted(event)) enqueue(async () => {
+				validationDatasetId ??= ensureDataset({
+					name: dataset,
+					rows: validationSet
+				});
+				await writeExperiment({
+					projectName: `${experimentPrefix}/cand-${String(event.candidateId)}`,
+					datasetId: await validationDatasetId,
+					datasetName: dataset,
+					rows: validationSet,
+					scores: event.instanceScores,
+					outputs: event.outputs,
+					metadata: experimentMetadata({
+						candidate: event.candidate,
+						candidateId: event.candidateId,
+						aggregateScore: event.aggregateScore,
+						search: searchSpecific({ ...event })
+					})
+				});
+			});
+			if (isRunFinished(event) && event.testInstanceScores !== void 0 && testSet !== void 0) {
+				const heldOut = event.testInstanceScores;
+				const outputs = event.testOutputs;
+				const { bestCandidateId, testScore } = event;
+				enqueue(async () => {
+					heldOutDatasetId ??= ensureDataset({
+						name: testDataset,
+						rows: testSet
+					});
+					await writeExperiment({
+						projectName: `${experimentPrefix}/held-out-cand-${String(bestCandidateId)}`,
+						datasetId: await heldOutDatasetId,
+						datasetName: testDataset,
+						rows: testSet,
+						scores: heldOut,
+						outputs,
+						metadata: {
+							textopt_candidate_id: bestCandidateId,
+							textopt_split: "test",
+							...testScore === void 0 ? {} : { textopt_score: testScore }
+						}
+					});
+				});
+			}
+		},
+		flush: async () => {
+			await pending;
+		}
+	};
+}
+function experimentMetadata(args) {
+	const search = Object.fromEntries(Object.entries(args.search).map(([key, value]) => [`textopt_${key}`, value]));
+	return {
+		...args.candidate,
+		textopt_candidate_id: args.candidateId,
+		textopt_score: args.aggregateScore,
+		...search
+	};
+}
+/**
+* The fields the emitting optimizer added to the shared acceptance payload:
+* GEPA's lineage, MIPRO's menu choices, SIMBA's step. Read structurally rather
+* than per optimizer so a new search reports its own vocabulary without this
+* package learning about it.
+*/
+function searchSpecific(event) {
+	const shared = /* @__PURE__ */ new Set([
+		"type",
+		"candidateId",
+		"candidate",
+		"aggregateScore",
+		"instanceScores",
+		"outputs"
+	]);
+	return Object.fromEntries(Object.entries(event).filter(([key]) => !shared.has(key)));
+}
+/**
+* A dataset row's id, derived from the dataset and the instance it names
+* rather than assigned by the server.
+*
+* Server-assigned ids would give a resumed or repeated run a fresh set of
+* rows, and every experiment logged before the restart would have nothing left
+* to compare against.
+*/
+async function exampleIdFor(args) {
+	return uuidV5({
+		namespace: TEXTOPT_NAMESPACE,
+		name: `${args.dataset}:${args.instance}`
+	});
+}
+/** RFC 4122 name-based UUID, SHA-1 flavour. LangSmith ids must be UUIDs. */
+async function uuidV5(args) {
+	const namespaceBytes = uuidToBytes(args.namespace);
+	const nameBytes = new TextEncoder().encode(args.name);
+	const payload = new Uint8Array(namespaceBytes.length + nameBytes.length);
+	payload.set(namespaceBytes);
+	payload.set(nameBytes, namespaceBytes.length);
+	const bytes = new Uint8Array(await crypto.subtle.digest("SHA-1", payload)).slice(0, 16);
+	bytes[6] = bytes[6] & 15 | 80;
+	bytes[8] = bytes[8] & 63 | 128;
+	const hex = [...bytes].map((byte) => byte.toString(16).padStart(2, "0")).join("");
+	return [
+		hex.slice(0, 8),
+		hex.slice(8, 12),
+		hex.slice(12, 16),
+		hex.slice(16, 20),
+		hex.slice(20)
+	].join("-");
+}
+function uuidToBytes(uuid) {
+	const hex = uuid.replaceAll("-", "");
+	const bytes = /* @__PURE__ */ new Uint8Array(16);
+	for (let index = 0; index < bytes.length; index += 1) bytes[index] = Number.parseInt(hex.slice(index * 2, index * 2 + 2), 16);
+	return bytes;
+}
+//#endregion
+export { createLangSmithReporter };

package/package.json

@@ -0,0 +1,58 @@
+{
+  "name": "@textopt/langsmith",
+  "version": "0.0.0",
+  "description": "LangSmith experiment reporting for textopt",
+  "keywords": [
+    "prompt-optimization",
+    "gepa",
+    "langsmith",
+    "evaluation",
+    "llm"
+  ],
+  "license": "MIT",
+  "homepage": "https://github.com/ctdio/textopt#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/ctdio/textopt.git",
+    "directory": "packages/langsmith"
+  },
+  "bugs": {
+    "url": "https://github.com/ctdio/textopt/issues"
+  },
+  "type": "module",
+  "sideEffects": false,
+  "engines": {
+    "node": ">=22"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "exports": {
+    ".": {
+      "import": {
+        "types": "./dist/index.d.mts",
+        "default": "./dist/index.mjs"
+      },
+      "require": {
+        "types": "./dist/index.d.cts",
+        "default": "./dist/index.cjs"
+      }
+    }
+  },
+  "main": "./dist/index.cjs",
+  "module": "./dist/index.mjs",
+  "types": "./dist/index.d.cts",
+  "files": [
+    "dist"
+  ],
+  "dependencies": {
+    "textopt": "^0.1.0"
+  },
+  "devDependencies": {
+    "langsmith": "0.8.11"
+  },
+  "scripts": {
+    "build": "tsdown",
+    "typecheck": "tsc --noEmit"
+  }
+}