@workglow/eval 0.3.26

package/README.md

@@ -0,0 +1,179 @@
+# Workglow Eval Example
+
+A command-line harness for comparing AI models on Workglow task workflows, using
+HuggingFace datasets as ground truth and Workglow storage for every artifact —
+dataset rows, eval runs, and per-row results all live in a queryable SQLite
+database.
+
+## Overview
+
+The [`@workglow/sec`](https://github.com/workglow-dev/sec) project ships an eval
+harness specialized for SEC filing extraction. This example is the
+general-purpose version of that idea:
+
+1. **Pull** a dataset split from HuggingFace into tabular storage.
+2. **Run** each stored row through a task workflow, once per candidate model.
+3. **Score** the stored results and rank the models on quality and latency.
+
+Because every step reads from and writes to `ITabularStorage`, runs are
+inspectable and repeatable: re-score without re-running, diff two runs, or point
+another tool at the same SQLite file.
+
+## Getting Started
+
+```bash
+bun install
+bun run build-example
+
+# 1. Pull a dataset split into storage
+./dist/workglow-eval.js dataset pull dair-ai/emotion --split test --limit 100
+./dist/workglow-eval.js dataset list
+
+# 2. Run an eval workflow across models (per-row workflow × models)
+./dist/workglow-eval.js run-classify --dataset dair-ai/emotion --split test \
+  --models "onnx-community/LFM2.5-350M-ONNX,claude-haiku-4-5,gpt-5.4-mini"
+
+# 3. Re-score any stored run later
+./dist/workglow-eval.js runs
+./dist/workglow-eval.js report [run-id] --format json
+```
+
+Everything is stored under `~/.workglow/eval` (override with
+`WORKGLOW_EVAL_HOME`). Re-pulling a split replaces it; pulling with `--offset`
+extends the stored split, so large splits can be paged in incrementally.
+
+## Eval workflows
+
+### `run-classify` — label accuracy
+
+Each row runs a one-task workflow: `StructuredGenerationTask` with an output
+schema whose `label` property is an enum of the candidate labels. Any model
+with text generation + JSON mode is comparable — local ONNX models and cloud
+models compete on the same rows. Scored by accuracy (case/punctuation
+insensitive).
+
+Works out of the box with classification datasets such as `dair-ai/emotion` or
+`SetFit/sst2`. Integer-coded label columns are mapped through the dataset's
+`ClassLabel` names automatically; use `--labels "a,b,c"` when the dataset
+doesn't declare them (integer gold labels are then mapped through that list,
+in order).
+
+### `run-similarity` — embedding quality
+
+Each row runs a one-task workflow: `TextEmbeddingTask` embeds a sentence pair,
+and the cosine similarity of the two vectors is the model's predicted score.
+Scored by Pearson/Spearman correlation against the dataset's gold ratings
+(correlation is scale-invariant, so 0–5 ratings work as-is).
+
+Works out of the box with `mteb/stsbenchmark-sts`:
+
+```bash
+./dist/workglow-eval.js dataset pull mteb/stsbenchmark-sts --split test --limit 200
+./dist/workglow-eval.js run-similarity --dataset mteb/stsbenchmark-sts --split test \
+  --models "Xenova/all-MiniLM-L6-v2,text-embedding-3-small"
+```
+
+### `run-extract` — structured extraction quality
+
+Each row runs a one-task workflow: `StructuredGenerationTask` with an `items`
+array output schema, extracting structured records (people, deals, entities…)
+from the row's prose. The gold column (`--expected-column`, default
+`expected`) holds an array of objects — parquet struct/list columns and JSON
+strings both work. Candidate items are aligned to gold rows by
+`--key-field` (default `name`, case/punctuation-insensitive) and scored on
+three axes, micro-averaged across the split:
+
+- **score** — field-level agreement over matched rows, excluding the key
+  field itself (a matched pair agrees on it by construction). The ranking
+  metric; falls back to **found** when no fields were scored.
+- **found** — entity recall (gold rows matched by key)
+- **prec** — precision over distinct candidate rows (1 − hallucinated rows)
+
+`--fields "a,b,c"` limits which fields are requested and scored (default: the
+keys seen in the gold rows); `--instruction "…"` replaces the task sentence at
+the top of the prompt.
+
+```bash
+./dist/workglow-eval.js run-extract --dataset my-org/people-extraction --split test \
+  --key-field name --fields "name,role" \
+  --instruction "Extract every person mentioned in the text." \
+  --models "claude-haiku-4-5,onnx-community/LFM2.5-350M-ONNX,gguf:prism-ml/Bonsai-27B-gguf:Q1_0"
+```
+
+This is the generalized form of the SEC repo's extraction eval: same
+alignment-by-key scoring (a model that emits the same entity twice is
+over-producing rows, not hallucinating, so precision is computed over
+distinct rows), but driven by any HuggingFace dataset instead of committed
+fixtures.
+
+## Model ids
+
+Model ids resolve to an inline `ModelConfig` by shape — no model repository
+setup needed:
+
+| Shape                                    | Provider                            |
+| ---------------------------------------- | ----------------------------------- |
+| `claude-*`                               | Anthropic (`ANTHROPIC_API_KEY`)     |
+| `gpt-*`, `o<n>*`, `text-embedding-*`     | OpenAI (`OPENAI_API_KEY`)           |
+| `gemini-*`                               | Google Gemini (`GEMINI_API_KEY`)    |
+| `grok-*`                                 | xAI (`XAI_API_KEY`)                 |
+| `org/name` (optionally `org/name:dtype`) | Local HuggingFace Transformers ONNX |
+| `gguf:org/repo:Quant` or `gguf:*.gguf`   | Local node-llama-cpp (GGUF)         |
+
+Local models download on first use into the eval home's cache and run in a
+worker; no API key required. Embedding dimensions for ONNX models are read from
+the model's `config.json` on the hub; GGUF references resolve through
+node-llama-cpp's `hf:` URIs (an explicit `gguf:` prefix is required because a
+GGUF repo id is indistinguishable from an ONNX one). Note that small instruct
+models vary widely in JSON mode reliability — `onnx-community/LFM2.5-350M-ONNX`
+(the classify default) follows the schema; e.g. SmolLM2-360M echoes the schema
+back instead of an answer. A model that fails a row is recorded as a failed
+result (visible in the report), never a crashed sweep.
+
+### Bonsai 27B
+
+The Bonsai 27B release (2026-07, Qwen3.6-27B base) is the kind of model this
+harness is for — a 1-bit/ternary model whose claim is cloud-class quality at
+local cost. Compare it against a cloud model on the same rows:
+
+```bash
+./dist/workglow-eval.js run-classify --dataset dair-ai/emotion --split test \
+  --models "gguf:prism-ml/Bonsai-27B-gguf:Q1_0,claude-haiku-4-5"
+```
+
+The ternary variant is `gguf:prism-ml/Ternary-Bonsai-27B-gguf:Q2_0`. As of the
+release only GGUF/MLX/AWQ conversions exist — when `onnx-community` publishes
+the 27B ONNX conversion it slots straight in as
+`onnx-community/Bonsai-27B-ONNX` (the smaller Bonsai ONNX sizes,
+e.g. `onnx-community/Bonsai-8B-ONNX`, work today).
+
+## Dataset fetching
+
+`dataset pull` prefers the HuggingFace datasets viewer API
+(`datasets-server.huggingface.co/rows`), which provides typed features and
+pagination. When that endpoint is unreachable it falls back to downloading the
+repo's data files directly from the hub, parsing parquet (including the
+`ClassLabel` names embedded in the parquet footer), `jsonl`, `jsonl.gz`,
+`ndjson`, and `json`. Set `HF_TOKEN` for gated or private datasets.
+
+## Storage layout
+
+| Table              | Key                         | Contents                              |
+| ------------------ | --------------------------- | ------------------------------------- |
+| `eval_dataset`     | (dataset, split)            | columns, ClassLabel names, row count  |
+| `eval_dataset_row` | (dataset, split, row_index) | raw row JSON                          |
+| `eval_run`         | run_id (uuid)               | kind, dataset, models, column options |
+| `eval_result`      | (run_id, model, row_index)  | expected/predicted, latency, ok/error |
+
+The same schemas run on any `ITabularStorage` backend — the unit tests exercise
+them with `InMemoryTabularStorage`, the CLI uses `SqliteTabularStorage`.
+
+## Tests
+
+```bash
+bun run test          # unit tests (scorers, parsers, storage round-trip)
+```
+
+`src/test/liveSimilarity.e2e.test.ts` (ONNX) and `src/test/ggufSmoke.e2e.test.ts`
+(GGUF) are live end-to-end checks (network + model download) and are excluded
+from the default vitest run like the rest of the repo's `.e2e.test.ts` files.

package/dist/commands/dataset.d.ts

@@ -0,0 +1,9 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { Command } from "commander";
+import type { EvalStores } from "../storage";
+export declare function registerDatasetCommand(program: Command, openStores: () => Promise<EvalStores>): void;
+//# sourceMappingURL=dataset.d.ts.map

package/dist/commands/dataset.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"dataset.d.ts","sourceRoot":"","sources":["../../src/commands/dataset.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAEzC,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAG7C,wBAAgB,sBAAsB,CACpC,OAAO,EAAE,OAAO,EAChB,UAAU,EAAE,MAAM,OAAO,CAAC,UAAU,CAAC,GACpC,IAAI,CA6EN"}

package/dist/commands/report.d.ts

@@ -0,0 +1,11 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { Command } from "commander";
+import type { EvalStores } from "../storage";
+/** Render the report for one stored run to stdout (table or json). */
+export declare function printReport(stores: EvalStores, runId: string, format: string): Promise<void>;
+export declare function registerReportCommand(program: Command, openStores: () => Promise<EvalStores>): void;
+//# sourceMappingURL=report.d.ts.map

package/dist/commands/report.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"report.d.ts","sourceRoot":"","sources":["../../src/commands/report.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAGzC,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAG7C,sEAAsE;AACtE,wBAAsB,WAAW,CAC/B,MAAM,EAAE,UAAU,EAClB,KAAK,EAAE,MAAM,EACb,MAAM,EAAE,MAAM,GACb,OAAO,CAAC,IAAI,CAAC,CAyDf;AAED,wBAAgB,qBAAqB,CACnC,OAAO,EAAE,OAAO,EAChB,UAAU,EAAE,MAAM,OAAO,CAAC,UAAU,CAAC,GACpC,IAAI,CA2CN"}

package/dist/commands/run.d.ts

@@ -0,0 +1,9 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { Command } from "commander";
+import type { EvalStores } from "../storage";
+export declare function registerRunCommand(program: Command, openStores: () => Promise<EvalStores>, ensureProviders: () => Promise<void>): void;
+//# sourceMappingURL=run.d.ts.map

package/dist/commands/run.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"run.d.ts","sourceRoot":"","sources":["../../src/commands/run.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAMzC,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AA0C7C,wBAAgB,kBAAkB,CAChC,OAAO,EAAE,OAAO,EAChB,UAAU,EAAE,MAAM,OAAO,CAAC,UAAU,CAAC,EACrC,eAAe,EAAE,MAAM,OAAO,CAAC,IAAI,CAAC,GACnC,IAAI,CA0CN"}

package/dist/config.d.ts

@@ -0,0 +1,29 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+export interface EvalConfig {
+    /** Directory holding the SQLite database and model cache. */
+    readonly home: string;
+    /** SQLite database file with dataset rows, runs, and results. */
+    readonly dbPath: string;
+    /** ONNX model cache directory for the HuggingFace Transformers worker. */
+    readonly modelCache: string;
+}
+/**
+ * The eval home directory (no side effects; safe to call from pure helpers).
+ * WORKGLOW_EVAL_HOME is set by the invoking user for their own machine — the
+ * same trust domain as the process itself, like HOME or XDG_DATA_HOME — and is
+ * normalized to an absolute path so a relative value anchors predictably.
+ */
+export declare function evalHome(): string;
+/** GGUF weights directory for the node-llama-cpp worker (used by the model resolver). */
+export declare function ggufCacheDir(): string;
+/**
+ * Resolve the eval home directory. Everything the harness persists (dataset
+ * rows, eval runs, per-row results, downloaded local models) lives under this
+ * one directory so it is easy to inspect and easy to delete.
+ */
+export declare function loadConfig(): EvalConfig;
+//# sourceMappingURL=config.d.ts.map

package/dist/config.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAMH,MAAM,WAAW,UAAU;IACzB,6DAA6D;IAC7D,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,iEAAiE;IACjE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,0EAA0E;IAC1E,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;CAC7B;AAED;;;;;GAKG;AACH,wBAAgB,QAAQ,IAAI,MAAM,CAGjC;AAED,yFAAyF;AACzF,wBAAgB,YAAY,IAAI,MAAM,CAErC;AAED;;;;GAIG;AACH,wBAAgB,UAAU,IAAI,UAAU,CAQvC"}

package/dist/evals/classify.d.ts

@@ -0,0 +1,25 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { ModelConfig } from "@workglow/ai";
+import type { DatasetRow, LabelNames } from "../hf/types";
+import type { ColumnOptions, DatasetContext, RowExecutor } from "./types";
+/**
+ * Resolve the gold label for a row: integer-coded labels (HuggingFace
+ * `ClassLabel`) are mapped through the dataset's label names, falling back to
+ * the candidate label list (`--labels`) when the dataset doesn't declare
+ * names; string labels pass through unchanged.
+ */
+export declare function expectedLabel(row: DatasetRow, labelColumn: string, labelNames: LabelNames, candidateLabels?: readonly string[]): string;
+/** The candidate label set: explicit flag, else the dataset's ClassLabel names. */
+export declare function resolveCandidateLabels(options: ColumnOptions, context: DatasetContext): readonly string[];
+export declare function buildClassifyPrompt(text: string, labels: readonly string[]): string;
+/**
+ * Per-row classification workflow: a single StructuredGenerationTask whose
+ * output schema constrains the label to the candidate set, so any model with
+ * text generation + JSON mode (cloud or local ONNX) is comparable.
+ */
+export declare function makeClassifyExecutor(model: ModelConfig, options: ColumnOptions, context: DatasetContext): RowExecutor;
+//# sourceMappingURL=classify.d.ts.map

package/dist/evals/classify.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"classify.d.ts","sourceRoot":"","sources":["../../src/evals/classify.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAEhD,OAAO,KAAK,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAE1D,OAAO,KAAK,EAAE,aAAa,EAAE,cAAc,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAE1E;;;;;GAKG;AACH,wBAAgB,aAAa,CAC3B,GAAG,EAAE,UAAU,EACf,WAAW,EAAE,MAAM,EACnB,UAAU,EAAE,UAAU,EACtB,eAAe,GAAE,SAAS,MAAM,EAAO,GACtC,MAAM,CAOR;AAED,mFAAmF;AACnF,wBAAgB,sBAAsB,CACpC,OAAO,EAAE,aAAa,EACtB,OAAO,EAAE,cAAc,GACtB,SAAS,MAAM,EAAE,CAQnB;AAED,wBAAgB,mBAAmB,CAAC,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,SAAS,MAAM,EAAE,GAAG,MAAM,CAOnF;AAED;;;;GAIG;AACH,wBAAgB,oBAAoB,CAClC,KAAK,EAAE,WAAW,EAClB,OAAO,EAAE,aAAa,EACtB,OAAO,EAAE,cAAc,GACtB,WAAW,CAkCb"}

package/dist/evals/extract.d.ts

@@ -0,0 +1,26 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { ModelConfig } from "@workglow/ai";
+import type { ExtractedRow } from "../score/extraction";
+import type { ColumnOptions, DatasetContext, RowExecutor } from "./types";
+/**
+ * Parse a gold extraction column: parquet struct/list columns and
+ * datasets-server JSON arrive as real arrays of objects; string columns hold
+ * JSON. Anything else is a data error surfaced as a per-row failure.
+ */
+export declare function parseExpectedRows(value: unknown, column: string): ExtractedRow[];
+/** The fields to extract and score: explicit flag, else keys seen in the gold rows. */
+export declare function resolveExtractionFields(expectedRows: readonly ExtractedRow[], keyField: string, fields: readonly string[] | undefined): string[];
+export declare function buildExtractPrompt(text: string, instruction: string, keyField: string, fields: readonly string[]): string;
+/**
+ * Per-row extraction workflow: a single StructuredGenerationTask whose output
+ * schema is an `items` array of objects keyed by the alignment field, so any
+ * model with text generation + JSON mode is comparable. The gold column
+ * supplies the expected rows; scoring aligns candidate items to them by
+ * `keyField` (see {@link scoreExtraction}).
+ */
+export declare function makeExtractExecutor(model: ModelConfig, options: ColumnOptions, context: DatasetContext): RowExecutor;
+//# sourceMappingURL=extract.d.ts.map

package/dist/evals/extract.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"extract.d.ts","sourceRoot":"","sources":["../../src/evals/extract.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAEhD,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,qBAAqB,CAAC;AAExD,OAAO,KAAK,EAAE,aAAa,EAAE,cAAc,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAW1E;;;;GAIG;AACH,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,GAAG,YAAY,EAAE,CAShF;AAED,uFAAuF;AACvF,wBAAgB,uBAAuB,CACrC,YAAY,EAAE,SAAS,YAAY,EAAE,EACrC,QAAQ,EAAE,MAAM,EAChB,MAAM,EAAE,SAAS,MAAM,EAAE,GAAG,SAAS,GACpC,MAAM,EAAE,CAaV;AAED,wBAAgB,kBAAkB,CAChC,IAAI,EAAE,MAAM,EACZ,WAAW,EAAE,MAAM,EACnB,QAAQ,EAAE,MAAM,EAChB,MAAM,EAAE,SAAS,MAAM,EAAE,GACxB,MAAM,CASR;AA+BD;;;;;;GAMG;AACH,wBAAgB,mBAAmB,CACjC,KAAK,EAAE,WAAW,EAClB,OAAO,EAAE,aAAa,EACtB,OAAO,EAAE,cAAc,GACtB,WAAW,CA4Cb"}

package/dist/evals/prompt.d.ts

@@ -0,0 +1,12 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+/**
+ * Fence untrusted dataset text for inclusion in a prompt. The quote delimiter
+ * grows until it does not occur in the text, so a row that itself contains
+ * `"""` cannot terminate the fence early and smuggle instructions.
+ */
+export declare function fenceText(text: string): string;
+//# sourceMappingURL=prompt.d.ts.map

package/dist/evals/prompt.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"prompt.d.ts","sourceRoot":"","sources":["../../src/evals/prompt.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH;;;;GAIG;AACH,wBAAgB,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAI9C"}

package/dist/evals/runner.d.ts

@@ -0,0 +1,25 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { EvalKind } from "../models";
+import type { DatasetRowRecord, EvalStores } from "../storage";
+import type { ColumnOptions, DatasetContext } from "./types";
+export interface SweepOptions {
+    readonly kind: EvalKind;
+    readonly dataset: string;
+    readonly split: string;
+    readonly models: readonly string[];
+    readonly columns: ColumnOptions;
+    readonly context: DatasetContext;
+    readonly onProgress?: ((done: number, total: number, model: string, ok: boolean) => void) | undefined;
+}
+/**
+ * Run every stored dataset row through the eval workflow for each model,
+ * persisting one result row per (run, model, dataset row). Failures are
+ * recorded (`ok: 0` + error) rather than aborting the sweep, so one bad model
+ * or one bad row never loses the rest of the run.
+ */
+export declare function runSweep(stores: EvalStores, rows: readonly DatasetRowRecord[], options: SweepOptions): Promise<string>;
+//# sourceMappingURL=runner.d.ts.map

package/dist/evals/runner.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"runner.d.ts","sourceRoot":"","sources":["../../src/evals/runner.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAIH,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAE1C,OAAO,KAAK,EAAE,gBAAgB,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAI/D,OAAO,KAAK,EAAE,aAAa,EAAE,cAAc,EAAe,MAAM,SAAS,CAAC;AAE1E,MAAM,WAAW,YAAY;IAC3B,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC;IACxB,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,MAAM,EAAE,SAAS,MAAM,EAAE,CAAC;IACnC,QAAQ,CAAC,OAAO,EAAE,aAAa,CAAC;IAChC,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAC;IACjC,QAAQ,CAAC,UAAU,CAAC,EAClB,CAAC,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,KAAK,IAAI,CAAC,GAAG,SAAS,CAAC;CACnF;AAoBD;;;;;GAKG;AACH,wBAAsB,QAAQ,CAC5B,MAAM,EAAE,UAAU,EAClB,IAAI,EAAE,SAAS,gBAAgB,EAAE,EACjC,OAAO,EAAE,YAAY,GACpB,OAAO,CAAC,MAAM,CAAC,CAwEjB"}

package/dist/evals/similarity.d.ts

@@ -0,0 +1,15 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { ModelConfig } from "@workglow/ai";
+import type { ColumnOptions, DatasetContext, RowExecutor } from "./types";
+/**
+ * Per-row sentence-similarity workflow: one TextEmbeddingTask embeds both
+ * sentences, and cosine similarity of the two vectors is the model's predicted
+ * score. Correlation against the dataset's gold score (e.g. STS-Benchmark's
+ * 0–5 ratings) is scale-invariant, so no normalization is needed.
+ */
+export declare function makeSimilarityExecutor(model: ModelConfig, options: ColumnOptions, context: DatasetContext): RowExecutor;
+//# sourceMappingURL=similarity.d.ts.map

package/dist/evals/similarity.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"similarity.d.ts","sourceRoot":"","sources":["../../src/evals/similarity.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAIhD,OAAO,KAAK,EAAE,aAAa,EAAE,cAAc,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAE1E;;;;;GAKG;AACH,wBAAgB,sBAAsB,CACpC,KAAK,EAAE,WAAW,EAClB,OAAO,EAAE,aAAa,EACtB,OAAO,EAAE,cAAc,GACtB,WAAW,CAiCb"}

package/dist/evals/types.d.ts

@@ -0,0 +1,39 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { DatasetRow, LabelNames } from "../hf/types";
+/** What one workflow execution produced for one dataset row. */
+export interface RowPrediction {
+    readonly expected?: string | undefined;
+    readonly predicted?: string | undefined;
+    readonly expectedValue?: number | undefined;
+    readonly predictedValue?: number | undefined;
+}
+/** Runs the eval workflow for a single dataset row against one model. */
+export type RowExecutor = (row: DatasetRow) => Promise<RowPrediction>;
+export interface ColumnOptions {
+    readonly textColumn: string;
+    /** classify: gold label column. */
+    readonly labelColumn: string;
+    /** classify: candidate labels; defaults to the dataset's ClassLabel names. */
+    readonly labels?: readonly string[] | undefined;
+    /** similarity: second sentence column. */
+    readonly pairColumn: string;
+    /** similarity: gold similarity score column. */
+    readonly scoreColumn: string;
+    /** extract: gold column holding an array of objects (or its JSON string). */
+    readonly expectedColumn: string;
+    /** extract: field that identifies an entity when aligning rows. */
+    readonly keyField: string;
+    /** extract: fields to request and score; defaults to keys seen in the gold rows. */
+    readonly fields?: readonly string[] | undefined;
+    /** extract: task sentence at the top of the prompt. */
+    readonly instruction?: string | undefined;
+}
+export interface DatasetContext {
+    readonly columns: readonly string[];
+    readonly labelNames: LabelNames;
+}
+//# sourceMappingURL=types.d.ts.map

package/dist/evals/types.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/evals/types.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAE1D,gEAAgE;AAChE,MAAM,WAAW,aAAa;IAC5B,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACvC,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACxC,QAAQ,CAAC,aAAa,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC5C,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAC9C;AAED,yEAAyE;AACzE,MAAM,MAAM,WAAW,GAAG,CAAC,GAAG,EAAE,UAAU,KAAK,OAAO,CAAC,aAAa,CAAC,CAAC;AAEtE,MAAM,WAAW,aAAa;IAC5B,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;IAC5B,mCAAmC;IACnC,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,8EAA8E;IAC9E,QAAQ,CAAC,MAAM,CAAC,EAAE,SAAS,MAAM,EAAE,GAAG,SAAS,CAAC;IAChD,0CAA0C;IAC1C,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;IAC5B,gDAAgD;IAChD,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,6EAA6E;IAC7E,QAAQ,CAAC,cAAc,EAAE,MAAM,CAAC;IAChC,mEAAmE;IACnE,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,oFAAoF;IACpF,QAAQ,CAAC,MAAM,CAAC,EAAE,SAAS,MAAM,EAAE,GAAG,SAAS,CAAC;IAChD,uDAAuD;IACvD,QAAQ,CAAC,WAAW,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAC3C;AAED,MAAM,WAAW,cAAc;IAC7B,QAAQ,CAAC,OAAO,EAAE,SAAS,MAAM,EAAE,CAAC;IACpC,QAAQ,CAAC,UAAU,EAAE,UAAU,CAAC;CACjC"}

package/dist/hf/auth.d.ts

@@ -0,0 +1,8 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+/** Bearer-token headers for HuggingFace requests (gated/private repos). */
+export declare function hfAuthHeaders(token?: string | undefined): Record<string, string>;
+//# sourceMappingURL=auth.d.ts.map

package/dist/hf/auth.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"auth.d.ts","sourceRoot":"","sources":["../../src/hf/auth.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,2EAA2E;AAC3E,wBAAgB,aAAa,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,SAAS,GAAG,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAGhF"}

package/dist/hf/datasetsServer.d.ts

@@ -0,0 +1,14 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { FetchDatasetOptions, FetchedDataset } from "./types";
+/**
+ * Fetch rows through the HuggingFace datasets viewer API
+ * (`datasets-server.huggingface.co/rows`). Works for any public dataset the
+ * viewer has processed and includes `ClassLabel` names so integer label
+ * columns can be mapped back to strings.
+ */
+export declare function fetchViaDatasetsServer(options: FetchDatasetOptions): Promise<FetchedDataset>;
+//# sourceMappingURL=datasetsServer.d.ts.map

package/dist/hf/datasetsServer.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"datasetsServer.d.ts","sourceRoot":"","sources":["../../src/hf/datasetsServer.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAIH,OAAO,KAAK,EAAc,mBAAmB,EAAE,cAAc,EAAc,MAAM,SAAS,CAAC;AAmD3F;;;;;GAKG;AACH,wBAAsB,sBAAsB,CAC1C,OAAO,EAAE,mBAAmB,GAC3B,OAAO,CAAC,cAAc,CAAC,CAqCzB"}

package/dist/hf/hubFiles.d.ts

@@ -0,0 +1,38 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { DatasetRow, FetchDatasetOptions, FetchedDataset, LabelNames } from "./types";
+export interface HubTreeEntry {
+    readonly type: "file" | "directory";
+    readonly path: string;
+}
+/**
+ * Rank a repo file for a requested split. Data files on the hub conventionally
+ * embed the split in the file name (`test-00000-of-00001.parquet`, `test.jsonl.gz`)
+ * or live in a split-named directory. Returns undefined when the file cannot
+ * belong to the split.
+ */
+export declare function scoreFileForSplit(path: string, split: string): number | undefined;
+/** Pick the data files (shards, sorted) that best match the requested split. */
+export declare function selectSplitFiles(files: readonly string[], split: string): string[];
+/** JSON-safe copy of a row; see {@link sanitizeValue}. */
+export declare function sanitizeRow(row: DatasetRow): DatasetRow;
+export declare function parseJsonLines(text: string): DatasetRow[];
+/**
+ * Parse one downloaded dataset file into rows (+ any `ClassLabel` names the
+ * parquet footer carries). `limit` bounds how many rows are decoded.
+ */
+export declare function parseDatasetFile(path: string, bytes: ArrayBuffer, limit: number): Promise<{
+    rows: DatasetRow[];
+    labelNames: LabelNames;
+}>;
+/**
+ * Fetch rows by downloading the dataset's data files straight from the hub
+ * (`huggingface.co/datasets/<id>/resolve/main/...`). Used when the datasets
+ * viewer API is unreachable (e.g. restricted egress). Supports parquet,
+ * jsonl(.gz), ndjson, and json(.gz) files.
+ */
+export declare function fetchViaHubFiles(options: FetchDatasetOptions): Promise<FetchedDataset>;
+//# sourceMappingURL=hubFiles.d.ts.map

package/dist/hf/hubFiles.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"hubFiles.d.ts","sourceRoot":"","sources":["../../src/hf/hubFiles.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAOH,OAAO,KAAK,EAAE,UAAU,EAAE,mBAAmB,EAAE,cAAc,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AAI3F,MAAM,WAAW,YAAY;IAC3B,QAAQ,CAAC,IAAI,EAAE,MAAM,GAAG,WAAW,CAAC;IACpC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;CACvB;AAaD;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,GAAG,SAAS,CASjF;AAED,gFAAgF;AAChF,wBAAgB,gBAAgB,CAAC,KAAK,EAAE,SAAS,MAAM,EAAE,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,EAAE,CAalF;AAyBD,0DAA0D;AAC1D,wBAAgB,WAAW,CAAC,GAAG,EAAE,UAAU,GAAG,UAAU,CAEvD;AAED,wBAAgB,cAAc,CAAC,IAAI,EAAE,MAAM,GAAG,UAAU,EAAE,CAMzD;AAED;;;GAGG;AACH,wBAAsB,gBAAgB,CACpC,IAAI,EAAE,MAAM,EACZ,KAAK,EAAE,WAAW,EAClB,KAAK,EAAE,MAAM,GACZ,OAAO,CAAC;IAAE,IAAI,EAAE,UAAU,EAAE,CAAC;IAAC,UAAU,EAAE,UAAU,CAAA;CAAE,CAAC,CAwCzD;AAyBD;;;;;GAKG;AACH,wBAAsB,gBAAgB,CAAC,OAAO,EAAE,mBAAmB,GAAG,OAAO,CAAC,cAAc,CAAC,CA6B5F"}

package/dist/hf/ids.d.ts

@@ -0,0 +1,20 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+/**
+ * Validate and re-encode a HuggingFace repo id (`name` or `org/name`) for use
+ * in a hub URL. Ids come from CLI flags, so each segment is checked against
+ * the hub's id alphabet, dot-only segments are rejected, and the returned
+ * value is rebuilt from URL-encoded segments — user input can neither change
+ * the request host nor traverse to a different API path.
+ */
+export declare function sanitizeHubRepoId(id: string): string;
+/**
+ * Validate and re-encode a file path inside a hub repo (e.g. from the repo's
+ * tree listing) for use in a `resolve/` download URL. Rejects empty, `.`, and
+ * `..` segments and URL-encodes the rest.
+ */
+export declare function sanitizeHubFilePath(path: string): string;
+//# sourceMappingURL=ids.d.ts.map

package/dist/hf/ids.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"ids.d.ts","sourceRoot":"","sources":["../../src/hf/ids.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAOH;;;;;;GAMG;AACH,wBAAgB,iBAAiB,CAAC,EAAE,EAAE,MAAM,GAAG,MAAM,CAMpD;AAED;;;;GAIG;AACH,wBAAgB,mBAAmB,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAWxD"}

package/dist/hf/pullDataset.d.ts

@@ -0,0 +1,26 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { EvalStores } from "../storage";
+import type { FetchDatasetOptions, FetchedDataset } from "./types";
+/**
+ * Fetch dataset rows, preferring the datasets viewer API (typed features,
+ * server-side pagination) and falling back to downloading the repo's data
+ * files directly when the viewer is unreachable.
+ */
+export declare function fetchDataset(options: FetchDatasetOptions): Promise<FetchedDataset>;
+/**
+ * Fetch a dataset split and persist it into tabular storage: one metadata row
+ * per (dataset, split) plus one row per example. A plain re-pull replaces the
+ * stored split; a pull with `--offset` extends it (only rows from the offset
+ * onward are replaced), so a split can be paged in incrementally. The replace
+ * runs in a transaction so a failed pull cannot leave the split half-wiped
+ * (real rollback on SQLite; best-effort on backends without transactions).
+ */
+export declare function pullDatasetIntoStorage(stores: EvalStores, options: FetchDatasetOptions): Promise<{
+    numRows: number;
+    source: string;
+}>;
+//# sourceMappingURL=pullDataset.d.ts.map

package/dist/hf/pullDataset.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"pullDataset.d.ts","sourceRoot":"","sources":["../../src/hf/pullDataset.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAG7C,OAAO,KAAK,EAAE,mBAAmB,EAAE,cAAc,EAAE,MAAM,SAAS,CAAC;AAEnE;;;;GAIG;AACH,wBAAsB,YAAY,CAAC,OAAO,EAAE,mBAAmB,GAAG,OAAO,CAAC,cAAc,CAAC,CAUxF;AAED;;;;;;;GAOG;AACH,wBAAsB,sBAAsB,CAC1C,MAAM,EAAE,UAAU,EAClB,OAAO,EAAE,mBAAmB,GAC3B,OAAO,CAAC;IAAE,OAAO,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,CAAC,CAmC9C"}

package/dist/hf/types.d.ts

@@ -0,0 +1,33 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+/** A dataset row as a plain JSON object keyed by column name. */
+export type DatasetRow = Record<string, unknown>;
+/**
+ * Integer-coded label columns and their string names, as declared by the
+ * dataset (HuggingFace `ClassLabel` features). Keyed by column name.
+ */
+export type LabelNames = Record<string, readonly string[]>;
+export interface FetchedDataset {
+    readonly rows: readonly DatasetRow[];
+    readonly columns: readonly string[];
+    readonly labelNames: LabelNames;
+    /** The config the rows came from ("" when unknown, e.g. unfiltered hub files). */
+    readonly config: string;
+    /** Which fetch path produced the rows. */
+    readonly source: "datasets-server" | "hub-files";
+}
+export interface FetchDatasetOptions {
+    readonly dataset: string;
+    /** Dataset config name; defaults to the first config the server reports. */
+    readonly config?: string | undefined;
+    readonly split: string;
+    /** Maximum number of rows to fetch. */
+    readonly limit: number;
+    readonly offset?: number | undefined;
+    /** HF access token for gated/private datasets (defaults to HF_TOKEN env). */
+    readonly token?: string | undefined;
+}
+//# sourceMappingURL=types.d.ts.map

package/dist/hf/types.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/hf/types.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,iEAAiE;AACjE,MAAM,MAAM,UAAU,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;AAEjD;;;GAGG;AACH,MAAM,MAAM,UAAU,GAAG,MAAM,CAAC,MAAM,EAAE,SAAS,MAAM,EAAE,CAAC,CAAC;AAE3D,MAAM,WAAW,cAAc;IAC7B,QAAQ,CAAC,IAAI,EAAE,SAAS,UAAU,EAAE,CAAC;IACrC,QAAQ,CAAC,OAAO,EAAE,SAAS,MAAM,EAAE,CAAC;IACpC,QAAQ,CAAC,UAAU,EAAE,UAAU,CAAC;IAChC,kFAAkF;IAClF,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,0CAA0C;IAC1C,QAAQ,CAAC,MAAM,EAAE,iBAAiB,GAAG,WAAW,CAAC;CAClD;AAED,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB,4EAA4E;IAC5E,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACrC,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,uCAAuC;IACvC,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACrC,6EAA6E;IAC7E,QAAQ,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CACrC"}