prism-evals 0.9.0__tar.gz

prism_evals-0.9.0/.github/workflows/publish-python.yml

@@ -0,0 +1,80 @@
+name: Publish Python Package
+
+on:
+  push:
+    tags:
+      - "v*"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: Test
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+
+      - name: Install package
+        run: python -m pip install --upgrade pip && python -m pip install -e ".[dev]"
+
+      - name: Run tests
+        run: python -m pytest
+
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    needs: test
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+
+      - name: Install build tools
+        run: python -m pip install --upgrade pip build twine
+
+      - name: Build distributions
+        run: python -m build
+
+      - name: Check distributions
+        run: python -m twine check dist/*
+
+      - name: Upload distributions
+        uses: actions/upload-artifact@v4
+        with:
+          name: python-package-distributions
+          path: dist/
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build
+    if: startsWith(github.ref, 'refs/tags/v')
+    environment:
+      name: pypi
+      url: https://pypi.org/p/prism-evals
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - name: Download distributions
+        uses: actions/download-artifact@v4
+        with:
+          name: python-package-distributions
+          path: dist/
+
+      - name: Publish distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

prism_evals-0.9.0/.gitignore

@@ -0,0 +1,24 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+
+.env
+.venv/
+venv/
+
+build/
+dist/
+
+/runs/
+/examples/runs/
+
+viewer/.next/
+viewer/node_modules/
+viewer/coverage/
+viewer/tsconfig.tsbuildinfo

prism_evals-0.9.0/AGENTS.md

@@ -0,0 +1,276 @@
+# AGENTS.md
+
+This repo provides Prism Evals, a local Python framework for executable OpenAI
+API experiments. The package distribution is `prism-evals`, and the Python
+import is `prism_evals`.
+
+Use this file as the quick orientation for LLM agents working in this package
+repo, and as a template for consuming repos that install Prism Evals.
+
+To seed Prism Evals instructions into a consuming repo, install the package and
+run this from the consuming repo root:
+
+```bash
+python -m prism_evals init
+```
+
+The installed console scripts also work:
+
+```bash
+prism init
+prism-evals init
+pe init
+```
+
+## What An Eval Experiment Is
+
+An experiment is a normal Python file that configures:
+
+- A CSV dataset, JSONL file, or folder of JSON/YAML scenario files.
+- One or more `ModelConfig` entries or named model variants.
+- A workflow callable assigned to `exp.workflow`.
+- Item-level evals registered with `exp.eval(...)`.
+- Optional step-level evals inside `ctx.step(...)`.
+
+Run an experiment with the Prism CLI:
+
+```bash
+prism run path/to/experiment.py
+```
+
+The minimal shape is:
+
+```python
+from openai import AsyncOpenAI
+
+from prism_evals import Experiment, ModelConfig, TaskOutput
+
+exp = Experiment(name="my_eval", dataset="datasets/my_eval.csv", output_dir="runs")
+exp.model(ModelConfig(key="gpt5_low", model="gpt-5", params={"reasoning": {"effort": "low"}}))
+client = AsyncOpenAI()
+
+async def workflow(item, model, ctx):
+    response = await client.responses.create(
+        model=model.model,
+        **model.params,
+        input=item["prompt"],
+    )
+    return TaskOutput(text=response.output_text)
+
+exp.workflow = workflow
+```
+
+Direct `python path/to/experiment.py` execution is still supported if the file
+includes an explicit `exp.run()` block.
+
+## Where To Make Changes
+
+When changing eval behavior in a consuming repo, edit the experiment Python files
+first. Those files are the source of truth for datasets, model choices,
+workflow logic, scoring rules, concurrency, retries, and output settings.
+
+Common change points:
+
+- Change prompts, tool calls, multi-step logic, or response parsing in the
+  workflow assigned to `exp.workflow`.
+- Change model coverage by editing `exp.model(...)`, `exp.models(...)`, or
+  `exp.variant(...)` for multi-agent role/model configurations.
+- Change pass/fail or scoring logic by editing `exp.eval(...)` or the `evals=`
+  list passed to `ctx.step(...)`.
+- Change data coverage by editing the CSV/JSONL file or scenario folder
+  referenced by `Experiment(dataset=...)`.
+- Change output placement by editing `Experiment(output_dir=...)`.
+- Change resume behavior with `resume=True` or `resume=False`.
+- Change repeated sampling with `repetitions=...`.
+- Change parallelism with `concurrency=...`.
+
+If prompts, rubrics, schemas, or other files are part of the experiment, keep
+them near the experiment file and pass them through `artifacts=[...]` so each run
+copies them into the run directory.
+
+## Paths Are Relative To The Experiment File
+
+Relative `dataset`, `output_dir`, and `artifacts` paths are resolved relative to
+the Python file that creates `Experiment(...)`, not necessarily the process
+working directory.
+
+For example, in `experiments/qa.py`:
+
+```python
+Experiment(
+    name="qa",
+    dataset="datasets/qa.csv",
+    output_dir="runs",
+    artifacts=["prompts/system.md"],
+)
+```
+
+This reads `experiments/datasets/qa.csv`, writes under `experiments/runs/`, and
+copies `experiments/prompts/system.md` into the run artifacts.
+
+## Result Storage
+
+Each `Experiment` instance chooses one run directory. With default settings, the
+timestamp is created when `Experiment(...)` is constructed:
+
+```text
+<output_dir>/<YYYYMMDD-HHMMSS>_<experiment_name>/
+```
+
+If `timestamp_output_dir=False`, the run directory is:
+
+```text
+<output_dir>/<experiment_name>/
+```
+
+Run directories contain:
+
+- `manifest.json`: run metadata, dataset hash, experiment file hash, settings,
+  model configs, git commit, copied artifact metadata, and output path.
+- `results.jsonl`: append-only item-run records. This is the most complete
+  machine-readable output.
+- `results.csv`: one row per item/model/repetition with flattened item fields,
+  final output text, usage, errors, item-level scores, and step score columns.
+- `scores.csv`: one row per score, including both item-level and step-level
+  scores.
+- `steps.csv`: one row per recorded workflow step, including step output text,
+  usage, errors, media columns, and step scores.
+- `turns.csv`: one row per recorded conversation turn.
+- `tool_calls.csv`: one row per recorded tool call.
+- `artifacts/`: optional copied prompt/rubric/schema files listed in
+  `artifacts=[...]`.
+- `media/`: generated outputs saved with `ctx.media`.
+
+`runs/` and `examples/runs/` are ignored by git in this repo. Treat run outputs
+as generated artifacts unless the consuming repo explicitly chooses to version
+selected reports.
+
+## How To Read Results
+
+Start with `manifest.json` to confirm the experiment file, dataset, model
+configs, and settings that produced the run.
+
+Use `results.csv` for quick spreadsheet-style inspection. Useful columns include:
+
+- `item_id`, `item_index`, `model_key`, `repetition`, and `status`.
+- `output_text` for the final workflow output.
+- `media_count`, `media_paths_json`, and `primary_media_path` for generated
+  output files.
+- `score:<eval_key>` and `score_error:<eval_key>` for item-level evals.
+- `step:<step_key>.score:<eval_key>` for step evals flattened into the item row.
+- `input_tokens`, `output_tokens`, `reasoning_tokens`, `total_tokens`, and
+  `latency_s` for usage and timing.
+- `error_type` and `error_message` for failed item runs.
+
+Use `scores.csv` when comparing eval metrics across models, repetitions, or
+steps. Filter by:
+
+- `scope=item_run` for final-output evals.
+- `scope=step` and `step_key=<name>` for step evals.
+- `score_key=<eval>` for a specific metric.
+
+Use `steps.csv` when debugging multi-step workflows. It shows each step's
+status, output text, media paths, token usage, latency, response ID, and
+`scores_json`.
+
+Use `results.jsonl` when full record structure matters. It preserves nested
+records for items, final `TaskOutput` values, evals, optional legacy generation
+records, steps, errors, usage, and media metadata.
+
+## Resume Behavior
+
+`resume=True` skips item/model/repetition records that already have
+`status == "success"` in the current run directory's `results.jsonl`.
+
+The item-run identity is based on experiment name, dataset hash, per-item hash,
+item id/index, model or variant key, and repetition. Changing dataset contents or
+a scenario file creates different item-run IDs.
+
+For a clean rerun, launch a fresh process with `timestamp_output_dir=True`,
+change the output directory or experiment name, or delete the old generated run
+directory.
+
+## Datasets
+
+Datasets can be CSV files, JSONL files, structured JSON/YAML scenario files, or
+folders containing one JSON/YAML scenario file per item.
+
+- Every item is passed to the workflow as `item`; CSV values remain strings,
+  while JSON/YAML items preserve nested lists and objects.
+- If the CSV has an `id` column, that value is used as `item_id`.
+- If `id` is missing or blank, the zero-based row index is used as `item_id`.
+- Empty CSV values are normalized to empty strings.
+- If `dataset` is a directory, Prism recursively reads `*.json`, `*.yaml`, and
+  `*.yml` files in stable path order. Files beginning with `_` are ignored.
+- Scenario turns support shorthand keys such as `user`, `assistant_seed`,
+  `assistant_expect`, and `action`.
+
+Keep dataset columns explicit and stable. Evals often use selectors such as
+`item("expected")`, so renaming columns can silently change scoring behavior.
+
+CSV can point to scenario files with `scenario_path`, or store compact turn lists
+in `turns_json`.
+
+## Workflows And Steps
+
+A workflow receives `(item, model, ctx)` and may be sync or async. It must
+return `TaskOutput`.
+
+Import and use the OpenAI SDK directly inside experiment files. Prism owns eval
+orchestration and output storage, not provider SDK calls.
+
+Use `ctx.step("step_key", callable_or_value, evals=[...])` for multi-step
+workflows. Step callables must return `TaskOutput`. Step outputs and step evals
+are written to both `results.jsonl` and the flattened CSV files.
+
+Use `ctx.conversation(...)`, `ctx.user(...)`, `ctx.assistant_seed(...)`,
+`ctx.action_seed(...)`, and `ctx.turn(...)` for multi-turn scenarios. Seeded
+turns record context only; generated turns also write a normal step with key
+`turn:<turn_id>`.
+
+Use `ctx.record_tool_call(...)` to capture app/tool invocations that matter for
+scoring. Built-ins such as `ToolCalled`, `ToolNotCalled`, and `ToolArgsEqual`
+can score those calls.
+
+Return `TaskOutput(text=..., value=..., media=[...])` for display text,
+structured data, and generated media. Built-in selectors such as `text()`,
+`out("path")`, `step("step_key.path")`, and `step_text("step_key")` depend on
+that structure. Use `ctx.media.from_base64(...)`, `ctx.media.from_bytes(...)`,
+or `ctx.media.from_path(...)` to save generated outputs into `media/`.
+
+Use `ctx.realtime.run_text(...)` or `ctx.realtime.run_audio(...)` for Realtime
+API evals. Realtime helpers still return data through `TaskOutput`, and audio
+outputs should be attached as run-local media.
+
+## Viewer UI Preferences
+
+When editing the local Next.js viewer, optimize for dense eval inspection rather
+than marketing-style presentation:
+
+- Use the full viewport width for run tables, compare views, and charts.
+- Keep visible copy short and product-facing; avoid implementation details in
+  the UI.
+- Put related controls in the same toolbar row when space allows, with secondary
+  controls aligned to the right.
+- Prefer familiar icon-only controls for common actions such as close and remove.
+  Use a small, light `×` with an accessible label rather than bordered text
+  buttons.
+- Keep chart previews compact. Avoid legends or section labels in small cards
+  when the surrounding context already names the metric.
+- For run trend charts, show runs chronologically left to right so the newest
+  run is on the right.
+- Preserve user filters across summaries and charts; model filters should affect
+  both the table and chart previews.
+
+## Testing Changes
+
+For this package repo:
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest
+```
+
+For a consuming repo, run the smallest relevant experiment first, then inspect
+the generated run directory before broadening concurrency, model coverage, or
+dataset size.

prism_evals-0.9.0/CHANGELOG.md

@@ -0,0 +1,159 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+This project follows Semantic Versioning. While the project is pre-1.0, minor
+versions may include API changes as the experiment framework settles.
+
+## [0.9.0] - 2026-05-21
+
+### Added
+
+- Added folder-backed JSON/YAML scenario datasets, JSONL datasets, CSV
+  `scenario_path`, and CSV `turns_json` expansion for multi-turn eval inputs.
+- Added named model variants for multi-agent workflows, with `ctx.model(role)`
+  access to role-specific `ModelConfig` entries.
+- Added conversation turn recording helpers, seeded user/assistant/action turns,
+  tool-call recording, `turns.csv`, `tool_calls.csv`, and tool-call built-in
+  evaluators.
+
+## [0.8.0] - 2026-05-11
+
+### Added
+
+- Added first-class Realtime workflow support through `ctx.realtime`, including
+  text and audio helpers for `gpt-realtime-2`.
+- Parsed Realtime tool calls into `RealtimeRunResult.tool_calls` and
+  `TaskOutput.value["tool_calls"]` for scoring.
+- Added Realtime text and voice-agent smoke examples, plus viewer playback for
+  audio media.
+
+## [0.7.0] - 2026-05-04
+
+### Changed
+
+- `TaskOutput` is now the required workflow and step output contract.
+- Workflows should import provider SDKs directly instead of relying on Prism to
+  proxy OpenAI calls.
+
+### Added
+
+- Added `TaskOutput.media`, `MediaArtifact`, run-local `media/` storage,
+  compact media columns in CSV outputs, and viewer media previews.
+- Added `ctx.media.from_base64(...)`, `ctx.media.from_bytes(...)`, and
+  `ctx.media.from_path(...)` helpers for generated outputs.
+
+## [0.6.8] - 2026-04-16
+
+### Added
+
+- `prism run <experiment_file>` discovers and runs module-level `Experiment`
+  instances, so eval files no longer need an explicit `exp.run()` block.
+
+## [0.6.7] - 2026-04-16
+
+### Added
+
+- Compact inline `data:` URLs in captured raw payloads by default with
+  `redact_raw_data_urls=True`.
+
+## [0.6.0] - 2026-04-16
+
+### Changed
+
+- Rebranded the package to Prism Evals.
+- Renamed the Python distribution to `prism-evals` and the import package to
+  `prism_evals`.
+- Added `prism`, `prism-evals`, and `pe` console scripts.
+- Updated the local viewer, scaffolded instructions, docs, examples, and tests
+  to use Prism Evals naming.
+
+## [0.5.1] - 2026-04-15
+
+### Fixed
+
+- Bundle the local viewer with Python wheels so installed packages can launch `prism view`.
+- Install viewer npm dependencies automatically on first launch when they are missing.
+
+## [0.5.0] - 2026-04-15
+
+### Added
+
+- `prism view <runs_dir>` opens a read-only local Next.js viewer for parent directories of eval runs.
+- Viewer pages for all runs, run detail, score matrices, artifact downloads, and lane comparisons across `run + model_key` pairs.
+
+## [0.4.5] - 2026-04-15
+
+### Added
+
+- `Experiment(..., artifacts=[...])` can copy prompt files, configs, and other user files into the run output folder.
+
+## [0.4.4] - 2026-04-15
+
+### Added
+
+- Console output now includes a model-column score pivot table by eval key.
+
+## [0.4.3] - 2026-04-15
+
+### Added
+
+- Console token usage now shows average per item run and totals for input, cached, output, reasoning, and total tokens.
+
+## [0.4.2] - 2026-04-15
+
+### Fixed
+
+- Declared the `src/prism_evals` package for Hatchling wheel builds so `prism-evals` installs correctly from Git tags.
+
+## [0.4.1] - 2026-04-15
+
+### Added
+
+- Timestamp-prefixed output directories by default, with `timestamp_output_dir=False` to keep stable experiment folders.
+
+## [0.4.0] - 2026-04-15
+
+### Added
+
+- Multi-step workflows via `ctx.step(...)`, with step-owned outputs, evals, generations, usage, latency, and errors.
+- Step selectors: `step(...)` and `step_text(...)`.
+- `steps.csv` artifact and scoped score rows in `scores.csv`.
+
+### Changed
+
+- Renamed public terminology from row/execution/task to item/item-run/workflow.
+- Replaced `exp.task = callable` with `exp.workflow = callable`.
+- Replaced the dataset selector `row(...)` with `item(...)`.
+- Renamed `ExecutionRecord` to `ItemRunRecord` and public result fields to item/item-run names.
+
+## [0.3.0] - 2026-04-15
+
+### Changed
+
+- Replaced task decorators with direct task assignment via `exp.task = callable`.
+- Removed eval decorator registration in favor of `exp.eval("key", evaluator)`.
+- Task callables may now be sync or async, including callable task objects.
+
+## [0.2.0] - 2026-04-15
+
+### Added
+
+- Unified `exp.eval("key", evaluator)` registration for custom functions and built-in evaluators.
+- Built-in deterministic evaluators for equality, approximate equality, containment, regex, non-empty values, length bounds, and JSON path checks.
+- Selector helpers: `row(...)`, `out(...)`, and `text(...)`.
+
+## [0.1.0] - 2026-04-15
+
+### Added
+
+- Initial Prism Evals Python package with decorator-style experiments.
+- `Experiment`, `ModelConfig`, `TaskOutput`, and `EvalResult` public APIs.
+- OpenAI Responses API wrapper for automatic raw request/response capture.
+- Token usage capture for input, cached, output, reasoning, and total tokens.
+- Per-call latency and per-execution duration tracking.
+- CSV dataset loading with row/model/repetition execution matrix.
+- Bounded async concurrency, retries, resume support, and fail-fast option.
+- Local artifacts: `manifest.json`, `results.jsonl`, `results.csv`, and `scores.csv`.
+- Rich terminal progress and summary tables.
+- Example QA experiment and pytest coverage.

prism_evals-0.9.0/MIGRATION.md

@@ -0,0 +1,86 @@
+# Migration Guide
+
+## 0.7.0: `TaskOutput` Is Required
+
+Prism workflows and step callables must now return `TaskOutput`. Strings and
+dicts are no longer normalized as workflow outputs.
+
+Prism also no longer owns OpenAI SDK calls for normal workflows. Import and use
+the OpenAI SDK directly in your experiment file:
+
+```python
+from openai import AsyncOpenAI
+
+client = AsyncOpenAI()
+```
+
+Use that client for Responses API, Image API, or any other OpenAI call, then
+return a `TaskOutput` with the data Prism should evaluate and store.
+
+Text output:
+
+```python
+from openai import AsyncOpenAI
+from prism_evals import TaskOutput
+
+client = AsyncOpenAI()
+
+response = await client.responses.create(...)
+
+# Old
+return response.output_text
+
+# New
+return TaskOutput(text=response.output_text)
+```
+
+Structured output:
+
+```python
+# Old
+return {"answer": parsed}
+
+# New
+return TaskOutput(text=json.dumps(parsed), value=parsed)
+```
+
+Step output:
+
+```python
+# Old
+draft = await ctx.step("draft", lambda: "hello")
+
+# New
+draft = await ctx.step("draft", lambda: TaskOutput(text="hello"))
+```
+
+Generated media:
+
+```python
+from openai import AsyncOpenAI
+from prism_evals import TaskOutput
+
+client = AsyncOpenAI()
+
+async def workflow(item, model, ctx):
+    response = await client.images.generate(
+        model=model.model,
+        prompt=item["prompt"],
+        response_format="b64_json",
+        **model.params,
+    )
+    image = ctx.media.from_base64(response.data[0].b64_json, format="png")
+    return TaskOutput(text="Generated image", media=[image])
+```
+
+Generated files now live in `media/`. The `artifacts/` directory remains for
+files copied from the experiment source tree with `Experiment(...,
+artifacts=[...])`.
+
+Custom JSONL consumers should read generated media from `output.media` or
+`steps[].output.media`. Prism no longer expects raw provider responses to be the
+source of truth for generated images.
+
+Direct SDK calls are not automatically recorded as Prism generation records, and
+raw OpenAI request/response payloads are not captured by default. Store any
+important call details in `TaskOutput.metadata` when your eval needs them.