mistralai-workflows-plugins-evaluations 0.5.0__tar.gz

mistralai_workflows_plugins_evaluations-0.5.0/.gitignore

@@ -0,0 +1,32 @@
+# python sdk
+build
+*.egg-info
+*.egg
+
+# Env files
+.env
+.env.*
+!.env.example
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+.pytest_cache/
+.coverage
+htmlcov/
+
+# IDEs
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# docs
+docs/node_modules/**
+docs/.docusaurus/**
+docs/pnpm-lock.yaml
+.docs.temporal.io/
+
+#test
+/.test_durations

mistralai_workflows_plugins_evaluations-0.5.0/CHANGELOG.md

@@ -0,0 +1,167 @@
+# Changelog
+
+## Unreleased
+
+### Breaking
+
+- **Direction moved from `Goal` to the evaluator.** `Goal.higher_is_better()` and `Goal.lower_is_better()` are removed, and `Goal` no longer carries a direction — it's a pure gate. Set `direction="maximize"` / `"minimize"` on the `Evaluator`/`RunEvaluator` instead (or rely on inference from a `gte`/`lte` goal). See [Migrating](docs/docs/guides/migrating.mdx).
+
+  ```python
+  # Before
+  Evaluator(name="latency", scorer=latency_scorer, goal=Goal.lower_is_better())
+  # After
+  Evaluator(name="latency", scorer=latency_scorer, direction="minimize")
+  ```
+
+  **Default direction is `maximize`.** An unset direction with no `lte` goal resolves to higher-is-better. Staying on an older plugin means evaluators are sent without a direction, so minimize-metrics (latency, cost…) are treated as maximize — upgrade and set `direction="minimize"`.
+
+- **`MutatorProposal` fields renamed** (custom mutators only). `overrides` → `changed`, `reasoning` → `hypothesis`. If you write a `GEPA(mutator=...)`, return the new field names — an un-updated mutator fails silently.
+
+  ```python
+  # Before
+  return MutatorProposal(overrides={"temperature": 0.2}, reasoning="cooler decoding")
+  # After
+  return MutatorProposal(changed={"temperature": 0.2}, hypothesis="cooler decoding")
+  ```
+
+- **`optimize()` return type redesigned.** `evaluation.optimize()` now returns the new `OptimizeResult` (was `OptimizationResult`): `optimized` → `winner` (plus `best_attempt`), `improved`/`improvement` → `verdict` + `winner.gain`, `candidates` → `trajectory`. Update code that reads the result; calling `optimize()` is unchanged.
+
+### Added
+
+- **Optimizer v0 persistence & trials.** Optimizations and their trials are now persisted (create-first, with a detail link back to AI Studio); each run's objective and spec are persisted alongside. Includes weighted evaluators and score normalization.
+
+- **Metadata callbacks** for runs and records, plus `beta.update_run` for post-hoc run metadata updates.
+
+- **Metric semantics on evaluators.** `Evaluator` accepts `direction`, `weight`, `min_value`, `max_value`; `RunEvaluator` accepts `direction`, `min_value`, `max_value` (no `weight` — run evaluators aren't part of the optimizer objective). These are persisted per-evaluator and drive AI Studio's comparison colouring and score-graph bounds.
+
+### Fixed
+
+- **Raise the `mistralai-evaluations` floor to `>=0.6.0`.** The plugin now imports SDK symbols introduced in 0.6.0 (metric-semantics on evaluators, optimization result models, run/record metadata contexts, create/patch optimization endpoints). Resolving this plugin against an older SDK raised `ImportError` on load.
+
+- Raise the `mistralai-workflows` floor to `>=3.8.0`. Since 0.3.0 the plugin contributes the `eval-record` child workflow to the worker via the `get_worker_interceptors` hook, but forwarding plugin-contributed workflows to the registration payload only landed in core 3.8.0. Resolving this plugin against an older core produced a worker that *ran* `eval-record` without ever registering it, so every event it emitted was rejected by the API and its runs never appeared in run history.
+
+## 0.3.0
+
+### Breaking
+
+- **Workflow-as-task param contract changed.** When using a workflow class as the task, the entrypoint now receives a serialized `TaskContext` (`{"input_record": ..., "system": ..., "metadata": ...}`) instead of `params = <input_record>` directly.
+
+  ```python
+  # Before
+  @workflow.define(name="my-task")
+  class MyTask:
+      @workflow.entrypoint
+      async def run(self, params: dict) -> str:
+          text = params["text"]  # params WAS the input record
+          ...
+
+  # After
+  @workflow.define(name="my-task")
+  class MyTask:
+      @workflow.entrypoint
+      async def run(self, params: dict) -> str:
+          # params is a serialized TaskContext
+          input_record = params["input_record"]
+          system = params.get("system")
+          metadata = params.get("metadata")
+          text = input_record["text"]
+          ...
+
+          # Or, for typed access:
+          ctx = TaskContext.model_validate(params)
+          text = ctx.input_record["text"]
+  ```
+
+### Added
+
+- **`TaskContext` and `ScorerContext` context objects.** New typed context objects that bundle all inputs for eval callbacks. Detected via type annotation — legacy positional-arg signatures still work unchanged.
+
+  ```python
+  from mistralai.workflows.plugins.evaluations.types import TaskContext, ScorerContext, Score
+
+  # New style — access to system, metadata, and more
+  @evaluation.task
+  async def my_task(context: TaskContext) -> str:
+      prompt = context.system.params["system_prompt"]
+      verbose = context.metadata.get("verbose", False)
+      return context.input_record["text"]
+
+  @evaluation.scorer
+  async def judge(context: ScorerContext) -> Score:
+      model = context.system.params["judge_model"]
+      return Score(value=score_output(context.output, model))
+
+  # Legacy style — still works, no changes needed
+  @evaluation.task
+  async def my_task(input_record: dict) -> str: ...
+
+  @evaluation.scorer
+  async def accuracy(input_record: dict, output: str) -> Score: ...
+  ```
+
+- `TaskContext`, `ScorerContext` re-exported from `mistralai.workflows.plugins.evaluations.types`.
+
+- **Goal support.** `Evaluator` and `RunEvaluator` `goal` / `aggregate_goal` fields are now forwarded to the backend when creating evaluation runs. Use `Goal.gte()`, `Goal.lte()`, `Goal.between()`, etc.
+
+  ```python
+  from mistralai.workflows.plugins.evaluations.types import Evaluator, Goal
+
+  Evaluator(name="accuracy", scorer=my_scorer, goal=Goal.gte(0.8))
+  Evaluator(name="quality", scorer=my_scorer, aggregate_goal=Goal.between(0.5, 1.0))
+  ```
+
+- `Goal` and `GoalSpec` re-exported from `mistralai.workflows.plugins.evaluations.types`.
+
+- **System & metadata passthrough.** `system` and `metadata` from `evaluation.run()` are now forwarded to tasks, scorers, and run evaluators:
+  - Activity tasks: via `TaskContext` (new) or as a second positional arg (legacy)
+  - Scorers: via `ScorerContext` (new) — legacy scorers don't receive them
+  - Workflow-as-task: via the serialized `TaskContext` params dict
+  - `RunEvaluatorContext.system` new field
+
+- **Metadata forwarded to run evaluators in local mode.** `metadata` passed to `evaluation.run()` is now correctly forwarded to `RunEvaluatorContext.metadata` when `local=True`.
+
+## 0.2.0
+
+### Breaking
+
+- **`start_to_close_timeout` removed** from `@evaluation.task`, `@evaluation.scorer`, and `@evaluation.run_scorer`. Use `execution_timeout` instead, which aligns with the `@workflow.define` convention.
+
+  ```python
+  # Before
+  @evaluation.task(start_to_close_timeout=timedelta(minutes=30))
+  @evaluation.scorer(start_to_close_timeout=timedelta(minutes=10))
+
+  # After
+  @evaluation.task(execution_timeout=timedelta(minutes=30))
+  @evaluation.scorer(execution_timeout=timedelta(minutes=10))
+  ```
+
+- **Default timeouts and retries lowered.** Previous defaults were too generous for typical LLM evaluation workloads (a single API call rarely needs 10 minutes or 3 retries). If your tasks or scorers need more time, override explicitly.
+
+  | Decorator | `execution_timeout` | `retry_policy_max_attempts` |
+  |---|---|---|
+  | `@evaluation.task` | 10min → **2min** | 3 → **2** |
+  | `@evaluation.scorer` | 5min → **1min** | 3 → **2** |
+  | `@evaluation.run_scorer` | 5min → **1min** | 3 → **2** |
+
+- **`max_concurrent_tasks` and `max_concurrent_scorers` removed** from `evaluation.run()` and `evaluation.score()`. Use `max_concurrency` on the decorators instead.
+
+  ```python
+  # Before
+  await evaluation.run(dataset=..., task=my_task, evaluators=..., max_concurrent_tasks=10, max_concurrent_scorers=3)
+
+  # After
+  @evaluation.task(max_concurrency=10)
+  async def my_task(input: dict) -> dict: ...
+
+  @evaluation.scorer(max_concurrency=3)
+  async def my_scorer(input: dict, output: Any) -> Score: ...
+
+  await evaluation.run(dataset=..., task=my_task, evaluators=...)
+  ```
+
+### Added
+
+- `execution_timeout` on `eval-record` child workflows — auto-computed from task and scorer timeouts, retries, and a 60s buffer. Prevents a single hanging record from blocking the entire evaluation run.
+- `execution_timeout` on task child workflows (workflow mode) — uses the task's declared `@workflow.define(execution_timeout=...)` instead of a hardcoded 600s.
+- `max_concurrency` parameter on `@evaluation.task`, `@evaluation.scorer`, and `@evaluation.run_scorer` decorators (default: 5). Concurrency config now lives alongside timeouts and retries on the decorators, instead of being passed to `run()`.

mistralai_workflows_plugins_evaluations-0.5.0/LICENSE

@@ -0,0 +1,190 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+   "License" shall mean the terms and conditions for use, reproduction,
+   and distribution as defined by Sections 1 through 9 of this document.
+
+   "Licensor" shall mean the copyright owner or entity authorized by
+   the copyright owner that is granting the License.
+
+   "Legal Entity" shall mean the union of the acting entity and all
+   other entities that control, are controlled by, or are under common
+   control with that entity. For the purposes of this definition,
+   "control" means (i) the power, direct or indirect, to cause the
+   direction or management of such entity, whether by contract or
+   otherwise, or (ii) ownership of fifty percent (50%) or more of the
+   outstanding shares, or (iii) beneficial ownership of such entity.
+
+   "You" (or "Your") shall mean an individual or Legal Entity
+   exercising permissions granted by this License.
+
+   "Source" form shall mean the preferred form for making modifications,
+   including but not limited to software source code, documentation
+   source, and configuration files.
+
+   "Object" form shall mean any form resulting from mechanical
+   transformation or translation of a Source form, including but
+   not limited to compiled object code, generated documentation,
+   and conversions to other media types.
+
+   "Work" shall mean the work of authorship, whether in Source or
+   Object form, made available under the License, as indicated by a
+   copyright notice that is included in or attached to the work
+   (an example is provided in the Appendix below).
+
+   "Derivative Works" shall mean any work, whether in Source or Object
+   form, that is based on (or derived from) the Work and for which the
+   editorial revisions, annotations, elaborations, or other modifications
+   represent, as a whole, an original work of authorship. For the purposes
+   of this License, Derivative Works shall not include works that remain
+   separable from, or merely link (or bind by name) to the interfaces of,
+   the Work and Derivative Works thereof.
+
+   "Contribution" shall mean any work of authorship, including
+   the original version of the Work and any modifications or additions
+   to that Work or Derivative Works thereof, that is intentionally
+   submitted to Licensor for inclusion in the Work by the copyright owner
+   or by an individual or Legal Entity authorized to submit on behalf of
+   the copyright owner. For the purposes of this definition, "submitted"
+   means any form of electronic, verbal, or written communication sent
+   to the Licensor or its representatives, including but not limited to
+   communication on electronic mailing lists, source code control systems,
+   and issue tracking systems that are managed by, or on behalf of, the
+   Licensor for the purpose of discussing and improving the Work, but
+   excluding communication that is conspicuously marked or otherwise
+   designated in writing by the copyright owner as "Not a Contribution."
+
+   "Contributor" shall mean Licensor and any individual or Legal Entity
+   on behalf of whom a Contribution has been received by Licensor and
+   subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   copyright license to reproduce, prepare Derivative Works of,
+   publicly display, publicly perform, sublicense, and distribute the
+   Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   (except as stated in this section) patent license to make, have made,
+   use, offer to sell, sell, import, and otherwise transfer the Work,
+   where such license applies only to those patent claims licensable
+   by such Contributor that are necessarily infringed by their
+   Contribution(s) alone or by combination of their Contribution(s)
+   with the Work to which such Contribution(s) was submitted. If You
+   institute patent litigation against any entity (including a
+   cross-claim or counterclaim in a lawsuit) alleging that the Work
+   or a Contribution incorporated within the Work constitutes direct
+   or contributory patent infringement, then any patent licenses
+   granted to You under this License for that Work shall terminate
+   as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+   Work or Derivative Works thereof in any medium, with or without
+   modifications, and in Source or Object form, provided that You
+   meet the following conditions:
+
+   (a) You must give any other recipients of the Work or
+       Derivative Works a copy of this License; and
+
+   (b) You must cause any modified files to carry prominent notices
+       stating that You changed the files; and
+
+   (c) You must retain, in the Source form of any Derivative Works
+       that You distribute, all copyright, patent, trademark, and
+       attribution notices from the Source form of the Work,
+       excluding those notices that do not pertain to any part of
+       the Derivative Works; and
+
+   (d) If the Work includes a "NOTICE" text file as part of its
+       distribution, then any Derivative Works that You distribute must
+       include a readable copy of the attribution notices contained
+       within such NOTICE file, excluding those notices that do not
+       pertain to any part of the Derivative Works, in at least one
+       of the following places: within a NOTICE text file distributed
+       as part of the Derivative Works; within the Source form or
+       documentation, if provided along with the Derivative Works; or,
+       within a display generated by the Derivative Works, if and
+       wherever such third-party notices normally appear. The contents
+       of the NOTICE file are for informational purposes only and
+       do not modify the License. You may add Your own attribution
+       notices within Derivative Works that You distribute, alongside
+       or as an addendum to the NOTICE text from the Work, provided
+       that such additional attribution notices cannot be construed
+       as modifying the License.
+
+   You may add Your own copyright statement to Your modifications and
+   may provide additional or different license terms and conditions
+   for use, reproduction, or distribution of Your modifications, or
+   for any such Derivative Works as a whole, provided Your use,
+   reproduction, and distribution of the Work otherwise complies with
+   the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+   any Contribution intentionally submitted for inclusion in the Work
+   by You to the Licensor shall be under the terms and conditions of
+   this License, without any additional terms or conditions.
+   Notwithstanding the above, nothing herein shall supersede or modify
+   the terms of any separate license agreement you may have executed
+   with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+   names, trademarks, service marks, or product names of the Licensor,
+   except as required for reasonable and customary use in describing the
+   origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+   agreed to in writing, Licensor provides the Work (and each
+   Contributor provides its Contributions) on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+   implied, including, without limitation, any warranties or conditions
+   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+   PARTICULAR PURPOSE. You are solely responsible for determining the
+   appropriateness of using or redistributing the Work and assume any
+   risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+   whether in tort (including negligence), contract, or otherwise,
+   unless required by applicable law (such as deliberate and grossly
+   negligent acts) or agreed to in writing, shall any Contributor be
+   liable to You for damages, including any direct, indirect, special,
+   incidental, or consequential damages of any character arising as a
+   result of this License or out of the use or inability to use the
+   Work (including but not limited to damages for loss of goodwill,
+   work stoppage, computer failure or malfunction, or any and all
+   other commercial damages or losses), even if such Contributor
+   has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+   the Work or Derivative Works thereof, You may choose to offer,
+   and charge a fee for, acceptance of support, warranty, indemnity,
+   or other liability obligations and/or rights consistent with this
+   License. However, in accepting such obligations, You may act only
+   on Your own behalf and on Your sole responsibility, not on behalf
+   of any other Contributor, and only if You agree to indemnify,
+   defend, and hold each Contributor harmless for any liability
+   incurred by, or claims asserted against, such Contributor by reason
+   of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+Copyright 2026 Mistral AI
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

mistralai_workflows_plugins_evaluations-0.5.0/Makefile

@@ -0,0 +1,18 @@
+.PHONY: lint format typecheck test installdeps
+
+lint:
+	uv run ruff format --check .
+	uv run ruff check .
+
+format:
+	uv run ruff check --fix .
+	uv run ruff format .
+
+typecheck:
+	uv run mypy mistralai/
+
+test:
+	uv run pytest tests/ -v
+
+installdeps:
+	uv sync

mistralai_workflows_plugins_evaluations-0.5.0/PKG-INFO

@@ -0,0 +1,21 @@
+Metadata-Version: 2.5
+Name: mistralai-workflows-plugins-evaluations
+Version: 0.5.0
+Summary: Evaluation plugin for Mistral Workflows - run evaluations as parallel Temporal activities
+Project-URL: Homepage, https://mistral.ai
+Project-URL: Documentation, https://docs.mistral.ai/workflows
+Author-email: Mistral AI <support@mistral.ai>
+License: Apache-2.0
+License-File: LICENSE
+Keywords: ai,evaluation,mistral,observability,plugin,workflows
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: <3.15,>=3.12
+Requires-Dist: mistralai-evaluations>=0.6.0
+Requires-Dist: mistralai-workflows<4.0.0,>=3.8.0
+Requires-Dist: structlog<26,>=24

mistralai_workflows_plugins_evaluations-0.5.0/docs/docs/getting-started/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Getting Started",
+  "position": 1,
+  "link": {
+    "type": "generated-index",
+    "description": "Get up and running with the Workflow Evaluation Plugin"
+  }
+}

mistralai_workflows_plugins_evaluations-0.5.0/docs/docs/getting-started/installation.mdx

@@ -0,0 +1,82 @@
+---
+id: installation
+sidebar_label: Installation
+title: Installation
+sidebar_position: 2
+---
+
+## Prerequisites
+
+- Python 3.12+
+- [uv](https://docs.astral.sh/uv/) for dependency management
+- A Mistral API key
+- A working [Mistral Workflows](/workflows/) setup
+- Access to the Mistral private package registry (Gemfury)
+
+## Install
+
+Create a new project:
+
+```bash
+uv init my-eval-workflow
+cd my-eval-workflow
+```
+
+Add the plugin to your `pyproject.toml`:
+
+```toml
+[project]
+name = "my-eval-workflow"
+version = "0.1.0"
+dependencies = [
+    "mistralai",
+    "mistralai-workflows[mistralai]",
+    "mistralai-workflows-plugins-evaluations>=0.1.0",
+]
+
+[[tool.uv.index]]
+name = "gemfury"
+url = "https://pypi.fury.io/mistralai/"
+
+[tool.uv]
+exclude-newer = "7 days"
+exclude-newer-package = {
+    "mistralai-workflows-plugins-evaluations" = false,
+    "mistralai-evaluations" = false,
+    "mistralai" = false,
+    "mistralai-workflows" = false,
+}
+
+[tool.uv.sources]
+mistralai-workflows-plugins-evaluations = { index = "gemfury" }
+```
+
+Then install:
+
+```bash
+uv sync
+```
+
+:::tip Gemfury credentials
+Make sure Gemfury credentials are set in your `.zshrc` / `.bashrc`. If you don't have them yet, see the internal Gemfury setup guide.
+:::
+
+## Environment variables
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `MISTRAL_API_KEY` | Yes | Your Mistral API key for model calls and AI Studio uploads |
+| `DEPLOYMENT_NAME` | Yes | Your workflow deployment name |
+
+```bash
+export MISTRAL_API_KEY=<your-api-key>
+export DEPLOYMENT_NAME=my-eval-workflow
+```
+
+## Verify installation
+
+```python
+from mistralai.workflows.plugins.evaluations import evaluation
+from mistralai.workflows.plugins.evaluations.types import Evaluator, Score
+print("Plugin installed successfully!")
+```

mistralai_workflows_plugins_evaluations-0.5.0/docs/docs/getting-started/introduction.mdx

@@ -0,0 +1,140 @@
+---
+id: introduction
+sidebar_label: Introduction
+title: Workflow Evaluation Plugin
+sidebar_position: 1
+slug: /
+---
+
+:::caution Internal Use Only
+This plugin is available for Mistral teams but **not for external clients yet**. The product is under active development — share your feedback on [#eng-offline-evaluations](https://mistralai.slack.com/archives/C0AGP0NVA67).
+:::
+
+The Workflow Evaluation Plugin (`mistralai-workflows-plugins-evaluations`) lets you run evaluations inside [Mistral Workflows](/workflows/), with automatic parallel execution via Temporal. It provides the same `evaluation.run()` API as the [Evaluation SDK](/evaluation-sdk/), but splits task execution and scoring into parallel Temporal activities for scalable, production-grade evaluation pipelines.
+
+## Overview
+
+The plugin follows the same mental model as the Evaluation SDK:
+
+```
+Dataset  -->  Task  -->  Evaluators  -->  Results
+```
+
+The key difference: instead of running everything in a single process, the plugin fans out each record as a **Temporal child workflow** that runs the task then the scorers in parallel. Results are uploaded to AI Studio automatically.
+
+## When to use this vs. the Evaluation SDK
+
+| | Evaluation SDK | Workflow Evaluation Plugin |
+|---|---|---|
+| **Runs in** | Any Python script | A Mistral Workflow |
+| **Parallelism** | `asyncio.gather` in one process | Temporal activities across workers |
+| **Retries** | Manual | Automatic per-record via Temporal |
+| **Failure isolation** | One failure can crash the run | Each record is isolated in a child workflow |
+| **Best for** | Quick iteration, notebooks, CI scripts | Production pipelines, large datasets, complex tasks |
+
+## Quickstart
+
+### 1. Install
+
+Add to your workflow project's `pyproject.toml`:
+
+```toml
+[project]
+dependencies = [
+    "mistralai",
+    "mistralai-workflows[mistralai]",
+    "mistralai-workflows-plugins-evaluations>=0.1.0",
+]
+
+[[tool.uv.index]]
+name = "gemfury"
+url = "https://pypi.fury.io/mistralai/"
+
+[tool.uv.sources]
+mistralai-workflows-plugins-evaluations = { index = "gemfury" }
+```
+
+Then install:
+
+```bash
+uv sync
+```
+
+### 2. Write your first evaluation workflow
+
+```python
+from mistralai.workflows import workflow
+from mistralai.workflows.plugins.evaluations import evaluation
+from mistralai.workflows.plugins.evaluations.types import (
+    Evaluator,
+    Goal,
+    Score,
+    ScorerContext,
+    System,
+    TaskContext,
+)
+
+dataset = [
+    {"prompt": "Say hello", "expected": "hello"},
+    {"prompt": "Say goodbye", "expected": "goodbye"},
+]
+
+@evaluation.task
+async def my_task(ctx: TaskContext) -> str:
+    from mistralai.client import Mistral
+    import os
+
+    client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
+    resp = await client.chat.complete_async(
+        model=ctx.system.params["model"],
+        messages=[
+            {"role": "system", "content": ctx.system.params["system_prompt"]},
+            {"role": "user", "content": ctx.input_record["prompt"]},
+        ],
+    )
+    return str(resp.choices[0].message.content)
+
+@evaluation.scorer
+async def accuracy(ctx: ScorerContext) -> Score:
+    expected = ctx.input_record["expected"]
+    return Score(value=1 if expected in ctx.output.lower() else 0)
+
+@workflow.define(name="my-evaluation")
+class MyEvaluationWorkflow:
+    @workflow.entrypoint
+    async def run(self, params: dict) -> dict:
+        result = await evaluation.run(
+            dataset=dataset,
+            task=my_task,
+            evaluators=[Evaluator(
+                name="accuracy",
+                description="Checks if the expected word appears in the model output.",
+                scorer=accuracy,
+                goal=Goal.gte(0.8),
+            )],
+            system=System(name="default", params={
+                "model": "mistral-small-latest",
+                "system_prompt": "Reply with only the last word of the user's sentence.",
+            }),
+            tags=["workflow"],
+            local=True,
+        )
+        return result.model_dump()
+```
+
+### 3. Run it
+
+Deploy and execute the workflow as you would any Mistral Workflow. Results appear in your terminal and in AI Studio.
+
+## Key features
+
+- **Parallel execution** -- tasks and scorers run as concurrent Temporal activities with configurable concurrency
+- **Failure isolation** -- each record runs in its own child workflow; one failure doesn't crash the run
+- **Automatic retries** -- configurable retry policies per task and scorer
+- **Two task modes** -- `@evaluation.task` (activity) or workflow-as-task (child workflow)
+- **Context objects** -- `TaskContext` and `ScorerContext` give typed access to input, system params, and metadata
+- **Goals** -- set pass/fail thresholds with `Goal.gte()`, `Goal.lte()`, `Goal.between()`
+- **Run evaluators** -- compute aggregate metrics (F1, global accuracy) after all records complete
+- **[Optimization](/workflow-evaluation/guides/optimization/)** -- automatically search for better prompts and parameters, with the loop running as a workflow
+- **Local mode** -- `local=True` skips AI Studio uploads for fast iteration
+- **AI Studio integration** -- results auto-upload with workflow cross-linking

mistralai_workflows_plugins_evaluations-0.5.0/docs/docs/guides/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Guides",
+  "position": 2,
+  "link": {
+    "type": "generated-index",
+    "description": "Step-by-step guides for the Workflow Evaluation Plugin"
+  }
+}