mas-lab-library-eval 0.2.0__tar.gz

mas_lab_library_eval-0.2.0/.gitignore

@@ -0,0 +1,86 @@
+*.pyc
+__pycache__/
+*.egg-info/
+.DS_Store
+dist/
+build/
+.venv/
+.demo-ui/
+.demo-ui-mock/
+.env
+.env.local
+.env.*.local
+config.yaml
+!lab/src/mas/lab/templates/**/config.yaml
+!docs/tutorials/**/config.yaml
+!labs/**/config.yaml
+!library-samples/sample-workspace/config.yaml
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.test-venv/
+node_modules/
+*.log
+docs/tutorials/**/logs/
+.hypothesis/
+compliance-report.json
+COMPLIANCE_REPORT.md
+compliance/results/
+
+# MkDocs build output
+site/
+
+.vscode/
+.cursor/
+.claude/
+
+# Test / coverage artifacts
+.coverage
+coverage.xml
+coverage.json
+**/coverage.json
+
+# LaTeX build artifacts
+*.fdb_latexmk
+*.fls
+*.aux
+*.out
+*.toc
+*.synctex.gz
+
+# Run output / artifacts
+**/output/*.jsonl
+**/output/benchmark
+**/output/meta.json
+**/output/current_run_id.txt
+**/output/*/
+.tutorial-cache/
+tests/.tutorial-engine-cache/
+.demo-cache/
+# Local mas-ctl chat traces next to an agent (fixtures live under fixtures/ or T01 traces/)
+docs/tutorials/**/agents/**/traces/
+
+# Backup files
+*.bak
+*.bak2
+
+# Legacy dead code (removed from repo)
+lab/src/mas/lab/_cli_legacy.py
+.deepeval/
+~/.mas/
+
+
+# Docker local secrets + data (never commit)
+docker/.env
+docker/data/*
+!docker/data/.gitkeep
+
+# Local protocol compliance checkouts and reports
+compliance/vendor/
+
+# Local harness playbooks (not published)
+AGENTS.md
+# .agents/
+COVERAGE_REPORT.md
+MANIFEST_SCHEMA_MAPPING.md
+.lycheecache

mas_lab_library_eval-0.2.0/PKG-INFO

@@ -0,0 +1,192 @@
+Metadata-Version: 2.5
+Name: mas-lab-library-eval
+Version: 0.2.0
+Summary: Evaluation and quality metrics library for MAS Lab — MCE integration.
+License: Apache-2.0
+Requires-Python: >=3.11
+Requires-Dist: click>=8.0
+Requires-Dist: deepeval>=4.0.0
+Requires-Dist: mas-lab-component-bench
+Requires-Dist: mas-lab-component-core
+Requires-Dist: mas-lab-runtime
+Requires-Dist: mce-metrics-plugin>=0.1.0
+Requires-Dist: metrics-computation-engine>=0.1.0
+Requires-Dist: pandas>=2.0.0
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
+Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+<!--
+  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+  SPDX-License-Identifier: Apache-2.0
+-->
+# library-eval
+
+Scores MAS runs with quality metrics and LLM-as-judge prompts. It wraps the Metrics Computation Engine (MCE), an open-source set of LLM-as-judge metrics from the AGNTCY `telemetry-hub` project, for MAS traces.
+
+Use it when you want a number for "did the system reach the goal" or "was the answer grounded" for one trace or a whole experiment.
+
+## Quick start
+
+```bash
+uv pip install -e library-eval          # from the repository root; needs Python 3.11+
+mas-lab eval --list-metrics
+mas-lab eval path/to/events.jsonl --metric goal_success_rate --metric groundedness
+```
+
+Batch scoring over a benchmark output tree:
+
+```bash
+mas-lab eval path/to/experiment/ --metric goal_success_rate --recursive
+```
+
+`mas-lab eval SOURCE` flags:
+
+| Flag | Meaning |
+|------|---------|
+| `--metric` | Metric to compute; repeatable. |
+| `--model` | Judge model name (default `azure/gpt-4o`). |
+| `--api-base` | LLM proxy base URL (default `https://api.openai.com/v1`). |
+| `--api-key-env` | Environment variable holding the API key (default `OPENAI_API_KEY`). |
+| `--json` | Print results as JSON. |
+| `--list-metrics` | List available metrics and exit; `SOURCE` is then optional. |
+| `--recursive` | Score every trace under the `SOURCE` directory. |
+
+The package depends on `lab/components/core`, `lab/components/bench` and `mas-runtime` from this repository, plus the public MCE packages:
+
+- `metrics-computation-engine`: MCE core.
+- `mce-metrics-plugin`: quality metrics (GoalSuccessRate, Groundedness, ResponseCompleteness, ...).
+- `deepeval`: continuous-score implementations of `answer_relevancy` and `goal_success_rate`.
+
+## What the library provides
+
+1. Library code (`mas.library.eval.*`): MCE integration, session entity construction, metric computation, and the `eval_metric` plugin type.
+2. A CLI component (`mas.library.eval.cli`) that registers `mas-lab eval` through the `mas.lab.cli.components` entry point:
+
+```toml
+[project.entry-points."mas.lab.cli.components"]
+eval = "mas.library.eval.cli:EvalCliComponent"
+```
+
+The `eval_mce` pipeline step, which scores a run into `metrics.json`, ships in `library-lab` (`mas.library.lab.steps.eval.mce`) and calls into this library. `mas.library.eval.steps` holds an older CamelCase-metric step class that is not registered.
+
+```
+src/mas/library/eval/
++-- evaluator.py     EvalProvider + get_provider("mce")
++-- providers/       mce.py: built-in MCE provider (stock + registered metrics);
+|                    trip_planner_gt.py, adversarial.py: sample ground-truth / adversarial providers
++-- mce/
+|   +-- catalog.py       one catalog: METRIC_MAP + METRIC_REGISTRY
+|   +-- runner.py        native-trace MCE/deepeval compute
+|   +-- registry_api.py  SessionEntity CamelCase API
++-- metrics/         EvalMetric, plugins, prompt_metrics, stock wrappers, toy example
++-- steps/           older CamelCase-metric step class (not registered)
++-- cli/             mas-lab eval
+```
+
+## Score runs in a benchmark
+
+Attach the `eval_mce` step to a run. Metric ids are snake_case and the judge model comes from the experiment (see [Tutorial 10](../docs/tutorials/10-evaluation-metrics/README.md)):
+
+```yaml
+# pipeline attached with `mas-lab benchmark run ... --pipeline run:post:eval.yaml`
+apiVersion: mas/v1
+kind: Pipeline
+spec:
+  steps:
+    - name: eval
+      type: eval_mce
+      per_run: true
+      in: trace
+      out: metrics
+      config:
+        metrics: [goal_success_rate, groundedness, response_completeness]
+```
+
+## Metric names
+
+The CLI, the `eval_mce` step `metrics:` and the `metrics.json` keys use snake_case ids. The class names used by the `SessionEntity` API (`compute_session_metrics`) are CamelCase.
+
+| Id (CLI and YAML) | Class name |
+|-------------------|------------|
+| `goal_success_rate` | `GoalSuccessRate` |
+| `answer_relevancy` | `AnswerRelevancy` |
+
+`mas-lab eval --list-metrics` lists the ids. The 11 stock session metrics are `answer_relevancy`, `component_conflict_rate`, `consistency`, `context_preservation`, `goal_success_rate`, `groundedness`, `information_retention`, `intent_recognition_accuracy`, `response_completeness`, `workflow_cohesion_index` and `workflow_efficiency`. The rule-based core metrics (`ToolError`, `ToolErrorRate`, `ToolUtilizationAccuracy`, `CyclesCount`, `AgentToAgentInteractions`, `AgentToToolInteractions`) are available through the CamelCase API only.
+
+## Register your own metric (`eval_metric` plugin)
+
+A library registers session metrics on the MCE eval provider through the `eval_metric` plugin type (`EvalProvider.register_metric`). `eval_mce` then runs any mix of stock MCE ids and registered ids into one `metrics.json`. Each metric declares the input it needs (`unit` / `evidence`). Stock MCE ids score MAS input and output, not the raw trajectory.
+
+```yaml
+# library.yaml
+types:
+  - eval_metric
+plugins:
+  - type: eval_metric
+    name: toy_echo
+    module: mas.library.eval.metrics.toy
+    class: ToyEchoMetric
+    attributes:
+      provider: mce          # default
+  # A family that yields many ids from one factory:
+  - type: eval_metric
+    name: my_family
+    factory: mas.library.example.metrics:build_metrics
+    attributes:
+      provider: mce
+```
+
+Rules:
+
+- Duplicate metric ids raise at registration.
+- Stock MCE session ids are `EvalMetric`s on `get_provider("mce")` (one batch per run).
+- Plugins and inline `prompt_metrics` run through the same `compute_metrics` engine.
+- Unknown ids error with the list of available ids.
+- Metrics that share `batch_key` are computed once per run via `compute_batch`.
+- A failing metric writes `{value: null, error: "..."}` for that id only.
+- Optional `details` on each score is additive in `metrics.json` schema v1.
+
+List registered ids in the `metrics:` list of an `eval_mce` step, or split families across steps with distinct `metrics_filename` values (`metrics.json`, `metrics_judge2.json`). For a one-off LLM-as-judge question, add `prompt_metrics: [{id, prompt, unit, evidence}]` to the same step without a plugin; ids must not collide with a stock or registered metric. `collect_metrics` and `metrics_to_dataframe` accept a filename, a glob or a list, and copy `details` into a JSON column.
+
+## Programmatic use
+
+```python
+import os
+
+from mas.library.eval.mce import build_session_from_trace, compute_session_metrics
+
+session_entity = build_session_from_trace("path/to/events.jsonl")
+
+results = await compute_session_metrics(      # async
+    session=session_entity,
+    metrics=["GoalSuccessRate", "Groundedness"],
+    llm_config={
+        "LLM_MODEL_NAME": "azure/gpt-4o",
+        "LLM_BASE_MODEL_URL": "https://api.openai.com/v1",
+        "LLM_API_KEY": os.environ["OPENAI_API_KEY"],
+    },
+)
+```
+
+`llm_config` is a dict as above, or an `LLMJudgeConfig` with the same three fields:
+
+```python
+from metrics_computation_engine.models.requests import LLMJudgeConfig
+
+llm_config = LLMJudgeConfig(
+    LLM_MODEL_NAME="azure/gpt-4o",
+    LLM_BASE_MODEL_URL="https://api.openai.com/v1",
+    LLM_API_KEY=os.environ["OPENAI_API_KEY"],
+)
+```
+
+## See also
+
+- [Tutorial 10, Evaluation metrics](../docs/tutorials/10-evaluation-metrics/README.md): how stock MCE, prompts, plugins, judge infra and trajectory wrappers fit together.
+- [Tutorial 3, Run an experiment](../docs/tutorials/03-experiments-and-analysis/README.md): first MCE scoring.
+- Feature example (not a sample app): [examples/mce/judge-override/](examples/mce/judge-override/)
+- [summarization.md, MCE judge](../docs/manifests/summarization.md#mce-judge-model)
+- [MCE documentation](https://github.com/agntcy/telemetry-hub/tree/main/metrics_computation_engine)
+- [mce_metrics_plugin](https://github.com/agntcy/telemetry-hub/tree/main/metrics_computation_engine/plugins/mce_metrics_plugin)

mas_lab_library_eval-0.2.0/README.md

@@ -0,0 +1,173 @@
+<!--
+  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+  SPDX-License-Identifier: Apache-2.0
+-->
+# library-eval
+
+Scores MAS runs with quality metrics and LLM-as-judge prompts. It wraps the Metrics Computation Engine (MCE), an open-source set of LLM-as-judge metrics from the AGNTCY `telemetry-hub` project, for MAS traces.
+
+Use it when you want a number for "did the system reach the goal" or "was the answer grounded" for one trace or a whole experiment.
+
+## Quick start
+
+```bash
+uv pip install -e library-eval          # from the repository root; needs Python 3.11+
+mas-lab eval --list-metrics
+mas-lab eval path/to/events.jsonl --metric goal_success_rate --metric groundedness
+```
+
+Batch scoring over a benchmark output tree:
+
+```bash
+mas-lab eval path/to/experiment/ --metric goal_success_rate --recursive
+```
+
+`mas-lab eval SOURCE` flags:
+
+| Flag | Meaning |
+|------|---------|
+| `--metric` | Metric to compute; repeatable. |
+| `--model` | Judge model name (default `azure/gpt-4o`). |
+| `--api-base` | LLM proxy base URL (default `https://api.openai.com/v1`). |
+| `--api-key-env` | Environment variable holding the API key (default `OPENAI_API_KEY`). |
+| `--json` | Print results as JSON. |
+| `--list-metrics` | List available metrics and exit; `SOURCE` is then optional. |
+| `--recursive` | Score every trace under the `SOURCE` directory. |
+
+The package depends on `lab/components/core`, `lab/components/bench` and `mas-runtime` from this repository, plus the public MCE packages:
+
+- `metrics-computation-engine`: MCE core.
+- `mce-metrics-plugin`: quality metrics (GoalSuccessRate, Groundedness, ResponseCompleteness, ...).
+- `deepeval`: continuous-score implementations of `answer_relevancy` and `goal_success_rate`.
+
+## What the library provides
+
+1. Library code (`mas.library.eval.*`): MCE integration, session entity construction, metric computation, and the `eval_metric` plugin type.
+2. A CLI component (`mas.library.eval.cli`) that registers `mas-lab eval` through the `mas.lab.cli.components` entry point:
+
+```toml
+[project.entry-points."mas.lab.cli.components"]
+eval = "mas.library.eval.cli:EvalCliComponent"
+```
+
+The `eval_mce` pipeline step, which scores a run into `metrics.json`, ships in `library-lab` (`mas.library.lab.steps.eval.mce`) and calls into this library. `mas.library.eval.steps` holds an older CamelCase-metric step class that is not registered.
+
+```
+src/mas/library/eval/
++-- evaluator.py     EvalProvider + get_provider("mce")
++-- providers/       mce.py: built-in MCE provider (stock + registered metrics);
+|                    trip_planner_gt.py, adversarial.py: sample ground-truth / adversarial providers
++-- mce/
+|   +-- catalog.py       one catalog: METRIC_MAP + METRIC_REGISTRY
+|   +-- runner.py        native-trace MCE/deepeval compute
+|   +-- registry_api.py  SessionEntity CamelCase API
++-- metrics/         EvalMetric, plugins, prompt_metrics, stock wrappers, toy example
++-- steps/           older CamelCase-metric step class (not registered)
++-- cli/             mas-lab eval
+```
+
+## Score runs in a benchmark
+
+Attach the `eval_mce` step to a run. Metric ids are snake_case and the judge model comes from the experiment (see [Tutorial 10](../docs/tutorials/10-evaluation-metrics/README.md)):
+
+```yaml
+# pipeline attached with `mas-lab benchmark run ... --pipeline run:post:eval.yaml`
+apiVersion: mas/v1
+kind: Pipeline
+spec:
+  steps:
+    - name: eval
+      type: eval_mce
+      per_run: true
+      in: trace
+      out: metrics
+      config:
+        metrics: [goal_success_rate, groundedness, response_completeness]
+```
+
+## Metric names
+
+The CLI, the `eval_mce` step `metrics:` and the `metrics.json` keys use snake_case ids. The class names used by the `SessionEntity` API (`compute_session_metrics`) are CamelCase.
+
+| Id (CLI and YAML) | Class name |
+|-------------------|------------|
+| `goal_success_rate` | `GoalSuccessRate` |
+| `answer_relevancy` | `AnswerRelevancy` |
+
+`mas-lab eval --list-metrics` lists the ids. The 11 stock session metrics are `answer_relevancy`, `component_conflict_rate`, `consistency`, `context_preservation`, `goal_success_rate`, `groundedness`, `information_retention`, `intent_recognition_accuracy`, `response_completeness`, `workflow_cohesion_index` and `workflow_efficiency`. The rule-based core metrics (`ToolError`, `ToolErrorRate`, `ToolUtilizationAccuracy`, `CyclesCount`, `AgentToAgentInteractions`, `AgentToToolInteractions`) are available through the CamelCase API only.
+
+## Register your own metric (`eval_metric` plugin)
+
+A library registers session metrics on the MCE eval provider through the `eval_metric` plugin type (`EvalProvider.register_metric`). `eval_mce` then runs any mix of stock MCE ids and registered ids into one `metrics.json`. Each metric declares the input it needs (`unit` / `evidence`). Stock MCE ids score MAS input and output, not the raw trajectory.
+
+```yaml
+# library.yaml
+types:
+  - eval_metric
+plugins:
+  - type: eval_metric
+    name: toy_echo
+    module: mas.library.eval.metrics.toy
+    class: ToyEchoMetric
+    attributes:
+      provider: mce          # default
+  # A family that yields many ids from one factory:
+  - type: eval_metric
+    name: my_family
+    factory: mas.library.example.metrics:build_metrics
+    attributes:
+      provider: mce
+```
+
+Rules:
+
+- Duplicate metric ids raise at registration.
+- Stock MCE session ids are `EvalMetric`s on `get_provider("mce")` (one batch per run).
+- Plugins and inline `prompt_metrics` run through the same `compute_metrics` engine.
+- Unknown ids error with the list of available ids.
+- Metrics that share `batch_key` are computed once per run via `compute_batch`.
+- A failing metric writes `{value: null, error: "..."}` for that id only.
+- Optional `details` on each score is additive in `metrics.json` schema v1.
+
+List registered ids in the `metrics:` list of an `eval_mce` step, or split families across steps with distinct `metrics_filename` values (`metrics.json`, `metrics_judge2.json`). For a one-off LLM-as-judge question, add `prompt_metrics: [{id, prompt, unit, evidence}]` to the same step without a plugin; ids must not collide with a stock or registered metric. `collect_metrics` and `metrics_to_dataframe` accept a filename, a glob or a list, and copy `details` into a JSON column.
+
+## Programmatic use
+
+```python
+import os
+
+from mas.library.eval.mce import build_session_from_trace, compute_session_metrics
+
+session_entity = build_session_from_trace("path/to/events.jsonl")
+
+results = await compute_session_metrics(      # async
+    session=session_entity,
+    metrics=["GoalSuccessRate", "Groundedness"],
+    llm_config={
+        "LLM_MODEL_NAME": "azure/gpt-4o",
+        "LLM_BASE_MODEL_URL": "https://api.openai.com/v1",
+        "LLM_API_KEY": os.environ["OPENAI_API_KEY"],
+    },
+)
+```
+
+`llm_config` is a dict as above, or an `LLMJudgeConfig` with the same three fields:
+
+```python
+from metrics_computation_engine.models.requests import LLMJudgeConfig
+
+llm_config = LLMJudgeConfig(
+    LLM_MODEL_NAME="azure/gpt-4o",
+    LLM_BASE_MODEL_URL="https://api.openai.com/v1",
+    LLM_API_KEY=os.environ["OPENAI_API_KEY"],
+)
+```
+
+## See also
+
+- [Tutorial 10, Evaluation metrics](../docs/tutorials/10-evaluation-metrics/README.md): how stock MCE, prompts, plugins, judge infra and trajectory wrappers fit together.
+- [Tutorial 3, Run an experiment](../docs/tutorials/03-experiments-and-analysis/README.md): first MCE scoring.
+- Feature example (not a sample app): [examples/mce/judge-override/](examples/mce/judge-override/)
+- [summarization.md, MCE judge](../docs/manifests/summarization.md#mce-judge-model)
+- [MCE documentation](https://github.com/agntcy/telemetry-hub/tree/main/metrics_computation_engine)
+- [mce_metrics_plugin](https://github.com/agntcy/telemetry-hub/tree/main/metrics_computation_engine/plugins/mce_metrics_plugin)

mas_lab_library_eval-0.2.0/docs/cli/index.md

@@ -0,0 +1,45 @@
+<!--
+  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+  SPDX-License-Identifier: Apache-2.0
+-->
+# mas-lab-eval(1)
+
+## NAME
+
+`mas-lab eval` — Score traces with MCE LLM-as-judge metrics
+
+## SYNOPSIS
+
+```text
+mas-lab eval [OPTIONS] [SOURCE]
+```
+
+## DESCRIPTION
+
+Library CLI component (entry point `mas.lab.cli.components`). Calls the same
+MCE runner as the `eval_mce` pipeline step (`mas.library.eval.mce.runner`).
+
+Metric ids are snake_case (`goal_success_rate`). `--list-metrics` prints the
+available ids. When `--model` is omitted, the workspace infra default is used,
+else `gpt-4o`.
+
+## OPTIONS
+
+| Flag | Meaning |
+|------|------|
+| `SOURCE` | Positional argument. |
+| `--fixture` | Path to a fixture YAML that provides ground truth. |
+| `--metric` | MCE metric(s) to compute (repeatable). Snake_case ids only (goal_success_rate). Defaults to ['goal_success_rate'] when omitted. Use --list-metrics to show the available metrics. |
+| `--response-agent` | agent_id whose last execution_end is the final response. Auto-detected when omitted. |
+| `--model` | Judge model. Default: workspace infra default, else gpt-4o. |
+| `--list-metrics` | List the available metrics and exit. |
+| `--json` | Output results as JSON instead of a human-readable table. |
+| `--overlay` | Label for display (e.g. inject-C7-backend). |
+| `--engine` | Metric computation engine: mce-v1. |
+| `--api-key-env` | Name of the environment variable holding the API key (default: OPENAI_API_KEY). |
+| `--recursive` | Score every events.jsonl under SOURCE when SOURCE is a directory. |
+
+## SEE ALSO
+
+- `mas-lab --help`
+- [library-eval README](../../README.md)

mas_lab_library_eval-0.2.0/examples/README.md

@@ -0,0 +1,18 @@
+<!--
+  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+  SPDX-License-Identifier: Apache-2.0
+-->
+# Examples (library-eval)
+
+Small feature scenarios for MCE evaluation — closer to functional tests than
+to sample apps. Layout: `examples/<category>/<name>/`.
+
+Tests load these as fixtures. `mas-ctl validate` works from the repo root.
+
+**Not here:** use-case apps such as the trip planner. Those live in `library-samples/apps/`.
+
+## Index
+
+| Category | Example | Kind | What it shows |
+|----------|---------|------|----------------|
+| [mce](mce/) | [judge-override](mce/judge-override/) | Experiment | `experiment.models` judge vs main + per-step `config.model` |

mas_lab_library_eval-0.2.0/examples/mce/README.md

@@ -0,0 +1,11 @@
+<!--
+  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+  SPDX-License-Identifier: Apache-2.0
+-->
+# MCE examples
+
+Runnable experiment snippets for `eval_mce` (LLM-as-judge). Not sample apps.
+
+| Example | Kind | What it shows |
+|---------|------|----------------|
+| [judge-override](judge-override/) | Experiment | `experiment.models` (main vs judge) + per-step `config.model` |

mas_lab_library_eval-0.2.0/examples/mce/judge-override/README.md

@@ -0,0 +1,65 @@
+<!--
+  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+  SPDX-License-Identifier: Apache-2.0
+-->
+# Example: MCE judge model (`experiment.models` + `eval_mce.config.model`)
+
+Eval feature example. Not a sample app.
+
+There is **no committed global model in `config.yaml`**. Pin slots on the
+experiment (`experiment.models`), the MAS, or the Agent. Scalar
+`experiment.model` is shorthand for `models.main`. Omitted / `any` is filled
+at engine time from local `config.yaml` `defaults.model`.
+
+This example pins **turn vs judge** with the slot map, then a per-step
+override:
+
+- `experiment.models.main` — turn default (`gpt-4o`)
+- `experiment.models.judge` — default for every `eval_mce` that omitted `config.model`
+- per-step `config.model` — still wins
+- `experiment.evaluation.model` — optional judge override (wins over `models.judge`)
+
+Metric *prompts* live inside MCE (`mce_metrics_plugin`); this library does
+not override them. Only the model id is configurable here.
+
+| File | Role |
+|------|------|
+| `experiment.yaml` | Lab spec: `models.judge` + one strict step `config.model` |
+| `mas.yaml` | Stub Agent (`spec.models: gpt-4o`) so `applications.manifest` resolves |
+
+## Quickstart
+
+```bash
+mas-ctl validate library-eval/examples/mce/judge-override/experiment.yaml
+```
+
+To run for real, point `applications` at a MAS that produced traces, then:
+
+```bash
+mas-lab benchmark run library-eval/examples/mce/judge-override/experiment.yaml --progress
+```
+
+INFO logs: `EvalMceStep … judge model=… source=experiment.models.judge`
+(or `eval_mce.config.model` on the strict step). Omit `models.judge` and
+the source is `experiment.model` / `application.spec.models`.
+
+## What is overridable
+
+Existing `eval_mce` config (unchanged) plus **`model`**:
+
+| Key | Default | Meaning |
+|-----|---------|---------|
+| `runs_dir` | pipeline output dir | Tree of `item*/r*/traces/events.jsonl` |
+| `metrics` | all session MCE metrics | Subset to score |
+| `overwrite` | `false` | Recompute `metrics.json` |
+| `validate` | `true` | Schema-check artefacts |
+| `max_workers` | `2` | Parallel judge calls |
+| `fail_threshold` | `1.0` | Fraction of items that may fail |
+| **`model`** | `experiment.models.judge` then `models.main` then application `spec.models[]` | LLM-as-judge model |
+
+## Docs
+
+- [summarization.md § MCE judge](../../../../docs/manifests/summarization.md#mce-judge-model)
+- [experiment.md](../../../../docs/manifests/experiment.md)
+- [pipeline-steps.md](../../../../lab/docs/pipeline-steps.md)
+- Category: [MCE examples](../README.md)

mas_lab_library_eval-0.2.0/examples/mce/judge-override/experiment.yaml

@@ -0,0 +1,28 @@
+#  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+#  SPDX-License-Identifier: Apache-2.0
+# Feature example for experiment.models slots and eval_mce config.model.
+# Not a sample app. validate-only unless you point applications at a real MAS.
+experiment:
+  name: judge-override
+  description: "Pin turn vs judge via experiment.models; per-step config.model still wins."
+  models:
+    main: gpt-4o
+    judge: gpt-4o-mini
+  application:
+    manifest: ./mas.yaml
+  evaluation:
+    method: llm_judge
+  post:
+    - type: extract_trajectories
+    - type: eval_mce
+      depends_on: [extract_trajectories]
+    - type: eval_mce
+      name: eval_mce_strict
+      depends_on: [extract_trajectories]
+      config:
+        model: gpt-4o
+        max_workers: 2
+        fail_threshold: 0.5
+        metrics:
+          - goal_success_rate
+          - answer_relevancy

mas_lab_library_eval-0.2.0/examples/mce/judge-override/mas.yaml

@@ -0,0 +1,15 @@
+#  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+#  SPDX-License-Identifier: Apache-2.0
+apiVersion: mas/v1
+kind: Agent
+
+metadata:
+  name: qa-agent
+
+spec:
+  models:
+    - model: gpt-4o
+  description: Stub agent for the eval_mce judge-override example.
+  context:
+    role: |
+      Answer questions clearly.

mas_lab_library_eval-0.2.0/library.yaml

@@ -0,0 +1,29 @@
+#  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+#  SPDX-License-Identifier: Apache-2.0
+# MAS Library Metadata — Eval
+#
+# Declares the eval_metric plugin type and a toy metric used by tests and docs.
+# Libraries register metrics onto the MCE eval provider.
+
+apiVersion: mas/v1
+kind: Library
+
+name: eval
+description: >
+  Evaluation metrics and LLM-as-judge scoring. Hosts the eval_metric plugin
+  type so other libraries can register session metrics onto an eval provider.
+version: "0.2.0"
+
+module_base: mas.library.eval
+
+types:
+  - eval_metric
+
+plugins:
+  - type: eval_metric
+    name: toy_echo
+    module: mas.library.eval.metrics.toy
+    class: ToyEchoMetric
+    description: Example session metric that scores 1.0 when a native trace exists.
+    attributes:
+      provider: mce