mas-lab-component-bench 0.2.0__tar.gz

mas_lab_component_bench-0.2.0/.gitignore

@@ -0,0 +1,86 @@
+*.pyc
+__pycache__/
+*.egg-info/
+.DS_Store
+dist/
+build/
+.venv/
+.demo-ui/
+.demo-ui-mock/
+.env
+.env.local
+.env.*.local
+config.yaml
+!lab/src/mas/lab/templates/**/config.yaml
+!docs/tutorials/**/config.yaml
+!labs/**/config.yaml
+!library-samples/sample-workspace/config.yaml
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.test-venv/
+node_modules/
+*.log
+docs/tutorials/**/logs/
+.hypothesis/
+compliance-report.json
+COMPLIANCE_REPORT.md
+compliance/results/
+
+# MkDocs build output
+site/
+
+.vscode/
+.cursor/
+.claude/
+
+# Test / coverage artifacts
+.coverage
+coverage.xml
+coverage.json
+**/coverage.json
+
+# LaTeX build artifacts
+*.fdb_latexmk
+*.fls
+*.aux
+*.out
+*.toc
+*.synctex.gz
+
+# Run output / artifacts
+**/output/*.jsonl
+**/output/benchmark
+**/output/meta.json
+**/output/current_run_id.txt
+**/output/*/
+.tutorial-cache/
+tests/.tutorial-engine-cache/
+.demo-cache/
+# Local mas-ctl chat traces next to an agent (fixtures live under fixtures/ or T01 traces/)
+docs/tutorials/**/agents/**/traces/
+
+# Backup files
+*.bak
+*.bak2
+
+# Legacy dead code (removed from repo)
+lab/src/mas/lab/_cli_legacy.py
+.deepeval/
+~/.mas/
+
+
+# Docker local secrets + data (never commit)
+docker/.env
+docker/data/*
+!docker/data/.gitkeep
+
+# Local protocol compliance checkouts and reports
+compliance/vendor/
+
+# Local harness playbooks (not published)
+AGENTS.md
+# .agents/
+COVERAGE_REPORT.md
+MANIFEST_SCHEMA_MAPPING.md
+.lycheecache

mas_lab_component_bench-0.2.0/PKG-INFO

@@ -0,0 +1,19 @@
+Metadata-Version: 2.5
+Name: mas-lab-component-bench
+Version: 0.2.0
+Summary: Benchmark engine, pipeline execution, plots and validation for MAS Lab.
+License: Apache-2.0
+Requires-Python: >=3.11
+Requires-Dist: click>=8.0
+Requires-Dist: huggingface-hub>=0.30.0
+Requires-Dist: jsonschema>=4.0
+Requires-Dist: mas-lab-component-core
+Requires-Dist: mas-lab-ctl
+Requires-Dist: mas-lab-library-eval
+Requires-Dist: mas-lab-runtime
+Requires-Dist: openai>=1.0.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: playwright>=1.40
+Requires-Dist: plotnine>=0.12.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: scipy>=1.10.0

mas_lab_component_bench-0.2.0/README.md

@@ -0,0 +1,232 @@
+<!--
+  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+  SPDX-License-Identifier: Apache-2.0
+-->
+# lab/components/bench
+
+Runs reproducible benchmark experiments: scenarios x dataset items x runs, then pipeline steps on the resulting `events.jsonl` logs. Runs resume from the trace cache when inputs match.
+
+Use it when you compare multi-agent system (MAS) variants, for example an ablation, and need statistics and plots. Terms are in the [glossary](../../../docs/glossary.md). The user guide is [lab/docs/benchmark.md](../../docs/benchmark.md).
+
+## Quick start
+
+```bash
+uv pip install mas-lab                      # full stack (meta-package)
+uv pip install -e lab/components/bench      # or this component alone
+
+mas-lab benchmark run my-experiment.yaml --dry-run    # validate config, show plan
+mas-lab benchmark run my-experiment.yaml --progress   # execute (resumes a prior run)
+mas-lab benchmark follow my-experiment.yaml           # watch progress from a second terminal
+mas-lab benchmark show last                           # inspect results
+```
+
+## Experiment YAML
+
+```yaml
+experiment:
+  name: my-trip-planner-ablation
+  default_flavour: local
+  application:
+    manifest: path/to/mas.yaml
+    configs_dir: ./overlays
+  scenarios:
+    - id: baseline
+      overlays: {logic: [], control: [], infra: []}
+    - id: with-mitigation
+      overlays: {logic: [], control: [full_mitigation], infra: []}
+  dataset:
+    path: ./datasets/queries.yaml
+    limit: 20
+  run:
+    n_runs: 5
+    artifacts:
+      trace: trace
+      df: dataframe
+    post:
+      - {name: extract-trace-stats, type: extract_trace_stats, in: trace, out: df}
+  item:
+    artifacts: {df: dataframe}
+    post:
+      - {name: gather-item, type: gather_level, in: df, out: df, depends_on: [extract-trace-stats]}
+  scenario:
+    artifacts: {df: dataframe}
+    post:
+      - {name: gather-scenario, type: gather_level, in: df, out: df, depends_on: [gather-item]}
+  post:
+    - {name: gather-experiment, type: gather_level, in: df, out: df, depends_on: [gather-scenario]}
+  execution:
+    strategy: coverage
+```
+
+## Commands
+
+### `benchmark run`
+
+Executes an experiment. A run with `run.n_runs: 5` resumes from where it stopped. Results go under the lab output root derived from `experiment.name` (override with `-o`).
+
+```bash
+mas-lab benchmark run experiment.yaml
+mas-lab benchmark run experiment.yaml --force          # always start fresh
+mas-lab benchmark run experiment.yaml --dry-run        # validate + plan only
+mas-lab benchmark run experiment.yaml --single-run     # 1 scenario, handy for CI
+mas-lab benchmark run experiment.yaml --limit-scenarios 10
+mas-lab benchmark run experiment.yaml --sample-scenarios 20 --seed 42
+mas-lab benchmark run experiment.yaml --max-runs 3
+mas-lab benchmark run experiment.yaml --strategy depth # depth-first (default: coverage)
+mas-lab benchmark run experiment.yaml -o /tmp/results
+```
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--force` | off | Always create a new run |
+| `--benchmark-id` | - | Resume a specific benchmark by ID |
+| `--progress / --no-progress` | progress | Real-time progress bar |
+| `--dry-run` | off | Validate config + show plan, no execution |
+| `--max-runs` | from YAML `run.n_runs` | Override `n_runs` |
+| `--limit-scenarios` | - | Limit to first N scenarios |
+| `--sample-scenarios` | - | Randomly sample N scenarios |
+| `--single-run` | off | Run exactly 1 scenario |
+| `-o / --output-dir` | derived from `experiment.name` | Output directory |
+| `--trace-cache` | `$XDG_CACHE_HOME/mas/traces` | Override trace-cache directory |
+| `--force-lock` | off | Break existing lock |
+| `--flavour` | `local` | Runtime flavour YAML name |
+| `--strategy` | `coverage` | `coverage` (breadth-first) or `depth` (depth-first) |
+
+### Other `benchmark` commands
+
+```bash
+# list: --status running|completed|failed|partial, --limit (20), --format simple|markdown|csv|json|jsonl
+mas-lab benchmark list --status completed --limit 10 --format markdown
+
+# show: one run, or a section of it
+mas-lab benchmark show <id>
+mas-lab benchmark show last plots          # also: events-trace, otel-traces
+mas-lab benchmark show last --format json
+
+# follow: poll run_info.json files and stream a tally until all runs finish
+mas-lab benchmark follow experiment.yaml --interval 10
+mas-lab benchmark follow experiment.yaml --once        # single snapshot
+
+# analyze: regenerate plots and statistics from existing output (no re-execution)
+mas-lab benchmark analyze <id> --experiment-yaml path/to/experiment.yaml
+
+# update: edit metadata of a run
+mas-lab benchmark update <id> --name "Final ablation v2" --add-tag ablation --remove-tag wip
+
+# step: inspect and restart individual pipeline steps of a run
+mas-lab benchmark step list <id>
+mas-lab benchmark step show <id> <step_id>
+mas-lab benchmark step restart <id> <step_id>
+
+# pipeline: run a standalone post-processing pipeline YAML on existing output
+mas-lab benchmark pipeline validate pipeline.yaml   # schema check
+mas-lab benchmark pipeline show pipeline.yaml       # step graph
+mas-lab benchmark pipeline plan pipeline.yaml       # execution plan
+mas-lab benchmark pipeline run pipeline.yaml
+```
+
+## Post-processing pipeline
+
+A pipeline YAML declares steps and their dependencies. Steps run in topological order; independent steps run in parallel.
+
+```yaml
+name: post-process
+steps:
+  - name: extract
+    type: extract_trajectories
+
+  - name: stats
+    type: extract_trace_stats
+    depends_on: [extract]
+
+  - name: eval
+    type: eval_mce
+    scope: run              # scores ONE run per invocation; needs run-level context
+    depends_on: [extract]
+    config:
+      metrics: [goal_success_rate]
+
+  - name: plot
+    type: plot
+    depends_on: [eval]
+    config:
+      spec: shapley_bars
+```
+
+Step implementations are library plugins registered in the runtime registry through the `types:` / `plugins:` block of [`library-lab/library.yaml`](../../../library-lab/library.yaml). `mas-lab benchmark pipeline validate` lists the registered step types. The full catalog is in [pipeline-steps.md](../../docs/pipeline-steps.md). Steps such as `embed_states` and `list_clickhouse_sessions` can live in your own package and register through entry points.
+
+| Category | Step types |
+|----------|------------|
+| `extract/` | `extract_trace_stats`, `extract_mealy_stats`, `extract_trajectories` (builds `trajectories.jsonl` from `events.jsonl`) |
+| `eval/` | `eval_mce` (MCE LLM-as-judge scoring on `events.jsonl`), `collect_metrics`, `compute_ci` |
+| `viz/` | `plot` (declarative figure from a `plot_library/` spec: `shapley_bars`, `ablation_heatmap`, `trajectory_gantt`), `plot_trajectory` (per-run SVG), `plot_multilevel_trajectory` (swimlane HTML/SVG), `ci_plot` |
+| `data/` | `to_dataframe`, `serialize`, `processor` (run a registered trajectory processor by name) |
+| `services/` | `service_start`, `export_otel` |
+| other | `experiment` (run a MAS scenario via `mas-ctl`), `dataset` (load / filter a scenario dataset), `annotate_metrics` (attach scores to run metadata), `embed_trajectories` (embed trajectory strings for manifold analysis) |
+
+Shared helpers are in `pipeline/lib/` (`data_source`, `plot_lib`, `plot_specs/`).
+
+## Related commands
+
+Score one trace with MCE metrics (for batches, use the `eval_mce` step; see [library-eval](../../../library-eval/README.md)):
+
+```bash
+mas-lab eval path/to/traces/events.jsonl --metric groundedness --json
+```
+
+Draw visualisations directly from a trace file:
+
+```bash
+mas-lab plot trajectory events.jsonl -o trajectory.png
+mas-lab plot multilevel-trajectory events.jsonl -o multilevel.png
+mas-lab plot communication-flow events.jsonl -o flow.png
+mas-lab plot list output/benchmark/runs/
+```
+
+## Python API
+
+Prefer `mas-lab benchmark run experiment.yaml`. For an in-process run:
+
+```python
+from pathlib import Path
+from mas.lab.benchmark.worker import run_benchmark_sync
+
+ok = run_benchmark_sync(
+    Path("labs/design-space.lab/02-topologies/experiment.yaml"),
+    flavour_name="local",
+    output_dir=Path.home() / ".local/share/mas/labs/my-run",
+)
+```
+
+## Package layout
+
+```
+components/bench/src/mas/lab/benchmark/
++-- dataset.py              dataset loading, filtering, sampling
++-- experiment.py           ExperimentRunner + ExperimentConfig
++-- metadata.py             BenchmarkMetadata (run_info.json)
++-- lock.py                 advisory file lock (concurrent-run safety)
++-- deduplication.py        trace-cache dedup (skip already-seen prompts)
++-- analysis.py             statistical analysis helpers
++-- migrate.py              run directory migration utilities
++-- pipeline/
+    +-- pipeline.py         pipeline loader + validator
+    +-- executor.py         topological executor (parallel)
+    +-- registry.py         step-type registry
+    +-- cache.py            step output cache
+    +-- steps/              experiment.py, extract_trajectories.py, eval_mce.py,
+                            plot_multilevel_trajectory.py, ...
+```
+
+## Documentation
+
+| Document | Description |
+|----------|-------------|
+| [benchmark.md](../../docs/benchmark.md) | Benchmark guide: quickstart, concepts, lifecycle |
+| [pipeline.md](../../docs/pipeline.md) | Pipeline architecture and YAML schema |
+| [pipeline-steps.md](../../docs/pipeline-steps.md) | Step type reference |
+| [pipeline-processors.md](../../docs/pipeline-processors.md) | Processor registry and plot library |
+| [library-eval/README.md](../../../library-eval/README.md) | MCE metrics and `mas-lab eval` |
+| [designing-experiments.md](../../docs/designing-experiments.md) | Designing ablations and Shapley experiments |
+| [benchmark-state-architecture.md](../../docs/benchmark-state-architecture.md) | Run state machine and locking |
+| [multi-scenario-format.md](../../docs/multi-scenario-format.md) | Multi-scenario dataset format |

mas_lab_component_bench-0.2.0/pyproject.toml

@@ -0,0 +1,35 @@
+#  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+#  SPDX-License-Identifier: Apache-2.0
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "mas-lab-component-bench"
+version = "0.2.0"
+description = "Benchmark engine, pipeline execution, plots and validation for MAS Lab."
+requires-python = ">=3.11"
+license = {text = "Apache-2.0"}
+
+dependencies = [
+    "mas-lab-component-core",
+    "mas-lab-library-eval",
+    "mas-lab-runtime",
+    "mas-lab-ctl",
+    "jsonschema>=4.0",
+    "click>=8.0",
+    "huggingface_hub>=0.30.0",
+    "openai>=1.0.0",
+    "pandas>=2.0.0",
+    "plotnine>=0.12.0",
+    "scipy>=1.10.0",
+    "psutil>=5.9.0",
+    "playwright>=1.40",
+    # MCE evaluation metrics are provided by mas-lab-library-eval (workspace member)
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/mas"]
+
+[project.entry-points."mas.lab.runners"]
+native = "mas.lab.benchmark.plugins.mas:MasRuntimeRunner"

mas_lab_component_bench-0.2.0/src/mas/lab/benchmark/README.md

@@ -0,0 +1,323 @@
+<!--
+  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+  SPDX-License-Identifier: Apache-2.0
+-->
+# Benchmark Framework - Reusable Components
+
+**Composable primitives for agent evaluation.**
+
+## Philosophy
+
+This framework follows these principles:
+
+1. **Declarative over imperative** - Configure via YAML/JSON, not code
+2. **Incremental over monolithic** - Add runs without rerunning everything
+3. **Composable over all-in-one** - Mix and match components
+4. **Human-readable over optimized** - Folder names you can understand
+5. **DataFrame-native over custom formats** - Leverage pandas/polars ecosystem
+
+## Components
+
+### 1. Dataset (`dataset.py`)
+
+Loads and manipulates evaluation datasets.
+
+```python
+from mas.lab.benchmark import Dataset
+
+# Load from JSON
+dataset = Dataset.from_json("math_problems.json")
+
+# Filter by category
+arithmetic = dataset.filter(category="arithmetic")
+
+# Iterate
+for item in dataset:
+    print(item.id, item.prompt)
+```
+
+**JSON Format:**
+```json
+{
+  "name": "dataset_name",
+  "version": "v1",
+  "items": [
+    {
+      "id": "001",
+      "prompt": "Question text",
+      "category": "category_name",
+      "expected_answer": "Answer",
+      "metadata_key": "metadata_value"
+    }
+  ]
+}
+```
+
+### 2. ResultStorage (`storage.py`)
+
+Manages human-readable folder structure.
+
+```python
+from mas.lab.benchmark import ResultStorage, RunMetadata
+
+storage = ResultStorage("./results")
+
+# Generate run ID
+run_id = storage.generate_run_id("cot", run_number=1)
+# → "cot_run_001_20260218_143022"
+
+# Save run
+metadata = RunMetadata(
+    dataset_name="math",
+    item_id="001",
+    run_id=run_id,
+    pattern="cot",
+    timestamp="2026-02-18T14:30:22",
+    config={"model": "gpt-4"},
+    success=True,
+    latency_ms=1234.5
+)
+
+storage.save_run(
+    dataset_name="math",
+    item_id="001",
+    run_id=run_id,
+    metadata=metadata,
+    events_file=Path("events.jsonl"),
+    artifacts={"log": Path("agent.log")}
+)
+
+# List all runs
+runs = storage.list_runs("math")
+# → [("math", "001", "cot_run_001_20260218_143022"), ...]
+```
+
+**Folder Structure:**
+```
+results/
+  {dataset_name}/
+    {item_id}/
+      {run_id}/
+        metadata.json
+        events.jsonl
+        artifacts/
+    consolidated.parquet
+```
+
+### 3. MultiRunOrchestrator (`runner.py`)
+
+Coordinates N runs per item across patterns.
+
+```python
+from mas.lab.benchmark import MultiRunOrchestrator
+
+orchestrator = MultiRunOrchestrator(
+    storage=storage,
+    n_runs=3,
+    pause_between_runs=1.0
+)
+
+# Define agent factory
+async def agent_factory(pattern_name, pattern_config):
+    # Your agent creation logic
+    return agent
+
+# Run benchmark
+summary = await orchestrator.run_dataset(
+    dataset=dataset,
+    patterns={
+        "cot": {"manifest": "cot.yaml"},
+        "react": {"manifest": "react.yaml"}
+    },
+    agent_factory=agent_factory
+)
+```
+
+### 4. ResultAnalyzer (`analysis.py`)
+
+Consolidates results into DataFrame.
+
+```python
+from mas.lab.benchmark import ResultAnalyzer
+
+analyzer = ResultAnalyzer(storage)
+
+# Consolidate all runs
+df = analyzer.consolidate_results(
+    dataset_name="math",
+    include_events=True,  # Parse event logs
+    cache=True            # Save as Parquet
+)
+
+# Compute statistics
+stats = analyzer.compute_statistics(df)
+#    pattern  latency_ms_mean  success_rate  mean_tokens
+# 0  cot      1234.5           0.95          567
+# 1  react    890.2            0.92          423
+
+# Compare patterns
+comparison = analyzer.compare_patterns(df, ["cot", "react"], "latency_ms")
+```
+
+**DataFrame Schema:**
+```python
+df.columns
+# ['dataset_name', 'item_id', 'run_id', 'pattern',
+#  'timestamp', 'success', 'latency_ms', 'error',
+#  'total_tokens', 'prompt_tokens', 'completion_tokens',
+#  'tool_calls', 'llm_calls',
+#  'config_*']
+```
+
+### 5. Pipeline plots (`plotnine` / `plot` steps)
+
+Paper figures use **`type: plotnine`** in `experiment.yaml` experiment-level `post:`.
+Ad-hoc charts from step data use **`type: plot`** with **`config.spec`** (plot library).
+
+See [pipeline-steps.md](../../../../../../docs/pipeline-steps.md) and `labs/*/experiment.yaml` for examples.
+
+## Complete Example
+
+```python
+import asyncio
+from pathlib import Path
+from mas.lab.benchmark import (
+    Dataset, ResultStorage, MultiRunOrchestrator,
+    ResultAnalyzer,
+)
+
+async def my_agent_factory(pattern_name, pattern_config):
+    # Your agent creation logic
+    pass
+
+async def main():
+    # 1. Load dataset
+    dataset = Dataset.from_yaml("dataset.yaml")
+    
+    # 2. Setup storage
+    storage = ResultStorage("./results")
+    
+    # 3. Run benchmark
+    orchestrator = MultiRunOrchestrator(storage, n_runs=3)
+    await orchestrator.run_dataset(
+        dataset,
+        patterns={"cot": {}, "react": {}},
+        agent_factory=my_agent_factory
+    )
+    
+    # 4. Analyze
+    analyzer = ResultAnalyzer(storage)
+    df = analyzer.consolidate_results(dataset.name)
+    stats = analyzer.compute_statistics(df)
+    print(stats)
+    # Plots: declare plotnine / plot steps in experiment.yaml post:
+
+asyncio.run(main())
+```
+
+## Incremental Updates
+
+### Add Items
+```python
+# Load existing dataset
+dataset = Dataset.from_json("dataset.json")
+
+# Add new items
+dataset.items.append(DatasetItem(id="006", prompt="New question"))
+
+# Save
+dataset.to_json("dataset.json")
+
+# Run benchmark (only new items executed)
+await orchestrator.run_dataset(dataset, patterns, agent_factory)
+```
+
+### Add Runs
+```python
+# Increase n_runs
+orchestrator = MultiRunOrchestrator(storage, n_runs=5)  # Was 3
+
+# Run again - adds run_004, run_005 to existing items
+await orchestrator.run_dataset(dataset, patterns, agent_factory)
+```
+
+### Add Patterns
+```python
+# Add new pattern
+patterns = {
+    "cot": {...},
+    "react": {...},
+    "planner": {...}  # NEW
+}
+
+# Run again - only planner executed on all items
+await orchestrator.run_dataset(dataset, patterns, agent_factory)
+```
+
+### Regenerate Plots
+```python
+# Re-run pipeline post steps only (content-addressed trace cache skips LLM calls)
+# mas-lab benchmark run experiment.yaml --force-post
+```
+
+## Advanced Usage
+
+### Custom Metrics
+
+Extend `ResultAnalyzer._extract_event_metrics()`:
+
+```python
+def _extract_event_metrics(self, events_file: Path) -> Dict[str, Any]:
+    metrics = super()._extract_event_metrics(events_file)
+    
+    # Add custom metrics
+    metrics["my_custom_metric"] = ...
+    
+    return metrics
+```
+
+### Custom Plots
+
+Add a `plotnine` step in experiment-level `post:` or a plot-library YAML under
+`pipeline/steps/plot_library/`. See `labs/design-space.lab/01-design-patterns/experiment.yaml`.
+
+### Parallel Execution
+
+```python
+# Run patterns in parallel
+import asyncio
+
+async def run_pattern(pattern_name, pattern_config):
+    await orchestrator.run_dataset(
+        dataset,
+        patterns={pattern_name: pattern_config},
+        agent_factory=agent_factory
+    )
+
+await asyncio.gather(
+    run_pattern("cot", {...}),
+    run_pattern("react", {...}),
+)
+```
+
+## Dependencies
+
+```bash
+pip install pandas pyarrow plotnine pyyaml
+```
+
+## Testing
+
+```bash
+cd mas-lab
+pytest tests/benchmark/
+```
+
+## Contributing
+
+When adding new components:
+
+1. Follow existing patterns (DataClass for data, Class for behavior)
+2. Add docstrings with examples
+3. Write tests in `tests/benchmark/`
+4. Update this documentation