@tangle-network/starter-foundry 0.7.3 → 0.13.2

package/registry/families/agent-eval-harness-py/files/AGENTS.md

@@ -0,0 +1,135 @@
+---
+name: agent-eval-harness-py
+role: Python eval-harness operator — runs scenarios against an agent-under-test inside a tangle-sandbox, scores with multi-dim LLM judges, emits scorecard.json, gates regressions with bootstrap CI + Cohen's d + Welch's t-test
+domain: agent-evaluation
+runtime: python>=3.10
+allowedDomains:
+  - router.tangle.tools
+  - sandbox.tangle.tools
+allowedEnv:
+  - TANGLE_API_KEY
+  - TANGLE_SANDBOX_API_KEY
+  - EVAL_JUDGE_MODEL
+  - EVAL_OUT
+version: 0.1.0
+---
+
+## Role
+
+You are the Python eval-harness operator. The deliverable is a
+`scorecard.json` produced by `python -m eval run` that an operator (or CI)
+can compare against a baseline with `python -m eval compare`. The scorecard
+shape is identical to the TypeScript sibling (`agent-eval-harness-ts`) so a
+TS baseline can be compared against a Python head and vice-versa.
+
+You **do not** invent metrics, judges, or rubrics outside the scenario+judge
+contract. If a scenario does not declare a judge, the run records a `gap`
+status — never a fabricated score.
+
+## Success criteria
+
+A run is correct when **all** of these hold:
+
+1. Every declared scenario produced one `flow` entry in `scorecard.flows[]`.
+2. Every flow entry has `value`, `target`, `direction`, `status`,
+   `productValueClaim`. `status` is one of `pass | fail | unmeasured | gap`.
+3. The scorecard's `aggregate` is the unweighted mean of pass-rate across
+   measurable flows (`status in {pass, fail}`), rounded to 4 decimals.
+4. The scorecard validates against `src/eval/scorecard.py::Scorecard.model_validate`.
+5. `python -m eval compare baseline.json head.json` exits 0 when no flow
+   regressed beyond `--alpha 0.05` Welch's t-test on bootstrap samples
+   AND no flow's Cohen's d magnitude exceeds 0.2 in the worse direction.
+
+## Stop rules
+
+Stop and surface a `[blocked]` (literal token) when:
+
+- `TANGLE_API_KEY` is unset and a judge requires it.
+- `TANGLE_SANDBOX_API_KEY` is unset and `--driver sandbox` is selected.
+- A scenario module fails to import (`ImportError`); never auto-rewrite the
+  scenario.
+- Bootstrap CI math sees fewer than 8 samples (CI is meaningless below that).
+
+A `[blocked]` line names which env var or precondition is missing and what
+to set.
+
+## Workflow
+
+When the operator says "run the eval":
+
+1. `python -m eval run --scenarios scenarios/ --out scorecard.json`
+2. If `scorecard.aggregate < target` (default 0.85), summarise which flows
+   failed and propose a single fix.
+3. Never edit `src/eval/judges/rubric.py` unless asked — it's the
+   measurement substrate.
+
+When the operator says "compare":
+
+1. `python -m eval compare baseline.json head.json --alpha 0.05`
+2. If exit non-zero, list every regressed flow with `cohen_d`,
+   `welch_p`, and `diff_ci_low/high` (CI on `head − baseline`, not on the
+   head mean alone — the latter was the Gen-15 muffled-gate).
+
+## Tangle integration (sandbox-driven runs)
+
+Scenarios can run against three drivers, controlled by `--driver`:
+
+- `local` — agent is a Python callable in `scenarios.<name>.agent_fn`
+  (fastest, used in CI smoke).
+- `http` — agent is an HTTP endpoint conforming to the OpenAI-compatible
+  chat-completions shape; pass `--agent-url`.
+- `sandbox` — uses `tangle_sandbox.Sandbox` to spin an isolated container,
+  deploy the agent bundle from `--bundle <dir>`, run scenarios via
+  `box.task()`, then `box.delete()`. This is the production driver.
+
+The `sandbox` driver writes the bundle's `AGENTS.md` + resources at
+`/home/agent/`, matching the harness contract used by every Tangle agent.
+See `src/eval/sandbox_runner.py`.
+
+## Bias against fake signal
+
+- Never assert `>=` thresholds without a bootstrap CI (`scipy.stats` or the
+  hand-rolled `bootstrap_ci()` in `regression.py`).
+- Never set a flow `status: pass` when the judge timed out or returned
+  malformed JSON — record `unmeasured` with a structured reason.
+- Never coalesce missing scenarios silently. Missing == `gap`.
+
+## Scorecard shape (cross-language contract)
+
+```json
+{
+  "product": "<name>",
+  "timestamp": "<ISO-8601>",
+  "coverage": "<measured>/<declared> flows measured",
+  "aggregate": 0.0,
+  "inputs": [{ "path": "...", "mtime": "..." }],
+  "flows": [
+    {
+      "name": "<flow-name>",
+      "value": 0.0,
+      "target": 0.85,
+      "status": "pass | fail | unmeasured | gap",
+      "productValueClaim": "<one sentence>",
+      "direction": "higher-better | lower-better",
+      "notes": "<optional>"
+    }
+  ]
+}
+```
+
+Both the TS and Python harnesses round `value`, `target`, and `aggregate`
+to 4 decimals so byte-equal cross-language diffs are possible.
+
+## What you WILL do
+
+- Run the scenarios + judges as declared.
+- Persist the scorecard to `--out` (default `scorecard.json`).
+- Surface every `gap`/`unmeasured` flow in the run summary.
+- Suggest the single highest-leverage fix when `aggregate < target`.
+
+## What you will NOT do
+
+- Edit judge rubrics to make a flow pass.
+- Hide failed flows from the scorecard.
+- Use mock LLM responses in production runs.
+- Skip the regression gate "because the diff is small."

package/registry/families/agent-eval-harness-py/files/README.md

@@ -0,0 +1,145 @@
+# agent-eval-harness-py
+
+Python eval harness for agent-under-test scoring. Loads scenarios, runs them
+against an agent (local fn / HTTP endpoint / tangle-sandbox), scores with
+multi-dim LLM judges via `router.tangle.tools`, emits a `scorecard.json`
+that is byte-for-byte compatible with the TS sibling
+([`agent-eval-harness-ts`](../agent-eval-harness-ts)) so cross-language
+`compare` works.
+
+## Quick start
+
+```bash
+pip install -e ".[dev]"
+python -m eval run --scenarios scenarios --out scorecard.json
+```
+
+> **`tangle-sandbox` PyPI status (2026-04-27): not yet published.** The
+> SDK source is at `~/webb/agent-dev-container/products/sandbox/sdk-python/`
+> (version 0.1.0). Until `pip install tangle-sandbox` is publishable,
+> install from local checkout:
+>
+> ```bash
+> pip install -e ~/webb/agent-dev-container/products/sandbox/sdk-python/
+> ```
+>
+> The `--driver sandbox` mode requires this; `--driver local` and
+> `--driver http` work without it.
+
+The default scenario (`scenarios/example.py`) runs offline with a local
+agent + offline judge — useful for smoke tests.
+
+To run with the LLM-judge:
+
+```bash
+export TANGLE_API_KEY=sk-tan-...
+python -m eval run --scenarios scenarios --out scorecard.json
+```
+
+To run against a tangle-sandbox-deployed agent bundle:
+
+```bash
+export TANGLE_SANDBOX_API_KEY=sk-...
+export TANGLE_SANDBOX_BASE_URL=https://sandbox.tangle.tools
+python -m eval run \
+  --scenarios scenarios \
+  --bundle path/to/agent-bundle \
+  --driver sandbox \
+  --out scorecard.json
+```
+
+## Regression gate
+
+```bash
+python -m eval compare baseline.json head.json
+# Welch's t-test, alpha=0.05; Cohen's d threshold 0.2; 1000-resample bootstrap CI
+```
+
+For a meaningful gate you need ≥8 repeated samples per flow (we won't
+fabricate a CI from a single value). Provide them with `--samples
+samples.json` where `samples.json` looks like:
+
+```json
+{
+  "<flow-name>": {
+    "baseline": [0.83, 0.85, 0.84, 0.86, 0.85, 0.87, 0.85, 0.84],
+    "head":     [0.81, 0.79, 0.82, 0.80, 0.83, 0.81, 0.78, 0.80]
+  }
+}
+```
+
+## Scenario contract
+
+```python
+# scenarios/<name>.py
+from eval.judges.rubric import RubricJudge, RubricDimension
+
+JUDGE = RubricJudge(
+    name="quality",
+    dimensions=[
+        RubricDimension(name="correctness", description="...", weight=2.0),
+        RubricDimension(name="completeness", description="...", weight=1.0),
+    ],
+)
+
+def _agent(prompt: str) -> str:  # only required for --driver local
+    ...
+
+SCENARIO = {
+    "name": "unique-flow-name",
+    "input": "the prompt sent to the agent",
+    "expected_output": "reference (optional)",
+    "target": 0.85,
+    "direction": "higher-better",        # or "lower-better"
+    "productValueClaim": "one sentence — what does pass mean to the user",
+    "agent_fn": _agent,
+    "judges": [JUDGE],
+}
+```
+
+## Layout
+
+```
+src/eval/
+  __init__.py            public surface
+  __main__.py            python -m eval
+  cli.py                 argparse: run | compare
+  runner.py              scenario loader + dispatcher
+  scorecard.py           Pydantic models + cross-language JSON
+  sandbox_runner.py      tangle-sandbox lifecycle
+  regression.py          bootstrap CI / Cohen's d / Welch's t-test
+  judges/
+    __init__.py
+    rubric.py            LLM-as-judge over router.tangle.tools
+    example_judge.py     3-dim code-quality rubric
+scenarios/
+  example.py             smoke (offline agent + offline judge)
+tests/
+  test_runner.py         scenario loader + flow emission
+  test_regression.py     gate math (anchors)
+  test_sandbox.py        SandboxDriver lifecycle (mocked SDK)
+.github/workflows/eval.yml  CI: pytest + example run + regression gate
+```
+
+## Cross-language compatibility
+
+This harness emits the same `scorecard.json` shape as
+`agent-eval-harness-ts`. A baseline produced by either harness can be
+compared against a head from the other:
+
+```bash
+python -m eval compare ts-baseline.json py-head.json
+# or
+node ts/dist/cli.js compare py-baseline.json ts-head.json
+```
+
+Field rounding (4 decimals on `value` / `target` / `aggregate`), key
+order, and the Welch + Cohen + bootstrap CI math are all aligned.
+
+## What this is NOT
+
+- A general-purpose benchmark suite — write your own scenarios.
+- A judge marketplace — `RubricJudge` is one implementation; build domain
+  judges in `src/eval/judges/`.
+- A safe substitute for human review — judges are noisy. The regression
+  gate exists because LLM-judge variance is non-trivial.

package/registry/families/agent-eval-harness-py/files/agent.json

@@ -0,0 +1,30 @@
+{
+  "name": "agent-eval-harness-py",
+  "description": "Python eval-harness operator agent: drives `python -m eval run`, inspects scorecard.json, runs regression gate against a baseline, opens fix PRs when judge findings cluster on a specific scenario or dimension. Reads/writes scenarios/, src/eval/, and scorecard artifacts; never edits judges/ unless explicitly told.",
+  "version": "0.1.0",
+  "tags": [
+    "agent-eval",
+    "python",
+    "eval-harness"
+  ],
+  "prompt": {
+    "systemPromptFile": "AGENTS.md"
+  },
+  "tools": {
+    "filesystem": true,
+    "shell": true,
+    "router": true
+  },
+  "permissions": {
+    "filesystem.write": "ask",
+    "shell.exec": "ask",
+    "network": "ask"
+  },
+  "resources": {
+    "files": [
+      {
+        "source": "README.md"
+      }
+    ]
+  }
+}

package/registry/families/agent-eval-harness-py/files/pyproject.toml

@@ -0,0 +1,48 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "agent-eval-harness-py"
+version = "0.1.0"
+description = "Python eval harness for agent-under-test scoring; emits cross-language-compatible scorecard.json and gates regressions with bootstrap CI + Cohen's d + Welch's t-test."
+readme = "README.md"
+requires-python = ">=3.10"
+authors = [{ name = "Tangle" }]
+license = { text = "MIT" }
+keywords = ["agent-eval", "llm-as-judge", "regression-gate", "tangle-sandbox"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+]
+dependencies = [
+  "tangle-sandbox>=0.1",
+  "httpx>=0.27",
+  "pydantic>=2.6",
+  "numpy>=1.26",
+  "scipy>=1.11",
+]
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.0",
+  "pytest-asyncio>=0.23",
+]
+
+[project.scripts]
+eval-harness = "eval.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-dir]
+"" = "src"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"
+pythonpath = ["src", "."]

package/registry/families/agent-eval-harness-py/files/requirements.txt

@@ -0,0 +1,7 @@
+tangle-sandbox>=0.1
+httpx>=0.27
+pydantic>=2.6
+numpy>=1.26
+scipy>=1.11
+pytest>=8.0
+pytest-asyncio>=0.23

package/registry/families/agent-eval-harness-py/files/scenarios/__init__.py

@@ -0,0 +1 @@
+"""Scenario package — every ``*.py`` module exporting ``SCENARIO`` is auto-loaded."""

package/registry/families/agent-eval-harness-py/files/scenarios/example.py

@@ -0,0 +1,50 @@
+"""Example scenario — agent must respond with the literal word `pong` to a `ping`.
+
+This scenario uses a deterministic local agent (no LLM) so CI can run it
+offline. Replace ``agent_fn`` with an HTTP call or sandbox driver in real use.
+
+To enable the rubric judge, set ``TANGLE_API_KEY`` and pass
+``--driver local`` (the local driver still consults the judge — only the
+agent-under-test is local).
+"""
+
+from __future__ import annotations
+
+from eval.judges.rubric import JudgeInput, JudgeResult
+
+
+def _agent_fn(prompt: str) -> str:
+    """A trivial agent: replies 'pong' iff the prompt contains 'ping'."""
+    return "pong" if "ping" in prompt.lower() else "unknown"
+
+
+def _exact_match_judge(ji: JudgeInput) -> JudgeResult:
+    """Offline judge — case-insensitive exact match against expected_output.
+
+    Demonstrates the judge contract without needing the router. Real
+    deployments swap this for a ``RubricJudge``.
+    """
+    expected = (ji.expected_output or "").strip().lower()
+    actual = ji.actual_output.strip().lower()
+    score = 1.0 if expected and actual == expected else 0.0
+    return JudgeResult(
+        score=score,
+        ok=True,
+        rationale=f"expected={expected!r} actual={actual!r}",
+    )
+
+
+SCENARIO = {
+    "name": "example_ping_pong",
+    "input": "ping",
+    "expected_output": "pong",
+    "target": 0.85,
+    "direction": "higher-better",
+    "productValueClaim": (
+        "Smoke test — proves the harness can load a scenario, dispatch to a "
+        "local agent, score with a judge, and emit a flow. If this fails, "
+        "the harness is broken before any real eval can run."
+    ),
+    "agent_fn": _agent_fn,
+    "judges": [_exact_match_judge],
+}

package/registry/families/agent-eval-harness-py/files/src/eval/__init__.py

@@ -0,0 +1,57 @@
+"""Python eval harness for agent-under-test scoring.
+
+Public surface:
+
+- ``Scorecard``, ``ScorecardFlow`` — Pydantic v2 models matching the TS sibling
+  byte-for-byte (cross-language ``compare`` works).
+- ``run_scenarios`` — load scenarios, dispatch against the chosen driver,
+  collect flow results, build a ``Scorecard``.
+- ``bootstrap_ci``, ``cohens_d``, ``welch_t_test``, ``regression_gate`` —
+  statistical gate primitives.
+- ``RubricJudge`` — LLM-as-judge over router.tangle.tools, multi-dim rubrics.
+- ``SandboxDriver`` — tangle-sandbox-backed scenario runner.
+"""
+
+from __future__ import annotations
+
+from .judges.rubric import RubricJudge, RubricResult
+from .regression import (
+    RegressionVerdict,
+    bootstrap_ci,
+    cohens_d,
+    regression_gate,
+    welch_t_test,
+)
+from .runner import RunConfig, run_scenarios
+from .sandbox_runner import SandboxDriver
+from .scorecard import (
+    FlowDirection,
+    FlowStatus,
+    Scorecard,
+    ScorecardFlow,
+    ScorecardInput,
+    aggregate_score,
+    write_scorecard,
+)
+
+__all__ = [
+    "Scorecard",
+    "ScorecardFlow",
+    "ScorecardInput",
+    "FlowStatus",
+    "FlowDirection",
+    "aggregate_score",
+    "write_scorecard",
+    "RunConfig",
+    "run_scenarios",
+    "SandboxDriver",
+    "RubricJudge",
+    "RubricResult",
+    "bootstrap_ci",
+    "cohens_d",
+    "welch_t_test",
+    "regression_gate",
+    "RegressionVerdict",
+]
+
+__version__ = "0.1.0"

package/registry/families/agent-eval-harness-py/files/src/eval/__main__.py

@@ -0,0 +1,10 @@
+"""Entrypoint for ``python -m eval``."""
+
+from __future__ import annotations
+
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())