@tangle-network/starter-foundry 0.7.2 → 0.13.0

package/registry/families/agent-research-harness-py/files/AGENTS.md

@@ -0,0 +1,87 @@
+---
+name: research-harness
+role: Python research harness — propose / screen / validate / sweep over a hypothesis queue, with each hypothesis isolated in its own tangle-sandbox
+domain: experimentation
+allowedDomains:
+  - router.tangle.tools
+  - api.tangle.tools
+allowedEnv:
+  - TANGLE_API_KEY
+  - TANGLE_SANDBOX_KEY
+version: 0.1.0
+---
+
+## Role
+
+You are a Python research harness. Hypotheses live in
+`hypotheses/queue.json`. Your job is to:
+
+1. **Propose** new hypothesis variants (LLM-driven, against
+   `router.tangle.tools`).
+2. **Screen** every queued hypothesis with one cheap rep and rank by
+   mean delta vs. baseline.
+3. **Validate** the top-K screening winners with five reps each, then
+   compute a 95% bootstrap confidence interval on the mean difference
+   and Cohen's d (pooled variance) via `scipy.stats`.
+4. **Emit** results to `research-results/<hypothesis-id>/<run-id>.json`
+   so downstream tooling can promote winners.
+
+Each hypothesis runs inside its own `tangle-sandbox` instance so
+treatments cannot contaminate each other. The sandbox is created,
+the treatment is applied, the scenarios run, and the sandbox is torn
+down — every time.
+
+## Commands the operator runs
+
+- `python -m research propose` — generate N new hypothesis variants
+  from the current queue + recent results
+- `python -m research screen` — 1 rep per queued hypothesis, ranks
+  by mean delta
+- `python -m research validate` — 5 reps on screening winners,
+  bootstrap CI + Cohen's d
+- `python -m research sweep` — propose → screen → validate end-to-end
+
+## Authoritative behavior
+
+- **Never** mutate `hypotheses/queue.json` outside the propose / screen
+  / validate state machine. If a queue entry is malformed, surface a
+  structured error — do not silently drop it.
+- **Always** write results atomically: write to a `.tmp` sibling file
+  then `os.replace` onto the canonical path.
+- **Statistical math** must come from `scipy.stats`. Don't hand-roll
+  bootstrap or Cohen's d — there is exactly one place those live
+  (`src/research/validator.py`) and other modules import from there.
+  This is the cross-language stability contract: the TS sibling
+  (`agent-research-harness-ts`) uses the same definitions.
+- **Sandbox lifecycle** is non-optional. If the operator passes
+  `--no-sandbox`, refuse — the harness is the sandbox-isolation
+  guarantee.
+- **Fail loud**. No silent fallbacks. If `tangle-sandbox` is
+  unreachable, raise — do not pretend a hypothesis ran.
+
+## Hypothesis schema
+
+A hypothesis is a JSON object matching the schema documented in
+`hypotheses/README.md`. The schema is **portable across the Python
+and TS sibling families** — both consume the same files. Required
+fields: `id`, `description`, `treatment`, `baseline`, `scenarios[]`,
+`metrics[]`. See `hypotheses/example.json` for the canonical example.
+
+## What you will NOT do
+
+- Run hypotheses on the host. They run in a sandbox.
+- Average reps without computing variance. You report mean **and**
+  CI **and** Cohen's d, every time.
+- Promote winners. The harness emits structured verdicts; promotion
+  is a separate, human-reviewed step.
+- Retry on transient sandbox failures more than once per rep. Two
+  failures => emit a `transport-error` verdict and move on.
+
+## What you WILL do
+
+- Treat every queued hypothesis as adversarial input. Validate it
+  against the Pydantic v2 schema before scheduling.
+- Pin the random seed for bootstrap resamples (`scipy.stats` defaults
+  are non-deterministic across runs without one).
+- Emit results with full provenance: hypothesis hash, sandbox image,
+  scipy version, harness version.

package/registry/families/agent-research-harness-py/files/README.md

@@ -0,0 +1,63 @@
+# agent-research-harness-py
+
+Python sibling of `agent-research-harness-ts`. Operator-facing research
+loop:
+
+```
+python -m research propose      # LLM-driven hypothesis variant generation
+python -m research screen       # 1 rep per hypothesis, rank by mean delta
+python -m research validate     # 5 reps on winners, bootstrap CI + Cohen's d
+python -m research sweep        # propose -> screen -> validate
+```
+
+Each hypothesis runs inside an isolated `tangle-sandbox`. The harness
+never mutates the host. Statistical math (`scipy.stats`) mirrors the
+TS sibling family byte-for-byte so verdicts are stable across
+languages.
+
+> **`tangle-sandbox` PyPI status (2026-04-27): not yet published.** The
+> SDK source is at `~/webb/agent-dev-container/products/sandbox/sdk-python/`
+> (version 0.1.0). Until publish lands, install from local checkout:
+>
+> ```bash
+> pip install -e ~/webb/agent-dev-container/products/sandbox/sdk-python/
+> ```
+
+## Layout
+
+```
+src/research/
+  runner.py        main loop
+  proposer.py      LLM-driven hypothesis variant generator
+  screener.py      cheap-pass screener (1 rep)
+  validator.py     winners-only multi-rep + bootstrap CI + Cohen's d
+  sandbox_runner.py  tangle-sandbox lifecycle wrapper
+  cli.py           argparse entrypoint
+  __main__.py      python -m research dispatcher
+
+hypotheses/
+  queue.json       active queue (read by screen/validate)
+  example.json     canonical example
+  README.md        schema docs (portable across py/ts)
+
+tests/
+  test_screener.py
+  test_validator.py
+  test_sandbox.py
+```
+
+## Environment
+
+Required:
+
+- `TANGLE_API_KEY` — for the proposer's LLM calls (router.tangle.tools)
+- `TANGLE_SANDBOX_KEY` — for tangle-sandbox client
+
+## Quality contract
+
+- Every result records: hypothesis hash, sandbox image, scipy version,
+  harness version, RNG seed.
+- No silent fallbacks. Sandbox unreachable => raise. Validation errors
+  => raise.
+- Bootstrap CI: 95%, 1000 resamples, fixed seed.
+- Cohen's d: pooled-variance form.

package/registry/families/agent-research-harness-py/files/agent.json

@@ -0,0 +1,45 @@
+{
+  "name": "research-harness",
+  "description": "Python research harness — autonomous propose / screen / validate loop. Each hypothesis is materialized as an isolated tangle-sandbox run so treatments cannot leak into each other. Bootstrap CI + Cohen's d math from scipy.stats; verdicts mirror the TS sibling family.",
+  "version": "0.1.0",
+  "tags": [
+    "agent-runtime",
+    "research-harness",
+    "experimentation",
+    "tangle-sandbox",
+    "scipy",
+    "python"
+  ],
+  "prompt": {
+    "systemPromptFile": "AGENTS.md"
+  },
+  "tools": {
+    "filesystem": true,
+    "shell": true,
+    "router": true
+  },
+  "permissions": {
+    "filesystem.write": "allow",
+    "shell.exec": "ask",
+    "network": "ask"
+  },
+  "resources": {
+    "files": [
+      {
+        "source": "src/"
+      },
+      {
+        "source": "hypotheses/"
+      },
+      {
+        "source": "pyproject.toml"
+      },
+      {
+        "source": "requirements.txt"
+      },
+      {
+        "source": "README.md"
+      }
+    ]
+  }
+}

package/registry/families/agent-research-harness-py/files/hypotheses/README.md

@@ -0,0 +1,45 @@
+# Hypothesis schema
+
+The same JSON shape is consumed by both `agent-research-harness-py`
+and `agent-research-harness-ts`. Keep them in sync.
+
+## Required fields
+
+| field         | type                       | rules                                      |
+| ------------- | -------------------------- | ------------------------------------------ |
+| `id`          | string                     | kebab-case, unique across the queue        |
+| `description` | string                     | >= 20 chars                                |
+| `sandboxImage`| string                     | tangle-sandbox image (default `ubuntu`)    |
+| `treatment`   | `{description, files, env}`| files = `{path: content}`                  |
+| `baseline`    | `{description, files, env}`| same shape as treatment                    |
+| `scenarios`   | array (>= 1)               | each: `{id, description, command[], direction?}` |
+| `metrics`     | array of unique strings    | numeric fields the harness will extract    |
+| `tags`        | array of strings           | optional, free-form                        |
+
+## Scenario `command`
+
+`command` is an argv array. The harness joins it with spaces and
+runs it inside the sandbox via `box.exec(...)`. The command must
+print a single JSON line as its **last** stdout line — that line is
+parsed for `metrics[]`.
+
+Example last-line:
+
+```json
+{"throughput_qps": 1234, "p95_ms": 12.4}
+```
+
+## Direction
+
+- `higher-is-better` (default) — the screener / validator interpret a
+  positive `treatment - baseline` delta as desirable.
+- `lower-is-better` — applies to latency, error-rate, cost. The
+  hypothesis writer must invert the metric upstream (multiply by -1)
+  if they want the same verdict ladder. The harness deliberately
+  doesn't auto-invert — explicit > clever.
+
+## Worked example
+
+See `example.json`. Drop new hypotheses into `queue.json` (a top-level
+JSON array). The CLI's `propose` subcommand will append generated
+variants for you.

package/registry/families/agent-research-harness-py/files/hypotheses/example.json

@@ -0,0 +1,29 @@
+{
+  "id": "example-batch-size-tuning",
+  "description": "Example hypothesis: doubling the inference batch size improves throughput without regressing latency p95.",
+  "sandboxImage": "ubuntu",
+  "treatment": {
+    "description": "Set BATCH_SIZE=64 in the runtime config.",
+    "files": {
+      "config/runtime.env": "BATCH_SIZE=64\n"
+    },
+    "env": {}
+  },
+  "baseline": {
+    "description": "Default BATCH_SIZE=32.",
+    "files": {
+      "config/runtime.env": "BATCH_SIZE=32\n"
+    },
+    "env": {}
+  },
+  "scenarios": [
+    {
+      "id": "throughput-scenario",
+      "description": "Run the throughput benchmark and emit a JSON line with throughput_qps + p95_ms.",
+      "command": ["bash", "-c", "scripts/run-throughput-bench.sh --json"],
+      "direction": "higher-is-better"
+    }
+  ],
+  "metrics": ["throughput_qps", "p95_ms"],
+  "tags": ["performance", "inference"]
+}

package/registry/families/agent-research-harness-py/files/hypotheses/queue.json

@@ -0,0 +1 @@
+[]

package/registry/families/agent-research-harness-py/files/pyproject.toml

@@ -0,0 +1,33 @@
+[project]
+name = "agent-research-harness"
+version = "0.1.0"
+description = "Python research harness — propose/screen/validate hypotheses with tangle-sandbox isolation and scipy.stats verdicts."
+requires-python = ">=3.11"
+dependencies = [
+    "tangle-sandbox>=0.1",
+    "httpx>=0.27",
+    "pydantic>=2.6",
+    "numpy>=1.26",
+    "scipy>=1.12",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "pytest-asyncio>=0.23",
+]
+
+[project.scripts]
+research = "research.cli:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/research"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]
+addopts = "-q"

package/registry/families/agent-research-harness-py/files/requirements.txt

@@ -0,0 +1,6 @@
+tangle-sandbox>=0.1,<0.2
+httpx>=0.27,<0.29
+pydantic>=2.6,<3
+numpy>=1.26,<3
+scipy>=1.12,<2
+pytest>=8.0,<9

package/registry/families/agent-research-harness-py/files/src/research/__init__.py

@@ -0,0 +1,37 @@
+"""Python research harness — propose / screen / validate / sweep.
+
+Public surface:
+    Hypothesis              - Pydantic v2 model for a single hypothesis
+    HypothesisResult        - per-rep run result
+    ScreeningRanking        - screener output
+    ValidationVerdict       - validator output (mean delta, CI, Cohen's d)
+    cohens_d                - pooled-variance Cohen's d
+    bootstrap_mean_diff_ci  - 95% bootstrap CI on mean(treatment) - mean(baseline)
+    run_screen              - rank hypotheses by 1-rep mean delta
+    run_validate            - 5-rep validation with bootstrap CI + Cohen's d
+"""
+
+from .screener import ScreeningRanking, run_screen
+from .types import (
+    Hypothesis,
+    HypothesisResult,
+    ProposerConfig,
+    Scenario,
+    ValidationVerdict,
+)
+from .validator import bootstrap_mean_diff_ci, cohens_d, run_validate
+
+__all__ = [
+    "Hypothesis",
+    "HypothesisResult",
+    "ProposerConfig",
+    "Scenario",
+    "ScreeningRanking",
+    "ValidationVerdict",
+    "bootstrap_mean_diff_ci",
+    "cohens_d",
+    "run_screen",
+    "run_validate",
+]
+
+__version__ = "0.1.0"

package/registry/families/agent-research-harness-py/files/src/research/__main__.py

@@ -0,0 +1,6 @@
+"""`python -m research` dispatcher."""
+
+from .cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/registry/families/agent-research-harness-py/files/src/research/cli.py

@@ -0,0 +1,146 @@
+"""argparse-based CLI for the research-harness."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from . import __version__
+from .proposer import propose
+from .runner import (
+    DEFAULT_QUEUE_PATH,
+    DEFAULT_RESULTS_DIR,
+    DEFAULT_TOP_K,
+    DEFAULT_VALIDATE_REPS,
+    load_queue,
+    save_queue,
+    screen,
+    sweep,
+    validate,
+)
+from .types import ProposerConfig
+
+
+def _add_common(p: argparse.ArgumentParser) -> None:
+    p.add_argument(
+        "--queue",
+        type=Path,
+        default=DEFAULT_QUEUE_PATH,
+        help="path to hypotheses/queue.json",
+    )
+    p.add_argument(
+        "--results-dir",
+        type=Path,
+        default=DEFAULT_RESULTS_DIR,
+        help="directory for emitted result JSON",
+    )
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="research",
+        description="Python research harness — propose / screen / validate / sweep",
+    )
+    p.add_argument("--version", action="version", version=f"research {__version__}")
+    sub = p.add_subparsers(dest="cmd", required=True)
+
+    p_propose = sub.add_parser("propose", help="generate new hypothesis variants")
+    _add_common(p_propose)
+    p_propose.add_argument(
+        "-n", "--n-proposals", type=int, default=3, help="number of variants to draft"
+    )
+    p_propose.add_argument(
+        "--model",
+        default="anthropic/claude-sonnet-4",
+        help="model id on router.tangle.tools",
+    )
+    p_propose.add_argument(
+        "--temperature", type=float, default=0.7, help="sampling temperature"
+    )
+
+    p_screen = sub.add_parser("screen", help="rank queued hypotheses (1 rep each)")
+    _add_common(p_screen)
+
+    p_validate = sub.add_parser("validate", help="multi-rep validation of winners")
+    _add_common(p_validate)
+    p_validate.add_argument(
+        "--reps",
+        type=int,
+        default=DEFAULT_VALIDATE_REPS,
+        help="reps per winning hypothesis",
+    )
+    p_validate.add_argument(
+        "--top-k",
+        type=int,
+        default=DEFAULT_TOP_K,
+        help="how many screening winners to validate",
+    )
+
+    p_sweep = sub.add_parser("sweep", help="screen then validate end-to-end")
+    _add_common(p_sweep)
+    p_sweep.add_argument("--reps", type=int, default=DEFAULT_VALIDATE_REPS)
+    p_sweep.add_argument("--top-k", type=int, default=DEFAULT_TOP_K)
+
+    return p
+
+
+def _cmd_propose(args: argparse.Namespace) -> int:
+    queue = load_queue(args.queue)
+    cfg = ProposerConfig(
+        n_proposals=args.n_proposals,
+        model=args.model,
+        temperature=args.temperature,
+    )
+    new, errors = propose(queue, recent_verdicts=[], config=cfg)
+    if errors:
+        for e in errors:
+            print(f"[propose] rejected: {e}", file=sys.stderr)
+    if not new:
+        print("[propose] no valid proposals returned", file=sys.stderr)
+        return 1
+    save_queue(queue + new, args.queue)
+    print(json.dumps({"appended": [h.id for h in new], "rejected": len(errors)}))
+    return 0
+
+
+def _cmd_screen(args: argparse.Namespace) -> int:
+    ranking = screen(args.queue, args.results_dir)
+    print(json.dumps(ranking.model_dump(), indent=2))
+    return 0
+
+
+def _cmd_validate(args: argparse.Namespace) -> int:
+    verdicts = validate(
+        args.queue, args.results_dir, n_reps=args.reps, top_k=args.top_k
+    )
+    print(json.dumps([v.model_dump() for v in verdicts], indent=2))
+    return 0 if verdicts else 1
+
+
+def _cmd_sweep(args: argparse.Namespace) -> int:
+    verdicts = sweep(
+        args.queue, args.results_dir, top_k=args.top_k, n_reps=args.reps
+    )
+    print(json.dumps([v.model_dump() for v in verdicts], indent=2))
+    return 0 if verdicts else 1
+
+
+_DISPATCH = {
+    "propose": _cmd_propose,
+    "screen": _cmd_screen,
+    "validate": _cmd_validate,
+    "sweep": _cmd_sweep,
+}
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+    handler = _DISPATCH[args.cmd]
+    return handler(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/registry/families/agent-research-harness-py/files/src/research/proposer.py

@@ -0,0 +1,140 @@
+"""Proposer — LLM-driven hypothesis variant generator.
+
+Given the current queue + recent results, asks an OpenAI-compatible
+model (router.tangle.tools) to propose N new hypothesis variants.
+
+The proposer is intentionally minimal: it builds a strict prompt,
+expects a JSON-array response, validates each entry against the
+`Hypothesis` schema, and appends valid entries to the queue. Invalid
+entries are discarded with a structured error — never silently
+dropped without surfacing why.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from typing import Any
+
+import httpx
+from pydantic import ValidationError
+
+from .types import Hypothesis, ProposerConfig
+
+PROPOSER_SYSTEM_PROMPT = """You are the proposer in a research harness.
+Given the current hypothesis queue and recent validation verdicts,
+propose new hypothesis variants that explore the design space the
+existing queue has not yet covered.
+
+Output rules:
+- Reply with a single JSON array of hypothesis objects. Nothing else.
+- Each hypothesis object must include: id, description (>= 20 chars),
+  treatment {description, files, env}, baseline {description, files,
+  env}, scenarios (>=1, each with id/description/command), metrics
+  (>=1, unique).
+- ids must be kebab-case and unique across the queue.
+- Do not repeat any id already in the queue.
+"""
+
+
+def _client(api_key: str, base_url: str) -> httpx.Client:
+    return httpx.Client(
+        base_url=base_url,
+        headers={
+            "Authorization": f"Bearer {api_key}",
+            "Content-Type": "application/json",
+        },
+        timeout=httpx.Timeout(60.0, connect=10.0),
+    )
+
+
+def _build_user_prompt(
+    queue: list[Hypothesis],
+    recent_verdicts: list[dict[str, Any]],
+    n_proposals: int,
+) -> str:
+    return json.dumps(
+        {
+            "n_proposals": n_proposals,
+            "queue": [h.model_dump(by_alias=True) for h in queue],
+            "recent_verdicts": recent_verdicts,
+            "instruction": (
+                f"Propose {n_proposals} new hypotheses. Avoid duplicates "
+                "of any id in the queue."
+            ),
+        },
+        indent=2,
+    )
+
+
+def propose(
+    queue: list[Hypothesis],
+    recent_verdicts: list[dict[str, Any]] | None = None,
+    config: ProposerConfig | None = None,
+    *,
+    api_key: str | None = None,
+) -> tuple[list[Hypothesis], list[str]]:
+    """Generate `config.n_proposals` new hypothesis variants.
+
+    Returns a tuple `(valid, errors)`:
+
+    - `valid`  — `Hypothesis` instances that parsed cleanly and don't
+      collide with existing queue ids.
+    - `errors` — string descriptions of every entry that was rejected
+      (so the operator sees what was discarded and why).
+
+    The function does NOT mutate the queue. The caller decides whether
+    to persist the new entries.
+    """
+    cfg = config or ProposerConfig()
+    key = api_key or os.environ.get("TANGLE_API_KEY", "")
+    if not key:
+        raise RuntimeError("TANGLE_API_KEY is required for propose()")
+
+    body = {
+        "model": cfg.model,
+        "temperature": cfg.temperature,
+        "max_tokens": cfg.max_tokens,
+        "messages": [
+            {"role": "system", "content": PROPOSER_SYSTEM_PROMPT},
+            {
+                "role": "user",
+                "content": _build_user_prompt(
+                    queue, recent_verdicts or [], cfg.n_proposals
+                ),
+            },
+        ],
+    }
+
+    with _client(key, cfg.base_url) as client:
+        resp = client.post("/chat/completions", json=body)
+        resp.raise_for_status()
+        payload = resp.json()
+
+    text = payload["choices"][0]["message"]["content"].strip()
+    if text.startswith("```"):
+        # Strip a fenced code block if the model wrapped its reply.
+        lines = text.splitlines()
+        text = "\n".join(lines[1:-1] if lines[-1].startswith("```") else lines[1:])
+
+    try:
+        raw = json.loads(text)
+    except json.JSONDecodeError as e:
+        raise ValueError(f"proposer returned non-JSON content: {e}") from e
+    if not isinstance(raw, list):
+        raise ValueError(f"proposer returned non-array root: {type(raw).__name__}")
+
+    existing_ids = {h.id for h in queue}
+    valid: list[Hypothesis] = []
+    errors: list[str] = []
+    for i, entry in enumerate(raw):
+        try:
+            h = Hypothesis.model_validate(entry)
+        except ValidationError as e:
+            errors.append(f"entry[{i}]: {e}")
+            continue
+        if h.id in existing_ids or h.id in {v.id for v in valid}:
+            errors.append(f"entry[{i}]: duplicate id '{h.id}'")
+            continue
+        valid.append(h)
+    return valid, errors