benchcraft-agent 0.1.0__py3-none-any.whl

benchcraft_agent-0.1.0.dist-info/METADATA

@@ -0,0 +1,184 @@
+Metadata-Version: 2.4
+Name: benchcraft-agent
+Version: 0.1.0
+Summary: Benchcraft Agent: a bring-your-own-agent AgentAdapter pattern -- a rule-based reference agent executes file-manipulation tool-use tasks through the shared lazycore sandbox executor, scored for success/latency and reported via OTel GenAI telemetry.
+Author: Benchcraft
+License: MIT
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.11
+Requires-Dist: benchcraft-core
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# benchcraft-agent
+
+LazyAgent's signature capability at this scaffold's depth (architecture
+doc `Benchcraft_Unified_Architecture.md`, Part 3, "Module 8: LazyAgent"): a
+minimal, real, **bring-your-own-agent task-execution benchmark loop**. A
+plain Python callable -- standing in for a real framework-agnostic agent,
+per MASEval's `AgentAdapter` interface (smolagents/LangGraph/AutoGen/CAMEL,
+per the module survey) -- executes a small file-manipulation tool-use task
+*inside the shared `lazycore.sandbox` executor*, and the run is scored for
+success/failure with basic accuracy/latency metrics reported via
+`lazycore.telemetry`'s OTel GenAI helpers.
+
+## Scope
+
+**In scope for this pass:**
+
+- One `AgentAdapter` interface (`benchcraft_lazyagent.adapter`) with one
+  concrete implementation, `SandboxedAgentAdapter`, that always executes
+  the agent's chosen action through a caller-supplied
+  `lazycore.sandbox.BaseSandboxExecutor`.
+- One concrete task family (`benchcraft_lazyagent.tasks`): a
+  file-manipulation tool-use task -- "create a file named X with content Y
+  in the sandboxed working directory" -- with two fixed variants: a
+  pass-designed task (write target inside the allowed sandbox path) and a
+  fail-designed sandbox-escape-attempt task (write target deliberately
+  outside the allowed sandbox path).
+- One reference agent callable, `rule_based_agent`: a plain, deterministic
+  Python function (not an LLM, not a real framework integration) that
+  reads a task's structured fields and proposes a shell command. It exists
+  to exercise the loop end-to-end, not to be a capable agent.
+- One tiny multi-task benchmark runner (`benchcraft_lazyagent.benchmark`)
+  that runs a small, fixed task suite and reports an aggregate pass rate +
+  mean wall-clock latency (measured via `time.perf_counter()`).
+- OTel GenAI telemetry via `lazycore.telemetry` (`genai_span`,
+  `set_ml_metric` for `ml.metric.accuracy`, `add_transcript_event` for each
+  trajectory step) -- no parallel telemetry/reporting schema is built here.
+
+**Explicitly deferred (not in this pass), and why:**
+
+| Deferred | Why |
+|---|---|
+| Real agent framework integrations (smolagents, LangGraph, AutoGen, CAMEL) | "Bring your own agent" at this scaffold depth means the core loop accepts *any* Python callable matching `AgentFn`'s signature -- wiring in a specific framework's decision function is real future integration work, not core-loop plumbing. Per §2.8, this platform explicitly does not build a router/registry of supported frameworks. |
+| Multi-Objective Pareto RAG Optimization loop (accuracy vs. latency vs. cost) | This is LazyAgent's headline capability in Part 3, but it's a full optimization loop over a RAG pipeline's chunking/indexing/reranking/model-choice search space -- a substantially larger scope than "prove the sandboxed benchmark-eval loop works end-to-end," which is this pass's goal. |
+| DISCO-style sample condensation (~1% informative task subset) | A data-selection technique for the Pareto RAG optimization loop above; deferred along with it. |
+| SWE-bench-style heavyweight task suites | Per the architecture doc's own v1 rescope note for this module: "given ARM64/Apple-Silicon friction with Docker-based SWE-bench-style isolation, the initial benchmark suite should prioritize tasks compatible with the Mac-first sandbox strategy" -- this pass follows that guidance literally by using a Seatbelt-compatible file-manipulation task instead of a Docker-dependent suite. |
+| RAG pipeline tuning (chunking/indexing/reranking search space, reranker-latency tradeoffs, prod-vector-DB disconnects -- Appendix A) | Informational/deferred per the task brief; not implemented in this pass. |
+| Cloud/remote agent targets | Locked out of v1 scope platform-wide for LazyAgent (architecture doc Part 4, Part 6). |
+
+## Sandbox wiring
+
+This package **reuses `lazycore.sandbox` for all containment** -- it does
+not build a second sandbox mechanism (per CLAUDE.md's "fix what's there /
+no duplication" rule and architecture doc §2.3). `SandboxedAgentAdapter`
+always calls `executor.run_command(...)` on a caller-supplied
+`lazycore.sandbox.BaseSandboxExecutor` (typically
+`lazycore.sandbox.get_default_executor()`, which resolves to the real
+`SeatbeltSandboxExecutor` on macOS).
+
+LazyAgent layers its **own** mode-specific `SandboxPolicy` values on top of
+that shared executor, per architecture doc §2.3's description of this
+module's sandboxing research as "the platform's most rigorous":
+
+- **`allow_network=False`** (default-deny egress) on every task policy --
+  this module's benchmark tasks have no legitimate reason to reach the
+  network, and per CLAUDE.md's "local-only, v1" constraint, no network
+  calls belong in the core path.
+- **`allowed_write_paths` scoped to a single per-task temp workspace
+  directory** -- never the whole filesystem, never a shared directory
+  across tasks. The pass-designed task's write target is inside this
+  directory; the fail-designed task's write target is a sibling directory
+  deliberately *excluded* from `allowed_write_paths`, to prove containment
+  is real (see "Fail-designed task" below).
+- **`allowed_read_paths` left at its default (unrestricted read)** --
+  matching `lazycore.sandbox`'s own documented default behavior (the
+  write/network surfaces are the actual enforcement points for this task
+  family; there is no sensitive read-only data these tasks need to be
+  isolated from).
+- **`timeout_seconds` set** on every task policy, so a misbehaving agent
+  action can't hang the benchmark run indefinitely.
+
+Per architecture doc §2.3.1's split-trust architecture, this package never
+attempts to sandbox any GPU/Metal/MPS-bound process -- there is none in
+this scaffold; the only thing ever run inside the sandbox is the agent's
+proposed shell command.
+
+### Fail-designed task proves containment is real, not decorative
+
+`benchcraft_lazyagent.tasks.make_fail_task` builds a task whose target file
+lives in a `forbidden/` directory that is a sibling of (but not included
+in) the sandbox policy's `allowed_write_paths`. The rule-based reference
+agent still *attempts* the write (it doesn't know the sandbox will block
+it) -- the Seatbelt backend's default-deny write policy is what actually
+stops it. The test suite (`tests/test_tasks.py`) asserts on the real
+filesystem state after the run: not just that the task is scored `False`,
+but that the forbidden file (and even its parent directory, since `mkdir
+-p` on it is blocked too) was never created. This is the concrete
+demonstration that sandbox containment drives the scored benchmark
+outcome, not just decorates it.
+
+## Public API
+
+```python
+from benchcraft_lazyagent import (
+    # adapter.py
+    AgentAdapter, SandboxedAgentAdapter, AgentAction, AgentFn,
+    AgentTrajectory, TrajectoryStep, TaskSpec, TaskResult,
+    # tasks.py
+    FileTaskSpec, rule_based_agent, score_file_task,
+    make_pass_task, make_fail_task, default_task_suite,
+    # benchmark.py
+    BenchmarkReport, ScorerFn, run_task, run_benchmark,
+)
+```
+
+- `AgentFn = Callable[[TaskSpec], AgentAction]` is the bring-your-own-agent
+  seam: any Python callable with this signature can stand in for "the
+  agent" -- a real framework's step function could be adapted to match
+  this signature without touching `SandboxedAgentAdapter` or
+  `run_benchmark` at all.
+- `SandboxedAgentAdapter(agent_fn).run_task(task, executor)` runs one task
+  and returns an `AgentTrajectory` (a 3-step transcript: the task
+  description as a `"user"` turn, the agent's chosen action as an
+  `"assistant"` turn, and the sandbox's real `SandboxResult` as a `"tool"`
+  turn).
+- `run_benchmark(tasks, agent_fn)` runs a small task suite end-to-end and
+  returns a `BenchmarkReport` with per-task `TaskResult`s plus an aggregate
+  `pass_rate` and `mean_latency_seconds`.
+
+## Installation
+
+```bash
+pip install -e packages/lazycore
+pip install -e "packages/lazyagent[dev]"
+```
+
+`lazycore` is declared as a bare (unpinned) dependency in `pyproject.toml`,
+matching the convention already established by `packages/lazytune`,
+`packages/automl`, and `packages/lazyforecast`. It is a local sibling
+package, not published to PyPI, so it still must be installed (or
+otherwise made resolvable) first -- a plain `pip install -e
+"packages/lazyagent[dev]"` without `lazycore` already installed/resolvable
+will fail to resolve the dependency.
+
+This package's own dependency surface is **stdlib + lazycore only** -- no
+smolagents/langgraph/autogen/camel, per the "explicitly deferred" table
+above.
+
+## Running tests
+
+```bash
+pytest packages/lazyagent/tests
+```
+
+The suite exercises the **real** `SeatbeltSandboxExecutor` on macOS (this
+repo's reference platform) -- it is skipped, not mocked, on non-macOS
+hosts via `pytest.mark.skipif`. Tests assert on real filesystem state
+(files that should exist do; files that should have been blocked do not)
+and on a real, finite, non-zero mean latency computed from
+`time.perf_counter()` measurements.
+
+## Running the example
+
+```bash
+python packages/lazyagent/examples/agent_benchmark_example.py
+```
+
+Runs the two-task default suite (one pass-designed, one fail-designed) and
+prints each task's pass/fail status plus the aggregate pass rate and mean
+latency.

benchcraft_agent-0.1.0.dist-info/RECORD

@@ -0,0 +1,7 @@
+benchcraft_lazyagent/__init__.py,sha256=PITDkIJe_7QnX4WGTotn_lT34cmI7vSHiG_7u5wOAL4,1948
+benchcraft_lazyagent/adapter.py,sha256=nNf_M0_gKivUOSIqXaGiQ3H73mzwTZpNIGv0n2j-J0Y,10131
+benchcraft_lazyagent/benchmark.py,sha256=3JvZCu_rE5-_CTeNIXXxKLFZCs7hWM59otC_FrsfGUs,9473
+benchcraft_lazyagent/tasks.py,sha256=RYkr1tMksUqlhZ8Y8f4IhinDkwDn_rt5IAbADWFtx_g,10172
+benchcraft_agent-0.1.0.dist-info/METADATA,sha256=Wj_kmtPafsvt7rh9el7vtpRDWacpDurmfEsHdYbGhCg,9898
+benchcraft_agent-0.1.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
+benchcraft_agent-0.1.0.dist-info/RECORD,,

benchcraft_agent-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.31.0
+Root-Is-Purelib: true
+Tag: py3-none-any

benchcraft_lazyagent/__init__.py

@@ -0,0 +1,67 @@
+"""Benchcraft LazyAgent -- bring-your-own-agent task-execution benchmark loop.
+
+Implements exactly one signature capability from the architecture doc's
+"Module 8: LazyAgent" (see `Benchcraft_Unified_Architecture.md`, Part 3):
+a minimal, real, bring-your-own-agent task-execution benchmark loop where a
+plain Python callable (standing in for a real framework-agnostic agent, per
+MASEval's `AgentAdapter` interface) executes a small file-manipulation
+tool-use task inside the shared `lazycore.sandbox` executor, and the run is
+scored for success/failure with accuracy/latency metrics reported via
+`lazycore.telemetry`.
+
+See this package's README for full scope, the sandbox policy used, and
+everything explicitly deferred (real agent framework integrations, the
+Multi-Objective Pareto RAG Optimization loop, DISCO-style sample
+condensation, SWE-bench-style task suites).
+"""
+
+from __future__ import annotations
+
+from benchcraft_lazyagent.adapter import (
+    AgentAction,
+    AgentAdapter,
+    AgentFn,
+    AgentTrajectory,
+    SandboxedAgentAdapter,
+    TaskResult,
+    TaskSpec,
+    TrajectoryStep,
+)
+from benchcraft_lazyagent.benchmark import (
+    BenchmarkReport,
+    ScorerFn,
+    run_benchmark,
+    run_task,
+)
+from benchcraft_lazyagent.tasks import (
+    FileTaskSpec,
+    default_task_suite,
+    make_fail_task,
+    make_pass_task,
+    rule_based_agent,
+    score_file_task,
+)
+
+__all__ = [
+    # Alphabetically sorted (ruff RUF022) across adapter.py/benchmark.py/
+    # tasks.py -- see the import blocks above for which module each name
+    # actually comes from.
+    "AgentAction",
+    "AgentAdapter",
+    "AgentFn",
+    "AgentTrajectory",
+    "BenchmarkReport",
+    "FileTaskSpec",
+    "SandboxedAgentAdapter",
+    "ScorerFn",
+    "TaskResult",
+    "TaskSpec",
+    "TrajectoryStep",
+    "default_task_suite",
+    "make_fail_task",
+    "make_pass_task",
+    "rule_based_agent",
+    "run_benchmark",
+    "run_task",
+    "score_file_task",
+]

benchcraft_lazyagent/adapter.py

@@ -0,0 +1,244 @@
+"""Bring-your-own-agent ``AgentAdapter`` pattern (architecture doc Part 3, "Module 8: LazyAgent").
+
+The architecture doc's LazyAgent survey names MASEval's "framework-agnostic
+``AgentAdapter`` interface spanning smolagents/LangGraph/AutoGen/CAMEL" as
+the design this module should implement a Bring-Your-Own-Agent version of.
+Per §2.8 ("no LLM router, no multi-provider abstraction... each module
+takes a bare-minimum 'bring your own local model handle' approach"),
+"bring your own agent" here means concretely: :class:`AgentAdapter` accepts
+any Python callable matching the documented :data:`AgentFn` signature as
+"the agent" -- it is *not* a registry/router of supported agent frameworks.
+A caller could later plug in a real framework's decision function (e.g. a
+thin wrapper around a smolagents/LangGraph step) without changing this
+package's core loop at all.
+
+Nothing here duplicates the shared sandbox executor (`lazycore.sandbox`) --
+:class:`AgentAdapter` always executes the agent's chosen action *through*
+a caller-supplied `lazycore.sandbox.BaseSandboxExecutor` instance. This
+module only adds the LazyAgent-specific machinery layered on top: the task/
+trajectory/result data shapes, and the adapter that wires an arbitrary
+agent callable to the shared executor. Per §2.3, LazyAgent does not get its
+own executor class -- only its own mode-specific `SandboxPolicy` values
+(see `benchcraft_lazyagent.tasks`).
+"""
+
+from __future__ import annotations
+
+import abc
+import time
+from dataclasses import dataclass
+from typing import Callable, Sequence
+
+from lazycore.sandbox import BaseSandboxExecutor, SandboxPolicy, SandboxResult
+
+__all__ = [
+    "AgentAction",
+    "AgentAdapter",
+    "AgentFn",
+    "AgentTrajectory",
+    "SandboxedAgentAdapter",
+    "TaskResult",
+    "TaskSpec",
+    "TrajectoryStep",
+]
+
+
+@dataclass(frozen=True)
+class TaskSpec:
+    """A single benchmark task the agent is asked to complete.
+
+    Deliberately minimal at this scaffold's depth (see README "Scope") --
+    one concrete task family is implemented in ``benchcraft_lazyagent.tasks``
+    (file-manipulation tool-use), not a general task-description schema for
+    arbitrary tool-use suites (SWE-bench-style task suites are explicitly
+    out of scope for this pass).
+
+    Attributes:
+        name: Short, unique task identifier (used in reports/telemetry).
+        description: Natural-language task description handed to the agent
+            callable -- e.g. "create a file named out.txt with content
+            'hello' in the sandboxed working directory".
+        sandbox_policy: The :class:`~lazycore.sandbox.SandboxPolicy` this
+            task's action is executed under. LazyAgent's own mode-specific
+            policy values (default-deny egress, restrictive write paths)
+            are constructed by ``benchcraft_lazyagent.tasks``, not by
+            `lazycore` itself -- see that module and the README for why.
+        expect_success: Whether this task variant is *designed* to succeed
+            (``True``) or *designed* to fail via sandbox containment
+            (``False``) -- used only for documentation/test clarity; the
+            actual pass/fail determination always comes from ``scorer``.
+    """
+
+    name: str
+    description: str
+    sandbox_policy: SandboxPolicy
+    expect_success: bool = True
+
+
+@dataclass(frozen=True)
+class AgentAction:
+    """What the agent callable decided to do for a given :class:`TaskSpec`.
+
+    At this scaffold's depth the only supported action shape is a single
+    shell command (argv-style, matching
+    :meth:`~lazycore.sandbox.BaseSandboxExecutor.run_command`'s contract) --
+    a real framework-agnostic adapter would eventually need to support
+    multi-step tool-call sequences, but a single action per task is enough
+    to exercise the sandbox-wiring and scoring loop that is this pass's
+    signature capability.
+    """
+
+    command: Sequence[str]
+    rationale: str = ""
+
+
+#: The "bring your own agent" plug-in point. A caller supplies any Python
+#: callable matching this signature -- it receives the task and must return
+#: the action it wants executed. `benchcraft_lazyagent.tasks.rule_based_agent`
+#: is the one reference implementation provided by this package (a plain,
+#: deterministic, rule-based function -- not a real LLM-backed agent); a
+#: real framework's step function could be adapted to this exact signature
+#: without touching `AgentAdapter` itself.
+AgentFn = Callable[[TaskSpec], AgentAction]
+
+
+@dataclass(frozen=True)
+class TrajectoryStep:
+    """One recorded step of an agent's execution trajectory.
+
+    Mirrors the shape `lazycore.telemetry.add_transcript_event` expects
+    (a ``role`` + ``content`` pair), so trajectories reported by this
+    module reuse the exact same OTel transcript-event convention LazyRed
+    uses for its own conversational transcripts (architecture doc §2.6).
+    """
+
+    role: str
+    content: str
+
+
+@dataclass(frozen=True)
+class AgentTrajectory:
+    """Everything the agent "did" for one task: its steps plus the sandboxed result."""
+
+    task_name: str
+    steps: tuple[TrajectoryStep, ...]
+    sandbox_result: SandboxResult
+
+
+@dataclass(frozen=True)
+class TaskResult:
+    """The scored outcome of running one :class:`TaskSpec` through an adapter.
+
+    ``trajectory`` is ``None`` exactly when the task never produced one --
+    i.e. ``agent_fn``/``adapter.run_task`` raised before returning an
+    :class:`AgentTrajectory` at all. `benchcraft_lazyagent.benchmark.run_task`
+    catches such exceptions at a per-task boundary (so one task's crash
+    never aborts the whole benchmark run) and reports them here as a
+    failed result: ``success=False``, ``latency_seconds`` set to the
+    wall-clock time elapsed up to the point of failure, ``trajectory=None``,
+    and the exception's type/message captured in ``detail`` -- the same
+    field already used for a scorer's human-readable pass/fail explanation,
+    rather than inventing a second, parallel error-reporting field.
+    """
+
+    task_name: str
+    success: bool
+    latency_seconds: float
+    trajectory: AgentTrajectory | None
+    detail: str = ""
+
+
+class AgentAdapter(abc.ABC):
+    """Minimal bring-your-own-agent interface.
+
+    ``run_task`` is the one canonical entrypoint (per CLAUDE.md's "one
+    canonical adapter interface" rule) -- concrete subclasses decide *how*
+    the agent's chosen action actually gets executed. This pass provides
+    exactly one concrete implementation, :class:`SandboxedAgentAdapter`,
+    which always executes through the shared `lazycore.sandbox` executor.
+    """
+
+    @abc.abstractmethod
+    def run_task(self, task: TaskSpec, executor: BaseSandboxExecutor) -> AgentTrajectory:
+        """Run ``task`` via this adapter's agent, executing inside ``executor``.
+
+        Args:
+            task: the task to attempt.
+            executor: a `lazycore.sandbox.BaseSandboxExecutor` instance
+                (e.g. `lazycore.sandbox.get_default_executor()`) that the
+                adapter must use for any containment of the agent's chosen
+                action -- this package never builds a second sandbox
+                mechanism.
+        """
+        raise NotImplementedError
+
+
+class SandboxedAgentAdapter(AgentAdapter):
+    """The one concrete :class:`AgentAdapter`: wires an arbitrary agent
+    callable's chosen action through the shared sandbox executor.
+
+    Construction takes ``agent_fn`` (see :data:`AgentFn`) -- this is the
+    "bring your own agent" seam. This package supplies exactly one
+    reference ``agent_fn`` (``benchcraft_lazyagent.tasks.rule_based_agent``)
+    but any callable with the same signature works, including one that
+    wraps a real agent framework's decision step.
+    """
+
+    def __init__(self, agent_fn: AgentFn) -> None:
+        """Wrap ``agent_fn`` -- the "bring your own agent" callable to drive.
+
+        Args:
+            agent_fn: a callable matching :data:`AgentFn` (task in, action
+                out). Stored as-is; not validated or wrapped further, so any
+                exception it raises during :meth:`run_task` propagates
+                unchanged to the caller (`benchcraft_lazyagent.benchmark.run_task`
+                is what converts such exceptions into a failed result).
+        """
+        self._agent_fn = agent_fn
+
+    def run_task(self, task: TaskSpec, executor: BaseSandboxExecutor) -> AgentTrajectory:
+        """Ask ``agent_fn`` for one action, run it in ``executor``, and record the trajectory.
+
+        Builds a three-step :class:`AgentTrajectory` -- a ``user`` step with
+        the task description, an ``assistant`` step with the chosen action's
+        command/rationale, and a ``tool`` step with the real
+        `lazycore.sandbox.SandboxResult` (exit code, whether the policy
+        blocked it, stdout/stderr) -- regardless of whether the sandboxed
+        command actually succeeded. Scoring the outcome is the caller's
+        responsibility (see `benchcraft_lazyagent.tasks.score_file_task`),
+        not this method's.
+
+        Args:
+            task: the task to attempt.
+            executor: the sandbox executor to run the agent's chosen
+                command through, per :meth:`AgentAdapter.run_task`'s
+                contract.
+        """
+        steps: list[TrajectoryStep] = [
+            TrajectoryStep(role="user", content=task.description)
+        ]
+
+        action = self._agent_fn(task)
+        steps.append(
+            TrajectoryStep(
+                role="assistant",
+                content=f"command={list(action.command)!r} rationale={action.rationale!r}",
+            )
+        )
+
+        result = executor.run_command(action.command, policy=task.sandbox_policy)
+        steps.append(
+            TrajectoryStep(
+                role="tool",
+                content=(
+                    f"exit_code={result.exit_code} policy_blocked={result.policy_blocked} "
+                    f"stdout={result.stdout!r} stderr={result.stderr!r}"
+                ),
+            )
+        )
+
+        return AgentTrajectory(
+            task_name=task.name,
+            steps=tuple(steps),
+            sandbox_result=result,
+        )

benchcraft_lazyagent/benchmark.py

@@ -0,0 +1,217 @@
+"""A tiny multi-task benchmark runner (architecture doc Part 3, "Module 8: LazyAgent").
+
+This is a deliberately minimal stand-in for the full "leaderboard" style
+Multi-Objective Pareto RAG Optimization loop (accuracy vs. latency vs.
+cost) described in Part 3 -- that loop, along with DISCO-style sample
+condensation, is explicitly out of scope for this pass (see README). What
+*is* implemented here: run a small, fixed set of :class:`TaskSpec` variants
+through a :class:`~benchcraft_lazyagent.adapter.SandboxedAgentAdapter`,
+score each one, and aggregate a pass rate + mean wall-clock latency --
+enough to prove the sandboxed benchmark-eval loop end-to-end.
+
+Reports each run via `lazycore.telemetry`'s OTel GenAI helpers
+(`genai_span`, `set_ml_metric`, `add_transcript_event`) rather than a
+parallel telemetry/reporting schema, per architecture doc §2.6.
+
+**`add_transcript_event` call sites deliberately use the safe-by-default
+metadata-only path** (no ``include_raw_content=True``, no ``sanitizer``).
+This is a conscious choice, not an oversight: per "bring your own agent"
+(see `benchcraft_lazyagent.adapter`), ``agent_fn`` is an arbitrary
+caller-supplied callable, and its proposed command's real stdout/stderr
+(captured in the "tool" trajectory step) can contain whatever that
+command actually printed -- which, for a real agent (not just this
+package's synthetic `rule_based_agent` reference), could include secrets,
+credentials, or other sensitive tool output. Exporting that verbatim into
+an OTel span by default would be exactly the credential/PII-leak risk
+`lazycore.telemetry.add_transcript_event`'s safe-by-default contract
+exists to prevent. A caller who wants full transcript content in their own
+exported traces can wrap/re-emit these spans with ``include_raw_content=True``
+or a ``sanitizer`` at their own telemetry layer; this package does not
+opt in on their behalf.
+"""
+
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass
+from typing import Callable, Sequence
+
+from lazycore.sandbox import BaseSandboxExecutor, get_default_executor
+from lazycore.telemetry import add_transcript_event, genai_span, set_ml_metric
+
+from benchcraft_lazyagent.adapter import (
+    AgentFn,
+    AgentTrajectory,
+    SandboxedAgentAdapter,
+    TaskResult,
+    TaskSpec,
+)
+from benchcraft_lazyagent.tasks import score_file_task
+
+__all__ = ["BenchmarkReport", "ScorerFn", "run_benchmark", "run_task"]
+
+#: A scorer inspects the task and its recorded trajectory (which includes
+#: the real `lazycore.sandbox.SandboxResult`) and returns
+#: ``(success, human_readable_detail)``. `benchcraft_lazyagent.tasks.score_file_task`
+#: is the one reference scorer this package ships, matching the one task
+#: family it implements.
+ScorerFn = Callable[[TaskSpec, AgentTrajectory], tuple[bool, str]]
+
+
+@dataclass(frozen=True)
+class BenchmarkReport:
+    """Aggregate result of running a small task suite through the benchmark loop."""
+
+    results: tuple[TaskResult, ...]
+    pass_rate: float
+    mean_latency_seconds: float
+
+    @property
+    def task_count(self) -> int:
+        """Number of tasks included in this report (``len(self.results)``)."""
+        return len(self.results)
+
+
+def run_task(
+    task: TaskSpec,
+    *,
+    agent_fn: AgentFn,
+    executor: BaseSandboxExecutor,
+    scorer: ScorerFn = score_file_task,
+) -> TaskResult:
+    """Run one task through a fresh :class:`SandboxedAgentAdapter` and score it.
+
+    Latency is measured via ``time.perf_counter()`` around the adapter's
+    ``run_task`` call only (the actual agent-decide + sandboxed-execute
+    work), not around scoring/telemetry overhead.
+
+    This is the **one** per-task exception boundary for the whole benchmark
+    loop (`run_benchmark` deliberately does not add a second one around its
+    call to this function -- see that function's docstring). Any exception
+    raised by ``agent_fn`` (invoked indirectly via ``adapter.run_task``),
+    by ``adapter.run_task`` itself (e.g. a sandbox executor raising
+    `lazycore.sandbox.SandboxPolicyViolationError` for fail-fast policy
+    violations, or `lazycore.sandbox.SandboxBackendUnavailableError`), or
+    by ``scorer`` is caught here and converted into a failed
+    :class:`TaskResult` rather than propagating out and aborting every
+    other task in the suite. ``KeyboardInterrupt``/``SystemExit`` are
+    intentionally *not* caught (``except Exception``, not
+    ``except BaseException``) -- a user's Ctrl-C or an explicit process
+    exit should still stop the run.
+
+    On such a failure, ``latency_seconds`` records wall-clock time up to
+    the point of failure (not ``NaN``/``0``) so `run_benchmark`'s mean-
+    latency aggregation stays a meaningful number rather than needing
+    special-casing, and ``trajectory`` is ``None`` if the failure happened
+    before ``adapter.run_task`` returned one (e.g. ``agent_fn`` itself
+    raised), or the real trajectory if only ``scorer`` raised afterwards.
+    """
+    adapter = SandboxedAgentAdapter(agent_fn)
+
+    start = time.perf_counter()
+    trajectory: AgentTrajectory | None = None
+    try:
+        trajectory = adapter.run_task(task, executor)
+        latency_seconds = time.perf_counter() - start
+        success, detail = scorer(task, trajectory)
+    except Exception as exc:  # noqa: BLE001 - deliberate: one task's failure must not crash the suite.
+        latency_seconds = time.perf_counter() - start
+        success = False
+        detail = f"task raised {type(exc).__name__}: {exc}"
+
+        with genai_span(
+            f"lazyagent.task.{task.name}",
+            attributes={"lazyagent.task.name": task.name},
+        ) as span:
+            set_ml_metric(span, "accuracy", 0.0)
+            span.set_attribute("lazyagent.task.latency_seconds", latency_seconds)
+            span.set_attribute("lazyagent.task.detail", detail)
+            span.set_attribute("lazyagent.task.error", True)
+            if trajectory is not None:
+                for step in trajectory.steps:
+                    add_transcript_event(span, step.role, step.content)
+            add_transcript_event(span, "error", detail)
+
+        return TaskResult(
+            task_name=task.name,
+            success=False,
+            latency_seconds=latency_seconds,
+            trajectory=trajectory,
+            detail=detail,
+        )
+
+    with genai_span(
+        f"lazyagent.task.{task.name}",
+        attributes={"lazyagent.task.name": task.name},
+    ) as span:
+        set_ml_metric(span, "accuracy", 1.0 if success else 0.0)
+        span.set_attribute("lazyagent.task.latency_seconds", latency_seconds)
+        span.set_attribute("lazyagent.task.detail", detail)
+        for step in trajectory.steps:
+            add_transcript_event(span, step.role, step.content)
+
+    return TaskResult(
+        task_name=task.name,
+        success=success,
+        latency_seconds=latency_seconds,
+        trajectory=trajectory,
+        detail=detail,
+    )
+
+
+def run_benchmark(
+    tasks: Sequence[TaskSpec],
+    agent_fn: AgentFn,
+    *,
+    executor: BaseSandboxExecutor | None = None,
+    scorer: ScorerFn = score_file_task,
+) -> BenchmarkReport:
+    """Run ``tasks`` through ``agent_fn`` and report an aggregate pass rate + mean latency.
+
+    Args:
+        tasks: the small, fixed task suite to run (e.g.
+            `benchcraft_lazyagent.tasks.default_task_suite`).
+        agent_fn: the bring-your-own-agent callable (see
+            `benchcraft_lazyagent.adapter.AgentFn`).
+        executor: a `lazycore.sandbox.BaseSandboxExecutor`; defaults to
+            `lazycore.sandbox.get_default_executor()` (Seatbelt on macOS).
+        scorer: how to score each task's trajectory; defaults to
+            `benchcraft_lazyagent.tasks.score_file_task`, matching the one
+            task family this package implements.
+
+    A task whose ``agent_fn``/``adapter.run_task``/``scorer`` raises does
+    not abort the run: `run_task` is the one per-task exception boundary
+    (see its docstring) and converts such a failure into a failed
+    :class:`~benchcraft_lazyagent.adapter.TaskResult` before it ever
+    reaches this loop, so this function never needs a second try/except
+    around its call to `run_task`. That failed result's ``success=False``
+    is counted in ``pass_rate``'s denominator like any other failure, and
+    its ``latency_seconds`` (wall-clock time up to the point of failure)
+    is included in ``mean_latency_seconds`` like any other task's --
+    deliberately not excluded or treated as ``0``/``NaN``, so the
+    aggregate mean latency stays a single well-defined number regardless
+    of how many tasks in the suite raised.
+    """
+    if not tasks:
+        raise ValueError("run_benchmark requires at least one task")
+
+    active_executor = executor or get_default_executor()
+
+    results = tuple(
+        run_task(task, agent_fn=agent_fn, executor=active_executor, scorer=scorer)
+        for task in tasks
+    )
+
+    pass_rate = sum(1 for r in results if r.success) / len(results)
+    mean_latency_seconds = sum(r.latency_seconds for r in results) / len(results)
+
+    with genai_span("lazyagent.benchmark.run") as span:
+        set_ml_metric(span, "accuracy", pass_rate)
+        span.set_attribute("lazyagent.benchmark.task_count", len(results))
+        span.set_attribute("lazyagent.benchmark.mean_latency_seconds", mean_latency_seconds)
+
+    return BenchmarkReport(
+        results=results,
+        pass_rate=pass_rate,
+        mean_latency_seconds=mean_latency_seconds,
+    )

benchcraft_lazyagent/tasks.py

@@ -0,0 +1,237 @@
+"""The one concrete benchmark task family: file-manipulation tool-use.
+
+Per README "Scope", this pass implements exactly one task family -- create
+a file with expected content in a sandboxed working directory -- rather
+than a general task-description schema or a SWE-bench-style suite. Two
+task variants are provided:
+
+- :func:`make_pass_task` -- the agent's write target is inside the
+  sandbox's ``allowed_write_paths``. The rule-based reference agent
+  (:func:`rule_based_agent`) issues a shell command to create the file, the
+  shared `lazycore.sandbox` executor allows it, and the task scores as a
+  success.
+- :func:`make_fail_task` -- the agent (deliberately, to exercise
+  containment) attempts to write *outside* ``allowed_write_paths``. The
+  shared sandbox executor's default-deny write policy blocks this, the
+  file is never created, and the task scores as a failure -- proving the
+  sandbox's containment genuinely drives the scored outcome, not just
+  decoration around it.
+
+:func:`rule_based_agent` is the one reference "agent" callable this package
+ships: a plain, deterministic Python function (not an LLM, not a real
+framework) that reads a :class:`~benchcraft_lazyagent.adapter.FileTaskSpec`
+and decides on a shell command satisfying (or, for the fail variant,
+attempting to satisfy) the task description. Its signature matches
+:data:`~benchcraft_lazyagent.adapter.AgentFn` exactly, so a caller could
+swap in a real framework's decision function without changing
+`benchcraft_lazyagent.adapter` or `benchcraft_lazyagent.benchmark` at all.
+"""
+
+from __future__ import annotations
+
+import shlex
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path
+
+from lazycore.sandbox import SandboxPolicy
+
+from benchcraft_lazyagent.adapter import AgentAction, AgentTrajectory, TaskSpec
+
+__all__ = [
+    "FileTaskSpec",
+    "default_task_suite",
+    "make_fail_task",
+    "make_pass_task",
+    "rule_based_agent",
+    "score_file_task",
+]
+
+
+@dataclass(frozen=True)
+class FileTaskSpec(TaskSpec):
+    """A :class:`TaskSpec` for the file-manipulation task family.
+
+    Attributes:
+        target_path: Absolute path of the file the agent is asked to
+            create. For :func:`make_pass_task` this is inside the sandbox
+            policy's ``allowed_write_paths``; for :func:`make_fail_task` it
+            is deliberately outside that allowlist.
+        expected_content: Exact file content required for the task to
+            score as a success.
+    """
+
+    target_path: str = ""
+    expected_content: str = ""
+
+
+def rule_based_agent(task: TaskSpec) -> AgentAction:
+    """The one reference "bring your own agent" callable this package ships.
+
+    Deliberately simple and deterministic: it does not parse
+    ``task.description`` with any NLP -- it reads the structured fields off
+    a :class:`FileTaskSpec` directly and always proposes the same kind of
+    shell command (``mkdir -p`` the parent directory, then write the exact
+    expected content via ``printf``). This is intentional -- the point of
+    this scaffold is to exercise the sandbox-wiring and scoring loop, not
+    to build a capable agent. Whether the command actually succeeds is
+    entirely up to the sandbox policy the task executes under (see
+    ``benchcraft_lazyagent.tasks`` module docstring).
+
+    Raises:
+        TypeError: if ``task`` is not a :class:`FileTaskSpec` -- this
+            reference agent only knows how to handle this one task family.
+    """
+    if not isinstance(task, FileTaskSpec):
+        raise TypeError(
+            f"rule_based_agent only supports FileTaskSpec tasks, got {type(task).__name__}"
+        )
+
+    quoted_content = shlex.quote(task.expected_content)
+    quoted_path = shlex.quote(task.target_path)
+    parent_dir = shlex.quote(str(Path(task.target_path).parent))
+    shell_script = f"mkdir -p {parent_dir} && printf '%s' {quoted_content} > {quoted_path}"
+
+    return AgentAction(
+        command=["/bin/sh", "-c", shell_script],
+        rationale=f"write expected content to {task.target_path!r} via a shell one-liner",
+    )
+
+
+def score_file_task(task: TaskSpec, trajectory: AgentTrajectory) -> tuple[bool, str]:
+    """Score a :class:`FileTaskSpec` run by checking the real filesystem.
+
+    Success requires the target file to actually exist *and* contain
+    exactly ``task.expected_content`` -- deliberately re-checking the real
+    filesystem rather than trusting the sandbox's reported exit code, so
+    that a sandbox-blocked write (nonzero exit, or a zero exit that still
+    didn't actually create the file) is scored as a failure regardless of
+    what the agent or shell reported.
+    """
+    if not isinstance(task, FileTaskSpec):
+        raise TypeError(f"score_file_task only supports FileTaskSpec, got {type(task).__name__}")
+
+    path = Path(task.target_path)
+    if not path.is_file():
+        return False, f"expected file {task.target_path!r} does not exist"
+
+    actual_content = path.read_text(encoding="utf-8")
+    if actual_content != task.expected_content:
+        return False, (
+            f"file {task.target_path!r} exists but content mismatch: "
+            f"expected {task.expected_content!r}, got {actual_content!r}"
+        )
+
+    return True, f"file {task.target_path!r} created with expected content"
+
+
+def _fresh_task_root(base_dir: Path, name: str) -> Path:
+    """Allocate a genuinely fresh, unique task-scoped directory under ``base_dir``.
+
+    Regression fix (CodeRabbit): the previous implementation derived the
+    task root deterministically as ``base_dir / name``. Re-invoking a
+    factory (:func:`make_pass_task`/:func:`make_fail_task`) with the same
+    ``base_dir``/``name`` pair -- e.g. re-running a benchmark suite against
+    a persistent workspace directory instead of a brand-new
+    `tempfile.TemporaryDirectory` each time -- would silently reuse
+    whatever directory (and any stale files inside it) a prior run left
+    behind. Since :func:`score_file_task` scores by trusting real
+    filesystem state, a stale target file from a previous run could make a
+    task falsely "pass" without this run's agent ever having created it,
+    and could also mask a real containment regression in the fail-designed
+    task (a stale ``forbidden/`` directory from an old, unpatched sandbox
+    run could linger even after containment is fixed).
+
+    ``tempfile.mkdtemp`` guarantees a new, unique directory on every call
+    (even for repeated ``base_dir``/``name`` pairs), so each task instance
+    always starts from a genuinely empty workspace.
+    """
+    base_dir = Path(base_dir)
+    base_dir.mkdir(parents=True, exist_ok=True)
+    return Path(tempfile.mkdtemp(prefix=f"{name}-", dir=str(base_dir)))
+
+
+def make_pass_task(base_dir: Path, *, name: str = "create_file_pass") -> FileTaskSpec:
+    """A task the sandbox should allow: write inside ``allowed_write_paths``.
+
+    Args:
+        base_dir: A directory this task family creates its own
+            task-scoped subdirectory under (the caller typically passes a
+            fresh `tempfile.TemporaryDirectory` path). Each call gets a
+            fresh, unique subdirectory (see :func:`_fresh_task_root`) even
+            if ``base_dir``/``name`` are reused across calls.
+    """
+    task_root = _fresh_task_root(base_dir, name)
+    workspace = task_root / "workspace"
+    workspace.mkdir(parents=True, exist_ok=True)
+
+    target_path = workspace / "hello.txt"
+    expected_content = "hello from benchcraft lazyagent"
+
+    policy = SandboxPolicy(
+        allow_network=False,
+        allowed_write_paths=(str(workspace),),
+        working_directory=str(workspace),
+        timeout_seconds=15.0,
+    )
+
+    return FileTaskSpec(
+        name=name,
+        description=(
+            f"Create a file named {target_path.name!r} with content "
+            f"{expected_content!r} in the sandboxed working directory."
+        ),
+        sandbox_policy=policy,
+        expect_success=True,
+        target_path=str(target_path),
+        expected_content=expected_content,
+    )
+
+
+def make_fail_task(base_dir: Path, *, name: str = "create_file_escape_fail") -> FileTaskSpec:
+    """A task designed to fail: the target path is outside ``allowed_write_paths``.
+
+    This exists specifically to prove sandbox containment is exercised and
+    genuinely affects the scored outcome (README acceptance criteria): the
+    rule-based agent attempts to write to ``forbidden/escape.txt``, a
+    sibling directory of the sandbox's allowed workspace that is *not*
+    included in ``allowed_write_paths``. The shared sandbox executor's
+    default-deny write policy should block the write, the file should
+    never be created, and :func:`score_file_task` should therefore score
+    this task as a failure. Like :func:`make_pass_task`, each call gets a
+    fresh, unique task root (see :func:`_fresh_task_root`) so a stale
+    ``forbidden/`` directory from a previous run can never linger into
+    this run's containment check.
+    """
+    task_root = _fresh_task_root(base_dir, name)
+    workspace = task_root / "workspace"
+    workspace.mkdir(parents=True, exist_ok=True)
+    forbidden_dir = task_root / "forbidden"  # deliberately NOT in allowed_write_paths
+
+    target_path = forbidden_dir / "escape.txt"
+    expected_content = "this should never be written"
+
+    policy = SandboxPolicy(
+        allow_network=False,
+        allowed_write_paths=(str(workspace),),  # forbidden_dir is intentionally excluded
+        working_directory=str(workspace),
+        timeout_seconds=15.0,
+    )
+
+    return FileTaskSpec(
+        name=name,
+        description=(
+            f"Create a file at {str(target_path)!r} with content "
+            f"{expected_content!r} -- outside the sandboxed working "
+            "directory (a sandbox-escape attempt used to prove containment)."
+        ),
+        sandbox_policy=policy,
+        expect_success=False,
+        target_path=str(target_path),
+        expected_content=expected_content,
+    )
+
+
+def default_task_suite(base_dir: Path) -> list[FileTaskSpec]:
+    """The small, fixed set of task variants the benchmark runner exercises by default."""
+    return [make_pass_task(base_dir), make_fail_task(base_dir)]