dsh-eval-guard 0.7.2 → 0.7.3

package/assets/neural-pendulum/TASK.md

@@ -8,9 +8,9 @@
 
 - 训练时从随机小扰动、评测时从固定初始状态起摆
 - 保持三根杆全部向上至少 t_tgt = 10.0 秒
-- 同时保证神经网络基础设施通过固定头文件契约并通过隐藏测试
+- 同时保证神经网络基础设施通过 build/nn_task 固定 CLI 的数值评测
 
-开发阶段可以使用 Python/PyTorch 做离线数值对照和实验；最终提交的基础设施和控制器源码必须是纯 C++，不得出现 torch，评测环境不提供 Python/torch。
+开发阶段可以使用 Python/PyTorch 做离线数值对照和实验；最终提交的基础设施和控制器源码必须是纯 C++，不得出现 torch，候选运行时不提供 Python/torch。
 
 ## 环境
 
@@ -100,7 +100,7 @@ t_can = 三根杆同时处于全向上容差内的最长连续时间
 
 ~~~
 t_can >= t_tgt
-且基础设施测试全部通过
+且神经网络基础设施数值评测全部通过
 ~~~
 
 任务完成，计时停止。
@@ -152,34 +152,140 @@ cmake --build build -j4
 必须产出：
 
 ~~~
-build/libnn.a
+build/nn_task
 build/control
 ~~~
 
-### 1. 基础设施入口
+工作区模板 CMakeLists.txt 默认从 `src/core/*.cpp` 构建共享静态库，从 `src/nn_task.cpp` 和 `src/control.cpp` 构建两个可执行文件。可以按需调整目录结构和 CMakeLists.txt，但必须从零构建出上面两个二进制。
 
-基础设施实现放在 `src/nn/*.cpp`，由 CMake 构建为 `build/libnn.a`；公开头文件 `include/nn/nn.hpp` 是固定 API 契约，不得修改声明或语义。
+### 1. 神经网络基础设施入口 build/nn_task
 
-评测侧会独立编译隐藏测试并链接 `build/libnn.a`。
+基础设施通过固定 CLI 暴露，评测会逐操作调用。所有输入/输出文件都是连续 row-major float32，所有数值输出也必须写为 float32。
 
-必须包含的基础设施：
+#### 逐元素
 
-- fp32 tensor，形状检查与广播
-- elementwise add/sub/mul/div
-- matmul / linear
-- relu / softmax
-- layernorm
-- conv2d，支持 stride/padding/dilation/groups
-- MSE
-- matmul/linear/relu/layernorm/conv2d/MSE 反向
-- SGD momentum
-- AdamW
-- 模型保存/加载
-- 线程池
+~~~bash
+build/nn_task elementwise --op add|sub|mul|div --shape 64,256 --a A --b B --out O
+~~~
+
+支持广播：从尾部对齐，维度为 1 可广播。
+
+#### 矩阵乘法
+
+~~~bash
+build/nn_task matmul --a-shape M,K --b-shape K,N --a A --b B --out O
+~~~
+
+只要求 2D 输入。
+
+#### Linear
+
+~~~bash
+build/nn_task linear --x-shape B,IN --w-shape OUT,IN --b-shape OUT --x X --w W --bias B --out O
+~~~
+
+计算 `O = X @ W^T + bias`。
+
+#### 激活
+
+~~~bash
+build/nn_task relu --shape N --x X --out O
+build/nn_task softmax --shape B,C --x X --out O
+~~~
+
+Softmax 沿最后一维。
+
+#### LayerNorm
+
+~~~bash
+build/nn_task layernorm --shape B,C --x X --gamma G --beta B --eps 1e-5 --out O
+~~~
+
+对最后一维归一化；gamma/beta 形状为 `C`。
+
+#### MSE
+
+~~~bash
+build/nn_task mse --pred-shape S --target-shape S --pred P --target T --out O
+~~~
+
+输出 1 个 float32，为所有元素平均平方误差。
+
+#### Conv2d
+
+~~~bash
+build/nn_task conv2d --x-shape N,C,H,W --w-shape O,Cg,KH,KW --b-shape O --x X --w W --bias B --stride SH,SW --padding PH,PW --dilation DH,DW --groups G --out OUT
+~~~
+
+标准 NCHW 卷积，`Cg = C / groups`。
+
+#### 反向传播
+
+~~~bash
+build/nn_task matmul-backward --a-shape M,K --b-shape K,N --a A --b B --grad G --da DA --db DB
+build/nn_task linear-backward --x-shape B,IN --w-shape OUT,IN --b-shape OUT --x X --w W --grad G --dx DX --dw DW --db DB
+build/nn_task relu-backward --shape N --x X --grad G --dx DX
+build/nn_task layernorm-backward --shape B,C --x X --gamma G --grad G --eps 1e-5 --dx DX --dgamma DG
+build/nn_task mse-backward --pred-shape S --target-shape S --pred P --target T --grad G --dp DP --dtarget DT
+build/nn_task conv2d-backward --x-shape N,C,H,W --w-shape O,Cg,KH,KW --b-shape O --x X --w W --grad G --stride SH,SW --padding PH,PW --dilation DH,DW --groups G --dx DX --dw DW --db DB
+~~~
+
+反向传播定义为：上游梯度 `G` 对输入、权重、bias 的梯度；`G` 与 forward 输出同形状。
+
+#### 优化器
+
+~~~bash
+build/nn_task adamw --param P --grad G --m M --v V --lr LR --beta1 B1 --beta2 B2 --eps EPS --weight-decay WD --step T --out-param P2 --out-m M2 --out-v V2
+build/nn_task sgd-momentum --param P --grad G --momentum-buffer MB --lr LR --momentum MU --out-param P2 --out-buffer MB2
+~~~
+
+AdamW 使用 decoupled weight decay：
+
+~~~text
+m_t = beta1*m + (1-beta1)*g
+v_t = beta2*v + (1-beta2)*g*g
+m_hat = m_t / (1-beta1^t)
+v_hat = v_t / (1-beta2^t)
+param = param*(1-lr*wd) - lr*m_hat/(sqrt(v_hat)+eps)
+~~~
+
+SGD momentum：
+
+~~~text
+buffer = momentum*buffer + grad
+param = param - lr*buffer
+~~~
+
+#### 数值评测方式
+
+评测侧在独立进程中生成数据，并用 PyTorch 在相同输入/参数上计算参考值，逐操作比较。工作区不会收到期望输出张量。
+
+- 每个操作使用 3 组独立随机数据（固定种子），形状固定
+- 误差指标为加权平均相对误差：
+
+~~~
+mre = sum(|candidate - reference|) / sum(|reference|)
+~~~
+
+- 每个操作有独立阈值，单位与 fp32 精度匹配：
+
+| 操作 | 加权平均相对误差阈值 |
+| --- | --- |
+| elementwise.add / sub / mul / div | 1e-6 |
+| relu | 1e-6 |
+| mse | 1e-6 |
+| sgd-momentum | 1e-6 |
+| matmul / linear | 1e-5 |
+| softmax / layernorm | 1e-5 |
+| adamw | 1e-5 |
+| conv2d | 1e-4 |
+| matmul-backward / linear-backward / relu-backward / layernorm-backward / mse-backward / conv2d-backward | 1e-4 |
 
-占位符或空实现会被判为基础设施错误。
+- 任一操作不可执行、崩溃、超时（单次调用 60 秒）、输出文件缺失或元素数不符，或 mre 超过阈值，都会判为基础设施失败（stage = infra）
+- 速度（GFLOP/s）会同时报告，但速度不参与通过判定
+- 不做完整训练，不要求模型保存/加载格式；控制器需要的权重可以放在源码目录或控制器自行打包
 
-隐藏测试由评测侧注入，使用固定种子和独立数据，不复制到工作区。
+评测数据在测试进程内生成，不随工作区提供。不要伪造输出：评测直接执行你的二进制并检查它写出的文件。
 
 ### 2. 控制器入口
 
@@ -222,7 +328,7 @@ eval_test
 
 ### eval_test
 
-由宿主对最新提交快照进行构建和隐藏仿真评估，返回：
+由宿主对最新提交快照进行构建、算子数值评测和隐藏仿真评估，返回：
 
 ~~~text
 stage
@@ -248,8 +354,8 @@ completed
 阶段含义：
 
 - no_submission：还没有提交快照
-- compile：CMake 构建 `libnn.a` / `control` 失败
-- infra：隐藏基础设施测试编译、链接或运行失败
+- compile：CMake 构建失败，或没有产出 build/nn_task / build/control
+- infra：build/nn_task 算子数值评测失败（不可执行、崩溃、超时、输出缺失或误差超阈值）
 - controller：控制器启动、协议、崩溃、超时或输出非法
 - simulation：隐藏仿真推进异常、NaN 或超时
 - scoring：测试有效，但 `t_can < 10.0`
@@ -278,10 +384,9 @@ t_can >= 10.0
 
 - 只允许 C++17 + CMake + C++ 标准库
 - 仅 CPU，fp32
-- 提交源码中不得出现 torch
-- 评测环境无 Python/torch
+- 提交源码中不得出现 torch；候选运行时不提供 Python/torch（评测侧会用 PyTorch 计算数值参考值）
 - 无网络
-- 不得修改仿真器或公开头文件 `include/nn/nn.hpp`
+- 不得修改或替换 tools/sim；它只用于训练和自测
 - 不得提交编译产物
 - 赛后人工审核：控制器必须是神经网络，且必须实际调用你实现的神经网络基础设施；手写控制律、查表、固定轨迹或外部依赖会被判无效
 
@@ -361,9 +466,9 @@ T_done 是第一次返回 completed 的测试请求时间。
 
 ## 建议工作顺序
 
-1. 阅读 `include/nn/nn.hpp` 契约与 tools/sim 协议
-2. 先实现最小可用的 tensor/算子/反向/优化器
-3. 用自测和 `include/nn/nn.hpp` 契约检查基础设施
-4. 用 tools/sim 采集数据并训练控制器
+1. 阅读 build/nn_task 的 CLI 协议与 tools/sim 协议
+2. 先实现最小可用的 tensor/算子/反向/优化器/线程池，并让 build/nn_task 跑通
+3. 用 PyTorch 或独立 double 标量参考、有限差分梯度检查逐算子核对误差，边界形状/广播/groups/stride/dilation 都要覆盖
+4. 用 tools/sim 采集数据并训练控制器，导出权重供 build/control 使用
 5. 反复提交和测试，优先把 t_can 提升到 10 秒
 6. 完成后立即总结

package/eval/README.md

@@ -4,15 +4,25 @@
 
 | 任务 | 脚本 | 评分 |
 |---|---|---|
-| tank | eval/tank.py | 官方引擎 + BT，对基线 |
-| warehouse | eval/warehouse.py | 正确性 + 相对基线加速比 |
-| land | eval/land.py | 官方 round_robin/rate |
-| bomberman | eval/bomberman.py | 官方 tournament/run |
-| ctf | eval/ctf.py | 官方 arena plan/run/rate |
-| commit0 | eval/commit0.py | 单元测试通过率 |
-| swe-bench-pro | eval/swe-bench-pro.py | 隐藏 F2P/P2P 测试 |
-| featurebench | eval/featurebench.py | 隐藏 F2P/P2P 测试；全部通过才 resolved |
+| tank | eval/tank.py | 官方引擎 + BT，对基线（quick 20 局） |
+| warehouse | eval/warehouse.py | 正确性 + 相对基线加速比（quick 仅 pub_small） |
+| land | eval/land.py | 官方 round_robin/rate（quick 仅单挑 2 图） |
+| bomberman | eval/bomberman.py | 官方 tournament/run（quick 仅 2 人 1 种子） |
+| ctf | eval/ctf.py | 官方 arena plan/run/rate（quick 仅 2/3 方） |
+| commit0 | eval/commit0.py | 单元测试通过率（测试来自安装快照） |
+| swe-bench-pro | eval/swe-bench-pro.py | 隐藏 F2P/P2P 节点全部通过才 resolved |
+| featurebench | eval/featurebench.py | 隐藏 feature 测试 + 全部回归文件通过才 resolved |
+| neural-cpp | eval/neural-cpp.py | 隐藏 C++ 测试；accuracy / matrixAccuracy / speedGeomean / score |
+| neural-fp32 | eval/neural-fp32.py | 逐操作 MRE/阈值/速度；无统一 score |
+| neural-cifar | eval/neural-cifar.py | FP32 算子指标 + 隐藏验证集准确率/FPS（独立目录） |
+| neural-pendulum | eval/neural-pendulum.py | FP32 算子指标 + 固定初态控制 t_can（独立目录） |
 
 eval/legacy/ 保存从独立评测目录迁移过来的评测器；eval/arena/ 保存官方引擎、裁判和历史对手。common.py 负责临时工作区、结果解析和结果文件写入。
 
-后台任务由 host 的 JobManager 执行，日志和结果保存在 $DSH_HOME/eval-guard/jobs/<job-id>/。
+## 完整性约束（宿主注入）
+
+- `EVAL_BASELINE_DIR`：安装成功时快照的测试/配置目录。commit0/SWE/FeatureBench 的 wrapper 会在评测前用它覆盖候选副本，并删除候选新增的 `conftest.py`；同时校验 `.venv` 指纹（`venv.json`）。
+- `EVAL_RESULT_FILE` / `EVAL_JOB_DIR`：只由顶层 evaluator 使用；所有 evaluator 再启动子进程时会剥掉这些变量，候选二进制拿不到结果文件路径。
+- 隐藏资产（neural-cpp 的隐藏测试、CIFAR 的验证标签）在候选构建/推理开始前读入内存；CIFAR 的图像副本会放到独立临时目录，标签不落盘到候选可见路径。
+
+后台任务由 host 的 JobManager 执行，日志和结果保存在 $DSH_HOME/eval-guard/jobs/<job-id>/。Job 子进程的环境变量已收敛为白名单（PATH/HOME/LANG/LC_*/TMPDIR/SSL_* 等）。

package/eval/bomberman.py

@@ -1,12 +1,12 @@
 #!/usr/bin/env python3
-"""Single-workspace tank evaluator wrapper."""
+"""Single-workspace bomberman evaluator wrapper."""
 from __future__ import annotations
 import shutil, tempfile
 from pathlib import Path
 from common import EVAL_ROOT, candidate_row, emit, fail, parse_workspace, run_legacy
 
 workspace = parse_workspace()
-with tempfile.TemporaryDirectory(prefix='bomber-eval-') as td:
+with tempfile.TemporaryDirectory(prefix='bomberman-eval-') as td:
     temp = Path(td)
     bot = workspace / 'submission' / 'bot.cpp'
     if not bot.is_file():
@@ -17,4 +17,6 @@ with tempfile.TemporaryDirectory(prefix='bomber-eval-') as td:
     if not result.get('ok'):
         raise SystemExit(fail({'task': 'bomberman', 'ok': False, 'error': result.get('error', 'evaluation failed')}))
     row = candidate_row(result['summary'])
-    emit({'task': 'bomberman', 'ok': True, 'score': row.get('score', row.get('rating', row.get('theta', 0))), 'rating': row.get('rating', row.get('theta', 0)), 'wins': row.get('wins', 0), 'losses': row.get('losses', 0), 'draws': row.get('draws', 0)})
+    if not row:
+        raise SystemExit(fail({'task': 'bomberman', 'ok': False, 'error': 'candidate row missing from evaluator summary'}))
+    emit({'task': 'bomberman', 'ok': True, 'score': row.get('score', row.get('rating', 0)), 'rating': row.get('rating', row.get('theta', 0)), 'theta': row.get('theta', 0)})

package/eval/commit0.py

@@ -1,11 +1,14 @@
 #!/usr/bin/env python3
 """Single-workspace Commit0 evaluator wrapper."""
 from __future__ import annotations
-import os, shutil, tempfile
+import os, tempfile
 from pathlib import Path
-from common import EVAL_ROOT, candidate_row, clean_copy, emit, fail, parse_workspace, run_legacy
+from common import EVAL_ROOT, candidate_row, clean_copy, emit, fail, parse_workspace, restore_baseline, run_legacy, verify_venv
 
 workspace = parse_workspace()
+problem = verify_venv(workspace)
+if problem is not None:
+    raise SystemExit(fail({'task': 'commit0', 'ok': False, 'error': problem}))
 with tempfile.TemporaryDirectory(prefix='commit0-eval-') as td:
     temp = Path(td)
     candidate = temp / 'candidate' / 'workspace'
@@ -13,7 +16,11 @@ with tempfile.TemporaryDirectory(prefix='commit0-eval-') as td:
     clean_copy(workspace, candidate, ignore_venv=True)
     venv = workspace / '.venv'
     if venv.is_dir():
-        os.symlink(venv, candidate / '.venv')
+        try:
+            os.symlink(venv, candidate / '.venv')
+        except FileExistsError:
+            pass
+    restore_baseline(candidate)
     result = run_legacy(EVAL_ROOT / 'legacy' / 'commit0.py', ['--submissions', str(temp)], temp / 'out')
     if not result.get('ok'):
         raise SystemExit(fail({'task': 'commit0', 'ok': False, 'error': result.get('error', 'evaluation failed')}))

package/eval/common.py

@@ -2,6 +2,7 @@
 from __future__ import annotations
 
 import argparse
+import hashlib
 import json
 import os
 import shutil
@@ -12,6 +13,22 @@ from pathlib import Path
 
 EVAL_ROOT = Path(__file__).resolve().parent
 
+# Variables the host injects for the evaluator itself. They must not leak into
+# the legacy evaluator or into any candidate process it starts (a candidate
+# that learns EVAL_RESULT_FILE could overwrite the result).
+HOST_EVAL_KEYS = frozenset({
+    'EVAL_RESULT_FILE', 'EVAL_JOB_DIR', 'EVAL_INSTALL_TOKEN',
+    'EVAL_BASELINE_DIR', 'EVAL_VENV_MANIFEST',
+})
+
+
+def child_env() -> dict[str, str]:
+    """Environment for trusted legacy evaluators and their candidate children."""
+    return {
+        key: value for key, value in os.environ.items()
+        if key not in HOST_EVAL_KEYS and key not in {'PYTHONPATH', 'PYTHONHOME'}
+    }
+
 
 def parse_workspace() -> Path:
     parser = argparse.ArgumentParser(description='single-workspace evaluator')
@@ -28,18 +45,23 @@ def emit(payload: dict) -> None:
     Path(target).write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding='utf-8')
 
 
-def clean_copy(source: Path, target: Path, ignore_venv: bool = False) -> None:
+def clean_copy(source: Path, target: Path, ignore_venv: bool = False,
+               ignore_names: tuple[str, ...] = ()) -> None:
+    ignored = set(ignore_names)
+
     def ignore(directory: str, names: list[str]) -> set[str]:
-        if ignore_venv and Path(directory) == source:
-            return {name for name in names if name == '.venv'}
+        if Path(directory) == source:
+            skip = set(ignored)
+            if ignore_venv:
+                skip.add('.venv')
+            return {name for name in names if name in skip}
         return set()
     shutil.copytree(source, target, symlinks=True, ignore=ignore)
 
 
 def run_legacy(script: Path, args: list[str], out: Path) -> dict:
     cmd = [sys.executable, '-X', 'utf8', str(script), *args, '--out', str(out)]
-    env = {k: v for k, v in os.environ.items() if k not in {'PYTHONPATH', 'PYTHONHOME'}}
-    completed = subprocess.run(cmd, cwd=str(EVAL_ROOT), capture_output=True, text=True, encoding='utf-8', errors='replace', env=env, timeout=1800)
+    completed = subprocess.run(cmd, cwd=str(EVAL_ROOT), capture_output=True, text=True, encoding='utf-8', errors='replace', env=child_env(), timeout=1800)
     if completed.returncode != 0:
         return {'ok': False, 'error': (completed.stderr or completed.stdout or 'evaluator failed')[-4000:]}
     summary = out / 'summary.json'
@@ -59,3 +81,145 @@ def candidate_row(summary: dict, name: str = 'candidate') -> dict:
 def fail(payload: dict) -> int:
     emit(payload)
     return 1
+
+
+def parse_pytest_results(output: str) -> dict[str, str]:
+    """Map pytest node ids to PASSED/FAILED/ERROR/SKIPPED from -rA output."""
+    import re
+    pattern = re.compile(r'^(PASSED|FAILED|ERROR|SKIPPED|XFAIL|XPASS) (\S+)', re.MULTILINE)
+    results: dict[str, str] = {}
+    for match in pattern.finditer(output):
+        results[match.group(2)] = match.group(1)
+    return results
+
+
+def matches_node(node: str, selector: str) -> bool:
+    """Whether a pytest node id belongs to a file/node selector."""
+    if node == selector:
+        return True
+    return node.startswith(selector + '::') or node.startswith(selector + '/')
+
+
+
+# ---------------------------------------------------------------------------
+# Pristine test material and venv integrity
+# ---------------------------------------------------------------------------
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    digest.update(path.read_bytes())
+    return digest.hexdigest()
+
+
+def sha256_tree(root: Path) -> str:
+    digest = hashlib.sha256()
+
+    def walk(current: Path) -> None:
+        for entry in sorted(current.iterdir(), key=lambda item: item.name):
+            digest.update(str(entry.relative_to(root)).encode('utf-8'))
+            if entry.is_dir():
+                walk(entry)
+            elif entry.is_file():
+                digest.update(entry.read_bytes())
+            elif entry.is_symlink():
+                digest.update(b'link:' + os.readlink(entry).encode('utf-8'))
+
+    walk(root)
+    return digest.hexdigest()
+
+
+def baseline_dir() -> Path | None:
+    raw = os.environ.get('EVAL_BASELINE_DIR', '')
+    if raw == '':
+        return None
+    path = Path(raw)
+    return path if path.is_dir() else None
+
+
+def restore_baseline(candidate: Path) -> None:
+    """Overwrite candidate test material with the pristine install-time copy.
+
+    Also removes conftest.py files the agent added (the pristine snapshot keeps
+    the project's own ones), so a workspace hook cannot rewrite pytest results.
+    """
+    baseline = baseline_dir()
+    if baseline is None:
+        return
+    candidate = candidate.resolve()
+    baseline = baseline.resolve()
+    for hook in candidate.rglob('conftest.py'):
+        if hook.is_symlink():
+            hook.unlink()
+            continue
+        try:
+            relative = hook.relative_to(candidate)
+        except ValueError:
+            continue
+        if not (baseline / relative).is_file():
+            hook.unlink()
+    for entry in baseline.iterdir():
+        if entry.name in {'manifest.json', 'venv.json'}:
+            continue
+        destination = candidate / entry.name
+        if destination.exists() or destination.is_symlink():
+            if destination.is_dir() and not destination.is_symlink():
+                shutil.rmtree(destination, ignore_errors=True)
+            else:
+                destination.unlink()
+        if entry.is_dir():
+            shutil.copytree(entry, destination, symlinks=True)
+        else:
+            shutil.copy2(entry, destination)
+
+
+def verify_venv(workspace: Path) -> str | None:
+    """Verify the venv control plane against the install-time fingerprint.
+
+    Returns a human-readable reason when the venv was tampered with, else None.
+    """
+    baseline = baseline_dir()
+    if baseline is None:
+        return None
+    manifest_path = baseline / 'venv.json'
+    if not manifest_path.is_file():
+        return None
+    try:
+        manifest = json.loads(manifest_path.read_text(encoding='utf-8'))
+    except Exception:
+        return 'venv manifest is corrupt'
+    venv = workspace / '.venv'
+    if not venv.is_dir():
+        return 'the evaluator requires the installed .venv; it is missing'
+    expected_python = manifest.get('python')
+    if isinstance(expected_python, str) and expected_python != '':
+        python = venv / 'bin' / 'python'
+        try:
+            actual = str(python.resolve())
+        except OSError:
+            actual = ''
+        if actual != expected_python:
+            return 'the .venv python interpreter was replaced'
+    for relative, expected in (manifest.get('files') or {}).items():
+        target = venv / relative
+        if not target.is_file():
+            return f'the .venv file {relative} is missing'
+        try:
+            actual = sha256_file(target)
+        except OSError as error:
+            return f'cannot read the .venv file {relative}: {error}'
+        if actual != expected:
+            return f'the .venv file {relative} was modified'
+    expected_pytest = manifest.get('pytest')
+    if isinstance(expected_pytest, str) and expected_pytest != '':
+        found = False
+        for site in venv.glob('lib*/python*/site-packages/pytest'):
+            found = True
+            try:
+                actual = sha256_tree(site)
+            except OSError as error:
+                return f'cannot read the venv pytest package: {error}'
+            if actual != expected_pytest:
+                return 'the venv pytest package was modified'
+        if not found:
+            return 'the venv pytest package disappeared'
+    return None

package/eval/ctf.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Single-workspace tank evaluator wrapper."""
+"""Single-workspace ctf evaluator wrapper."""
 from __future__ import annotations
 import shutil, tempfile
 from pathlib import Path
@@ -17,4 +17,6 @@ with tempfile.TemporaryDirectory(prefix='ctf-eval-') as td:
     if not result.get('ok'):
         raise SystemExit(fail({'task': 'ctf', 'ok': False, 'error': result.get('error', 'evaluation failed')}))
     row = candidate_row(result['summary'])
-    emit({'task': 'ctf', 'ok': True, 'score': row.get('score', row.get('rating', row.get('theta', 0))), 'rating': row.get('rating', row.get('theta', 0)), 'wins': row.get('wins', 0), 'losses': row.get('losses', 0), 'draws': row.get('draws', 0)})
+    if not row:
+        raise SystemExit(fail({'task': 'ctf', 'ok': False, 'error': 'candidate row missing from evaluator summary'}))
+    emit({'task': 'ctf', 'ok': True, 'score': row.get('score', row.get('rating', 0)), 'rating': row.get('rating', row.get('theta', 0)), 'theta': row.get('theta', 0)})

package/eval/featurebench.py

@@ -1,11 +1,14 @@
 #!/usr/bin/env python3
-"""Single-workspace SWE-Bench Pro evaluator wrapper."""
+"""Single-workspace FeatureBench evaluator wrapper."""
 from __future__ import annotations
 import os, tempfile
 from pathlib import Path
-from common import EVAL_ROOT, candidate_row, clean_copy, emit, fail, parse_workspace, run_legacy
+from common import EVAL_ROOT, candidate_row, clean_copy, emit, fail, parse_workspace, restore_baseline, run_legacy, verify_venv
 
 workspace = parse_workspace()
+problem = verify_venv(workspace)
+if problem is not None:
+    raise SystemExit(fail({'task': 'featurebench', 'ok': False, 'error': problem}))
 with tempfile.TemporaryDirectory(prefix='feature-eval-') as td:
     temp = Path(td)
     candidate = temp / 'candidate' / 'workspace'
@@ -13,13 +16,14 @@ with tempfile.TemporaryDirectory(prefix='feature-eval-') as td:
     clean_copy(workspace, candidate, ignore_venv=True)
     venv = workspace / '.venv'
     if venv.is_dir():
-        os.symlink(venv, candidate / '.venv')
+        try:
+            os.symlink(venv, candidate / '.venv')
+        except FileExistsError:
+            pass
+    restore_baseline(candidate)
     result = run_legacy(EVAL_ROOT / 'legacy' / 'featurebench.py', ['--submissions', str(temp)], temp / 'out')
     if not result.get('ok'):
         raise SystemExit(fail({'task': 'featurebench', 'ok': False, 'error': result.get('error', 'evaluation failed')}))
     row = candidate_row(result['summary'])
-    passed = row.get('passed', 0)
-    failed = row.get('failed', 0)
-    errors = row.get('errors', 0)
-    resolved = failed == 0 and errors == 0 and passed > 0
-    emit({'task': 'featurebench', 'ok': True, 'score': 100.0 if resolved else 0.0, 'resolved': resolved, 'passRate': row.get('score', 0), 'passed': passed, 'failed': failed, 'errors': errors, 'status': row.get('status')})
+    resolved = row.get('resolved') is True
+    emit({'task': 'featurebench', 'ok': True, 'score': 100.0 if resolved else 0.0, 'resolved': resolved, 'passRate': row.get('score', 0), 'passed': row.get('passed', 0), 'failed': row.get('failed', 0), 'errors': row.get('errors', 0), 'f2pPassed': row.get('f2p_passed', 0), 'f2pTotal': row.get('f2p_total', 0), 'status': row.get('status')})

package/eval/land.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Single-workspace tank evaluator wrapper."""
+"""Single-workspace land evaluator wrapper."""
 from __future__ import annotations
 import shutil, tempfile
 from pathlib import Path
@@ -17,4 +17,6 @@ with tempfile.TemporaryDirectory(prefix='land-eval-') as td:
     if not result.get('ok'):
         raise SystemExit(fail({'task': 'land', 'ok': False, 'error': result.get('error', 'evaluation failed')}))
     row = candidate_row(result['summary'])
-    emit({'task': 'land', 'ok': True, 'score': row.get('score', row.get('rating', row.get('theta', 0))), 'rating': row.get('rating', row.get('theta', 0)), 'wins': row.get('wins', 0), 'losses': row.get('losses', 0), 'draws': row.get('draws', 0)})
+    if not row:
+        raise SystemExit(fail({'task': 'land', 'ok': False, 'error': 'candidate row missing from evaluator summary'}))
+    emit({'task': 'land', 'ok': True, 'score': row.get('score', row.get('rating', 0)), 'rating': row.get('rating', row.get('theta', 0)), 'theta': row.get('theta', 0)})