dsh-eval-guard 0.7.3 → 0.7.4

package/ISSUES.md

@@ -1,10 +1,14 @@
 # dsh-eval-guard 问题清单与修复记录
 
-> 版本：0.7.3 · 审计基线：本仓库工作树（HEAD 1e1f6b4 + 未提交改动）
+> 版本：0.7.4 · 审计基线：本仓库工作树（含 0.7.4 修复与 Neural-Pendulum 修正环境）
 > 对照 DSH：0.2.1-alpha.1 源码（并核对 git tag 0.1.2 / 0.1.5 / 0.1.7 / 0.2.0-rc.1 / rc.2 / 0.2.1-alpha.1）
 > 本文件先记录问题，再记录修复状态。状态：✅ 已修复 / 🟡 部分修复 / ⛔ 保留（有明确原因或需要架构级改动）
 >
 > **2026-10-22 更新**：本轮只修复“守护本体 + Neural-Pendulum”。已修复：fork/子代理会话识别与约束、子代理路径审计、sandbox 继承环境清洗、PTC 控制通道标记回填、allow 规则宿主绑定安全、写路径规则、设置页任务默认值原子保存、Job 异步失败清理与重启 PID 校验、Pendulum 按请求时刻计分与 done final_score。  
+>
+> **2026-10-23 会话回归修复（未发版）**：修复 Neural-Pendulum 仿真器 Coriolis 项多乘 cos 导致高转速下能量注入并产生 NaN；修正关节 3 在绝对角坐标下的广义力矩映射，由 (tau1,0,tau3) 改为 (tau1,-tau3,tau3)；TASK/SPEC/prompt 明确 theta 约定并说明工作区无 PyTorch/numpy；网络 syscall 改为 AF_INET/AF_INET6 的 EPERM 软拒绝，避免 pip/NSS 本地检查被误杀。
+
+> **0.7.4**：把上述修复连同固定物理的 RL 可行性环境一起发布。开发参考环境在 `assets/neural-pendulum/rl-feasibility/`（Python + CPU PyTorch，仅开发参考，不进入候选工作区）；其中报告/汇总/曲线已验证 full 36D + base reward + random init 可学、去掉手动差分 da 或 H=1 会失败、PPO/SAC 应交付 best checkpoint。
 > **下文仍有若干条目标记为 ✅ 但实际未完成，不要再按旧状态理解**：arena legacy 的 `std::thread` 静态规则未移除；`cifar_cache.py` 解压仍无 filter、缓存路径仍用 `Path.home()/.dsh`；CIFAR 隐藏标签仍以文件形式落盘；FP32“30 分钟整轮”仍是 op 间检查；无 baseline 时评测仍 fail-open。其余任务相关修复按用户要求暂不动。
 
 ## 0. 修复状态总览

package/README.md

@@ -3,7 +3,7 @@
 在 DSH 中启用后，约束**所有用户主对话**：
 
 - 文件类工具（read/write/edit/grep/glob/ops_*/drawio_* 等）做参数路径审计：只允许修改会话工作区；工作区之外允许读取“用户不可写但可读”的文件（系统文件、只读资料），用户可写的数据目录仍不可读写。进程沙箱内 `/tmp` 映射到工作区 `.tmp`；文件工具不会重写路径，读写 `/tmp` 请直接使用工作区内的 `.tmp` 路径。
-- 进程类工具（bash / run_code / 后台任务）由 bwrap + seccomp supervisor 二次约束：只读系统视图、工作区可写、私有 /tmp、无网络；进入沙箱前清空继承环境，只回填 PATH/HOME/TMPDIR/XDG/LANG 等非敏感变量，避免宿主 API key / 代理泄漏；越界访问会记录并终止进程组。
+- 进程类工具（bash / run_code / 后台任务）由 bwrap + seccomp supervisor 二次约束：只读系统视图、工作区可写、私有 /tmp、无网络；进入沙箱前清空继承环境，只回填 PATH/HOME/TMPDIR/XDG/LANG 等非敏感变量，避免宿主 API key / 代理泄漏。文件越界访问会记录并终止进程组；网络 `connect/bind/sendto` 对 AF_INET/AF_INET6 返回 EPERM 并记录，但不杀进程组（`socket()` 创建允许，避免 pip/NSS 等本地检查被误杀），真正的网络边界是 bwrap 的私有 network namespace。
 - 禁止联网、子代理、Agent Team/跨 agent 通信，以及可绕过单次评测的持久化工具（goal / schedule / drawio / canvas 等）。用户 fork 出的会话仍按主会话约束；子代理会话本身也同样受工具审计和进程沙箱约束（黑名单只是额外的一层）。
 - 工具黑名单可配置；未知工具默认放行。DSH 内置与 dsh-ops 的文件工具会做参数路径审计。
 
@@ -59,6 +59,8 @@ CIFAR-10 下载缓存固定在 DSH home 下（默认 `~/.dsh/eval-guard/cache/ci
 
 Neural-Pendulum 要求从零实现纯 C++17、仅 CPU、fp32 的神经网络基础设施，并用它训练控制器驱动预编译三节摆从接近全向下起摆、保持全向上 10 秒。后端评测与 Neural-FP32/Neural-CIFAR 使用同一套算子/阈值（单次调用 60 秒）。算子评测全部通过后才驱动 build/control，在固定初始状态上统计连续全向上时间 t_can。评测脚本在独立目录 `dsh-eval-guard/neural-pendulum-eval/`；宿主侧提交与测试工具为 eval_submit / eval_test，工具的注册依赖会话级 run 绑定。
 
+**开发参考环境**：修正后的 RL 可行性环境（Python + CPU PyTorch，仅开发参考，不进入候选工作区、不是提交依赖）放在 `assets/neural-pendulum/rl-feasibility/`。其 `sim.py` 与评测用 `sim_eval` 使用相同物理和物理关节力矩映射 `(tau1, -tau3, tau3)`；报告、汇总表和曲线在 `RESULTS.md` / `results/` 中。15 组实验确认：full 36D（cos/sin + 手动差分 da/5，H=4）+ base reward + random init 可学；去掉 da 或把 H 缩到 1 会失败；PPO/SAC 都应交付 best checkpoint。
+
 **分数**：eval_test 返回 stage / t_can / completed / error；时间积分 `final_score` 写入宿主侧事件与完成标记（`~/.dsh/eval-guard/neural-pendulum/<workspace>/done`），用于赛后排行，不返回给 agent。分数在**测试请求时刻**切换，当前测试结果也计入本次积分；宿主按事件重新计算最终值，而不是使用“评测完成时刻”的旧值。起算时间取首次提交写入的 meta.json。
 
 评测时在临时目录独立构建，然后从 assets/neural-cpp/hidden/ 注入 C++ 测试源码；隐藏测试在候选构建**开始前**读入内存，构建完成后再落盘编译，测试数据由固定种子在进程内生成，不依赖或复制外部数据文件。
@@ -174,7 +176,7 @@ DSH_CHECKOUT=/path/to/deepseek-harness scripts/install.sh web
 
 - 只支持 Linux；启用守护前会探测 bwrap 与 unprivileged user namespace，探测失败时设置页显示原因且不生效。
 - seccomp 用户通知的路径检查与 open 之间存在极小的 TOCTOU 窗口；bwrap 只读/无网络挂载是第二层内核边界。
-- io_uring 被拒绝（以 EPERM 软拒绝，不杀进程）；正常 syscall 不记录，进程审计只覆盖被拦截的文件/网络 syscall。
+- io_uring 被拒绝（以 EPERM 软拒绝，不杀进程）；正常 syscall 不记录，进程审计只覆盖被拦截的文件/网络 syscall。AF_INET/AF_INET6 的连接类 syscall 也是 EPERM 软拒绝；文件策略违规仍会终止进程组。
 - 工具路径审计只覆盖已知文件工具（DSH 内置、dsh-ops、drawio）；未知工具默认放行，这是产品选择，不是完整内核边界。
 - `forbiddenPaths` 的 `!` 例外只放宽审计；它不会把宿主 home、DSH 状态目录、共享 /tmp 或 workspace 的祖先 ro-bind 进沙箱。系统目录本身已只读可见，通常不需要 `!`。
 - 只读文件放行是产品选择：工作区之外“用户不可写但可读”的文件允许读取，因此只读安装的隐藏资产/数据不在 OS 权限层面被隐藏；真正需要隔离的隐藏数据仍需独立用户/容器。

package/assets/neural-pendulum/SPEC.md

@@ -25,7 +25,7 @@ agent 必须：
 4. 保证基础设施通过 `build/nn_task` 固定 CLI 的数值评测：评测侧用 PyTorch 在相同数据上计算参考值，逐操作比较加权平均相对误差并对照阈值。
 5. 提交的源码中不得出现 torch；候选运行时不提供 Python/torch。
 
-允许在开发阶段使用 PyTorch 做数值自查和参考，但提交源码必须完全独立。
+评测侧用 PyTorch 计算参考值；工作区/候选运行时不提供 PyTorch 或 numpy（也无网络），开发阶段请用 C++ 或独立 double 标量参考做自查。提交源码必须完全独立。
 
 ## 3. 被控对象
 
@@ -34,7 +34,7 @@ agent 必须：
 - 3 根均匀杆
 - 第一关节、第三关节受控；第二关节被动
 - 控制量: 两个归一化力矩 [u1, u3]，每个 in [-1, 1]
-- 物理力矩: tau_1 = u1 * tau_max，tau_3 = u3 * tau_max，tau_2 = 0
+- 物理力矩: 关节 1 电机力矩 tau_1 = u1 * tau_max；关节 3 电机力矩 tau_3 = u3 * tau_max（关节 3 是杆 2/杆 3 之间的内部电机，杆 2 受到 -tau_3 反作用）；关节 2 电机力矩为 0。在绝对角度坐标下广义驱动力向量为 (tau_1, -tau_3, tau_3)。
 - 控制频率: 100 Hz，控制周期 dt = 0.01 s
 - 每个控制周期内部用 4 个 RK4 子步
 - 最大仿真时长: 30.0 s
@@ -118,7 +118,8 @@ obs = [a1, a2, a3]  (rad)
 ~~~
 tau_1 = u1 * tau_max
 tau_3 = u3 * tau_max
-tau_2 = 0
+tau_2(joint motor) = 0
+absolute generalized force = (tau_1, -tau_3, tau_3)
 ~~~
 
 非有限值、超范围或格式错误：

package/assets/neural-pendulum/TASK.md

@@ -10,7 +10,7 @@
 - 保持三根杆全部向上至少 t_tgt = 10.0 秒
 - 同时保证神经网络基础设施通过 build/nn_task 固定 CLI 的数值评测
 
-开发阶段可以使用 Python/PyTorch 做离线数值对照和实验；最终提交的基础设施和控制器源码必须是纯 C++，不得出现 torch，候选运行时不提供 Python/torch。
+评测侧会用 PyTorch 计算算子参考值，但**工作区/候选运行时不提供 PyTorch 或 numpy**，也没有网络，无法 pip 安装。开发阶段请用 C++，或 Python 标准库/独立 double 实现做数值对照；最终提交的基础设施和控制器源码必须是纯 C++，不得出现 torch。
 
 ## 环境
 
@@ -47,7 +47,7 @@ joint 2：被动
 joint 3：驱动
 ~~~
 
-第一关节和第三关节各有一个归一化力矩输入。
+第一关节和第三关节各有一个归一化力矩输入。第三关节是杆 2/杆 3 之间的内部电机：它的力矩同时作用在杆 3 和杆 2 上（反作用），不是只作用在杆 3 上的外部力矩；关节 2 没有主动电机力矩。
 
 ### 观测
 
@@ -61,6 +61,7 @@ obs = [a1, a2, a3]  (rad)
 即 3 维观测，单位 rad，不除以 pi。
 
 - theta_i 是第 i 根杆的绝对角度
+- theta_i = 0 表示竖直向下，theta_i = pi 表示竖直向上
 - a_i 是相对竖直向上的角度误差
 - 不提供角速度
 - 不提供第三杆末端位置
@@ -468,7 +469,7 @@ T_done 是第一次返回 completed 的测试请求时间。
 
 1. 阅读 build/nn_task 的 CLI 协议与 tools/sim 协议
 2. 先实现最小可用的 tensor/算子/反向/优化器/线程池，并让 build/nn_task 跑通
-3. 用 PyTorch 或独立 double 标量参考、有限差分梯度检查逐算子核对误差，边界形状/广播/groups/stride/dilation 都要覆盖
+3. 用独立 double 标量参考、有限差分梯度检查逐算子核对误差，边界形状/广播/groups/stride/dilation 都要覆盖（工作区没有 PyTorch/numpy）
 4. 用 tools/sim 采集数据并训练控制器，导出权重供 build/control 使用
 5. 反复提交和测试，优先把 t_can 提升到 10 秒
 6. 完成后立即总结

package/assets/neural-pendulum/rl-feasibility/README.md

@@ -0,0 +1,60 @@
+# Neural-Pendulum RL feasibility harness (dev-only reference)
+
+This directory is a **development/reference** copy of the reinforcement-learning
+environment used to confirm that the corrected Neural-Pendulum dynamics are
+learnable.  It is **not** part of the candidate runtime, is not copied into the
+task workspace, and is not required by the evaluator.
+
+## Fixed physics
+
+The Python simulator here uses the same physical model as
+`assets/neural-pendulum/sim.cpp` / `sim_eval`:
+
+- three-link planar pendulum, 100 Hz ZOH, RK4 with 4 substeps;
+- physical joint-motor mapping in absolute-angle coordinates:
+  `(tau1, -tau3, tau3)` (joint 3 is the internal motor between links 2 and 3,
+  joint 2 is passive);
+- corrected Coriolis/centrifugal term
+  `C_i = sum_j a_ij sin(theta_i-theta_j) omega_j^2`;
+- 30 s episodes, success tolerance 0.05 rad, t_tgt = 10 s.
+
+The Python simulator was cross-checked against `../../sim_eval` on the fixed
+initial state and identical action sequences; maximum observation error was
+about 5e-6 (simulator output has 6 significant digits).
+
+## What was run
+
+15 formal runs (PPO/SAC, observation ablations, reward ablations, fixed vs
+random init; 3 PPO seeds) in the corrected environment.  Full report:
+`RESULTS.md`; machine table: `results/table.md`; aggregate:
+`results/summary.json`; selected curves: `results/curves_selected.png`.
+
+Core result:
+
+- PPO full 36D (cos/sin + manual-difference da/5, H=4), base reward, random
+  init: first success 11.33M env steps / 16 min; best t_can 26.85 s.
+- SAC full 36D, base reward, random init: first success 0.70M env steps;
+  best t_can 28.75 s.
+- Angle-only (no da) and H=1 observations fail; sparse/energy rewards fail in
+  this budget; PPO needs random init, SAC can also learn from fixed init.
+- A non-RL pole-placement reference with 0.5 low-pass holds 29.99 s from the
+  fixed upright offset and succeeds on 32/32 random upright offsets, proving
+  the hold is physically feasible.
+
+## Run
+
+Requires Python 3 with CPU PyTorch on the development host.  This is **not**
+available to the candidate sandbox and is not a submission dependency.
+
+```bash
+cd assets/neural-pendulum/rl-feasibility
+TORCH_THREADS=4 python3 test_contract.py
+TORCH_THREADS=4 python3 train_ppo.py --exp local_ppo --init rand --obs full \
+    --reward base --envs 64 --minutes 30 --tb-root tb --ckpt-dir checkpoints --out results
+tensorboard --logdir tb
+```
+
+TensorBoard event files are written live during training.  The experiment
+matrix and per-run JSON/TB/log files from the full sweep are kept outside the
+npm package; only the source, report, summary, table and selected curve are
+bundled here to keep the plugin package small.

package/assets/neural-pendulum/rl-feasibility/RESULTS.md

@@ -0,0 +1,370 @@
+# RESULTS — fixed-physics 3-link pendulum RL feasibility
+
+Work dir: `feasibility/neural-pendulum-fixedenv`.
+Started 2026-10-23 01:13 CST; experiment matrix completed 06:35 CST.
+All numbers below are new runs in the corrected environment; no old
+`(tau1, 0, tau3)` result is reused.
+
+## TL;DR
+
+* **PPO and SAC both learn the corrected dynamics from the mandated random
+  near-down resets and reach t_can >= 10 s at the fixed evaluation state.**
+  * PPO (64 env, full 36D, base reward): first success at **11.33M env steps /
+    16.0 min**; best **26.85 s**; final 26.61 s; best-checkpoint robustness
+    **8/8**.
+  * SAC (32 env, full 36D, base reward): first success at **0.70M env steps /
+    24.7 min**; best **28.75 s**; final 28.21 s; best-checkpoint robustness
+    **6/8** (final 8/8).
+* **The learnability boundary is set by observation information, not by the
+  algorithm alone.**
+  * `full` H=4 (36D: cos/sin + da/5) works for both PPO and SAC.
+  * `angle` H=4 (24D, no da): PPO best 0.97 s, SAC best 0.04 s — fails.
+  * `full_h1` (9D) and `angle_h1` (6D): fail completely (best <= 0.06 s);
+    even a 50% upright curriculum does not rescue H=1.
+  * `angle_h8` (48D): best 1.31 s — longer history helps a little but does
+    not replace `da/5`.
+* **Reward boundary:** base works; sparse and strong energy shaping fail.
+  * `sparse`: 19.12M steps, best t_can 0, policy converges to tiny actions.
+  * `energy`: two designs fail; energy chasing with manual `da` has a
+    degenerate high-kinetic-energy spin optimum.
+* **Init boundary is algorithm-dependent.** Random init is required for PPO
+  (fixed-init PPO: best 1.19 s, robust 0/8). SAC fixed-init *does* succeed
+  (best 28.86 s, robust 8/8) because its replay covers the state space; PPO's
+  on-policy batch of 64 identical states does not.
+* Three PPO seeds all reach >26 s at their best checkpoint (26.85 / 27.17 /
+  28.46 s) but the final checkpoint is unreliable (seed 2 final 0.00 s).
+  Use `_best.pt`, not the last model.
+* A non-RL pole-placement stabiliser with a 0.5 low-pass torque filter holds
+  **29.99 s** from the fixed upright offset and succeeds on 32/32 random
+  upright offsets.  The hold is physically feasible with the corrected
+  mapping; it is not an RL result (privileged omega).
+
+## 1. Environment and contract
+
+* Simulator returns angles only: `reset(theta,omega)->theta`,
+  `step(u)->theta`.  No reward, done, info, omega or tip is exposed.
+* Corrected physical torque mapping:
+  `generalized_torque = (tau1, -tau3, tau3)` (joint 3 is the internal motor
+  between links 2 and 3; joint 2 is passive).  `sim.generalized_torque` and
+  `sim.make_torch_consts` both implement it.  Do not revert to
+  `(tau1, 0, tau3)`.
+* 100 Hz ZOH, RK4 with 4 substeps, 30 s episodes (3000 steps),
+  tau_max = 18 N m, damping 0.01, success tolerance 0.05 rad, t_tgt = 10 s.
+* `test_contract.py` passes; the Python sim was cross-validated against
+  `assets/neural-pendulum/sim_eval` by the Lead (~5e-6).
+* Training reset (random arm): theta_i ~ U(-0.05,0.05),
+  omega_i ~ U(-0.02,0.02), independent, no fixed bias.
+  Fixed arm: theta = [0.013,-0.008,0.021], omega = 0.
+  Curriculum arm: 50% near down, 50% theta = pi + U(-0.05,0.05).
+* Evaluation: fixed theta = [0.013,-0.008,0.021], omega = 0, deterministic
+  actions, full 3000 steps.  t_can is the longest continuous interval with
+  max_i |wrap(theta_i - pi)| < 0.05 rad.  Eval always uses the **same** obs
+  mode and reward preset as training; no reward switch at eval.
+
+## 2. Code changes in this directory
+
+1. `obs.py`: switchable modes
+   `full` 36D, `angle` 24D, `full_h1` 9D, `angle_h1` 6D,
+   `full_h2` 18D, `angle_h2` 12D, plus `angle_h8` 48D.
+   `da` is always computed internally because the reward uses it;
+   `diff=False` only drops it from the policy input.
+2. `task.py`: reward presets `base`, `sparse`, `energy` (exact weights
+   below), bounded energy error, and `--init mix` curriculum support.
+3. `train_ppo.py` / `train_sac.py`: `--obs`, `--reward`, `--init`,
+   `--up-frac`; eval uses the same mode/reward; live TB flush every
+   iteration/log tick; 8 extra deterministic rollouts from the training
+   reset distribution for final and best checkpoints.
+4. `evaluate.py`: obs-mode aware, plus `evaluate_starts`.
+5. `reference_controller.py`: non-RL local stabilisability proof.
+6. `finalize_run.py`: rebuild a results JSON from checkpoints + log when a
+   trainer crashes after saving checkpoints (used once for PPO seed 0).
+7. `summarize.py`, `plot_selected.py`: aggregate table/JSON/curves.
+
+## 3. Observation modes
+
+| --obs | frame | H | dim | contents |
+|---|---|---|---|---|
+| full | [cos a_i, sin a_i, da_i/5] | 4 | 36 | baseline |
+| angle | [cos a_i, sin a_i] | 4 | 24 | no explicit velocity |
+| full_h1 | [cos a_i, sin a_i, da_i/5] | 1 | 9 | single frame |
+| angle_h1 | [cos a_i, sin a_i] | 1 | 6 | single frame, no velocity |
+| full_h2 | [cos a_i, sin a_i, da_i/5] | 2 | 18 | optional |
+| angle_h2 | [cos a_i, sin a_i] | 2 | 12 | optional |
+| angle_h8 | [cos a_i, sin a_i] | 8 | 48 | longer history follow-up |
+
+`a_i = wrap_to_pi(theta_i - pi)`, `da_i = wrap_to_pi(a_i(t)-a_i(t-1))/dt`,
+`da(0)=0`.  Theta is in radians, never divided by pi.  History is oldest
+first, reset repeats the first frame.  The eval builder is the same class and
+mode as training.
+
+## 4. Reward presets (exact)
+
+```
+r = exp    * exp(-2 s2)
+  + sharp  * exp(-50 s2)
+  - ang    * s2
+  - vel    * sum da^2
+  - act    * sum u^2
+  + up     * 1[max_i |a_i| < 0.15]
+  + tip    * (tip_y + L)/(2L)
+  - energy * |E - E_up| / (E_up - E_down)
+```
+
+All terms are trainer-side functions of theta, the manual difference da and
+the action u; no simulator omega is read.  `E` uses da as velocity.  The
+energy error is clamped at 3 to keep the reward bounded at high spin.
+
+```json
+base   = {"exp":1.0,"sharp":2.0,"ang":0.05,"vel":0.001,"act":0.01,"up":3.0,"tip":0.3,"energy":0.01}
+sparse = {"exp":0.2,"sharp":0.0,"ang":0.0,"vel":0.001,"act":0.01,"up":3.0,"tip":0.0,"energy":0.0}
+energy = {"exp":1.0,"sharp":2.0,"ang":0.05,"vel":0.0,"act":0.005,"up":3.0,"tip":0.0,"energy":0.5}
+```
+
+`sparse`: at the near-down reset the only dense term is ~1e-26, so it is an
+intentionally hard exploration test.  `energy`: base angle terms + strong
+energy shaping + no velocity penalty; an earlier energy-dominant design
+(`exp=.2, sharp=.5, ang=.01, vel=.0002`) is archived under
+`failed/energy_unbounded/`.
+
+## 5. Training protocol
+
+PPO: 2x128 tanh actor/critic, 64 envs, horizon 600, 8 epochs,
+minibatch 2048, lr 3e-4 linearly decayed to 20%, gamma .99, lambda .95,
+clip .2, entropy .003, vf .5, target KL .03, grad clip .5.
+
+SAC: 2x256 tanh squashed-Gaussian actor, shared-trunk twin Q, 32 envs,
+batch 256, replay 300k, auto-alpha (target entropy -2), gamma .99,
+tau .005, lr 3e-4, UTD 0.25 (8 gradient updates per 32-env vector step).
+
+Both: CPU PyTorch, 4 threads per process, at most 2 training processes
+concurrently.  Eval every 5 PPO iterations (192k steps) or every 20k SAC
+steps.  Best checkpoint by eval t_can (ties by return).  Final checkpoint at
+the time budget.  TensorBoard flushed during training (not only at the end).
+At the end, 8 deterministic rollouts from the training distribution are run
+for the final and best checkpoints.
+
+## 6. Complete experiment matrix (fixed-initial deterministic eval)
+
+`best trob` = success rate / mean t_can of the best checkpoint on the 8
+training-distribution starts.  `first succ` = first eval snapshot with
+t_can >= 10 s.
+
+| exp | algo | obs | reward | init | env steps | wall | best t_can | final t_can | ever_up | best trob | first succ |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| ppo_full_base_rand | PPO | full 36 | base | rand | 35.98M | 45.1 min | **26.85 s** | 26.61 s | yes | 8/8, 26.86 s | **11.33M** |
+| ppo_full_base_rand_s1 | PPO | full 36 | base | rand | 29.03M | 40.8 min | **27.17 s** | 0.23 s | yes | 8/8, 26.78 s | 13.63M |
+| ppo_full_base_rand_s2 | PPO | full 36 | base | rand | 34.21M | 40.3 min | **28.46 s** | 0.00 s | no | 8/8, 28.03 s | 14.40M |
+| sac_full_base_rand | SAC | full 36 | base | rand | 1.42M | 50.6 min | **28.75 s** | 28.21 s | yes | 6/8, 20.89 s | **0.70M** |
+| ppo_angle_base_rand | PPO | angle 24 | base | rand | 31.76M | 40.6 min | 0.97 s | 0.04 s | yes | 0/8 | - |
+| sac_angle_base_rand | SAC | angle 24 | base | rand | 1.42M | 50.6 min | 0.04 s | 0.00 s | no | 0/8 | - |
+| ppo_full_h1_rand | PPO | full_h1 9 | base | rand | 25.42M | 35.3 min | 0.06 s | 0.00 s | no | 0/8 | - |
+| ppo_angle_h1_rand | PPO | angle_h1 6 | base | rand | 20.97M | 30.4 min | 0.00 s | 0.00 s | no | 0/8 | - |
+| ppo_angle_h8_rand | PPO | angle_h8 48 | base | rand | 30.11M | 40.4 min | 1.31 s | 0.05 s | yes | 0/8 | - |
+| ppo_full_h1_mix | PPO | full_h1 9 | base | mix 50% up | 21.39M | 30.5 min | 0.02 s | 0.00 s | no | 0/8 | - |
+| ppo_angle_mix | PPO | angle 24 | base | mix 50% up | 23.50M | 30.5 min | 0.25 s | 0.00 s | yes | 0/8 | - |
+| ppo_full_sparse_rand | PPO | full 36 | sparse | rand | 19.12M | 30.5 min | 0.00 s | 0.00 s | no | 0/8 | - |
+| ppo_full_energy_rand | PPO | full 36 | energy | rand | 24.50M | 35.5 min | 0.00 s | 0.00 s | no | 0/8 | - |
+| ppo_full_base_fixed | PPO | full 36 | base | fixed | 21.93M | 30.5 min | 1.19 s | 0.00 s | no | 0/8 | - |
+| sac_full_base_fixed | SAC | full 36 | base | fixed | 1.41M | 50.4 min | **28.86 s** | 27.62 s | yes | 8/8, 23.45 s | **0.52M** |
+
+Raw machine-readable table: `results/table.md`; per-run JSON:
+`results/<exp>.json`; plots: `results/curves_selected.png`.
+
+## 7. Results by question
+
+### Q1. Are PPO and SAC learnable? Steps/time to t_can >= 10 s
+
+**Yes, both, with full 36D observations and base reward.**
+
+| algo | envs | first success | wall to first success | steps/s | budget | best | final | best-checkpoint robustness |
+|---|---|---|---|---|---|---|---|---|
+| PPO | 64 | 11.33M | 16.0 min | ~13.3k | 45 min / 35.98M | 26.85 s | 26.61 s | 8/8 (mean 26.86 s) |
+| SAC | 32 | 0.70M | 24.7 min | ~470 | 50 min / 1.42M | 28.75 s | 28.21 s | 6/8 (mean 20.89 s) |
+
+The first successful snapshots were 26.65 s (PPO at 11.33M) and 16.74 s
+(SAC at 0.70M).  SAC needs ~16x fewer environment steps but ~28x more
+wall-clock per step, so the time to first success is comparable (16 vs
+25 min).  PPO is more stable
+at the final snapshot; SAC's final policy is also strong (8/8, 28.21 s).
+PPO eval t_can is intermittent during learning (0 <-> 26 s between successive
+192k-step evals), so a single eval snapshot is not a reliable success
+criterion; both `_best.pt` and the success-snapshot count are reported.
+
+### Q2. Angle-only vs full
+
+**Removing the explicit da/5 channel is a hard failure in this budget.**
+
+| exp | algo | obs | best t_can | final t_can | best robust |
+|---|---|---|---|---|---|
+| ppo_full_base_rand | PPO | full 36 | **26.85 s** | 26.61 s | 8/8 |
+| ppo_angle_base_rand | PPO | angle 24 | 0.97 s | 0.04 s | 0/8 |
+| sac_full_base_rand | SAC | full 36 | **28.75 s** | 28.21 s | 6/8 |
+| sac_angle_base_rand | SAC | angle 24 | 0.04 s | 0.00 s | 0/8 |
+
+PPO angle-only can swing up (ever_up=1, first up ~1.2 s) but cannot hold;
+its best up_frac is 3.8%.  SAC angle-only only briefly touches upright
+(best t_can 0.04 s, a single 0.04 s window at ~9.23 s; final ever_up 0) and
+its final return is negative.  H=4 cos/sin history does not substitute for
+the manually differentiated velocity.  This is a budget
+statement, not an information-theoretic impossibility, but 40-50 min and
+20-32M steps is a generous budget in this study.
+
+### Q3. H=4 -> H=1 and reduced observations
+
+**H=1 destroys learnability even with explicit da; longer history only
+partially substitutes for da.**
+
+| exp | obs | dim | best t_can | final t_can | ever_up | best robust |
+|---|---|---|---|---|---|---|
+| ppo_full_base_rand | full H=4 | 36 | **26.85 s** | 26.61 s | yes | 8/8 |
+| ppo_full_h1_rand | full H=1 | 9 | 0.06 s | 0.00 s | no | 0/8 |
+| ppo_angle_base_rand | angle H=4 | 24 | 0.97 s | 0.04 s | yes | 0/8 |
+| ppo_angle_h8_rand | angle H=8 | 48 | 1.31 s | 0.05 s | yes | 0/8 |
+| ppo_angle_h1_rand | angle H=1 | 6 | 0.00 s | 0.00 s | no | 0/8 |
+| ppo_full_h1_mix | full H=1 + 50% upright | 9 | 0.02 s | 0.00 s | no | 0/8 |
+
+The 9D full_h1 observation has the instantaneous full state (cos/sin+da/5)
+but no temporal window.  It never reaches upright in 25.4M steps.  A
+curriculum with half the resets already near upright also fails (21.4M
+steps), so the H=1 failure is representational (no phase/acceleration
+history), not only exploration.  Angle H=8 improves the best t_can from 0.97
+to 1.31 s but is still far from success; explicit da + H=4 is the only
+working combination.
+
+### Q4. Reward preset
+
+| exp | reward | env steps | best t_can | final t_can | ever_up | note |
+|---|---|---|---|---|---|---|
+| ppo_full_base_rand | base | 35.98M | **26.85 s** | 26.61 s | yes | works |
+| ppo_full_sparse_rand | sparse | 19.12M | 0.00 s | 0.00 s | no | policy -> tiny actions |
+| ppo_full_energy_rand | energy v2 | 24.50M | 0.00 s | 0.00 s | no | spinning optimum |
+| (failed v1) | energy unbounded | 6.7M | 0.00 s | 0.00 s | no | value loss ~5e7 |
+
+**base** is the only working reward.  **sparse** has no usable gradient from
+the near-down reset: `0.2 exp(-2 s2)` is ~1e-26 at s2~30, and the 0.15 rad
+bonus is never reached by exploration; the policy converges to mean |u|~0.01
+and return -0.39.  This is an exploration failure, not a crash.
+
+**energy** failed in two designs.  v1 used an unbounded energy error and
+blew up the value loss; v2 clamped it and kept the base angle terms but still
+never reached upright in 24.5M steps (0/24.5M, final return -1948).  The
+reason is degenerate: total mechanical energy `E = T + V` can be driven to
+`E_up` by spinning fast at the wrong angles, and `T` computed from the
+manual `da` is noisy/large exactly there.  A strong energy term rewards that
+spin instead of swing-up.  The base reward keeps energy weight at 0.01;
+raising it is not a shortcut.
+
+### Q5. Fixed-init vs random-init (algorithm-dependent)
+
+| exp | algo | init | env steps | best t_can | final t_can | best-checkpoint robustness |
+|---|---|---|---|---|---|---|
+| ppo_full_base_rand | PPO | random | 35.98M | **26.85 s** | 26.61 s | 8/8 (26.86 s) |
+| ppo_full_base_fixed | PPO | fixed | 21.93M | **1.19 s** | 0.00 s | 0/8 (mean return -99,380) |
+| sac_full_base_rand | SAC | random | 1.42M | **28.75 s** | 28.21 s | 6/8 (20.89 s) |
+| sac_full_base_fixed | SAC | fixed | 1.41M | **28.86 s** | 27.62 s | 8/8 (23.45 s) |
+
+PPO fixed-init **fails**, even at the exact evaluation state (best 1.19 s),
+and is catastrophic on the 8 distribution starts (mean return -99k).  PPO's
+on-policy batch contains 64 copies of one state, so it never sees the
+balance-recovery states it needs.
+
+SAC fixed-init **succeeds** (best 28.86 s, final 27.62 s, 8/8 on the
+distribution starts, first success at 0.52M).  SAC's replay buffer is filled
+with trajectories generated by its stochastic policy from that one start, so
+it does see off-state recovery data.  This is the opposite of the old
+`(tau1,0,tau3)` round, where fixed-init SAC failed and fixed-init PPO looked
+better at the fixed state.  The corrected internal-motor dynamics changes
+which algorithm can exploit a single start.
+
+### Seed variance (PPO full/base/rand)
+
+| seed | env steps | first success | best t_can | final t_can | best-checkpoint robustness |
+|---|---|---|---|---|---|
+| 0 | 35.98M | 11.33M | 26.85 s | 26.61 s | 8/8 (26.86 s) |
+| 1 | 29.03M | 13.63M | 27.17 s | 0.23 s | 8/8 (26.78 s) |
+| 2 | 34.21M | 14.40M | 28.46 s | 0.00 s | 8/8 (28.03 s) |
+
+All three seeds solve the task at their best checkpoint (8/8 robust), but the
+final checkpoint is unreliable (seed 2 final 0.00 s).  First success ranges
+11.3-14.4M steps (16-20 min).  **Always evaluate and ship `_best.pt`; the
+last model is not the policy.**
+
+## 8. Non-RL feasibility reference
+
+`reference_controller.py` → `results/reference_controller.json`:
+
+* Linearise at upright, pole placement with poles -5..-10, full state
+  (theta + simulator omega), torque command low-pass filtered with alpha=0.5.
+* Fixed start theta = pi + [0.013,-0.008,0.021], omega=0:
+  **t_can = 29.99 s**, final angle error ~2e-14, saturation 0%.
+* 32 random upright offsets U(-0.05,0.05): **32/32 success**, mean t_can
+  24.79 s.
+* Without the low-pass filter the same controller chatters and holds only
+  ~1.1 s; smooth torque commands matter for the 0.05 rad tolerance.
+
+This proves the upright hold is physically reachable with
+`(tau1,-tau3,tau3)` and |u|<=1.  It uses privileged omega and starts near
+upright, so it is not comparable to the RL numbers; it is an existence check
+and a hint that action smoothness matters.
+
+## 9. Failures / implementation notes (honest list)
+
+* PPO seed 0's results JSON was reconstructed by `finalize_run.py`.  The
+  trainer crashed **after** saving checkpoints and TB, in the post-training
+  robustness code (passed `ActorCritic` instead of `ActorCritic.actor` to
+  `make_ppo_policy`).  Training was unaffected.  The bug is fixed in
+  `train_ppo.py`; all later PPO runs wrote JSON directly.
+* The first energy preset produced unbounded rewards and a value-loss blow-up
+  (~5e7).  It was killed at 6.7M steps and archived under
+  `failed/energy_unbounded/`.  The second energy design is the one in the
+  matrix; it fails for a genuine reward-design reason, not a crash.
+* Sparse reward's no-op policy is a real exploration failure, not a logging
+  artefact: final mean |u| ~0.01, return -0.39, ever_up 0.
+* Some runs show value-loss spikes (PPO fixed, PPO angle) without NaNs; the
+  policies still train/plateau.  These are recorded in TB.
+* No simulator physics was changed.  The torque mapping was not reverted.
+
+## 10. Artifacts
+
+* TensorBoard: `tb/<exp>/` (live event files, all 15 runs).
+* Logs: `logs/<exp>.log` (stdout, contains eval lines and DONE summary).
+* Results: `results/<exp>.json`, aggregate `results/summary.json`,
+  `results/table.md`, `results/curves.png`, `results/curves_selected.png`.
+* Checkpoints: `checkpoints/<exp>_{best,last,final}.pt`.
+* Smoke tests: `smoke_archive/`; 6-min probe: `probes/`.
+* Failed/abandoned runs: `failed/energy_unbounded/`, `failed/energy_collapse/`.
+* Reference controller: `results/reference_controller.json`.
+
+## 11. Reproduction
+
+```bash
+cd feasibility/neural-pendulum-fixedenv
+export OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 TORCH_THREADS=4
+python3 test_contract.py
+python3 train_ppo.py --exp ppo_full_base_rand --obs full --reward base \
+  --init rand --envs 64 --horizon 600 --minutes 45 --eval-every 5 --seed 0
+python3 train_sac.py --exp sac_full_base_rand --obs full --reward base \
+  --init rand --envs 32 --minutes 50 --eval-every 20000 --log-every 10 --seed 0
+python3 summarize.py
+python3 plot_selected.py
+```
+
+## 12. Recommendations / next steps
+
+1. **For the actual C++ task, the controller should estimate velocity from
+   angles and use a 4-frame history (or an explicit recurrent state).**  The
+   3D observation is not enough by itself; the agent must reconstruct phase.
+   `full` 36D is the strongest configuration in this study.
+2. **Train from random near-down resets, not one fixed state.**  For PPO this
+   is the difference between success and failure; even for SAC, random init
+   is more robust across the distribution.
+3. **Use base reward; do not chase sparse or strong energy shaping.**  If a
+   sparse objective is required, add a curriculum or demonstration warm start;
+   pure sparse PPO does nothing in 19M steps.
+4. **Select checkpoints by eval t_can and ship the best, not the last.**
+   Final models collapse in 2 of 3 PPO seeds.
+5. **If more compute is available**, try (a) a recurrent/GRU policy on the 3D
+   angle stream, (b) an action low-pass filter or action-rate penalty, and
+   (c) a two-stage curriculum (balance near upright, then swing-up) with
+   warm-started PPO.  These target the two observed bottlenecks: velocity
+   estimation and balance-phase coverage.

package/assets/neural-pendulum/rl-feasibility/evaluate.py

@@ -0,0 +1,107 @@
+"""Deterministic evaluation at the fixed initial state (and optional starts).
+
+The evaluation uses the simulator API exactly like training (angles only), the
+same observation builder / obs mode, and the same trainer-side reward.  There
+is no action noise.
+
+    default fixed state: theta = [0.013, -0.008, 0.021] rad, omega = 0
+    optional starts    : drawn from the training reset distribution
+
+The episode always runs the full 3000 steps so that t_can, ever_up, return and
+the final angle error are comparable across algorithms.
+"""
+from __future__ import annotations
+
+import numpy as np
+import torch
+
+from obs import DT, make_numpy_obs, wrap_to_pi
+from sim import N_ACT, N_LINKS, UP_TOL, PendulumSim
+from task import EVAL_OMEGA0, EVAL_THETA0, NumpyTask
+
+
+def evaluate(policy_fn, p, episode_steps=3000, reward_w=None, obs_mode="full"):
+    """Run one full deterministic episode at the fixed initial state.
+
+    policy_fn(obs: np.ndarray[obs_dim]) -> np.ndarray[2]
+    """
+    sim = PendulumSim(p, episode_steps=episode_steps)
+    task = NumpyTask(p, reward_w=reward_w, obs_mode=obs_mode)
+    theta = sim.reset(EVAL_THETA0, EVAL_OMEGA0)
+    obs = task.reset(theta)
+    acts = np.zeros((episode_steps, N_ACT), dtype=np.float64)
+    for k in range(episode_steps):
+        u = np.clip(np.asarray(policy_fn(obs), dtype=np.float64).reshape(N_ACT),
+                    -1.0, 1.0)
+        acts[k] = u
+        theta = sim.step(u)
+        obs, _ = task.step(theta, u)
+    summ = task.tracker.summary()
+    a = wrap_to_pi(np.asarray(theta, dtype=np.float64) - np.pi)
+    return dict(t_can=float(summ["t_can"]),
+                success=float(summ["t_can"] >= 10.0),
+                ever_up=float(summ["ever_up"]),
+                first_up_t=float(summ["first_up_t"]),
+                ep_return=float(summ["ep_return"]),
+                final_ang_err=float(np.abs(a).max()),
+                up_frac=float(summ["up_frac"]),
+                mean_abs_action=float(np.mean(np.abs(acts))),
+                action_std=float(np.std(acts)),
+                mean_u1=float(acts[:, 0].mean()),
+                mean_u3=float(acts[:, 1].mean()),
+                steps=int(episode_steps))
+
+
+def evaluate_starts(policy_fn, p, starts, episode_steps=3000, reward_w=None,
+                    obs_mode="full"):
+    """Deterministic rollouts from an explicit list of (theta, omega) starts."""
+    rows = []
+    for theta0, omega0 in starts:
+        sim = PendulumSim(p, episode_steps=episode_steps)
+        task = NumpyTask(p, reward_w=reward_w, obs_mode=obs_mode)
+        theta = sim.reset(theta0, omega0)
+        obs = task.reset(theta)
+        for _ in range(episode_steps):
+            u = np.clip(np.asarray(policy_fn(obs), dtype=np.float64).reshape(N_ACT),
+                        -1.0, 1.0)
+            theta = sim.step(u)
+            obs, _ = task.step(theta, u)
+        summ = task.tracker.summary()
+        a = wrap_to_pi(np.asarray(theta, dtype=np.float64) - np.pi)
+        rows.append(dict(t_can=float(summ["t_can"]),
+                         success=float(summ["t_can"] >= 10.0),
+                         ever_up=float(summ["ever_up"]),
+                         ep_return=float(summ["ep_return"]),
+                         final_ang_err=float(np.abs(a).max())))
+    tc = np.array([r["t_can"] for r in rows], dtype=np.float64)
+    su = np.array([r["success"] for r in rows], dtype=np.float64)
+    re = np.array([r["ep_return"] for r in rows], dtype=np.float64)
+    ae = np.array([r["final_ang_err"] for r in rows], dtype=np.float64)
+    eu = np.array([r["ever_up"] for r in rows], dtype=np.float64)
+    return dict(n=int(len(rows)),
+                success_rate=float(su.mean()) if len(rows) else 0.0,
+                ever_up_rate=float(eu.mean()) if len(rows) else 0.0,
+                mean_t_can=float(tc.mean()) if len(rows) else 0.0,
+                median_t_can=float(np.median(tc)) if len(rows) else 0.0,
+                min_t_can=float(tc.min()) if len(rows) else 0.0,
+                max_t_can=float(tc.max()) if len(rows) else 0.0,
+                mean_return=float(re.mean()) if len(rows) else 0.0,
+                mean_final_ang_err=float(ae.mean()) if len(rows) else 0.0,
+                rows=rows)
+
+
+def make_ppo_policy(actor):
+    def policy(obs):
+        with torch.no_grad():
+            x = torch.as_tensor(obs, dtype=torch.float32).unsqueeze(0)
+            return actor(x).squeeze(0).numpy()
+    return policy
+
+
+def make_sac_policy(actor):
+    def policy(obs):
+        with torch.no_grad():
+            x = torch.as_tensor(obs, dtype=torch.float32).unsqueeze(0)
+            a, _ = actor.sample(x, det=True)
+            return a.squeeze(0).numpy()
+    return policy