dsh-aris-panel 0.2.7 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (857) hide show
  1. package/README.md +117 -108
  2. package/README_CN.md +98 -89
  3. package/aris/.claude-plugin/marketplace.json +34 -0
  4. package/aris/.claude-plugin/plugin.json +19 -0
  5. package/aris/.codex-plugin/plugin.json +18 -0
  6. package/aris/.env.example +53 -0
  7. package/aris/.gitattributes +6 -0
  8. package/aris/.github/agents/aris-reviewer-claude.agent.md +10 -0
  9. package/aris/.github/agents/aris-reviewer-openai.agent.md +10 -0
  10. package/aris/.github/workflows/check-skills-inventory.yml +32 -0
  11. package/aris/.github/workflows/lint-skills-helpers.yml +38 -0
  12. package/aris/.github/workflows/test-install-aris-ps1.yml +31 -0
  13. package/aris/.github/workflows/test-suite.yml +59 -0
  14. package/aris/AGENT_GUIDE.md +222 -0
  15. package/aris/CONTRIBUTING.md +83 -0
  16. package/aris/CONTRIBUTING_CN.md +83 -0
  17. package/aris/LICENSE +21 -0
  18. package/aris/README.md +1739 -0
  19. package/aris/README_CN.md +1526 -0
  20. package/aris/SETUP_GUIDE.md +279 -0
  21. package/aris/SETUP_GUIDE_CN.md +279 -0
  22. package/aris/UPSTREAM.txt +2 -0
  23. package/aris/aris-monitor/README.md +158 -0
  24. package/aris/aris-monitor/assets/screenshot.png +0 -0
  25. package/aris/aris-monitor/focus-tty.sh +156 -0
  26. package/aris/aris-monitor/focus.py +93 -0
  27. package/aris/aris-monitor/run.sh +49 -0
  28. package/aris/aris-monitor/scanner.py +445 -0
  29. package/aris/aris-monitor/ticker.py +81 -0
  30. package/aris/aris-monitor/widget.py +372 -0
  31. package/aris/assets/claude-fleet-preview.png +0 -0
  32. package/aris/assets/community_showcase_7_10_codex.png +0 -0
  33. package/aris/assets/community_showcase_8_10.png +0 -0
  34. package/aris/assets/feishu_interactive.jpg +0 -0
  35. package/aris/assets/feishu_push.png +0 -0
  36. package/aris/assets/paper_illustration_demo.png +0 -0
  37. package/aris/commands/setup.md +77 -0
  38. package/aris/community_papers/UAV-CC.pdf +0 -0
  39. package/aris/docs/ALI_CODING_PLAN_GUIDE.md +324 -0
  40. package/aris/docs/ANTIGRAVITY_ADAPTATION.md +409 -0
  41. package/aris/docs/ANTIGRAVITY_ADAPTATION_CN.md +363 -0
  42. package/aris/docs/ARIS-Code-README_CN.md +8 -0
  43. package/aris/docs/ARIS-Code-README_EN.md +8 -0
  44. package/aris/docs/ARIS_INTRO.html +902 -0
  45. package/aris/docs/ARIS_INTRO.md +368 -0
  46. package/aris/docs/ARIS_INTRO.review.json +79 -0
  47. package/aris/docs/CODEX_CLAUDE_REVIEW_GUIDE.md +116 -0
  48. package/aris/docs/CODEX_CLAUDE_REVIEW_GUIDE_CN.md +116 -0
  49. package/aris/docs/CODEX_GEMINI_REVIEW_GUIDE.md +249 -0
  50. package/aris/docs/CODEX_GEMINI_REVIEW_GUIDE_CN.md +250 -0
  51. package/aris/docs/COPILOT_CLI_ADAPTATION.md +388 -0
  52. package/aris/docs/CURSOR_ADAPTATION.md +305 -0
  53. package/aris/docs/CUSTOMIZATION.md +155 -0
  54. package/aris/docs/CUSTOMIZATION_CN.md +154 -0
  55. package/aris/docs/GPU_SETUP.md +57 -0
  56. package/aris/docs/GPU_SETUP_CN.md +54 -0
  57. package/aris/docs/LLM_API_MIX_MATCH_GUIDE.md +256 -0
  58. package/aris/docs/MANUAL_REVIEW_GUIDE.md +128 -0
  59. package/aris/docs/MANUAL_REVIEW_GUIDE_CN.md +116 -0
  60. package/aris/docs/MINIMAX_MCP_GUIDE.md +198 -0
  61. package/aris/docs/MODELSCOPE_GUIDE.md +305 -0
  62. package/aris/docs/MODEL_COMBINATIONS.md +134 -0
  63. package/aris/docs/MODEL_COMBINATIONS_CN.md +134 -0
  64. package/aris/docs/MiniMax-GLM-Configuration.md +195 -0
  65. package/aris/docs/NARRATIVE_REPORT_EXAMPLE.md +143 -0
  66. package/aris/docs/OPENCLAW_ADAPTATION.md +59 -0
  67. package/aris/docs/OPENROUTER_GUIDE.md +255 -0
  68. package/aris/docs/OPENROUTER_GUIDE_CN.md +255 -0
  69. package/aris/docs/PROJECT_FILES_GUIDE.md +161 -0
  70. package/aris/docs/PROJECT_FILES_GUIDE_CN.md +161 -0
  71. package/aris/docs/RESUBMIT_AND_TALK.md +91 -0
  72. package/aris/docs/RESUBMIT_AND_TALK_CN.md +91 -0
  73. package/aris/docs/SESSION_RECOVERY_GUIDE.md +439 -0
  74. package/aris/docs/SESSION_RECOVERY_GUIDE_CN.md +431 -0
  75. package/aris/docs/SKILLS_CATALOG.md +204 -0
  76. package/aris/docs/TRAE_ARIS_RUNBOOK_CN.md +329 -0
  77. package/aris/docs/TRAE_ARIS_RUNBOOK_EN.md +329 -0
  78. package/aris/docs/WATCHDOG_GUIDE.md +200 -0
  79. package/aris/docs/WATCHDOG_GUIDE_CN.md +199 -0
  80. package/aris/docs/aris-code-banner.png +0 -0
  81. package/aris/docs/aris-code-screenshot.png +0 -0
  82. package/aris/docs/aris-movie-director-method.png +0 -0
  83. package/aris/docs/aris_intro_slides.pdf +0 -0
  84. package/aris/docs/aris_logo.svg +126 -0
  85. package/aris/docs/auto_review_score_curve.png +0 -0
  86. package/aris/docs/blogs/continuous_dlm_2026h1_survey.html +3262 -0
  87. package/aris/docs/hero_combined.svg +844 -0
  88. package/aris/docs/hero_idea.svg +158 -0
  89. package/aris/docs/hero_implement.svg +109 -0
  90. package/aris/docs/hero_interaction.svg +172 -0
  91. package/aris/docs/hf_daily_paper_1.svg +19 -0
  92. package/aris/docs/integrations/FEISHU.md +208 -0
  93. package/aris/docs/integrations/FEISHU_CN.md +208 -0
  94. package/aris/docs/integrations/OBSIDIAN.md +65 -0
  95. package/aris/docs/integrations/OBSIDIAN_CN.md +65 -0
  96. package/aris/docs/integrations/VAST_GPU_GUIDE.md +96 -0
  97. package/aris/docs/integrations/VAST_GPU_GUIDE_CN.md +96 -0
  98. package/aris/docs/integrations/ZOTERO.md +57 -0
  99. package/aris/docs/integrations/ZOTERO_CN.md +57 -0
  100. package/aris/docs/tutorials/3d_generation_tutorial.html +1018 -0
  101. package/aris/docs/tutorials/3d_generation_tutorial.md +1206 -0
  102. package/aris/docs/tutorials/3d_generation_tutorial.review.json +34 -0
  103. package/aris/docs/tutorials/3d_generation_tutorial_en.html +1018 -0
  104. package/aris/docs/tutorials/3d_generation_tutorial_en.md +1206 -0
  105. package/aris/docs/tutorials/README.md +77 -0
  106. package/aris/docs/tutorials/agent_foundations_tutorial.html +1081 -0
  107. package/aris/docs/tutorials/agent_foundations_tutorial.md +1229 -0
  108. package/aris/docs/tutorials/agent_foundations_tutorial.review.json +146 -0
  109. package/aris/docs/tutorials/agent_foundations_tutorial_en.html +1081 -0
  110. package/aris/docs/tutorials/agent_foundations_tutorial_en.md +1229 -0
  111. package/aris/docs/tutorials/agentic_rl_tutorial.html +1269 -0
  112. package/aris/docs/tutorials/agentic_rl_tutorial.md +1544 -0
  113. package/aris/docs/tutorials/agentic_rl_tutorial.review.json +74 -0
  114. package/aris/docs/tutorials/agentic_rl_tutorial_en.html +1269 -0
  115. package/aris/docs/tutorials/agentic_rl_tutorial_en.md +1544 -0
  116. package/aris/docs/tutorials/attention_tutorial.html +774 -0
  117. package/aris/docs/tutorials/attention_tutorial.md +812 -0
  118. package/aris/docs/tutorials/attention_tutorial.review.json +51 -0
  119. package/aris/docs/tutorials/attention_tutorial_en.html +774 -0
  120. package/aris/docs/tutorials/attention_tutorial_en.md +812 -0
  121. package/aris/docs/tutorials/diffusion_distillation_tutorial.html +1421 -0
  122. package/aris/docs/tutorials/diffusion_distillation_tutorial.md +1672 -0
  123. package/aris/docs/tutorials/diffusion_distillation_tutorial.review.json +114 -0
  124. package/aris/docs/tutorials/diffusion_distillation_tutorial_en.html +1421 -0
  125. package/aris/docs/tutorials/diffusion_distillation_tutorial_en.md +1672 -0
  126. package/aris/docs/tutorials/diffusion_foundations_tutorial.html +1239 -0
  127. package/aris/docs/tutorials/diffusion_foundations_tutorial.md +1481 -0
  128. package/aris/docs/tutorials/diffusion_foundations_tutorial.review.json +56 -0
  129. package/aris/docs/tutorials/diffusion_foundations_tutorial_en.html +1239 -0
  130. package/aris/docs/tutorials/diffusion_foundations_tutorial_en.md +1481 -0
  131. package/aris/docs/tutorials/diffusion_post_training_tutorial.html +1135 -0
  132. package/aris/docs/tutorials/diffusion_post_training_tutorial.md +1322 -0
  133. package/aris/docs/tutorials/diffusion_post_training_tutorial.review.json +63 -0
  134. package/aris/docs/tutorials/diffusion_post_training_tutorial_en.html +1138 -0
  135. package/aris/docs/tutorials/diffusion_post_training_tutorial_en.md +1325 -0
  136. package/aris/docs/tutorials/distributed_training_tutorial.html +1429 -0
  137. package/aris/docs/tutorials/distributed_training_tutorial.md +1587 -0
  138. package/aris/docs/tutorials/distributed_training_tutorial.review.json +98 -0
  139. package/aris/docs/tutorials/distributed_training_tutorial_en.html +1429 -0
  140. package/aris/docs/tutorials/distributed_training_tutorial_en.md +1587 -0
  141. package/aris/docs/tutorials/flow_matching_tutorial.html +755 -0
  142. package/aris/docs/tutorials/flow_matching_tutorial.md +537 -0
  143. package/aris/docs/tutorials/flow_matching_tutorial.review.json +37 -0
  144. package/aris/docs/tutorials/flow_matching_tutorial_en.html +755 -0
  145. package/aris/docs/tutorials/flow_matching_tutorial_en.md +537 -0
  146. package/aris/docs/tutorials/image_generation_systems_tutorial.html +1181 -0
  147. package/aris/docs/tutorials/image_generation_systems_tutorial.md +1448 -0
  148. package/aris/docs/tutorials/image_generation_systems_tutorial.review.json +49 -0
  149. package/aris/docs/tutorials/image_generation_systems_tutorial_en.html +1181 -0
  150. package/aris/docs/tutorials/image_generation_systems_tutorial_en.md +1448 -0
  151. package/aris/docs/tutorials/kl_divergence_rlhf_tutorial.html +1210 -0
  152. package/aris/docs/tutorials/kl_divergence_rlhf_tutorial.md +1502 -0
  153. package/aris/docs/tutorials/kl_divergence_rlhf_tutorial.review.json +55 -0
  154. package/aris/docs/tutorials/kl_divergence_rlhf_tutorial_en.html +1212 -0
  155. package/aris/docs/tutorials/kl_divergence_rlhf_tutorial_en.md +1504 -0
  156. package/aris/docs/tutorials/kv_cache_speculative_decoding_tutorial.html +1118 -0
  157. package/aris/docs/tutorials/kv_cache_speculative_decoding_tutorial.md +1258 -0
  158. package/aris/docs/tutorials/kv_cache_speculative_decoding_tutorial.review.json +91 -0
  159. package/aris/docs/tutorials/kv_cache_speculative_decoding_tutorial_en.html +1118 -0
  160. package/aris/docs/tutorials/kv_cache_speculative_decoding_tutorial_en.md +1258 -0
  161. package/aris/docs/tutorials/llm_opd_tutorial.html +1109 -0
  162. package/aris/docs/tutorials/llm_opd_tutorial.md +1195 -0
  163. package/aris/docs/tutorials/llm_opd_tutorial.review.json +125 -0
  164. package/aris/docs/tutorials/llm_opd_tutorial_en.html +1112 -0
  165. package/aris/docs/tutorials/llm_opd_tutorial_en.md +1198 -0
  166. package/aris/docs/tutorials/long_context_rope_yarn_mla_tutorial.html +1130 -0
  167. package/aris/docs/tutorials/long_context_rope_yarn_mla_tutorial.md +1209 -0
  168. package/aris/docs/tutorials/long_context_rope_yarn_mla_tutorial.review.json +79 -0
  169. package/aris/docs/tutorials/long_context_rope_yarn_mla_tutorial_en.html +1132 -0
  170. package/aris/docs/tutorials/long_context_rope_yarn_mla_tutorial_en.md +1211 -0
  171. package/aris/docs/tutorials/moe_tutorial.html +968 -0
  172. package/aris/docs/tutorials/moe_tutorial.md +1031 -0
  173. package/aris/docs/tutorials/moe_tutorial.review.json +54 -0
  174. package/aris/docs/tutorials/moe_tutorial_en.html +968 -0
  175. package/aris/docs/tutorials/moe_tutorial_en.md +1031 -0
  176. package/aris/docs/tutorials/multi_agent_long_horizon_tutorial.html +1409 -0
  177. package/aris/docs/tutorials/multi_agent_long_horizon_tutorial.md +1803 -0
  178. package/aris/docs/tutorials/multi_agent_long_horizon_tutorial.review.json +62 -0
  179. package/aris/docs/tutorials/multi_agent_long_horizon_tutorial_en.html +1413 -0
  180. package/aris/docs/tutorials/multi_agent_long_horizon_tutorial_en.md +1807 -0
  181. package/aris/docs/tutorials/quantization_tutorial.html +1117 -0
  182. package/aris/docs/tutorials/quantization_tutorial.md +1344 -0
  183. package/aris/docs/tutorials/quantization_tutorial.review.json +56 -0
  184. package/aris/docs/tutorials/quantization_tutorial_en.html +1117 -0
  185. package/aris/docs/tutorials/quantization_tutorial_en.md +1344 -0
  186. package/aris/docs/tutorials/reasoning_models_tutorial.html +1085 -0
  187. package/aris/docs/tutorials/reasoning_models_tutorial.md +1285 -0
  188. package/aris/docs/tutorials/reasoning_models_tutorial.review.json +66 -0
  189. package/aris/docs/tutorials/reasoning_models_tutorial_en.html +1085 -0
  190. package/aris/docs/tutorials/reasoning_models_tutorial_en.md +1285 -0
  191. package/aris/docs/tutorials/rlhf_dpo_grpo_ppo_tutorial.html +1015 -0
  192. package/aris/docs/tutorials/rlhf_dpo_grpo_ppo_tutorial.md +1125 -0
  193. package/aris/docs/tutorials/rlhf_dpo_grpo_ppo_tutorial.review.json +69 -0
  194. package/aris/docs/tutorials/rlhf_dpo_grpo_ppo_tutorial_en.html +1014 -0
  195. package/aris/docs/tutorials/rlhf_dpo_grpo_ppo_tutorial_en.md +1123 -0
  196. package/aris/docs/tutorials/self_evolving_agents_tutorial.html +1354 -0
  197. package/aris/docs/tutorials/self_evolving_agents_tutorial.md +1554 -0
  198. package/aris/docs/tutorials/self_evolving_agents_tutorial.review.json +64 -0
  199. package/aris/docs/tutorials/self_evolving_agents_tutorial_en.html +1356 -0
  200. package/aris/docs/tutorials/self_evolving_agents_tutorial_en.md +1556 -0
  201. package/aris/docs/tutorials/vae_vqvae_vqgan_tutorial.html +1197 -0
  202. package/aris/docs/tutorials/vae_vqvae_vqgan_tutorial.md +1419 -0
  203. package/aris/docs/tutorials/vae_vqvae_vqgan_tutorial.review.json +67 -0
  204. package/aris/docs/tutorials/vae_vqvae_vqgan_tutorial_en.html +1196 -0
  205. package/aris/docs/tutorials/vae_vqvae_vqgan_tutorial_en.md +1419 -0
  206. package/aris/docs/tutorials/video_generation_tutorial.html +1169 -0
  207. package/aris/docs/tutorials/video_generation_tutorial.md +1189 -0
  208. package/aris/docs/tutorials/video_generation_tutorial.review.json +72 -0
  209. package/aris/docs/tutorials/video_generation_tutorial_en.html +1170 -0
  210. package/aris/docs/tutorials/video_generation_tutorial_en.md +1190 -0
  211. package/aris/docs/tutorials/vlm_multimodal_tutorial.html +1208 -0
  212. package/aris/docs/tutorials/vlm_multimodal_tutorial.md +1446 -0
  213. package/aris/docs/tutorials/vlm_multimodal_tutorial.review.json +67 -0
  214. package/aris/docs/tutorials/vlm_multimodal_tutorial_en.html +1208 -0
  215. package/aris/docs/tutorials/vlm_multimodal_tutorial_en.md +1446 -0
  216. package/aris/docs/wechat_group.jpg +0 -0
  217. package/aris/mcp-servers/claude-review/README.md +96 -0
  218. package/aris/mcp-servers/claude-review/run_with_claude_aws.sh +49 -0
  219. package/aris/mcp-servers/claude-review/server.py +718 -0
  220. package/aris/mcp-servers/codex-exec/README.md +57 -0
  221. package/aris/mcp-servers/codex-exec/server.py +480 -0
  222. package/aris/mcp-servers/codex-image2/README.md +65 -0
  223. package/aris/mcp-servers/codex-image2/server.py +893 -0
  224. package/aris/mcp-servers/feishu-bridge/requirements.txt +1 -0
  225. package/aris/mcp-servers/feishu-bridge/server.py +240 -0
  226. package/aris/mcp-servers/gemini-review/README.md +171 -0
  227. package/aris/mcp-servers/gemini-review/server.py +1856 -0
  228. package/aris/mcp-servers/llm-chat/requirements.txt +1 -0
  229. package/aris/mcp-servers/llm-chat/server.py +664 -0
  230. package/aris/mcp-servers/manual-review/README.md +133 -0
  231. package/aris/mcp-servers/manual-review/server.py +910 -0
  232. package/aris/mcp-servers/manual-review/ui.html +279 -0
  233. package/aris/mcp-servers/minimax-chat/requirements.txt +1 -0
  234. package/aris/mcp-servers/minimax-chat/server.py +381 -0
  235. package/aris/skills/ablation-planner/SKILL.md +123 -0
  236. package/aris/skills/alphaxiv/SKILL.md +196 -0
  237. package/aris/skills/analyze-results/SKILL.md +46 -0
  238. package/aris/skills/arxiv/SKILL.md +248 -0
  239. package/aris/skills/auto-paper-improvement-loop/SKILL.md +695 -0
  240. package/aris/skills/auto-review-loop/SKILL.md +1137 -0
  241. package/aris/skills/auto-review-loop-llm/SKILL.md +259 -0
  242. package/aris/skills/auto-review-loop-minimax/SKILL.md +302 -0
  243. package/{skills/citation-audit-claude → aris/skills/citation-audit}/SKILL.md +11 -21
  244. package/aris/skills/claims-drafting/SKILL.md +227 -0
  245. package/aris/skills/comm-lit-review/SKILL.md +297 -0
  246. package/aris/skills/deepxiv/SKILL.md +263 -0
  247. package/aris/skills/dse-loop/SKILL.md +296 -0
  248. package/aris/skills/embodiment-description/SKILL.md +129 -0
  249. package/aris/skills/exa-search/SKILL.md +205 -0
  250. package/{skills/experiment-audit-claude → aris/skills/experiment-audit}/SKILL.md +16 -26
  251. package/aris/skills/experiment-bridge/SKILL.md +376 -0
  252. package/aris/skills/experiment-plan/SKILL.md +249 -0
  253. package/aris/skills/experiment-queue/SKILL.md +431 -0
  254. package/aris/skills/experiment-queue/scripts/build_manifest.py +142 -0
  255. package/aris/skills/experiment-queue/scripts/queue_manager.py +433 -0
  256. package/aris/skills/feishu-notify/SKILL.md +156 -0
  257. package/aris/skills/figure-description/SKILL.md +138 -0
  258. package/aris/skills/figure-spec/SKILL.md +262 -0
  259. package/aris/skills/figure-spec/scripts/figure_renderer.py +799 -0
  260. package/aris/skills/formula-derivation/SKILL.md +280 -0
  261. package/aris/skills/gemini-search/SKILL.md +231 -0
  262. package/aris/skills/grant-proposal/SKILL.md +698 -0
  263. package/aris/skills/idea-creator/SKILL.md +584 -0
  264. package/aris/skills/idea-discovery/SKILL.md +527 -0
  265. package/aris/skills/idea-discovery-robot/SKILL.md +363 -0
  266. package/aris/skills/integrity-forensics/SKILL.md +284 -0
  267. package/aris/skills/interview-cheatsheet/SKILL.md +245 -0
  268. package/aris/skills/invention-structuring/SKILL.md +188 -0
  269. package/aris/skills/jurisdiction-format/SKILL.md +192 -0
  270. package/aris/skills/kill-argument/SKILL.md +437 -0
  271. package/aris/skills/mermaid-diagram/SKILL.md +419 -0
  272. package/{skills/meta-apply-claude → aris/skills/meta-apply}/SKILL.md +5 -15
  273. package/aris/skills/meta-optimize/SKILL.md +437 -0
  274. package/aris/skills/monitor-experiment/SKILL.md +140 -0
  275. package/aris/skills/novelty-check/SKILL.md +142 -0
  276. package/aris/skills/openalex/SKILL.md +237 -0
  277. package/aris/skills/overleaf-sync/SKILL.md +220 -0
  278. package/{skills/paper-claim-audit-claude → aris/skills/paper-claim-audit}/SKILL.md +13 -23
  279. package/aris/skills/paper-compile/SKILL.md +266 -0
  280. package/aris/skills/paper-figure/SKILL.md +312 -0
  281. package/aris/skills/paper-illustration/SKILL.md +736 -0
  282. package/aris/skills/paper-illustration-image2/SKILL.md +391 -0
  283. package/aris/skills/paper-illustration-image2/scripts/paper_illustration_image2.py +255 -0
  284. package/aris/skills/paper-plan/SKILL.md +386 -0
  285. package/aris/skills/paper-poster/SKILL.md +19 -0
  286. package/aris/skills/paper-poster-html/DESIGN_FINAL.md +176 -0
  287. package/aris/skills/paper-poster-html/IMPLEMENTATION_CONVENTIONS.md +161 -0
  288. package/aris/skills/paper-poster-html/LICENSES/posterly-MIT.txt +21 -0
  289. package/aris/skills/paper-poster-html/NOTICE.md +57 -0
  290. package/aris/skills/paper-poster-html/SKILL.md +323 -0
  291. package/aris/skills/paper-poster-html/scripts/_posterly/__init__.py +0 -0
  292. package/aris/skills/paper-poster-html/scripts/_posterly/canvas.py +200 -0
  293. package/aris/skills/paper-poster-html/scripts/_posterly/measure.py +588 -0
  294. package/aris/skills/paper-poster-html/scripts/_posterly/polish.py +498 -0
  295. package/aris/skills/paper-poster-html/scripts/_posterly/preflight.py +489 -0
  296. package/aris/skills/paper-poster-html/scripts/_posterly/render.py +215 -0
  297. package/aris/skills/paper-poster-html/scripts/_posterly/textutil.py +16 -0
  298. package/aris/skills/paper-poster-html/scripts/_posterly/verify_final.py +171 -0
  299. package/aris/skills/paper-poster-html/scripts/asset_check.py +897 -0
  300. package/aris/skills/paper-poster-html/scripts/extract_pdf_figures.py +666 -0
  301. package/aris/skills/paper-poster-html/scripts/poster_check.py +251 -0
  302. package/aris/skills/paper-poster-html/scripts/preprocess_figures.py +238 -0
  303. package/aris/skills/paper-poster-html/scripts/render_preview.py +217 -0
  304. package/aris/skills/paper-poster-html/scripts/run_gates.py +556 -0
  305. package/aris/skills/paper-poster-html/scripts/style_check.py +1324 -0
  306. package/aris/skills/paper-poster-html/templates/COMPONENTS.md +462 -0
  307. package/aris/skills/paper-poster-html/templates/README.md +170 -0
  308. package/aris/skills/paper-poster-html/templates/landscape_4col.html +1032 -0
  309. package/aris/skills/paper-poster-html/templates/landscape_hero.html +1046 -0
  310. package/aris/skills/paper-poster-html/templates/portrait_2col.html +947 -0
  311. package/aris/skills/paper-poster-html/templates/tokens/acl.json +9 -0
  312. package/aris/skills/paper-poster-html/templates/tokens/cvpr.json +9 -0
  313. package/aris/skills/paper-poster-html/templates/tokens/generic.json +9 -0
  314. package/aris/skills/paper-poster-html/templates/tokens/iclr.json +9 -0
  315. package/aris/skills/paper-poster-html/templates/tokens/icml.json +9 -0
  316. package/aris/skills/paper-poster-html/templates/tokens/neurips.json +9 -0
  317. package/aris/skills/paper-slides/SKILL.md +635 -0
  318. package/aris/skills/paper-talk/SKILL.md +381 -0
  319. package/aris/skills/paper-write/SKILL.md +682 -0
  320. package/aris/skills/paper-write/templates/IEEEtran.bst +2409 -0
  321. package/aris/skills/paper-write/templates/IEEEtran.cls +6347 -0
  322. package/aris/skills/paper-write/templates/iclr2026.tex +84 -0
  323. package/aris/skills/paper-write/templates/icml2025.tex +87 -0
  324. package/aris/skills/paper-write/templates/ieee_conference.tex +89 -0
  325. package/aris/skills/paper-write/templates/ieee_journal.tex +93 -0
  326. package/aris/skills/paper-write/templates/math_commands.tex +48 -0
  327. package/aris/skills/paper-write/templates/neurips2025.tex +80 -0
  328. package/aris/skills/paper-writing/SKILL.md +929 -0
  329. package/aris/skills/patent-novelty-check/SKILL.md +153 -0
  330. package/aris/skills/patent-pipeline/SKILL.md +344 -0
  331. package/aris/skills/patent-review/SKILL.md +203 -0
  332. package/aris/skills/pixel-art/SKILL.md +137 -0
  333. package/aris/skills/prior-art-search/SKILL.md +146 -0
  334. package/aris/skills/proof-checker/SKILL.md +866 -0
  335. package/aris/skills/proof-orchestrator/NOTICE.md +24 -0
  336. package/aris/skills/proof-orchestrator/SKILL.md +254 -0
  337. package/aris/skills/proof-orchestrator/references/audit-output-contract.md +126 -0
  338. package/aris/skills/proof-orchestrator/references/deepseek-routing.md +74 -0
  339. package/aris/skills/proof-orchestrator/references/dispatch-prompts.md +227 -0
  340. package/aris/skills/proof-orchestrator/references/notation-audit.md +135 -0
  341. package/aris/skills/proof-orchestrator/references/proof-audit-rubric.md +70 -0
  342. package/aris/skills/proof-orchestrator/references/stress-tests.md +38 -0
  343. package/aris/skills/proof-writer/SKILL.md +223 -0
  344. package/aris/skills/qzcli/SKILL.md +324 -0
  345. package/aris/skills/rebuttal/SKILL.md +376 -0
  346. package/aris/skills/render-html/SKILL.md +316 -0
  347. package/aris/skills/render-html/scripts/render_html.py +1006 -0
  348. package/aris/skills/render-html/scripts/templates/academic.html +703 -0
  349. package/aris/skills/render-html/scripts/templates/dashboard.html +333 -0
  350. package/aris/skills/research-implement-feature/SKILL.md +583 -0
  351. package/aris/skills/research-lit/SKILL.md +756 -0
  352. package/aris/skills/research-pipeline/SKILL.md +390 -0
  353. package/aris/skills/research-refine/SKILL.md +770 -0
  354. package/aris/skills/research-refine-pipeline/SKILL.md +186 -0
  355. package/aris/skills/research-review/SKILL.md +223 -0
  356. package/aris/skills/research-wiki/SKILL.md +461 -0
  357. package/aris/skills/resubmit-pipeline/SKILL.md +447 -0
  358. package/{skills/result-to-claim-claude → aris/skills/result-to-claim}/SKILL.md +7 -17
  359. package/aris/skills/run-experiment/SKILL.md +313 -0
  360. package/aris/skills/semantic-scholar/SKILL.md +236 -0
  361. package/aris/skills/serverless-modal/SKILL.md +335 -0
  362. package/aris/skills/shared-references/acceptance-gate.md +324 -0
  363. package/aris/skills/shared-references/assurance-contract.md +248 -0
  364. package/aris/skills/shared-references/capture-antipatterns.md +78 -0
  365. package/aris/skills/shared-references/citation-discipline.md +583 -0
  366. package/aris/skills/shared-references/compute-env-contract.md +163 -0
  367. package/aris/skills/shared-references/effort-contract.md +183 -0
  368. package/aris/skills/shared-references/evidence-precheck.md +65 -0
  369. package/aris/skills/shared-references/experiment-integrity.md +49 -0
  370. package/aris/skills/shared-references/external-cadence.md +326 -0
  371. package/aris/skills/shared-references/fan-out-pattern.md +366 -0
  372. package/aris/skills/shared-references/injection-hygiene.md +127 -0
  373. package/aris/skills/shared-references/integration-contract.md +461 -0
  374. package/aris/skills/shared-references/output-composition.md +93 -0
  375. package/aris/skills/shared-references/output-language.md +45 -0
  376. package/aris/skills/shared-references/output-manifest.md +49 -0
  377. package/aris/skills/shared-references/output-versioning.md +111 -0
  378. package/aris/skills/shared-references/patent-format-cn.md +199 -0
  379. package/aris/skills/shared-references/patent-format-ep.md +173 -0
  380. package/aris/skills/shared-references/patent-format-us.md +161 -0
  381. package/aris/skills/shared-references/patent-writing-principles.md +197 -0
  382. package/aris/skills/shared-references/prior-art-databases.md +141 -0
  383. package/aris/skills/shared-references/resumable-runs.md +109 -0
  384. package/aris/skills/shared-references/review-scope-limits.md +81 -0
  385. package/aris/skills/shared-references/review-tracing.md +391 -0
  386. package/aris/skills/shared-references/reviewer-independence.md +79 -0
  387. package/aris/skills/shared-references/reviewer-routing.md +852 -0
  388. package/aris/skills/shared-references/skill-governance.md +104 -0
  389. package/aris/skills/shared-references/taste-calibration.md +85 -0
  390. package/aris/skills/shared-references/venue-checklists.md +114 -0
  391. package/aris/skills/shared-references/wiki-helper-resolution.md +134 -0
  392. package/aris/skills/shared-references/writing-principles.md +525 -0
  393. package/aris/skills/skills-codex/README.md +102 -0
  394. package/aris/skills/skills-codex/README_CN.md +100 -0
  395. package/aris/skills/skills-codex/ablation-planner/SKILL.md +126 -0
  396. package/aris/skills/skills-codex/alphaxiv/SKILL.md +186 -0
  397. package/aris/skills/skills-codex/analyze-results/SKILL.md +45 -0
  398. package/aris/skills/skills-codex/arxiv/SKILL.md +210 -0
  399. package/aris/skills/skills-codex/auto-paper-improvement-loop/SKILL.md +618 -0
  400. package/aris/skills/skills-codex/auto-review-loop/SKILL.md +500 -0
  401. package/aris/skills/skills-codex/auto-review-loop-llm/SKILL.md +247 -0
  402. package/aris/skills/skills-codex/auto-review-loop-minimax/SKILL.md +290 -0
  403. package/aris/skills/skills-codex/citation-audit/SKILL.md +504 -0
  404. package/aris/skills/skills-codex/claims-drafting/SKILL.md +239 -0
  405. package/aris/skills/skills-codex/comm-lit-review/SKILL.md +299 -0
  406. package/aris/skills/skills-codex/comm-lit-review/references/domain-taxonomy.md +57 -0
  407. package/aris/skills/skills-codex/comm-lit-review/references/output-template.md +37 -0
  408. package/aris/skills/skills-codex/comm-lit-review/references/source-policy.md +99 -0
  409. package/aris/skills/skills-codex/comm-lit-review/references/venue-tiering.md +112 -0
  410. package/aris/skills/skills-codex/deepxiv/SKILL.md +142 -0
  411. package/aris/skills/skills-codex/dse-loop/SKILL.md +285 -0
  412. package/aris/skills/skills-codex/embodiment-description/SKILL.md +129 -0
  413. package/aris/skills/skills-codex/exa-search/SKILL.md +192 -0
  414. package/aris/skills/skills-codex/experiment-audit/SKILL.md +286 -0
  415. package/aris/skills/skills-codex/experiment-bridge/SKILL.md +356 -0
  416. package/aris/skills/skills-codex/experiment-plan/SKILL.md +249 -0
  417. package/aris/skills/skills-codex/experiment-queue/SKILL.md +401 -0
  418. package/aris/skills/skills-codex/feishu-notify/SKILL.md +155 -0
  419. package/aris/skills/skills-codex/figure-description/SKILL.md +138 -0
  420. package/aris/skills/skills-codex/figure-spec/SKILL.md +252 -0
  421. package/aris/skills/skills-codex/formula-derivation/SKILL.md +280 -0
  422. package/aris/skills/skills-codex/gemini-search/SKILL.md +205 -0
  423. package/aris/skills/skills-codex/grant-proposal/SKILL.md +626 -0
  424. package/aris/skills/skills-codex/idea-creator/SKILL.md +438 -0
  425. package/aris/skills/skills-codex/idea-discovery/SKILL.md +477 -0
  426. package/aris/skills/skills-codex/idea-discovery-robot/SKILL.md +362 -0
  427. package/aris/skills/skills-codex/integrity-forensics/SKILL.md +108 -0
  428. package/aris/skills/skills-codex/interview-cheatsheet/SKILL.md +245 -0
  429. package/aris/skills/skills-codex/invention-structuring/SKILL.md +188 -0
  430. package/aris/skills/skills-codex/jurisdiction-format/SKILL.md +192 -0
  431. package/aris/skills/skills-codex/kill-argument/SKILL.md +403 -0
  432. package/aris/skills/skills-codex/mermaid-diagram/SKILL.md +379 -0
  433. package/aris/skills/skills-codex/meta-apply/SKILL.md +154 -0
  434. package/aris/skills/skills-codex/meta-optimize/SKILL.md +348 -0
  435. package/aris/skills/skills-codex/monitor-experiment/SKILL.md +98 -0
  436. package/aris/skills/skills-codex/novelty-check/SKILL.md +130 -0
  437. package/aris/skills/skills-codex/openalex/SKILL.md +228 -0
  438. package/aris/skills/skills-codex/overleaf-sync/SKILL.md +220 -0
  439. package/aris/skills/skills-codex/paper-claim-audit/SKILL.md +350 -0
  440. package/aris/skills/skills-codex/paper-compile/SKILL.md +253 -0
  441. package/aris/skills/skills-codex/paper-figure/SKILL.md +311 -0
  442. package/aris/skills/skills-codex/paper-illustration/SKILL.md +690 -0
  443. package/aris/skills/skills-codex/paper-illustration-image2/SKILL.md +383 -0
  444. package/aris/skills/skills-codex/paper-illustration-image2/scripts/paper_illustration_image2.py +255 -0
  445. package/aris/skills/skills-codex/paper-plan/SKILL.md +278 -0
  446. package/aris/skills/skills-codex/paper-poster/SKILL.md +19 -0
  447. package/aris/skills/skills-codex/paper-poster-html/SKILL.md +377 -0
  448. package/aris/skills/skills-codex/paper-slides/SKILL.md +571 -0
  449. package/aris/skills/skills-codex/paper-talk/SKILL.md +381 -0
  450. package/aris/skills/skills-codex/paper-write/SKILL.md +487 -0
  451. package/aris/skills/skills-codex/paper-write/templates/IEEEtran.bst +2409 -0
  452. package/aris/skills/skills-codex/paper-write/templates/IEEEtran.cls +6347 -0
  453. package/aris/skills/skills-codex/paper-write/templates/aaai2026.bst +1493 -0
  454. package/aris/skills/skills-codex/paper-write/templates/aaai2026.sty +315 -0
  455. package/aris/skills/skills-codex/paper-write/templates/aaai2026.tex +952 -0
  456. package/aris/skills/skills-codex/paper-write/templates/acl.sty +312 -0
  457. package/aris/skills/skills-codex/paper-write/templates/acl2026.tex +377 -0
  458. package/aris/skills/skills-codex/paper-write/templates/acl_natbib.bst +1940 -0
  459. package/aris/skills/skills-codex/paper-write/templates/acm.bst +3081 -0
  460. package/aris/skills/skills-codex/paper-write/templates/acm_mm2026.tex +204 -0
  461. package/aris/skills/skills-codex/paper-write/templates/acmart.cls +3520 -0
  462. package/aris/skills/skills-codex/paper-write/templates/cvpr.bst +1448 -0
  463. package/aris/skills/skills-codex/paper-write/templates/cvpr.sty +508 -0
  464. package/aris/skills/skills-codex/paper-write/templates/cvpr2026.tex +63 -0
  465. package/aris/skills/skills-codex/paper-write/templates/iclr2026.tex +84 -0
  466. package/aris/skills/skills-codex/paper-write/templates/iclr2026_conference.bst +1440 -0
  467. package/aris/skills/skills-codex/paper-write/templates/iclr2026_conference.sty +246 -0
  468. package/aris/skills/skills-codex/paper-write/templates/icml2026.sty +767 -0
  469. package/aris/skills/skills-codex/paper-write/templates/icml2026.tex +662 -0
  470. package/aris/skills/skills-codex/paper-write/templates/ieee_conference.tex +89 -0
  471. package/aris/skills/skills-codex/paper-write/templates/ieee_journal.tex +93 -0
  472. package/aris/skills/skills-codex/paper-write/templates/math_commands.tex +48 -0
  473. package/aris/skills/skills-codex/paper-write/templates/neurips2026.tex +493 -0
  474. package/aris/skills/skills-codex/paper-write/templates/neurips_2026.sty +437 -0
  475. package/aris/skills/skills-codex/paper-writing/SKILL.md +744 -0
  476. package/aris/skills/skills-codex/patent-novelty-check/SKILL.md +153 -0
  477. package/aris/skills/skills-codex/patent-pipeline/SKILL.md +344 -0
  478. package/aris/skills/skills-codex/patent-review/SKILL.md +202 -0
  479. package/aris/skills/skills-codex/pixel-art/SKILL.md +139 -0
  480. package/aris/skills/skills-codex/prior-art-search/SKILL.md +146 -0
  481. package/aris/skills/skills-codex/proof-checker/SKILL.md +554 -0
  482. package/aris/skills/skills-codex/proof-orchestrator/SKILL.md +260 -0
  483. package/aris/skills/skills-codex/proof-orchestrator/references/audit-output-contract.md +126 -0
  484. package/aris/skills/skills-codex/proof-orchestrator/references/deepseek-routing.md +76 -0
  485. package/aris/skills/skills-codex/proof-orchestrator/references/dispatch-prompts.md +227 -0
  486. package/aris/skills/skills-codex/proof-orchestrator/references/notation-audit.md +135 -0
  487. package/aris/skills/skills-codex/proof-orchestrator/references/proof-audit-rubric.md +70 -0
  488. package/aris/skills/skills-codex/proof-orchestrator/references/stress-tests.md +38 -0
  489. package/aris/skills/skills-codex/proof-writer/SKILL.md +222 -0
  490. package/aris/skills/skills-codex/qzcli/SKILL.md +324 -0
  491. package/aris/skills/skills-codex/rebuttal/SKILL.md +305 -0
  492. package/aris/skills/skills-codex/render-html/SKILL.md +305 -0
  493. package/aris/skills/skills-codex/render-html/scripts/render_html.py +909 -0
  494. package/aris/skills/skills-codex/render-html/scripts/templates/academic.html +342 -0
  495. package/aris/skills/skills-codex/render-html/scripts/templates/dashboard.html +333 -0
  496. package/aris/skills/skills-codex/research-implement-feature/SKILL.md +406 -0
  497. package/aris/skills/skills-codex/research-lit/SKILL.md +464 -0
  498. package/aris/skills/skills-codex/research-pipeline/SKILL.md +346 -0
  499. package/aris/skills/skills-codex/research-refine/SKILL.md +721 -0
  500. package/aris/skills/skills-codex/research-refine-pipeline/SKILL.md +186 -0
  501. package/aris/skills/skills-codex/research-review/SKILL.md +160 -0
  502. package/aris/skills/skills-codex/research-wiki/SKILL.md +421 -0
  503. package/aris/skills/skills-codex/resubmit-pipeline/SKILL.md +444 -0
  504. package/aris/skills/skills-codex/result-to-claim/SKILL.md +246 -0
  505. package/aris/skills/skills-codex/run-experiment/SKILL.md +236 -0
  506. package/aris/skills/skills-codex/semantic-scholar/SKILL.md +219 -0
  507. package/aris/skills/skills-codex/serverless-modal/SKILL.md +335 -0
  508. package/aris/skills/skills-codex/shared-references/acceptance-gate.md +336 -0
  509. package/aris/skills/skills-codex/shared-references/assurance-contract.md +139 -0
  510. package/aris/skills/skills-codex/shared-references/capture-antipatterns.md +84 -0
  511. package/aris/skills/skills-codex/shared-references/citation-discipline.md +452 -0
  512. package/aris/skills/skills-codex/shared-references/compute-env-contract.md +163 -0
  513. package/aris/skills/skills-codex/shared-references/effort-contract.md +143 -0
  514. package/aris/skills/skills-codex/shared-references/evidence-precheck.md +73 -0
  515. package/aris/skills/skills-codex/shared-references/experiment-integrity.md +49 -0
  516. package/aris/skills/skills-codex/shared-references/external-cadence.md +334 -0
  517. package/aris/skills/skills-codex/shared-references/fan-out-pattern.md +375 -0
  518. package/aris/skills/skills-codex/shared-references/injection-hygiene.md +132 -0
  519. package/aris/skills/skills-codex/shared-references/integration-contract.md +372 -0
  520. package/aris/skills/skills-codex/shared-references/output-composition.md +98 -0
  521. package/aris/skills/skills-codex/shared-references/output-language.md +45 -0
  522. package/aris/skills/skills-codex/shared-references/output-manifest.md +40 -0
  523. package/aris/skills/skills-codex/shared-references/output-versioning.md +111 -0
  524. package/aris/skills/skills-codex/shared-references/patent-format-cn.md +199 -0
  525. package/aris/skills/skills-codex/shared-references/patent-format-ep.md +173 -0
  526. package/aris/skills/skills-codex/shared-references/patent-format-us.md +161 -0
  527. package/aris/skills/skills-codex/shared-references/patent-writing-principles.md +197 -0
  528. package/aris/skills/skills-codex/shared-references/prior-art-databases.md +141 -0
  529. package/aris/skills/skills-codex/shared-references/resumable-runs.md +125 -0
  530. package/aris/skills/skills-codex/shared-references/review-scope-limits.md +81 -0
  531. package/aris/skills/skills-codex/shared-references/review-tracing.md +144 -0
  532. package/aris/skills/skills-codex/shared-references/reviewer-independence.md +66 -0
  533. package/aris/skills/skills-codex/shared-references/reviewer-routing.md +128 -0
  534. package/aris/skills/skills-codex/shared-references/skill-governance.md +119 -0
  535. package/aris/skills/skills-codex/shared-references/taste-calibration.md +90 -0
  536. package/aris/skills/skills-codex/shared-references/venue-checklists.md +73 -0
  537. package/aris/skills/skills-codex/shared-references/wiki-helper-resolution.md +69 -0
  538. package/aris/skills/skills-codex/shared-references/writing-principles.md +525 -0
  539. package/aris/skills/skills-codex/slides-polish/SKILL.md +563 -0
  540. package/aris/skills/skills-codex/specification-writing/SKILL.md +211 -0
  541. package/aris/skills/skills-codex/system-profile/SKILL.md +103 -0
  542. package/aris/skills/skills-codex/training-check/SKILL.md +83 -0
  543. package/aris/skills/skills-codex/vast-gpu/SKILL.md +394 -0
  544. package/aris/skills/skills-codex/web-debug-search/SKILL.md +334 -0
  545. package/aris/skills/skills-codex/wiki-enrich/SKILL.md +255 -0
  546. package/aris/skills/skills-codex/writing-systems-papers/SKILL.md +184 -0
  547. package/aris/skills/skills-codex-claude-review/README.md +79 -0
  548. package/aris/skills/skills-codex-claude-review/README_CN.md +78 -0
  549. package/aris/skills/skills-codex-claude-review/auto-paper-improvement-loop/SKILL.md +625 -0
  550. package/aris/skills/skills-codex-claude-review/auto-review-loop/SKILL.md +516 -0
  551. package/{skills/novelty-check-claude → aris/skills/skills-codex-claude-review/novelty-check}/SKILL.md +2 -10
  552. package/aris/skills/skills-codex-claude-review/paper-figure/SKILL.md +319 -0
  553. package/aris/skills/skills-codex-claude-review/paper-plan/SKILL.md +287 -0
  554. package/aris/skills/skills-codex-claude-review/paper-write/SKILL.md +496 -0
  555. package/aris/skills/skills-codex-claude-review/research-refine/SKILL.md +732 -0
  556. package/aris/skills/skills-codex-claude-review/research-review/SKILL.md +177 -0
  557. package/aris/skills/skills-codex-gemini-review/README.md +176 -0
  558. package/aris/skills/skills-codex-gemini-review/README_CN.md +175 -0
  559. package/aris/skills/skills-codex-gemini-review/auto-paper-improvement-loop/SKILL.md +376 -0
  560. package/aris/skills/skills-codex-gemini-review/auto-review-loop/SKILL.md +304 -0
  561. package/aris/skills/skills-codex-gemini-review/grant-proposal/SKILL.md +630 -0
  562. package/aris/skills/skills-codex-gemini-review/idea-creator/SKILL.md +287 -0
  563. package/aris/skills/skills-codex-gemini-review/idea-discovery/SKILL.md +275 -0
  564. package/aris/skills/skills-codex-gemini-review/idea-discovery-robot/SKILL.md +365 -0
  565. package/aris/skills/skills-codex-gemini-review/novelty-check/SKILL.md +133 -0
  566. package/aris/skills/skills-codex-gemini-review/paper-figure/SKILL.md +289 -0
  567. package/aris/skills/skills-codex-gemini-review/paper-plan/SKILL.md +265 -0
  568. package/aris/skills/skills-codex-gemini-review/paper-poster-html/SKILL.md +102 -0
  569. package/aris/skills/skills-codex-gemini-review/paper-slides/SKILL.md +582 -0
  570. package/aris/skills/skills-codex-gemini-review/paper-write/SKILL.md +422 -0
  571. package/aris/skills/skills-codex-gemini-review/paper-writing/SKILL.md +325 -0
  572. package/aris/skills/skills-codex-gemini-review/research-refine/SKILL.md +674 -0
  573. package/aris/skills/skills-codex-gemini-review/research-review/SKILL.md +137 -0
  574. package/aris/skills/slides-polish/SKILL.md +565 -0
  575. package/aris/skills/specification-writing/SKILL.md +211 -0
  576. package/aris/skills/system-profile/SKILL.md +103 -0
  577. package/aris/skills/training-check/SKILL.md +132 -0
  578. package/aris/skills/vast-gpu/SKILL.md +394 -0
  579. package/aris/skills/web-debug-search/SKILL.md +334 -0
  580. package/aris/skills/wiki-enrich/SKILL.md +257 -0
  581. package/aris/skills/writing-systems-papers/SKILL.md +184 -0
  582. package/aris/templates/CLAUDE_MD_TEMPLATE.md +29 -0
  583. package/aris/templates/EXPERIMENT_LOG_TEMPLATE.md +47 -0
  584. package/aris/templates/EXPERIMENT_PLAN_TEMPLATE.md +51 -0
  585. package/aris/templates/EXPERIMENT_PLAN_TEMPLATE_CN.md +53 -0
  586. package/aris/templates/FINDINGS_TEMPLATE.md +52 -0
  587. package/aris/templates/IDEA_CANDIDATES_TEMPLATE.md +47 -0
  588. package/aris/templates/IDEA_CANDIDATES_TEMPLATE_CN.md +47 -0
  589. package/aris/templates/INVENTION_BRIEF_TEMPLATE.md +87 -0
  590. package/aris/templates/MANIFEST_TEMPLATE.md +7 -0
  591. package/aris/templates/NARRATIVE_REPORT_TEMPLATE.md +49 -0
  592. package/aris/templates/PAPER_PLAN_TEMPLATE.md +47 -0
  593. package/aris/templates/PATENT_CLAIMS_TEMPLATE.md +78 -0
  594. package/aris/templates/PATENT_SPECIFICATION_TEMPLATE.md +68 -0
  595. package/aris/templates/README.md +57 -0
  596. package/aris/templates/RESEARCH_BRIEF_TEMPLATE.md +35 -0
  597. package/aris/templates/RESEARCH_BRIEF_TEMPLATE_CN.md +41 -0
  598. package/aris/templates/RESEARCH_CONTRACT_TEMPLATE.md +60 -0
  599. package/aris/templates/claude-hooks/corpus_write_guard.json +16 -0
  600. package/aris/templates/claude-hooks/corpus_write_guard.py +85 -0
  601. package/aris/templates/claude-hooks/meta_logging.json +74 -0
  602. package/aris/templates/gitignore-trace.txt +3 -0
  603. package/aris/tests/__init__.py +0 -0
  604. package/aris/tests/_feishu_bridge_helpers.py +82 -0
  605. package/aris/tests/_llm_chat_helpers.py +179 -0
  606. package/aris/tests/_minimax_helpers.py +189 -0
  607. package/aris/tests/test_argument_hint_lint.py +183 -0
  608. package/aris/tests/test_arxiv_fetch.py +266 -0
  609. package/aris/tests/test_auto_proceed_contract.py +106 -0
  610. package/aris/tests/test_capture_filter.py +86 -0
  611. package/aris/tests/test_claude_review_artifact_tools.py +171 -0
  612. package/aris/tests/test_claude_review_generator.py +51 -0
  613. package/aris/tests/test_claude_review_overlay.py +64 -0
  614. package/aris/tests/test_claude_review_parse.py +290 -0
  615. package/aris/tests/test_codex_exec_bridge.py +306 -0
  616. package/aris/tests/test_codex_image2_server.py +424 -0
  617. package/aris/tests/test_codex_install_update.py +511 -0
  618. package/aris/tests/test_codex_skill_mirror.py +547 -0
  619. package/aris/tests/test_copilot_install.py +1202 -0
  620. package/aris/tests/test_copilot_native_evidence.py +562 -0
  621. package/aris/tests/test_corpus_write_guard.py +76 -0
  622. package/aris/tests/test_deepxiv_fetch.py +180 -0
  623. package/aris/tests/test_evidence_check.py +189 -0
  624. package/aris/tests/test_exa_search.py +220 -0
  625. package/aris/tests/test_feishu_bridge_server.py +289 -0
  626. package/aris/tests/test_forensics_gate.py +565 -0
  627. package/aris/tests/test_http_reviewer_fallback_contract.py +43 -0
  628. package/aris/tests/test_http_reviewer_provenance_alignment.py +25 -0
  629. package/aris/tests/test_http_reviewer_review_gate.py +112 -0
  630. package/aris/tests/test_idea_creator_query_pack_scan.py +238 -0
  631. package/aris/tests/test_idea_discovery_gate.py +400 -0
  632. package/aris/tests/test_install_aris_agent_profiles.py +98 -0
  633. package/aris/tests/test_install_aris_ps1.py +359 -0
  634. package/aris/tests/test_install_aris_replace_link.py +119 -0
  635. package/aris/tests/test_install_aris_selective.py +235 -0
  636. package/aris/tests/test_install_aris_tools_symlink.py +196 -0
  637. package/aris/tests/test_iteration_log.py +85 -0
  638. package/aris/tests/test_llm_chat_converter.py +61 -0
  639. package/aris/tests/test_llm_chat_review_fallback.py +516 -0
  640. package/aris/tests/test_llm_chat_server.py +414 -0
  641. package/aris/tests/test_mainline_large_payload_prompts.py +72 -0
  642. package/aris/tests/test_manual_review.py +804 -0
  643. package/aris/tests/test_manual_review_integration.py +305 -0
  644. package/aris/tests/test_minimax_chat_server.py +464 -0
  645. package/aris/tests/test_minimax_integration.py +241 -0
  646. package/aris/tests/test_openalex_fetch.py +197 -0
  647. package/aris/tests/test_plugin_manifest.py +40 -0
  648. package/aris/tests/test_proof_suite_integration.py +66 -0
  649. package/aris/tests/test_provenance.py +284 -0
  650. package/aris/tests/test_research_wiki_claims.py +93 -0
  651. package/aris/tests/test_research_wiki_encoding.py +74 -0
  652. package/aris/tests/test_research_wiki_experiments.py +88 -0
  653. package/aris/tests/test_research_wiki_fetch_arxiv_metadata.py +242 -0
  654. package/aris/tests/test_research_wiki_helper_resolution.py +253 -0
  655. package/aris/tests/test_research_wiki_ideas.py +113 -0
  656. package/aris/tests/test_review_gate.py +358 -0
  657. package/aris/tests/test_reviewer_pins.py +90 -0
  658. package/aris/tests/test_run_state.py +313 -0
  659. package/aris/tests/test_skill_groups.py +111 -0
  660. package/aris/tests/test_skill_picker.py +207 -0
  661. package/aris/tests/test_threat_scan.py +139 -0
  662. package/aris/tests/test_trigger_eval.py +171 -0
  663. package/aris/tests/test_verify_paper_audits.py +198 -0
  664. package/aris/tests/test_watchdog.py +496 -0
  665. package/aris/tests/test_watchdog_loop.py +148 -0
  666. package/aris/tests/test_web_debug_search.py +157 -0
  667. package/aris/tools/arxiv_fetch.py +311 -0
  668. package/aris/tools/capture_filter.py +126 -0
  669. package/aris/tools/check_skills_inventory.py +287 -0
  670. package/aris/tools/convert_skills_to_llm_chat.py +282 -0
  671. package/aris/tools/copilot_native_evidence.py +818 -0
  672. package/aris/tools/deepxiv_fetch.py +213 -0
  673. package/aris/tools/evidence_check.py +212 -0
  674. package/aris/tools/exa_search.py +425 -0
  675. package/aris/tools/experiment_queue/README.md +118 -0
  676. package/aris/tools/experiment_queue/build_manifest.py +44 -0
  677. package/aris/tools/experiment_queue/queue_manager.py +44 -0
  678. package/aris/tools/extract_paper_style.py +560 -0
  679. package/aris/tools/figure_renderer.py +69 -0
  680. package/aris/tools/forensics_gate.py +669 -0
  681. package/aris/tools/generate_codex_claude_review_overrides.py +355 -0
  682. package/aris/tools/idea_discovery_gate.py +256 -0
  683. package/aris/tools/install_aris.ps1 +1372 -0
  684. package/aris/tools/install_aris.sh +1416 -0
  685. package/aris/tools/install_aris_codex.sh +1023 -0
  686. package/aris/tools/install_aris_copilot.sh +1052 -0
  687. package/aris/tools/iteration_log.py +143 -0
  688. package/aris/tools/lint_skills_helpers.sh +84 -0
  689. package/aris/tools/meta_opt/check_ready.sh +80 -0
  690. package/aris/tools/meta_opt/log_event.sh +91 -0
  691. package/aris/tools/meta_opt/trigger_eval.py +280 -0
  692. package/aris/tools/meta_opt/trigger_evals.sample.json +28 -0
  693. package/aris/tools/openalex_fetch.py +326 -0
  694. package/aris/tools/overleaf_audit.sh +104 -0
  695. package/aris/tools/overleaf_setup.sh +150 -0
  696. package/aris/tools/paper_illustration_image2.py +62 -0
  697. package/aris/tools/provenance.py +294 -0
  698. package/aris/tools/research_wiki.py +1720 -0
  699. package/aris/tools/review_gate.py +502 -0
  700. package/aris/tools/run_state.py +399 -0
  701. package/aris/tools/save_trace.sh +477 -0
  702. package/aris/tools/semantic_scholar_fetch.py +438 -0
  703. package/aris/tools/skill-groups.tsv +117 -0
  704. package/aris/tools/skill_picker.py +238 -0
  705. package/aris/tools/smart_update.ps1 +521 -0
  706. package/aris/tools/smart_update.sh +591 -0
  707. package/aris/tools/smart_update_codex.sh +419 -0
  708. package/aris/tools/smart_update_copilot.sh +605 -0
  709. package/aris/tools/threat_scan.py +222 -0
  710. package/aris/tools/verify_paper_audits.sh +487 -0
  711. package/aris/tools/verify_papers.py +613 -0
  712. package/aris/tools/verify_wiki_coverage.sh +176 -0
  713. package/aris/tools/watchdog.py +485 -0
  714. package/aris/xhs_post.md +125 -0
  715. package/dsh/client.js +43 -14
  716. package/dsh/http-rpc.mjs +8 -7
  717. package/mcp-servers/codex-exec/README.md +4 -4
  718. package/mcp-servers/codex-exec/server.py +1 -1
  719. package/package.json +4 -5
  720. package/skills/ablation-planner/SKILL.md +1 -1
  721. package/skills/auto-paper-improvement-loop/SKILL.md +10 -10
  722. package/skills/auto-paper-improvement-loop-claude/SKILL.md +25 -612
  723. package/skills/auto-review-loop/SKILL.md +4 -4
  724. package/skills/auto-review-loop-claude/SKILL.md +25 -503
  725. package/skills/citation-audit/SKILL.md +3 -3
  726. package/skills/claims-drafting/SKILL.md +2 -2
  727. package/skills/experiment-audit/SKILL.md +4 -4
  728. package/skills/experiment-bridge/SKILL.md +14 -12
  729. package/skills/figure-spec/SKILL.md +1 -1
  730. package/skills/grant-proposal/SKILL.md +7 -7
  731. package/skills/human-in-loop-refine/SKILL.md +119 -0
  732. package/skills/human-in-loop-refine/scripts/refine_round.py +449 -0
  733. package/skills/human-in-loop-refine/scripts/test_refine_round.py +200 -0
  734. package/skills/human-in-loop-refine/templates/INPUTS.md +20 -0
  735. package/skills/human-in-loop-refine/templates/PAPER_NOTE.md +29 -0
  736. package/skills/human-in-loop-refine/templates/REVISION.md +27 -0
  737. package/skills/idea-creator/SKILL.md +2 -2
  738. package/skills/idea-discovery/SKILL.md +15 -13
  739. package/skills/idea-discovery-robot/SKILL.md +11 -9
  740. package/skills/integrity-forensics/SKILL.md +1 -1
  741. package/skills/interview-cheatsheet/SKILL.md +3 -3
  742. package/skills/invention-structuring/SKILL.md +2 -2
  743. package/skills/kill-argument/SKILL.md +8 -8
  744. package/skills/kill-argument-claude/SKILL.md +30 -431
  745. package/skills/meta-apply/SKILL.md +1 -1
  746. package/skills/meta-optimize/SKILL.md +2 -2
  747. package/skills/novelty-check/SKILL.md +2 -2
  748. package/skills/paper-claim-audit/SKILL.md +5 -5
  749. package/skills/paper-figure/SKILL.md +4 -3
  750. package/skills/paper-figure-claude/SKILL.md +24 -305
  751. package/skills/paper-plan/SKILL.md +3 -3
  752. package/skills/paper-plan-claude/SKILL.md +24 -273
  753. package/skills/paper-poster-html/SKILL.md +11 -9
  754. package/skills/paper-slides/SKILL.md +5 -5
  755. package/skills/paper-talk/SKILL.md +10 -8
  756. package/skills/paper-write/SKILL.md +3 -3
  757. package/skills/paper-write-claude/SKILL.md +24 -482
  758. package/skills/paper-writing/SKILL.md +19 -17
  759. package/skills/patent-novelty-check/SKILL.md +3 -3
  760. package/skills/patent-pipeline/SKILL.md +2 -2
  761. package/skills/patent-review/SKILL.md +3 -3
  762. package/skills/proof-checker/SKILL.md +12 -12
  763. package/skills/proof-checker-claude/SKILL.md +30 -860
  764. package/skills/rebuttal/SKILL.md +12 -10
  765. package/skills/render-html/SKILL.md +1 -1
  766. package/skills/research-implement-feature/SKILL.md +1 -1
  767. package/skills/research-pipeline/SKILL.md +15 -13
  768. package/skills/research-refine/SKILL.md +6 -6
  769. package/skills/research-refine-claude/SKILL.md +24 -719
  770. package/skills/research-refine-pipeline/SKILL.md +10 -8
  771. package/skills/research-review/SKILL.md +7 -7
  772. package/skills/research-review-claude/SKILL.md +25 -164
  773. package/skills/resubmit-pipeline/SKILL.md +14 -12
  774. package/skills/result-to-claim/SKILL.md +2 -2
  775. package/skills/shared-references/assurance-contract.md +1 -1
  776. package/skills/shared-references/effort-contract.md +1 -1
  777. package/skills/shared-references/fan-out-pattern.md +4 -4
  778. package/skills/shared-references/resumable-runs.md +2 -2
  779. package/skills/shared-references/review-scope-limits.md +1 -1
  780. package/skills/shared-references/review-tracing.md +2 -2
  781. package/skills/shared-references/reviewer-routing.md +16 -14
  782. package/skills/shared-references/skill-governance.md +3 -3
  783. package/skills/skills-codex/ablation-planner/SKILL.md +1 -1
  784. package/skills/skills-codex/auto-paper-improvement-loop/SKILL.md +8 -8
  785. package/skills/skills-codex/auto-review-loop/SKILL.md +2 -2
  786. package/skills/skills-codex/citation-audit/SKILL.md +4 -4
  787. package/skills/skills-codex/claims-drafting/SKILL.md +2 -2
  788. package/skills/skills-codex/experiment-audit/SKILL.md +9 -9
  789. package/skills/skills-codex/experiment-bridge/SKILL.md +1 -1
  790. package/skills/skills-codex/figure-spec/SKILL.md +1 -1
  791. package/skills/skills-codex/grant-proposal/SKILL.md +7 -7
  792. package/skills/skills-codex/idea-creator/SKILL.md +4 -4
  793. package/skills/skills-codex/idea-discovery/SKILL.md +6 -6
  794. package/skills/skills-codex/idea-discovery-robot/SKILL.md +1 -1
  795. package/skills/skills-codex/integrity-forensics/SKILL.md +4 -2
  796. package/skills/skills-codex/interview-cheatsheet/SKILL.md +3 -3
  797. package/skills/skills-codex/invention-structuring/SKILL.md +2 -2
  798. package/skills/skills-codex/kill-argument/SKILL.md +7 -7
  799. package/skills/skills-codex/meta-apply/SKILL.md +1 -1
  800. package/skills/skills-codex/meta-optimize/SKILL.md +2 -2
  801. package/skills/skills-codex/novelty-check/SKILL.md +1 -1
  802. package/skills/skills-codex/paper-claim-audit/SKILL.md +6 -6
  803. package/skills/skills-codex/paper-figure/SKILL.md +3 -3
  804. package/skills/skills-codex/paper-plan/SKILL.md +3 -3
  805. package/skills/skills-codex/paper-poster-html/SKILL.md +3 -3
  806. package/skills/skills-codex/paper-slides/SKILL.md +4 -4
  807. package/skills/skills-codex/paper-write/SKILL.md +3 -3
  808. package/skills/skills-codex/paper-writing/SKILL.md +7 -7
  809. package/skills/skills-codex/patent-novelty-check/SKILL.md +3 -3
  810. package/skills/skills-codex/patent-pipeline/SKILL.md +2 -2
  811. package/skills/skills-codex/patent-review/SKILL.md +3 -3
  812. package/skills/skills-codex/proof-checker/SKILL.md +11 -11
  813. package/skills/skills-codex/rebuttal/SKILL.md +2 -2
  814. package/skills/skills-codex/render-html/SKILL.md +1 -1
  815. package/skills/skills-codex/research-implement-feature/SKILL.md +1 -1
  816. package/skills/skills-codex/research-pipeline/SKILL.md +5 -5
  817. package/skills/skills-codex/research-refine/SKILL.md +6 -6
  818. package/skills/skills-codex/research-review/SKILL.md +6 -6
  819. package/skills/skills-codex/resubmit-pipeline/SKILL.md +4 -4
  820. package/skills/skills-codex/result-to-claim/SKILL.md +2 -2
  821. package/skills/skills-codex/shared-references/assurance-contract.md +2 -2
  822. package/skills/skills-codex/shared-references/resumable-runs.md +3 -3
  823. package/skills/skills-codex/shared-references/review-scope-limits.md +1 -1
  824. package/skills/skills-codex/shared-references/review-tracing.md +3 -3
  825. package/skills/skills-codex/shared-references/reviewer-independence.md +2 -2
  826. package/skills/skills-codex/shared-references/reviewer-routing.md +10 -10
  827. package/skills/skills-codex/slides-polish/SKILL.md +3 -3
  828. package/skills/skills-codex/specification-writing/SKILL.md +2 -2
  829. package/skills/slides-polish/SKILL.md +3 -3
  830. package/skills/specification-writing/SKILL.md +2 -2
  831. package/skills/training-check/SKILL.md +2 -2
  832. package/skills/verify-manager/SKILL.md +88 -0
  833. package/skills/verify-manager/phases/1-specify.md +17 -0
  834. package/skills/verify-manager/phases/2-plan.md +14 -0
  835. package/skills/verify-manager/phases/3-tasks.md +19 -0
  836. package/skills/verify-manager/phases/4-run.md +12 -0
  837. package/skills/verify-manager/phases/5-analyze.md +21 -0
  838. package/skills/verify-manager/phases/6-next.md +19 -0
  839. package/skills/verify-manager/scripts/test_verify_space.py +202 -0
  840. package/skills/verify-manager/scripts/verify_space.py +451 -0
  841. package/skills/verify-manager/templates/HYPOTHESIS.md +37 -0
  842. package/skills/verify-manager/templates/NEXT_IDEAS.md +20 -0
  843. package/skills/verify-manager/templates/PILOT_PLAN.md +56 -0
  844. package/skills/verify-manager/templates/PILOT_TASKS.md +28 -0
  845. package/skills/verify-manager/templates/PROMOTION.md +17 -0
  846. package/skills/verify-manager/templates/RESULT.md +36 -0
  847. package/test/client-render-smoke.mjs +143 -0
  848. package/test/test_claude_overlays.py +99 -0
  849. package/test/transport-smoke.mjs +16 -4
  850. package/tools/__pycache__/build_claude_overlays.cpython-314.pyc +0 -0
  851. package/tools/__pycache__/verify_lane.cpython-314.pyc +0 -0
  852. package/tools/build_claude_overlays.py +187 -0
  853. package/tools/check_skills_inventory.py +17 -3
  854. package/tools/convert_skills_to_llm_chat.py +15 -15
  855. package/tools/generate_codex_claude_review_overrides.py +12 -12
  856. package/tools/run_state.py +1 -1
  857. package/tools/save_trace.sh +2 -2
package/aris/docs/tutorials/kl_divergence_rlhf_tutorial_en.html ADDED
@@ -0,0 +1,1212 @@
1
+ <!DOCTYPE html>
2
+ <html lang="zh-CN">
3
+ <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>Kl Divergence Rlhf Tutorial En</title>
7
+
8
+ <meta name="generator" content="ARIS render-html (academic, v1)">
9
+ <meta name="aris:source-path" content="docs/tutorials/kl_divergence_rlhf_tutorial_en.md">
10
+ <meta name="aris:source-sha256" content="4477782e0f503450db748cc46a12ddd9361df1fbcdcf2f996d8fee58f4761b73">
11
+ <meta name="aris:generated-at" content="2026-05-19 19:29 UTC">
12
+
13
+ <!-- MathJax 3 -->
14
+ <script>
15
+ window.MathJax = {
16
+ tex: { inlineMath: [['$', '$'], ['\\(', '\\)']], displayMath: [['$$', '$$'], ['\\[', '\\]']], processEscapes: true },
17
+ options: { skipHtmlTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code'] }
18
+ };
19
+ </script>
20
+ <script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" async></script>
21
+
22
+ <!-- highlight.js -->
23
+ <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.9.0/build/styles/atom-one-light.min.css">
24
+ <script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.9.0/build/highlight.min.js"></script>
25
+ <script>document.addEventListener('DOMContentLoaded', () => hljs.highlightAll());</script>
26
+
27
+
28
+ <style>
29
+ :root {
30
+ --bg: #fdfcf7;
31
+ --bg-soft: #f4f1ea;
32
+ --bg-code: #f8f5ec;
33
+ --ink: #1a1a1a;
34
+ --ink-soft: #4a4a4a;
35
+ --ink-muted: #6b6b6b;
36
+ --primary: #1a4a8c;
37
+ --primary-soft: #2d6cb8;
38
+ --accent: #b8390e;
39
+ --warn: #b45309;
40
+ --warn-bg: #fef3c7;
41
+ --info-bg: #dbeafe;
42
+ --good-bg: #d1fae5;
43
+ --good: #065f46;
44
+ --bad-bg: #fee2e2;
45
+ --bad: #991b1b;
46
+ --border: #d6d0c0;
47
+ --border-soft: #e8e3d5;
48
+ }
49
+
50
+ * { box-sizing: border-box; }
51
+ html { scroll-behavior: smooth; }
52
+
53
+ body {
54
+ font-family: "Source Serif Pro", "Source Serif 4", "Crimson Pro", "Georgia", "Songti SC", "STSong", serif;
55
+ line-height: 1.65;
56
+ color: var(--ink);
57
+ background: var(--bg);
58
+ margin: 0;
59
+ padding: 0;
60
+ font-size: 16px;
61
+ }
62
+
63
+ .layout {
64
+ max-width: 1280px;
65
+ margin: 0 auto;
66
+ display: grid;
67
+ grid-template-columns: 260px 1fr;
68
+ gap: 48px;
69
+ padding: 40px 32px;
70
+ }
71
+
72
+ nav.toc {
73
+ position: sticky;
74
+ top: 24px;
75
+ align-self: start;
76
+ font-size: 13px;
77
+ max-height: calc(100vh - 48px);
78
+ overflow-y: auto;
79
+ border-right: 1px solid var(--border-soft);
80
+ padding-right: 16px;
81
+ }
82
+ nav.toc h3 {
83
+ margin: 0 0 12px;
84
+ font-size: 12px;
85
+ text-transform: uppercase;
86
+ letter-spacing: 0.08em;
87
+ color: var(--ink-muted);
88
+ font-weight: 600;
89
+ }
90
+ nav.toc ol { list-style: none; padding: 0; margin: 0; counter-reset: toc; }
91
+ nav.toc ol li { margin: 5px 0; counter-increment: toc; }
92
+ nav.toc ol li::before { content: counter(toc) ". "; color: var(--ink-muted); margin-right: 4px; }
93
+ nav.toc a {
94
+ color: var(--ink-soft);
95
+ text-decoration: none;
96
+ border-bottom: 1px dotted transparent;
97
+ }
98
+ nav.toc a:hover { color: var(--primary); border-bottom-color: var(--primary); }
99
+ nav.toc ul { list-style: none; padding-left: 14px; margin: 3px 0; font-size: 12px; }
100
+ nav.toc ul li::before { content: "→ "; color: var(--border); }
101
+
102
+ main { min-width: 0; }
103
+
104
+ header.hero {
105
+ border-bottom: 3px double var(--primary);
106
+ padding-bottom: 24px;
107
+ margin-bottom: 32px;
108
+ }
109
+ header.hero .eyebrow {
110
+ color: var(--accent);
111
+ font-size: 13px;
112
+ text-transform: uppercase;
113
+ letter-spacing: 0.12em;
114
+ font-weight: 600;
115
+ margin-bottom: 8px;
116
+ }
117
+ header.hero h1 {
118
+ font-size: 32px;
119
+ line-height: 1.2;
120
+ margin: 0 0 12px;
121
+ color: var(--ink);
122
+ font-weight: 700;
123
+ letter-spacing: -0.01em;
124
+ }
125
+ header.hero .subtitle {
126
+ font-size: 16px;
127
+ color: var(--ink-soft);
128
+ margin: 0 0 8px;
129
+ font-style: italic;
130
+ }
131
+ header.hero .byline {
132
+ font-size: 14px;
133
+ color: var(--ink-soft);
134
+ margin: 0 0 20px;
135
+ }
136
+ header.hero .byline strong {
137
+ color: var(--ink);
138
+ font-weight: 600;
139
+ }
140
+ header.hero .meta {
141
+ display: flex;
142
+ gap: 20px;
143
+ flex-wrap: wrap;
144
+ font-size: 12px;
145
+ color: var(--ink-muted);
146
+ border-top: 1px solid var(--border-soft);
147
+ padding-top: 14px;
148
+ }
149
+ header.hero .meta span strong { color: var(--ink-soft); }
150
+ header.hero .meta code {
151
+ font-family: "JetBrains Mono", "SF Mono", "Menlo", "Consolas", monospace;
152
+ font-size: 11px;
153
+ background: var(--bg-soft);
154
+ padding: 1px 5px;
155
+ border-radius: 3px;
156
+ border: 1px solid var(--border-soft);
157
+ }
158
+
159
+ h2 {
160
+ font-size: 24px;
161
+ margin: 44px 0 14px;
162
+ padding-bottom: 8px;
163
+ border-bottom: 1px solid var(--border);
164
+ color: var(--ink);
165
+ font-weight: 700;
166
+ }
167
+ h2 .num { color: var(--primary); font-weight: 600; margin-right: 8px; }
168
+ h3 { font-size: 19px; margin: 28px 0 10px; color: var(--primary); font-weight: 600; }
169
+ h4 { font-size: 16px; margin: 20px 0 8px; color: var(--ink); font-weight: 600; }
170
+
171
+ p { margin: 10px 0; }
172
+ ul, ol { padding-left: 22px; margin: 10px 0; }
173
+ ul li, ol li { margin: 4px 0; }
174
+ ul li::marker { color: var(--primary); }
175
+
176
+ strong { color: var(--accent); font-weight: 600; }
177
+ em { color: var(--ink-soft); }
178
+
179
+ a { color: var(--primary); }
180
+ a:hover { color: var(--accent); }
181
+
182
+ code:not(.hljs) {
183
+ font-family: "JetBrains Mono", "SF Mono", "Menlo", "Consolas", monospace;
184
+ font-size: 0.86em;
185
+ background: var(--bg-code);
186
+ padding: 1px 5px;
187
+ border-radius: 3px;
188
+ border: 1px solid var(--border-soft);
189
+ color: var(--accent);
190
+ }
191
+
192
+ pre {
193
+ background: #fafaf6;
194
+ border: 1px solid var(--border);
195
+ border-left: 4px solid var(--primary);
196
+ padding: 0;
197
+ overflow-x: auto;
198
+ border-radius: 4px;
199
+ margin: 14px 0;
200
+ }
201
+ pre code, pre code.hljs {
202
+ background: transparent !important;
203
+ display: block;
204
+ padding: 14px 18px !important;
205
+ font-size: 13px;
206
+ line-height: 1.55;
207
+ font-family: "JetBrains Mono", "SF Mono", "Menlo", monospace;
208
+ color: var(--ink);
209
+ }
210
+ pre.diagram {
211
+ background: #f9f6ed;
212
+ border-left: 4px solid var(--accent);
213
+ font-size: 12.5px;
214
+ line-height: 1.4;
215
+ }
216
+
217
+ .callout {
218
+ margin: 16px 0;
219
+ padding: 12px 16px;
220
+ border-radius: 4px;
221
+ border-left: 4px solid;
222
+ font-size: 15px;
223
+ }
224
+ .callout-title {
225
+ font-weight: 600;
226
+ margin-bottom: 6px;
227
+ font-size: 12px;
228
+ text-transform: uppercase;
229
+ letter-spacing: 0.06em;
230
+ }
231
+ .callout-info { background: var(--info-bg); border-left-color: var(--primary); }
232
+ .callout-info .callout-title { color: var(--primary); }
233
+ .callout-warn { background: var(--warn-bg); border-left-color: var(--warn); }
234
+ .callout-warn .callout-title { color: var(--warn); }
235
+ .callout-good { background: var(--good-bg); border-left-color: var(--good); }
236
+ .callout-good .callout-title { color: var(--good); }
237
+ .callout-bad { background: var(--bad-bg); border-left-color: var(--bad); }
238
+ .callout-bad .callout-title { color: var(--bad); }
239
+
240
+ table {
241
+ width: 100%;
242
+ border-collapse: collapse;
243
+ margin: 16px 0;
244
+ font-size: 14px;
245
+ border: 1px solid var(--border);
246
+ border-radius: 4px;
247
+ overflow: hidden;
248
+ }
249
+ thead { background: var(--primary); color: white; }
250
+ th, td {
251
+ text-align: left;
252
+ padding: 9px 12px;
253
+ border-bottom: 1px solid var(--border-soft);
254
+ vertical-align: top;
255
+ }
256
+ th { font-weight: 600; font-size: 13px; letter-spacing: 0.02em; }
257
+ tr:last-child td { border-bottom: none; }
258
+ tbody tr:nth-child(even) { background: var(--bg-soft); }
259
+
260
+ details.qa, details {
261
+ background: white;
262
+ border: 1px solid var(--border-soft);
263
+ border-radius: 6px;
264
+ margin: 10px 0;
265
+ padding: 0;
266
+ }
267
+ details summary {
268
+ cursor: pointer;
269
+ padding: 10px 14px;
270
+ font-weight: 600;
271
+ font-size: 14px;
272
+ color: var(--primary);
273
+ list-style: none;
274
+ user-select: none;
275
+ }
276
+ details summary::-webkit-details-marker { display: none; }
277
+ details summary::before {
278
+ content: "▸ ";
279
+ margin-right: 4px;
280
+ display: inline-block;
281
+ transition: transform 0.15s;
282
+ }
283
+ details[open] summary::before { transform: rotate(90deg); }
284
+ details[open] summary { border-bottom: 1px solid var(--border-soft); }
285
+ details > :not(summary) { padding: 10px 14px; }
286
+ details p:first-of-type { margin-top: 8px; }
287
+
288
+ mjx-container[display="true"] { margin: 12px 0 !important; }
289
+
290
+ footer.aris-footer {
291
+ margin-top: 60px;
292
+ padding-top: 20px;
293
+ border-top: 1px solid var(--border);
294
+ font-size: 12px;
295
+ color: var(--ink-muted);
296
+ }
297
+ footer.aris-footer a { color: var(--ink-muted); border-bottom: 1px dotted var(--border); }
298
+
299
+ @media (max-width: 900px) {
300
+ .layout { grid-template-columns: 1fr; gap: 20px; padding: 20px 16px; }
301
+ nav.toc {
302
+ position: static;
303
+ max-height: none;
304
+ border-right: none;
305
+ border-bottom: 1px solid var(--border-soft);
306
+ padding-right: 0;
307
+ padding-bottom: 14px;
308
+ }
309
+ header.hero h1 { font-size: 24px; }
310
+ h2 { font-size: 20px; }
311
+ }
312
+ @media print {
313
+ nav.toc { display: none; }
314
+ .layout { grid-template-columns: 1fr; padding: 0; }
315
+ body { background: white; }
316
+ header.hero { border-bottom-color: var(--ink); }
317
+ }
318
+ </style>
319
+ </head>
320
+ <body>
321
+ <div class="layout">
322
+
323
+ <nav class="toc">
324
+ <h3>Contents</h3>
325
+ <ol>
326
+ <li><a href="#0-tldr-cheat-sheet">§0 TL;DR Cheat Sheet</a>
327
+ </li>
328
+ <li><a href="#1-kl-fundamentals">§1 KL Fundamentals</a>
329
+ <ul>
330
+ <li><a href="#11-definition-properties-notation">1.1　Definition, properties, notation</a></li>
331
+ <li><a href="#12-forward-kl-vs-reverse-kl-mass-covering-or-mode-seeking">1.2　Forward KL vs Reverse KL: mass-covering or mode-seeking?</a></li>
332
+ <li><a href="#13-relation-to-other-divergences">1.3　Relation to other divergences</a></li>
333
+ <li><a href="#14-why-does-rlhf-add-kl">1.4　Why does RLHF add KL?</a></li>
334
+ </ul>
335
+ </li>
336
+ <li><a href="#2-kl-estimators-k1--k2--k3-interview-core">§2 KL Estimators (k1 / k2 / k3): Interview Core</a>
337
+ <ul>
338
+ <li><a href="#21-problem-setup">2.1　Problem setup</a></li>
339
+ <li><a href="#22-k1-estimator--plug-definition-in-directly">2.2　k1 estimator — &quot;plug definition in directly&quot;</a></li>
340
+ <li><a href="#23-k2-estimator--l2-form">2.3　k2 estimator — &quot;$L^2$ form&quot;</a></li>
341
+ <li><a href="#24-k3-estimator-schulman-2020-blog--non-negative-unbiased-value-estimator-but-as-a-loss-term-the-gradient-is-biased--see-36">2.4　k3 estimator (Schulman 2020 blog) — **non-negative unbiased value estimator** (but as a loss term, the gradient is biased — see §3.6)</a></li>
342
+ <li><a href="#25-comparison-table-for-the-three">2.5　Comparison table for the three</a></li>
343
+ <li><a href="#26-code-three-estimators--variance-simulation">2.6　Code: three estimators + variance simulation</a></li>
344
+ </ul>
345
+ </li>
346
+ <li><a href="#3-two-placements-of-kl-in-rlhf">§3 Two placements of KL in RLHF</a>
347
+ <ul>
348
+ <li><a href="#31-option-a-in-reward-shaping-ppo-rlhf--instructgpt-standard">3.1　Option A: In-reward shaping (PPO RLHF / InstructGPT standard)</a></li>
349
+ <li><a href="#32-option-b-in-loss-regularization-grpo--dapo">3.2　Option B: In-loss regularization (GRPO / DAPO)</a></li>
350
+ <li><a href="#33-mathematically-equivalent-engineering-wise-not-equivalent">3.3　Mathematically equivalent? Engineering-wise not equivalent</a></li>
351
+ <li><a href="#34-practice-which-goes-where">3.4　Practice: which goes where?</a></li>
352
+ <li><a href="#36-gradient-bias-analysis-of-estimator-placement-rethinking-kl-regularization-in-rlhf">3.6　Gradient bias analysis of estimator placement (Rethinking KL Regularization in RLHF)</a></li>
353
+ </ul>
354
+ </li>
355
+ <li><a href="#4--schedule">§4 β schedule</a>
356
+ <ul>
357
+ <li><a href="#41-fixed---baseline">4.1　Fixed β — baseline</a></li>
358
+ <li><a href="#42-adaptive---schulman-ppo-penalty-original">4.2　Adaptive β — Schulman PPO-Penalty original</a></li>
359
+ <li><a href="#43--annealing-schedule--analogous-to-learning-rate-schedule">4.3　β annealing schedule — analogous to learning rate schedule</a></li>
360
+ <li><a href="#44--failure-modes">4.4　β failure modes</a></li>
361
+ <li><a href="#45-adaptive--code-example">4.5　Adaptive β code example</a></li>
362
+ </ul>
363
+ </li>
364
+ <li><a href="#5-kls-relation-to-dpo--grpo--simpo--kto--ipo">§5 KL&#x27;s relation to DPO / GRPO / SimPO / KTO / IPO</a>
365
+ <ul>
366
+ <li><a href="#51-dpo-implicit-reward----sequence-log-density-ratio">5.1　DPO: implicit reward = β × sequence log-density ratio</a></li>
367
+ <li><a href="#52-grpo-k3-kl-in-loss">5.2　GRPO: k3 KL in loss</a></li>
368
+ <li><a href="#53-simpo-drops-reference-no-kl-anchor">5.3　SimPO: drops reference, no KL anchor</a></li>
369
+ <li><a href="#54-kl-handling-in-kto--ipo--orpo">5.4　KL handling in KTO / IPO / ORPO</a></li>
370
+ <li><a href="#55-recent-papers-2024-2026">5.5　Recent papers (2024-2026)</a></li>
371
+ </ul>
372
+ </li>
373
+ <li><a href="#6-theoretical-optimal-policy-of-kl-regularized-rl--reward-overoptimization">§6 Theoretical: optimal policy of KL-regularized RL + reward overoptimization</a>
374
+ <ul>
375
+ <li><a href="#61-closed-form-solution-of-kl-regularized-rl-the-mathematical-basis-for-dpo">6.1　Closed-form solution of KL-regularized RL (the mathematical basis for DPO)</a></li>
376
+ <li><a href="#62-inverting-the-optimal-policy-to-extract-implicit-reward">6.2　Inverting the optimal policy to extract implicit reward</a></li>
377
+ <li><a href="#63-rl-under-bradley-terry-preference-is-equivalent-to-dpo">6.3　RL under Bradley-Terry preference is equivalent to DPO</a></li>
378
+ <li><a href="#64-reward-overoptimization-gao-schulman-hilton-2023-icml">6.4　Reward Overoptimization (Gao, Schulman, Hilton 2023 ICML)</a></li>
379
+ <li><a href="#65-relation-between-kl-and-chi2-gap">6.5　Relation between KL and $\chi^2$ gap</a></li>
380
+ <li><a href="#66-sequence-level-vs-token-level-kl">6.6　Sequence-level vs token-level KL</a></li>
381
+ </ul>
382
+ </li>
383
+ <li><a href="#7-practice--code">§7 Practice + code</a>
384
+ <ul>
385
+ <li><a href="#71-ppo-style-in-reward-shaping-implementation">7.1　PPO style: In-reward shaping implementation</a></li>
386
+ <li><a href="#72-grpo-style-in-loss-regularization-implementation-k3">7.2　GRPO style: In-loss regularization implementation (k3)</a></li>
387
+ <li><a href="#73-dpo-closed-form-loss--implicit-reward-monitoring">7.3　DPO closed-form loss + implicit-reward monitoring</a></li>
388
+ <li><a href="#74-reward-overoptimization-monitoring">7.4　Reward overoptimization monitoring</a></li>
389
+ <li><a href="#75-engineering-checklist-debug-checklist">7.5　Engineering checklist (debug checklist)</a></li>
390
+ </ul>
391
+ </li>
392
+ <li><a href="#8-failure-modes-kl-collapse--runaway--reward-hacking">§8 Failure modes: KL Collapse / Runaway / Reward Hacking</a>
393
+ <ul>
394
+ <li><a href="#81-kl-collapse--too-large-or-entropy-too-low">8.1　KL Collapse (β too large or entropy too low)</a></li>
395
+ <li><a href="#82-kl-runaway--too-small-or-reward-signal-too-strong">8.2　KL Runaway (β too small or reward signal too strong)</a></li>
396
+ <li><a href="#83-reward-overoptimization-the-inverted-u-over-kl-distance">8.3　Reward Overoptimization (the inverted-U over KL distance)</a></li>
397
+ <li><a href="#84-length-bias-dpo-specific-also-a-form-of-insufficient-kl-anchor">8.4　Length bias (DPO-specific, also a form of insufficient KL anchor)</a></li>
398
+ <li><a href="#85-reference-policy-wrong-checkpoint-failure">8.5　Reference policy &quot;wrong checkpoint&quot; failure</a></li>
399
+ </ul>
400
+ </li>
401
+ <li><a href="#9-25-high-frequency-interview-questions">§9 25 high-frequency interview questions</a>
402
+ <ul>
403
+ <li><a href="#l1-must-know-10-questions">L1 must-know (10 questions)</a></li>
404
+ <li><a href="#l2-advanced-10-questions">L2 advanced (10 questions)</a></li>
405
+ <li><a href="#l3-top-lab-questions-5-questions">L3 top-lab questions (5 questions)</a></li>
406
+ </ul>
407
+ </li>
408
+ <li><a href="#a-appendix-reference-list">§A Appendix: reference list</a>
409
+ </li>
410
+ </ol>
411
+ </nav>
412
+
413
+ <main>
414
+ <header class="hero">
415
+
416
+ <h1>Kl Divergence Rlhf Tutorial En</h1>
417
+
418
+
419
+ <div class="meta">
420
+ <span><strong>Source:</strong> <code>docs/tutorials/kl_divergence_rlhf_tutorial_en.md</code></span>
421
+ <span><strong>SHA256:</strong> <code>4477782e0f50</code></span>
422
+ <span><strong>Rendered:</strong> 2026-05-19 19:29 UTC</span>
423
+
424
+ </div>
425
+ </header>
426
+
427
+ <h2 id="0-tldr-cheat-sheet">§0 TL;DR Cheat Sheet</h2>
428
+ <div class="callout callout-info"><div class="callout-title">8 sentences to nail KL in RLHF</div><p>one page covering interview essentials (see §1–§8 for derivations).</p></div>
429
+ <ol><li><strong>Definition</strong>: $\text{KL}(p \| q) = \mathbb{E}_{x \sim p}[\log(p(x)/q(x))]$ — non-negative, asymmetric, non-metric. In RLHF, $p = \pi_\theta$, $q = \pi_\text{ref}$, and the role is to anchor the post-RL policy near SFT to prevent reward hacking.</li><li><strong>Forward vs Reverse</strong>: convention — $p$ is the data/target, $q_\theta$ is the variational/optimization distribution. <strong>Forward KL</strong> = $\text{KL}(p\|q_\theta)$ (mass-covering); <strong>Reverse KL</strong> = $\text{KL}(q_\theta\|p)$ (mode-seeking). RLHF uses $\text{KL}(\pi_\theta \| \pi_\text{ref})$, which under this convention is <strong>reverse KL</strong> ($\pi_\theta$ is the variational side). Its mode-seeking behavior fits the RL goal — make $\pi_\theta$ pick reward-high modes inside high-density regions of $\pi_\text{ref}$ — and is also engineering-feasible (just estimate from rollout samples). Note that naming varies slightly across communities, but DPO / RLOO / "Rethinking KL" / "Comedy of Estimators" and other 2024-2026 RLHF papers all call $\text{KL}(\pi_\theta \| \pi_\text{ref})$ reverse KL.</li><li><p><strong>Three KL estimators (Schulman 2020 blog <code>joschu.net/blog/kl-approx.html</code>)</strong>:</p>
430
+ <ul><li><strong>k1</strong> = $\log(\pi_\theta/\pi_\text{ref})$ — unbiased but high variance, can be negative (cannot be read as a "distance").</li><li><strong>k2</strong> = $\tfrac{1}{2}(\log(\pi_\theta/\pi_\text{ref}))^2$ — always non-negative but <strong>biased</strong> (second-order Taylor approximation).</li><li><strong>k3</strong> = $(\pi_\text{ref}/\pi_\theta) - \log(\pi_\text{ref}/\pi_\theta) - 1$ — <strong>unbiased + non-negative + low variance</strong>, derived from the identity $\mathbb{E}_q[f(\log p/q)]$ with $f(x)=e^x-x-1$.</li></ul></li><li><strong>Two placements</strong>: (a) <strong>In-reward shaping</strong>: $\tilde{r}_t = r_t - \beta \cdot \text{KL}_t$, computed along with advantage / GAE; (b) <strong>In-loss regularization</strong>: $\mathcal{L} = \mathcal{L}_\text{PG} + \beta \cdot \mathbb{E}[\text{KL}]$. <strong>InstructGPT/Anthropic PPO use (a); GRPO uses (b) + k3 estimator</strong>. <strong>The objective is the same, but the gradient path differs</strong> — under a principled estimator (e.g. (a) k1-in-reward or (b) k2-as-loss) the two are gradient-equivalent on-policy; GRPO's (b) k3-as-loss actually carries an $O(\Delta^2)$ first-order Taylor bias (see §3.6 + Rethinking KL 2510.01555). The two placements also respond differently to PPO clip / importance-ratio truncation.</li><li><strong>β schedule</strong>: fixed β (simplest), <strong>Adaptive β</strong> (PPO-Penalty original — pulling β based on measured-KL-vs-target), Annealing β (tight early, loose later, like an SFT-to-RL transition). <strong>Too-large β fails to learn</strong> (policy stuck near ref); <strong>too-small β leads to reward hacking</strong> (policy drifts away, with long / sycophantic answers).</li><li><strong>Closed-form optimal policy of KL-regularized RL</strong>: solving the BT-style reward objective $\max_\pi \mathbb{E}[r] - \beta \cdot \text{KL}(\pi\|\pi_\text{ref})$ has the unique closed-form solution $\pi^*(y|x) \propto \pi_\text{ref}(y|x) \exp(r(x,y)/\beta)$. <strong>DPO is exactly this inverse plugged into Bradley-Terry</strong>, yielding $r = \beta \log(\pi^*/\pi_\text{ref}) + \beta\log Z$, where the partition function $\log Z$ <strong>cancels</strong> in the pairwise difference.</li><li><strong>KL's place in DPO/GRPO/SimPO</strong>: DPO's implicit reward $\hat r_\theta = \beta\log(\pi_\theta/\pi_\text{ref})$ is a sequence-level log-ratio (taking expectation under $y\sim\pi_\theta$ would equal $\beta\cdot$KL, but in DPO training $y$ comes from preference data so this is not KL itself — yet the reference appearing in the denominator still provides implicit anchoring); GRPO puts k3 KL in the loss (note that k3-as-loss yields a first-order gradient approximation, see §3.6); SimPO outright <strong>drops the reference</strong>, so it has <strong>no KL constraint</strong> — making it more sensitive to β/γ/length-norm.</li><li><strong>Failure modes</strong>: KL explosion (β too small, importance ratio too large), KL collapse (β too large or entropy too low, policy stuck), <strong>Reward Overoptimization</strong> (Gao 2023 ICML): proxy reward rises monotonically with KL, but gold reward rises then falls (inverted-U) — KL distance is the natural axis for overoptimization.</li></ol>
431
+ <h2 id="1-kl-fundamentals">§1 KL Fundamentals</h2>
432
+ <h3 id="11-definition-properties-notation">1.1　Definition, properties, notation</h3>
433
+ <p><strong>Definition</strong> (discrete):</p>
434
+ <p>$$\boxed{\;\text{KL}(p \| q) \triangleq \sum_x p(x) \log \frac{p(x)}{q(x)} = \mathbb{E}_{x \sim p}\!\left[\log \frac{p(x)}{q(x)}\right]\;}$$</p>
435
+ <p>Conventions: $0 \log 0 = 0$, $0 \log(0/0) = 0$, $p \log(p/0) = +\infty$. Continuous distributions replace $\sum$ with $\int$.</p>
436
+ <p><strong>Core properties</strong> (frequently asked together in interviews):</p>
437
+ <table><thead><tr><th>Property</th><th>One-line statement</th><th>Brief proof</th></tr></thead><tbody><tr><td><strong>Non-negative</strong></td><td>$\text{KL} \ge 0$</td><td>Jensen + $-\log$ convex: $\text{KL} = -\mathbb{E}_p \log(q/p) \ge -\log \mathbb{E}_p(q/p) = 0$</td></tr><tr><td><strong>Equality</strong></td><td>$\text{KL} = 0$ iff $p = q$ a.e.</td><td>Jensen equality condition</td></tr><tr><td><strong>Asymmetric</strong></td><td>$\text{KL}(p,q) \ne \text{KL}(q,p)$ (in KL's asymmetric arguments)</td><td>Direct example construction</td></tr><tr><td><strong>No triangle inequality</strong></td><td>Not a metric</td><td>So "KL distance" is informal/imprecise</td></tr><tr><td><strong>Convexity</strong></td><td>$\text{KL}(\cdot,\cdot)$ jointly convex in $(p,q)$</td><td>Log-sum inequality</td></tr><tr><td><strong>Chain rule</strong></td><td>Joint KL = marginal KL + expected conditional KL (see display below)</td><td>Direct log decomposition</td></tr><tr><td><strong>Reparameterization invariance</strong></td><td>Invariant under same transformation $x \mapsto T(x)$ (equality only when $T$ invertible)</td><td>Measure change + Jacobian cancellation</td></tr></tbody></table>
438
+ <p>Full chain-rule form:</p>
439
+ <p>$$\text{KL}(p(x,y) \,\|\, q(x,y)) = \text{KL}(p(x) \,\|\, q(x)) + \mathbb{E}_{p(x)}\!\big[\text{KL}(p(y|x) \,\|\, q(y|x))\big]$$</p>
440
+ <div class="callout callout-good"><div class="callout-title">Chain rule in RLHF = per-token KL summed</div><p>sequence-level KL is the <strong>expectation</strong> of the sum of trajectory log-ratios: $\text{KL}_\text{seq} = \mathbb{E}_{y\sim\pi_\theta}[\sum_t \log\pi_\theta(y_t|s_t)/\pi_\text{ref}(y_t|s_t)]$. In implementation, computing $\sum_t \log r_t$ on a single rollout is the <strong>single-MC estimator</strong> (the sequence form of k1); to get true expected KL one averages over a batch of rollouts. The <strong>true token-level KL</strong> $D_\text{KL}(\pi_\theta(\cdot|s_t)\|\pi_\text{ref}(\cdot|s_t))$ further requires a full-vocab sum (full-vocab KL) at that prefix rather than a log-ratio only at the sampled token. The two <strong>have the same expectation</strong> but use different estimator forms: sum-over-rollout of sampled log-ratios is the cheap-but-noisy estimator.</p></div>
441
+ <h3 id="12-forward-kl-vs-reverse-kl-mass-covering-or-mode-seeking">1.2　Forward KL vs Reverse KL: mass-covering or mode-seeking?</h3>
442
+ <p>Let $p$ be the data/true distribution and $q_\theta$ the parametric model. The two directions behave completely differently in fitting policies:</p>
443
+ <table><thead><tr><th>Direction</th><th>Form (display below)</th><th>Expectation under</th><th>Behavior</th><th>Classic uses</th></tr></thead><tbody><tr><td><strong>Forward KL</strong></td><td>$\text{KL}(p\,\</td><td>\,q_\theta) = \mathbb{E}<em>{p}[\log(p/q</em>\theta)]$</td><td>$p$ (data/target)</td><td><strong>mass-covering / mean-seeking</strong>: wherever $p > 0$, $q_\theta$ must be $> 0$, otherwise $\log(p/q) \to \infty$</td><td>MLE / distillation (student covers teacher)</td></tr><tr><td><strong>Reverse KL</strong></td><td>$\text{KL}(q_\theta\,\</td><td>\,p) = \mathbb{E}<em>{q</em>\theta}[\log(q_\theta/p)]$</td><td>$q_\theta$ (variational/optimization)</td><td><strong>mode-seeking / minorization</strong>: wherever $q_\theta > 0$, $p$ must be $> 0$; $q_\theta$ tends to collapse into a single mode</td><td>VI, RLHF, GAN-like training</td></tr></tbody></table>
444
+ <p>Notation conventions:</p>
445
+ <p>$$\text{Forward KL}: \text{KL}(p \,\|\, q_\theta) = \mathbb{E}_{p}\!\left[\log\frac{p}{q_\theta}\right],\qquad \text{Reverse KL}: \text{KL}(q_\theta \,\|\, p) = \mathbb{E}_{q_\theta}\!\left[\log\frac{q_\theta}{p}\right]$$</p>
446
+ <p>Classic picture (bimodal $p$, unimodal $q_\theta$):</p>
447
+ <ul><li>Forward KL fit → $q_\theta$ widens and bridges both modes (<strong>mass-covering</strong>).</li><li>Reverse KL fit → $q_\theta$ picks one mode and shrinks into it (<strong>mode-seeking</strong>).</li></ul>
448
+ <div class="callout callout-warn"><div class="callout-title">Modern consensus on the naming convention</div><p>Variational inference, DPO, RLOO, "Rethinking KL Regularization in RLHF" (arXiv 2510.01555), "A Comedy of Estimators" (arXiv 2512.21852), and other 2024-2026 RLHF papers all use the unified convention: $\text{KL}(q_\theta\|p)$ is called reverse KL ($q_\theta$ is the variational/optimization side), and $\text{KL}(p\|q_\theta)$ is called forward KL. This tutorial follows that standard convention. A few earlier RL textbooks name it by the sampling distribution — in interviews, simply state the formula to avoid label ambiguity.</p></div>
449
+ <p><strong>Which one does RLHF use?</strong> Almost all mainstream implementations (InstructGPT / Anthropic PPO / DeepSeekMath GRPO) use $\text{KL}(\pi_\theta \| \pi_\text{ref})$ (<strong>reverse-KL form</strong>: $\pi_\theta$ is the variational side, $\pi_\text{ref}$ is the target). <strong>Fundamental reason</strong>: training already samples from $\pi_\theta$ (rollout), so $\mathbb{E}_{\pi_\theta}[\log \pi_\theta/\pi_\text{ref}]$ is directly estimable from samples; meanwhile reverse KL's mode-seeking behavior matches the RL goal exactly — find the reward-high mode in the high-density region of $\pi_\text{ref}$, rather than "cover all of $\pi_\text{ref}$".</p>
450
+ <div class="callout callout-info"><div class="callout-title">Tip</div><p>Note: if we swapped roles of $\pi_\theta$ and $\pi_\text{ref}$ (forward KL = $\text{KL}(\pi_\text{ref}\|\pi_\theta)$), we would need to sample from $\pi_\text{ref}$, which engineering-wise requires IS, is expensive, and is semantically off-target (we are training $\pi_\theta$, not $\pi_\text{ref}$). So RLHF rarely uses the forward direction.</p></div>
451
+ <h3 id="13-relation-to-other-divergences">1.3　Relation to other divergences</h3>
452
+ <table><thead><tr><th>Divergence</th><th>Definition (display below)</th><th>Properties</th><th>Notes</th></tr></thead><tbody><tr><td><strong>JS</strong></td><td>Symmetric average of KL to mixture $m = (p+q)/2$</td><td>Symmetric, bounded ($\le \log 2$), square root is a metric</td><td>$\sqrt{\text{JS}}$ is a metric (Endres-Schindelin 2003 IEEE TIT 49(7)); the original GAN discriminator loss is equivalent to $2\cdot\text{JSD} - \log 4$; rarely used in RLHF</td></tr><tr><td><strong>α-divergence</strong></td><td>$\frac{1}{\alpha(1-\alpha)}(1 - \int p^\alpha q^{1-\alpha})$</td><td>Contains KL as a limit</td><td>Unified framework; $\alpha \to 1$ gives forward KL, $\alpha \to 0$ gives reverse KL</td></tr><tr><td><strong>Hellinger</strong></td><td>$H^2(p,q) = \tfrac{1}{2}\int(\sqrt{p}-\sqrt{q})^2$</td><td>Symmetric, bounded, $0 \le H^2 \le 1$</td><td>Relation to KL: $H^2 \le \tfrac{1}{2}\text{KL}$</td></tr><tr><td><strong>$\chi^2$</strong></td><td>$\chi^2(p,q) = \int \frac{(p-q)^2}{q}$</td><td>$f$-divergence at $f(t) = (t-1)^2$</td><td>Relation to KL: $\text{KL} \le \log(1 + \chi^2)$</td></tr><tr><td><strong>TV</strong></td><td>$\tfrac{1}{2}\int \lvert p-q\rvert$</td><td>A metric, $\in [0,1]$</td><td><strong>Pinsker</strong>: $\text{TV} \le \sqrt{\text{KL}/2}$</td></tr></tbody></table>
453
+ <p>Full forms of four common $f$-divergences:</p>
454
+ <p>$$\text{JS}(p, q) = \tfrac{1}{2}\text{KL}(p \,\|\, m) + \tfrac{1}{2}\text{KL}(q \,\|\, m),\quad m = (p+q)/2$$</p>
455
+ <p>$$\chi^2(p, q) = \int \frac{(p(x) - q(x))^2}{q(x)}\,dx,\qquad \text{Pinsker: } \mathrm{TV}(p,q) \le \sqrt{\text{KL}(p \,\|\, q) / 2}$$</p>
456
+ <div class="callout callout-info"><div class="callout-title">Why doesn&#x27;t RLHF use JS / α-divergence?</div><p>Mostly engineering inertia + KL's closed-form advantages: in the PPO/DPO framework, KL admits clean per-token decomposition (chain rule), has a closed-form optimum (softmax-style), and likelihood ratios directly give per-token KL increments — so no extra density-ratio estimator is needed. JS / α-divergence either lack such clean math or require an extra density-ratio estimator.</p></div>
457
+ <h3 id="14-why-does-rlhf-add-kl">1.4　Why does RLHF add KL?</h3>
458
+ <p>Put "without KL" and "with KL" objectives side by side:</p>
459
+ <p>$$\text{Without KL}:\quad \max_\pi \mathbb{E}_{x,\,y \sim \pi(\cdot|x)}[r(x,y)]$$</p>
460
+ <p>$$\text{With KL}:\quad \max_\pi \mathbb{E}_{x,\,y \sim \pi(\cdot|x)}[r(x,y)] - \beta\,\mathbb{E}_x\,\text{KL}\!\big(\pi(\cdot|x)\,\big\|\,\pi_\text{ref}(\cdot|x)\big)$$</p>
461
+ <p><strong>What happens without KL?</strong></p>
462
+ <ol><li><strong>Reward hacking</strong>: the policy finds RM blind spots (longer answers / "As an AI..." boilerplate / sycophancy / format hacks) to gain high RM scores while degrading human-perceived quality.</li><li><strong>Linguistic fluency collapses</strong>: the policy outputs token sequences the RM "likes" but humans find unintelligible (in extreme cases, degenerating to repeating a single token / completely ungrammatical).</li><li><strong>Distribution shift</strong>: the policy strays far from $\pi_\text{ref}$, to the point that even the RM cannot reliably score it (off-distribution — the more uncertain the RM, the more random the reward).</li></ol>
463
+ <p>After adding KL:</p>
464
+ <ul><li><strong>β provides implicit regularization against RM error</strong>: when the RM is unreliable, KL pulls the policy back to the SFT's known-good distribution.</li><li><strong>A closed-form optimal policy exists</strong> (derived in §6.1), giving the whole RLHF stack a mathematical foundation.</li><li><strong>DPO / KTO / GRPO all depend on this KL anchor</strong>: SimPO, which drops the reference, has empirically been observed to be more hyperparameter-sensitive.</li></ul>
465
+ <h2 id="2-kl-estimators-k1--k2--k3-interview-core">§2 KL Estimators (k1 / k2 / k3): Interview Core</h2>
466
+ <h3 id="21-problem-setup">2.1　Problem setup</h3>
467
+ <p>In practice, on every mini-batch we must estimate $\text{KL}(\pi_\theta \| \pi_\text{ref})$. But a full sum $\sum_y \pi_\theta(y) \log \pi_\theta(y)/\pi_\text{ref}(y)$ is infeasible ($y$ is a full response — combinatorial blowup). <strong>Only Monte Carlo is viable</strong>: estimate KL using samples drawn from $\pi_\theta$.</p>
468
+ <p>Let $\log r = \log(\pi_\theta(y)/\pi_\text{ref}(y))$ (not the importance ratio, but the policy log-ratio). With samples $y \sim \pi_\theta$, we want to estimate $\mathbb{E}_{\pi_\theta}[\log r] = \text{KL}(\pi_\theta \| \pi_\text{ref})$.</p>
469
+ <div class="callout callout-warn"><div class="callout-title">Notation unification</div><p>In this section $\log r$ always denotes the <strong>policy log-ratio</strong> $\log \pi_\theta - \log \pi_\text{ref}$ (<strong>not</strong> the PPO importance ratio $\pi_\theta / \pi_\text{old}$). The two have similar form but different meaning: the former measures "how far from reference", the latter measures "how far from the sampling policy".</p></div>
470
+ <h3 id="22-k1-estimator--plug-definition-in-directly">2.2　k1 estimator — "plug definition in directly"</h3>
471
+ <p>$$\boxed{\;\widehat{\text{KL}}_1 = \log\frac{\pi_\theta(y)}{\pi_\text{ref}(y)} = \log r\;}$$</p>
472
+ <ul><li><strong>Unbiased</strong>: $\mathbb{E}_{y\sim\pi_\theta}[\log r] = \text{KL}(\pi_\theta\|\pi_\text{ref})$ by definition.</li><li><strong>Can be negative</strong>: on a single sample, $\log r$ can be positive or negative (the log-ratio has no "non-negative" constraint).</li><li><strong>High variance</strong>: in the tails, $\log r$ can be very large or very small, especially in regions where $\pi_\theta$ and $\pi_\text{ref}$ have little overlap.</li></ul>
473
+ <p><strong>Problem</strong>: using k1 as a "KL monitoring metric" yields <strong>negative values</strong>; engineering loggers may display a negative KL, confusing newcomers (isn't KL supposed to be non-negative?). This is because <strong>non-negativity of the expectation does not imply non-negativity of every sample</strong>. But as per-token KL in reward shaping it is acceptable (what matters is unbiased mean).</p>
474
+ <h3 id="23-k2-estimator--l2-form">2.3　k2 estimator — "$L^2$ form"</h3>
475
+ <p>$$\boxed{\;\widehat{\text{KL}}_2 = \tfrac{1}{2}\!\left(\log\frac{\pi_\theta(y)}{\pi_\text{ref}(y)}\right)^2 = \tfrac{1}{2}(\log r)^2\;}$$</p>
476
+ <ul><li><strong>Always non-negative</strong> ✓</li><li><strong>Biased</strong>: $\mathbb{E}[\tfrac{1}{2}(\log r)^2] \ne \text{KL}$.</li><li><strong>Small-KL limit</strong>: Taylor expansion $\log r = (r - 1) - \tfrac{1}{2}(r-1)^2 + O((r-1)^3)$; when $\pi_\theta \approx \pi_\text{ref}$, $\log r$ is small, and $\tfrac{1}{2}(\log r)^2$ equals KL to second order (near $p = q$, KL = Fisher-information quadratic form).</li><li><strong>Variance smaller than k1</strong>: because squaring eliminates sign cancellation, but it is still not optimal.</li></ul>
477
+ <p>In practice k2 is mostly used for <strong>monitoring</strong> (providing a "non-negative but biased" visualization metric); it is rarely used for reward shaping.</p>
478
+ <h3 id="24-k3-estimator-schulman-2020-blog--non-negative-unbiased-value-estimator-but-as-a-loss-term-the-gradient-is-biased--see-36">2.4　k3 estimator (Schulman 2020 blog) — <strong>non-negative unbiased value estimator</strong> (but as a loss term, the gradient is biased — see §3.6)</h3>
479
+ <h4 id="241-construction">2.4.1　Construction</h4>
480
+ <p>Consider $f(x) = e^x - x - 1$. From $e^x \ge 1 + x$ (for real $x$), we get $f(x) \ge 0$, with $f(0) = 0$.</p>
481
+ <p>Substituting $x = \log(\pi_\text{ref}(y)/\pi_\theta(y)) = -\log r$:</p>
482
+ <p>$$f(-\log r) = e^{-\log r} - (-\log r) - 1 = \frac{1}{r} + \log r - 1 = \frac{\pi_\text{ref}(y)}{\pi_\theta(y)} + \log\frac{\pi_\theta(y)}{\pi_\text{ref}(y)} - 1$$</p>
483
+ <p>Equivalently (writing it with $\Delta = -\log r = \log(\pi_\text{ref}/\pi_\theta)$): $\frac{\pi_\text{ref}}{\pi_\theta} - \log\frac{\pi_\text{ref}}{\pi_\theta} - 1$ (note $-\log(\pi_\text{ref}/\pi_\theta) = \log(\pi_\theta/\pi_\text{ref})$, the two forms are equivalent).</p>
484
+ <p>Equivalently (this is the standard form in Schulman's blog):</p>
485
+ <p>$$\boxed{\;\widehat{\text{KL}}_3 = \frac{\pi_\text{ref}(y)}{\pi_\theta(y)} - \log\frac{\pi_\text{ref}(y)}{\pi_\theta(y)} - 1 = e^{\Delta} - \Delta - 1,\quad \Delta = \log\frac{\pi_\text{ref}(y)}{\pi_\theta(y)} = -\log r\;}$$</p>
486
+ <h4 id="242-three-core-properties-must-know">2.4.2　Three core properties (must-know)</h4>
487
+ <p><strong>Property 1 (non-negative)</strong>:</p>
488
+ <p>Since $e^\Delta - \Delta - 1 \ge 0$ for all $\Delta \in \mathbb{R}$ (the convex $e^\Delta$ lies above its tangent line $1 + \Delta$ at $\Delta = 0$), $\widehat{\text{KL}}_3 \ge 0$ always holds.</p>
489
+ <p><strong>Property 2 (unbiased)</strong>:</p>
490
+ <p>We show $\mathbb{E}_{y \sim \pi_\theta}[\widehat{\text{KL}}_3] = \text{KL}(\pi_\theta\|\pi_\text{ref})$.</p>
491
+ <p>$$\mathbb{E}_{\pi_\theta}\!\left[\frac{\pi_\text{ref}}{\pi_\theta}\right] = \sum_y \pi_\theta(y) \cdot \frac{\pi_\text{ref}(y)}{\pi_\theta(y)} = \sum_y \pi_\text{ref}(y) = 1$$</p>
492
+ <p>$$\mathbb{E}_{\pi_\theta}\!\left[-\log\frac{\pi_\text{ref}}{\pi_\theta}\right] = \mathbb{E}_{\pi_\theta}\!\left[\log\frac{\pi_\theta}{\pi_\text{ref}}\right] = \text{KL}(\pi_\theta\|\pi_\text{ref})$$</p>
493
+ <p>Therefore:</p>
494
+ <p>$$\mathbb{E}_{\pi_\theta}[\widehat{\text{KL}}_3] = 1 + \text{KL}(\pi_\theta\|\pi_\text{ref}) - 1 = \text{KL}(\pi_\theta\|\pi_\text{ref}) \quad \checkmark$$</p>
495
+ <p><strong>Property 3 (variance typically lower than k1)</strong>:</p>
496
+ <p>Intuition: $\widehat{\text{KL}}_3$ is $\log r$'s linear term (the $-\log r/\pi_\theta$ part, i.e. the equivalent form of k1, $\mathbb{E}_{\pi_\theta}[-\log(\pi_\text{ref}/\pi_\theta)]$) plus a <strong>mean-1 control variate</strong> $\pi_\text{ref}/\pi_\theta - 1$. The control variate reduces variance: when $r$ is large, $\log r$ is large but $1/r$ is small, and vice versa, so <strong>the two are negatively correlated</strong> and summing them gives lower variance than k1 alone.</p>
497
+ <p>Formally: $\widehat{\text{KL}}_3 = \widehat{\text{KL}}_1 + (\frac{\pi_\text{ref}}{\pi_\theta} - 1)$; the added term $(\frac{\pi_\text{ref}}{\pi_\theta} - 1)$ has expectation 0 but is strongly negatively correlated with $\log r$ → variance reduction.</p>
498
+ <div class="callout callout-info"><div class="callout-title">The elegance of k3</div><p>The derivation in Schulman's 2020 blog is equivalent to the above: he <strong>first looks for $f(x) \ge 0$ such that $\mathbb{E}_q f(\log p/q) = \text{KL}$</strong>, then picks $f(x) = e^x - x - 1$ (the simplest non-trivial non-negative differentiable choice). So k3 is not "inspired" — it is the most natural construction under the "non-negative + unbiased" constraint.</p></div>
499
+ <h4 id="243-variants-and-simplifications">2.4.3　Variants and simplifications</h4>
500
+ <p>Real-world code often writes (this also applies to the PPO ratio $r = \pi_\theta / \pi_\text{old}$, but with different semantics — that is an importance-ratio approximation, not a policy log-ratio):</p>
501
+ <pre><code class="language-python">approx_kl_to_old = ((ratio - 1) - torch.log(ratio.clamp_min(1e-8))).mean()</code></pre>
502
+ <p>I.e. $\widehat{\text{KL}}_3 \approx (r - 1) - \log r$, which is the equivalent dual of $e^\Delta - \Delta - 1$ above (treat $r = e^{\log r}$ as $e^\Delta$ with sign convention swapped). <strong>TRL / OpenRLHF / verl all default <code>approx_kl</code> to this form</strong>.</p>
503
+ <h3 id="25-comparison-table-for-the-three">2.5　Comparison table for the three</h3>
504
+ <table><thead><tr><th>Estimator</th><th>Form</th><th>Unbiased?</th><th>Non-negative?</th><th>Variance</th><th>Typical RLHF use</th></tr></thead><tbody><tr><td><strong>k1</strong></td><td>$\log r$</td><td>✅</td><td>❌</td><td>High</td><td>InstructGPT PPO reward shaping (per-token KL)</td></tr><tr><td><strong>k2</strong></td><td>$\tfrac{1}{2}(\log r)^2$</td><td>❌ value-estimator biased (second-order approximation of KL)</td><td>✅</td><td>Mid</td><td>Often used for monitoring from the value perspective; <strong>from the loss-gradient perspective, k2-as-loss is principled</strong> — on-policy it is gradient-equivalent to k1-in-reward, see §3.6</td></tr><tr><td><strong>k3</strong></td><td>$e^{-\log r} + \log r - 1$</td><td>✅ (as a <strong>value estimator</strong>)</td><td>✅</td><td>Low</td><td>Historical usage in DeepSeekMath GRPO / DAPO (when used <strong>as loss</strong>, its gradient is a biased first-order approximation, see §3.6 + the Rethinking KL paper)</td></tr></tbody></table>
505
+ <h3 id="26-code-three-estimators--variance-simulation">2.6　Code: three estimators + variance simulation</h3>
506
+ <pre><code class="language-python">import torch
507
+ import torch.nn.functional as F
508
+
509
+ def k1_estimator(logp_theta, logp_ref):
510
+ &quot;&quot;&quot;k1: log(π_θ / π_ref) — unbiased, but can be negative, high variance.&quot;&quot;&quot;
511
+ return logp_theta - logp_ref # [B, T]
512
+
513
+ def k2_estimator(logp_theta, logp_ref):
514
+ &quot;&quot;&quot;k2: 0.5 (log(π_θ / π_ref))^2 — biased, non-negative, mid variance.&quot;&quot;&quot;
515
+ return 0.5 * (logp_theta - logp_ref) ** 2
516
+
517
+ def k3_estimator(logp_theta, logp_ref):
518
+ &quot;&quot;&quot;k3 (Schulman 2020): exp(log π_ref - log π_θ) + log(π_θ/π_ref) - 1
519
+ = (π_ref/π_θ) - log(π_ref/π_θ) - 1 [letting Δ = log π_ref - log π_θ]
520
+
521
+ Value-estimator properties: unbiased, non-negative, low variance.
522
+ ⚠️ Recommended for KL value MONITORING, NOT as a loss term — k3-as-loss
523
+ gives gradient (1 - e^{-Δ})∇logπ_θ = (Δ - ½Δ² + O(Δ³))∇logπ_θ, a
524
+ first-order Taylor approximation of reverse-KL gradient with O(Δ²) bias.
525
+ For principled reverse-KL gradient, use k2-as-loss (gradient-equivalent
526
+ to k1-in-reward on-policy) or k1-in-reward via score function. See §3.6.
527
+ &quot;&quot;&quot;
528
+ log_r = logp_theta - logp_ref # log(π_θ/π_ref)
529
+ log_ratio_rev = -log_r # Δ = log(π_ref/π_θ)
530
+ return torch.exp(log_ratio_rev) - log_ratio_rev - 1
531
+
532
+ # Variance comparison: 1D synthetic
533
+ def compare_kl_estimators(n_samples=10_000, seed=0):
534
+ &quot;&quot;&quot;
535
+ Synthetic: π_θ = N(0, 1), π_ref = N(μ, 1). True KL = μ^2 / 2.
536
+ Sample y ~ π_θ; evaluate the three estimators.
537
+ &quot;&quot;&quot;
538
+ torch.manual_seed(seed)
539
+ mu = 0.5
540
+ true_kl = mu ** 2 / 2.0 # closed-form for Gaussians w/ same σ
541
+
542
+ y = torch.randn(n_samples) # y ~ N(0, 1) = π_θ
543
+ # log-pdf of N(0,1) vs N(μ,1) at y (drop common -0.5 log 2π):
544
+ logp_theta = -0.5 * y ** 2
545
+ logp_ref = -0.5 * (y - mu) ** 2
546
+
547
+ k1 = k1_estimator(logp_theta, logp_ref)
548
+ k2 = k2_estimator(logp_theta, logp_ref)
549
+ k3 = k3_estimator(logp_theta, logp_ref)
550
+
551
+ print(f&quot;true KL = {true_kl:.4f}&quot;)
552
+ for name, vals in [(&quot;k1&quot;, k1), (&quot;k2&quot;, k2), (&quot;k3&quot;, k3)]:
553
+ print(f&quot; {name}: mean={vals.mean().item():+.4f} var={vals.var().item():.4f} &quot;
554
+ f&quot;min={vals.min().item():+.3f} max={vals.max().item():+.3f}&quot;)
555
+ # Expected output: k1 mean ≈ 0.125 (unbiased), but min &lt; 0;
556
+ # k2 mean &gt; 0.125 (biased upward);
557
+ # k3 mean ≈ 0.125 (unbiased), min ≥ 0, var(k3) &lt; var(k1).</code></pre>
558
+ <div class="callout callout-good"><div class="callout-title">Running the simulation you will see</div></div>
559
+ <ul><li>k1 mean ≈ 0.125, min around −1.5 (negative possible), variance large.</li><li>k2 mean ≈ 0.18 (biased high), min ≥ 0, mid variance.</li><li>k3 mean ≈ 0.125 (unbiased), min ≥ 0, var <strong>distinctly smaller than</strong> k1.</li></ul>
560
+ <p>This matches the takeaway from Schulman's blog: <strong>k3 simultaneously delivers unbiased + non-negative + lower variance</strong>.</p>
561
+ <h2 id="3-two-placements-of-kl-in-rlhf">§3 Two placements of KL in RLHF</h2>
562
+ <h3 id="31-option-a-in-reward-shaping-ppo-rlhf--instructgpt-standard">3.1　Option A: In-reward shaping (PPO RLHF / InstructGPT standard)</h3>
563
+ <p>Bake KL <strong>into per-token reward</strong>, then run normal PPO + GAE:</p>
564
+ <p>$$\boxed{\;\tilde{r}_t = \underbrace{\mathbb{1}[t = T] \cdot R(x, y)}_{\text{terminal RM reward}} - \beta \cdot \underbrace{\log\frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_\text{ref}(y_t \mid x, y_{<t})}}_{\text{per-token KL (k1)}}\;}$$</p>
565
+ <p>Details to watch:</p>
566
+ <ul><li><strong>Per-token</strong>: every generated token contributes its log-prob ratio as part of that step's reward.</li><li><strong>k1 estimator</strong>: this directly uses $\log(\pi_\theta/\pi_\text{ref})$ (k1). Note <strong>a single token's value can be negative</strong>, but as reward shaping only the mean/sum needs to match the KL expectation.</li><li><strong>Terminal RM reward</strong>: the RM scalar for the entire answer is placed only on the last token (other tokens have RM reward = 0).</li></ul>
567
+ <p>After placement, run GAE:</p>
568
+ <p>$$\delta_t = \tilde{r}_t + \gamma V(s_{t+1}) - V(s_t),\quad A_t^{\text{GAE}} = \sum_{l \ge 0} (\gamma\lambda)^l \delta_{t+l}$$</p>
569
+ <p>The KL penalty naturally propagates into policy gradient through the advantage: <strong>at every step the "increase in action probability" is offset by KL's "pull back to reference"</strong>, and the final RL objective is expected $R$ − $\beta$ · KL.</p>
570
+ <h3 id="32-option-b-in-loss-regularization-grpo--dapo">3.2　Option B: In-loss regularization (GRPO / DAPO)</h3>
571
+ <p>Don't put KL in the reward; <strong>make it a separate loss term</strong>:</p>
572
+ <p>$$\boxed{\;\mathcal{L}_\text{full}(\theta) = -\underbrace{\mathbb{E}\!\left[\min(\rho_t A_t, \text{clip}(\rho_t, 1\!-\!\epsilon, 1\!+\!\epsilon) A_t)\right]}_{\text{PPO surrogate / GRPO surrogate}} + \beta \cdot \underbrace{\mathbb{E}\!\left[\widehat{\text{KL}}_\text{loss}(\pi_\theta \| \pi_\text{ref})\right]}_{\text{KL loss term}}\;}$$</p>
573
+ <p>GRPO's <strong>historical implementation</strong> (DeepSeekMath / DAPO) takes $\widehat{\text{KL}}_\text{loss}$ to be <strong>k3</strong> (per-token):</p>
574
+ <ul><li>Uses k3 estimator (per-token).</li><li>KL <strong>does not enter advantage computation</strong>; it is added to the loss directly as a regularizer.</li><li>Advantage comes from group-internal reward normalization ($\hat{A}_i = (r_i - \bar{r})/\sigma_r$), shared across all tokens.</li></ul>
575
+ <div class="callout callout-warn"><div class="callout-title">Important caveat</div><p>backpropagating k3 directly as a loss term is <strong>not</strong> the exact reverse-KL gradient — it is a biased first-order approximation (see §3.6 + the 2025 paper "Rethinking KL Regularization in RLHF"). The two more principled on-policy choices are: (1) <strong>k1 in reward</strong> (KL into reward, PPO/GRPO backprops via score function — gradient strictly unbiased); (2) <strong>k2 as loss</strong> ($\tfrac12 (\log r)^2$, $\nabla\mathcal L = \Delta\,\nabla\log\pi_\theta$, <strong>gradient-equivalent</strong> to (1) on-policy — both are the strict reverse-KL gradient). Off-policy still requires IS correction. §3.6 below gives the comparison table.</p></div>
576
+ <h3 id="33-mathematically-equivalent-engineering-wise-not-equivalent">3.3　Mathematically equivalent? Engineering-wise not equivalent</h3>
577
+ <p><strong>Math</strong>: both placements optimize the same objective:</p>
578
+ <p>$$J(\pi) = \mathbb{E}_{\pi}[R] - \beta \cdot \text{KL}(\pi \| \pi_\text{ref})$$</p>
579
+ <p>but the <strong>gradient propagation path differs</strong>:</p>
580
+ <ul><li>Option A: KL as part of reward → through GAE → into advantage → into PPO surrogate gradient.</li><li>Option B: KL as independent loss → direct gradient w.r.t. $\theta$.</li></ul>
581
+ <p><strong>Engineering differences</strong>:</p>
582
+ <table><thead><tr><th>Dimension</th><th>In-reward (Option A)</th><th>In-loss (Option B)</th></tr></thead><tbody><tr><td>KL estimator</td><td>k1 (per-token can be negative, acceptable as reward)</td><td>k3 (unbiased + non-negative, more stable)</td></tr><tr><td>Relation to PPO clip</td><td>KL is truncated by clip (importance-ratio truncation cuts KL along with it)</td><td>KL is independent, unaffected by clip</td></tr><tr><td>Advantage interpretation</td><td>Advantage contains KL → "net advantage"</td><td>Advantage is reward-only → "pure advantage"</td></tr><tr><td>Hyperparameter sensitivity</td><td>β directly affects reward scale → must coordinate with RM scale</td><td>β is independent of reward, but KL-vs-PG balance matters</td></tr><tr><td>Monitoring metrics</td><td>Watch token-level KL inside reward</td><td>Watch the KL loss curve independently</td></tr></tbody></table>
583
+ <div class="callout callout-warn"><div class="callout-title">How PPO clip masks the KL signal</div><p>In Option A, when the importance ratio $\rho_t$ falls outside the clip range ($\rho_t > 1+\epsilon$ with $\tilde A_t > 0$ or $\rho_t < 1-\epsilon$ with $\tilde A_t < 0$), PPO truncates the surrogate into a constant (zero gradient w.r.t. θ), and the KL term in $\tilde r_t$ at that step also becomes ineffective. This means <strong>when the policy drifts too far, PPO clip ironically disables the KL anchor</strong> — this is part of the motivation for Option B (KL in loss is always active).</p></div>
584
+ <h3 id="34-practice-which-goes-where">3.4　Practice: which goes where?</h3>
585
+ <table><thead><tr><th>Algorithm</th><th>KL placement</th><th>Estimator</th><th>Source</th></tr></thead><tbody><tr><td>InstructGPT PPO</td><td><strong>In-reward</strong></td><td>k1</td><td>Ouyang 2022 NeurIPS</td></tr><tr><td>Anthropic RLHF</td><td><strong>In-reward</strong></td><td>k1 + adaptive β</td><td>Bai 2022 arXiv 2204.05862</td></tr><tr><td>GRPO / DeepSeekMath</td><td><strong>In-loss</strong></td><td>k3</td><td>Shao 2024 arXiv 2402.03300</td></tr><tr><td>DAPO</td><td><strong>In-loss</strong> (often KL turned off or low-weighted in real configs)</td><td>k3</td><td>Yu 2025 arXiv 2503.14476</td></tr><tr><td>DPO</td><td><strong>Implicit in-loss</strong> (via $\pi_\text{ref}$ in the log-ratio denominator)</td><td>—</td><td>Rafailov 2023 NeurIPS</td></tr><tr><td>SimPO</td><td><strong>No reference, no KL</strong></td><td>—</td><td>Meng 2024 NeurIPS</td></tr></tbody></table>
586
+ <div class="callout callout-info"><div class="callout-title">DAPO field experience</div><p>ByteDance's verl team has reported in engineering write-ups that the KL term is <strong>often turned off or set to a very small β</strong> ($10^{-4}$ order) for large-scale math/code RL training. Reasons: the reward is already rule-based (close to ground truth), there is no RM hacking to worry about, and the KL anchor actually slows down training. This is the extreme "Option B + β ≈ 0" version.</p></div>
587
+ <h3 id="36-gradient-bias-analysis-of-estimator-placement-rethinking-kl-regularization-in-rlhf">3.6　Gradient bias analysis of estimator placement (Rethinking KL Regularization in RLHF)</h3>
588
+ <div class="callout callout-info"><div class="callout-title">Key reference</div><p>Kezhao Liu et al., "Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization", arXiv 2510.01555 (2025-10-02). This paper systematically distinguishes <strong>value estimation</strong> ("is the KL value itself estimated correctly?") from <strong>gradient optimization</strong> ("after differentiating w.r.t. θ, is the gradient actually the reverse-KL gradient?"), and its conclusions are not fully aligned with the historical GRPO implementation.</p></div>
589
+ <h4 id="setup-and-three-estimator-placements">Setup and three estimator placements</h4>
590
+ <p>For brevity, let $\Delta_t = \log\!\frac{\pi_\theta(y_t|s_t)}{\pi_\text{ref}(y_t|s_t)}$ (per-token log-ratio, <strong>with θ gradient</strong>), $\hat\Delta_t = \text{stop\_grad}(\Delta_t)$ (detached copy recorded at rollout time). Three common placements:</p>
591
+ <table><thead><tr><th>Placement</th><th>Where KL appears</th><th>Per-token form</th><th>Gradient contribution to θ</th></tr></thead><tbody><tr><td><strong>(P1) k1 in reward</strong></td><td>Enter reward / advantage</td><td>$\hat r_t \leftarrow r_t - \beta\,\hat\Delta_t$, then PPO surrogate</td><td>$-\beta\cdot \mathbb{E}[\nabla_\theta\log\pi_\theta\cdot \hat\Delta_t]$, <strong>the strict reverse-KL score-function gradient</strong> (on-policy)</td></tr><tr><td><strong>(P2) k2 as loss</strong></td><td>Direct loss term</td><td>$\mathcal{L}_\text{KL} = \tfrac12 \Delta_t^2$</td><td>$\nabla\mathcal{L}_\text{KL} = \Delta_t\,\nabla_\theta\log\pi_\theta$ (using $\nabla_\theta\Delta_t = \nabla_\theta\log\pi_\theta$). On-policy, $\mathbb{E}_{y\sim\pi_\theta}[\Delta\,\nabla\log\pi_\theta] = \nabla\text{KL}$ — <strong>gradient-equivalent</strong> to (P1), strictly principled</td></tr><tr><td><strong>(P3) k3 as loss</strong></td><td>Direct loss term</td><td>$\mathcal{L}_\text{KL} = e^{-\Delta_t} + \Delta_t - 1$</td><td>$\nabla\mathcal{L}_\text{KL} = (1 - e^{-\Delta_t})\nabla_\theta\log\pi_\theta$; this is a <strong>first-order Taylor approximation</strong> of the reverse-KL gradient, since $1-e^{-\Delta} = \Delta - \tfrac12\Delta^2 + O(\Delta^3)$, with $O(\Delta^2)$ bias</td></tr></tbody></table>
592
+ <h4 id="why-k3-as-loss-yields-a-biased-gradient-key-intuition">Why k3 as loss yields a biased gradient (key intuition)</h4>
593
+ <p>k3 as a <strong>value estimator</strong> (computing the KL numeric value) is unbiased: $\mathbb{E}_{\pi_\theta}[\widehat{\text{KL}}_3] = \text{KL}(\pi_\theta\|\pi_\text{ref})$ (see §2.4.2 Property 2). But when <strong>backpropagating w.r.t. θ</strong>, autograd computes $\nabla_\theta\widehat{\text{KL}}_3$, <strong>not</strong> $\nabla_\theta\,\mathbb{E}_{\pi_\theta}[\widehat{\text{KL}}_3]$. The latter is given by the score-function trick:</p>
594
+ <p>$$\nabla_\theta\,\text{KL}(\pi_\theta\|\pi_\text{ref}) = \mathbb{E}_{y\sim\pi_\theta}\!\big[\nabla_\theta\log\pi_\theta(y) \cdot \log(\pi_\theta(y)/\pi_\text{ref}(y))\big] \;+\; \mathbb{E}_{\pi_\theta}[\nabla_\theta\log\pi_\theta]\,\quad\text{(=0)}$$</p>
595
+ <p>= $\mathbb{E}[\nabla\log\pi_\theta \cdot \Delta]$ (<strong>this is the gradient given by (P1)</strong>, equivalent to treating $\Delta$ as a detached reward via the score function).</p>
596
+ <p>Meanwhile (P3) <code>loss.backward()</code> gives $\nabla_\theta(e^{-\Delta} + \Delta - 1) = (1 - e^{-\Delta})\nabla_\theta\Delta = (1 - 1/r)\nabla_\theta\log\pi_\theta$. Taylor expansion: $1 - 1/r = \Delta - \tfrac12\Delta^2 + O(\Delta^3)$. So (P3) ≈ (P1) only in the $\Delta\to 0$ neighborhood; far from 0 there is $O(\Delta^2)$ bias.</p>
597
+ <h4 id="practical-rules-combining-rethinking-kl--comedy-of-estimators">Practical rules (combining Rethinking KL + Comedy of Estimators)</h4>
598
+ <table><thead><tr><th>Scenario</th><th>Recommended placement</th><th>Notes</th></tr></thead><tbody><tr><td><strong>On-policy (rollout-step backprop, no PPO mini-batch multi-step updates)</strong></td><td><strong>(P1) k1 in reward</strong> or <strong>(P2) k2 as loss</strong></td><td>The two are <strong>gradient-equivalent</strong> on-policy, both being the strict reverse-KL gradient</td></tr><tr><td><strong>Off-policy (PPO mini-batch passes over the same data several times)</strong></td><td>(P1) k1 in reward + <strong>IS correction</strong>: $\rho\cdot\Delta$, $\rho = \pi_\theta^\text{new}/\pi_\theta^\text{old}$</td><td>Otherwise the reward mismatches the current policy</td></tr><tr><td><strong>Historical GRPO/DeepSeekMath/DAPO implementations</strong></td><td>(P3) k3 as loss</td><td>Carries $O(\Delta^2)$ gradient bias; in practice $\Delta$ is small (β anchors tight) and it still trains; but <strong>not theoretically principled</strong></td></tr><tr><td><strong>Value-side monitoring</strong> (looking at how large the KL is)</td><td>k3 estimator remains first choice</td><td>Unbiased, non-negative, low variance — exactly the three properties from §2.4</td></tr></tbody></table>
599
+ <div class="callout callout-info"><div class="callout-title">Summary phrasing</div><p>(interview-ready): k3 is an <strong>excellent KL value estimator</strong> (unbiased + non-negative + low variance); but the gradient produced by backpropagating it as a loss is a first-order Taylor approximation of the reverse-KL gradient, with bias. Rethinking KL recommends (P1) or (P2) on-policy; DAPO/GRPO historically use (P3) primarily as engineering convention + because the approximation holds in the small-β regime.</p></div>
600
+ <h2 id="4--schedule">§4 β schedule</h2>
601
+ <h3 id="41-fixed---baseline">4.1　Fixed β — baseline</h3>
602
+ <p>The most common. $\beta \in [0.01, 0.5]$; <strong>InstructGPT reports β = 0.02 as a reference value</strong> (per-token KL). Pros: simple and reproducible. Cons: KL typically grows from small to large during training; a fixed β may "fail to suppress" late in training or "over-suppress" early on.</p>
603
+ <h3 id="42-adaptive---schulman-ppo-penalty-original">4.2　Adaptive β — Schulman PPO-Penalty original</h3>
604
+ <p>The PPO paper (Schulman 2017 arXiv 1707.06347) originally proposed two versions: <strong>Clip</strong> (current mainstream) and <strong>Penalty</strong> (adaptive KL). The Penalty form:</p>
605
+ <p>$$\mathcal{L} = \mathbb{E}[r_t A_t] - \beta_k \cdot \text{KL}(\pi_{\theta_\text{old}} \| \pi_\theta)$$</p>
606
+ <p><strong>β adaptive rule</strong> (after each epoch, look at measured KL $d$):</p>
607
+ <ul><li>If $d < d_\text{target} / 1.5$: $\beta \leftarrow \beta / 2$ (KL below target, relax)</li><li>If $d > d_\text{target} \times 1.5$: $\beta \leftarrow \beta \times 2$ (KL above target, tighten)</li><li>Otherwise β unchanged</li></ul>
608
+ <p>Intuition: treat β as the P term of a PID controller with target = expected KL (e.g. $d_\text{target} = 0.01$). This approach also appeared in InstructGPT and later Anthropic work (Anthropic's helpfulness/harmlessness reports after 1707.06347 contain similar adaptive-β descriptions).</p>
609
+ <h3 id="43--annealing-schedule--analogous-to-learning-rate-schedule">4.3　β annealing schedule — analogous to learning rate schedule</h3>
610
+ <p>View β as a time function $\beta(t)$:</p>
611
+ <ul><li><strong>Tight early, loose late</strong>: $\beta(t) = \beta_0 \cdot \exp(-t / \tau)$. Intuition: early on, the policy is near ref — strengthen anchor to prevent instability; later the policy has learned — relax to let it explore reward.</li><li><strong>Loose early, tight late</strong>: opposite direction; explore RM signal early, anchor late. Rare.</li><li><strong>Cosine / linear decay</strong>: reference learning rate schedules.</li></ul>
612
+ <p>In RLHF engineering, annealing is uncommon — adaptive β is more robust than schedules (no schedule shape to tune).</p>
613
+ <h3 id="44--failure-modes">4.4　β failure modes</h3>
614
+ <table><thead><tr><th>β setting</th><th>Symptom</th><th>Diagnosis</th></tr></thead><tbody><tr><td><strong>β too large</strong> ($> 1$)</td><td>KL ≈ 0, policy stuck near ref, RM reward doesn't rise</td><td>Reward curve is flat / <code>chosen_logp - rejected_logp</code> doesn't separate</td></tr><tr><td><strong>β too small</strong> ($< 0.001$)</td><td>KL explodes (runaway), policy increasingly long / sycophantic / repetitive</td><td>KL curve climbing, generation length growing, human eval dropping</td></tr><tr><td><strong>β jumps</strong></td><td>Loss curve has discontinuities</td><td>Adaptive frequency too high / target_kl too strict</td></tr></tbody></table>
615
+ <div class="callout callout-warn"><div class="callout-title">Is InstructGPT&#x27;s β = 0.02 universally right?</div><p>Depends on task. <strong>Math/code RL</strong> typically needs smaller β (DeepSeekMath reports β ≈ 0.04, DAPO frequently uses much smaller β or 0) because reward is close to ground truth. <strong>Helpfulness/safety RL</strong> needs larger β (≥ 0.1) to prevent reward hacking, because neural RMs are easy to hack. That's why "β is not a universal hyperparameter".</p></div>
616
+ <h3 id="45-adaptive--code-example">4.5　Adaptive β code example</h3>
617
+ <pre><code class="language-python">class AdaptiveKLController:
618
+ &quot;&quot;&quot;
619
+ Schulman 2017 PPO-Penalty style adaptive β controller.
620
+ Call update(measured_kl) after every PPO epoch.
621
+ &quot;&quot;&quot;
622
+ def __init__(self, beta_init=0.02, target_kl=0.01, horizon=10000):
623
+ self.beta = beta_init
624
+ self.target_kl = target_kl
625
+ self.horizon = horizon # smoothing coefficient (larger = slower)
626
+
627
+ def update(self, measured_kl, n_steps):
628
+ # proportional update; clip to prevent extreme jumps
629
+ proportional_error = max(-0.2, min(0.2,
630
+ measured_kl / self.target_kl - 1.0))
631
+ mult = 1.0 + proportional_error * n_steps / self.horizon
632
+ self.beta *= mult
633
+ # safety clamp
634
+ self.beta = max(1e-4, min(1.0, self.beta))
635
+ return self.beta</code></pre>
636
+ <p>Usage:</p>
637
+ <pre><code class="language-python">kl_ctrl = AdaptiveKLController(beta_init=0.02, target_kl=0.01)
638
+ for epoch in range(num_epochs):
639
+ # ... rollout and PPO update ...
640
+ measured_kl = compute_mean_kl(policy, ref_policy, batch) # k3 recommended
641
+ kl_ctrl.update(measured_kl, n_steps=batch_size)
642
+ current_beta = kl_ctrl.beta</code></pre>
643
+ <div class="callout callout-info"><div class="callout-title">Analogy: adaptive β vs PID controller</div><p>Above is only a P term (proportional); HuggingFace TRL's implementation also uses only P. You could add I (integral: accumulated error) and D (derivative: rate of change) for full PID, but in practice P alone suffices; adding D tends to oscillate due to noisy KL estimates.</p></div>
644
+ <h2 id="5-kls-relation-to-dpo--grpo--simpo--kto--ipo">§5 KL's relation to DPO / GRPO / SimPO / KTO / IPO</h2>
645
+ <h3 id="51-dpo-implicit-reward----sequence-log-density-ratio">5.1　DPO: implicit reward = β × sequence log-density ratio</h3>
646
+ <p>DPO's implicit reward is $\hat{r}_\theta(x, y) = \beta \log(\pi_\theta(y|x) / \pi_\text{ref}(y|x))$, i.e. the sequence-level log-density ratio for the entire $y$:</p>
647
+ <p>$$\hat{r}_\theta(x, y) = \beta \sum_t \log\frac{\pi_\theta(y_t | x, y_{<t})}{\pi_\text{ref}(y_t | x, y_{<t})}$$</p>
648
+ <div class="callout callout-warn"><div class="callout-title">Important distinction: the implicit reward is not KL itself</div><p>$\hat{r}_\theta$ is <strong>a single sequence's pointwise log-ratio</strong>, <strong>not</strong> KL. Only when taking expectation over $y\sim\pi_\theta$ does $\mathbb{E}_{y\sim\pi_\theta}[\hat{r}_\theta(x,y)] = \beta\cdot\text{KL}(\pi_\theta(\cdot|x)\,\|\,\pi_\text{ref}(\cdot|x))$ (this is the sequence-level form of the k1 KL estimator). But <strong>during DPO training, $y_w, y_l$ do not come from $\pi_\theta$ — they come from fixed preference data</strong>, so the training-stage $\hat{r}_\theta$ cannot be read as KL; it is just a pairwise log-ratio difference.</p>
649
+ <p><strong>Correct framing</strong>: the dual RLHF interpretation of the DPO loss is to view $\hat{r}_\theta$ as a reward, with the BT model giving probability $\sigma(\hat{r}_w - \hat{r}_l)$; it corresponds to the reward expression obtained by inverting the "KL-regularized RL closed-form optimal policy $\pi^* \propto \pi_\text{ref}\exp(r/\beta)$". $\pi_\theta$ is implicitly KL-constrained under the implicit-RLHF view (because $\pi_\theta/\pi_\text{ref}$ appears in $\hat r_\theta$), but DPO optimizes a pairwise margin rather than an explicit KL margin.</p></div>
650
+ <h4 id="511-dpo-closed-form-derivation-review">5.1.1　DPO closed-form derivation review</h4>
651
+ <p>KL-regularized objective:</p>
652
+ <p>$$\max_\pi \mathbb{E}_{x,\, y \sim \pi}[r(x, y)] - \beta\, \text{KL}(\pi \| \pi_\text{ref})$$</p>
653
+ <p>For a single $x$, using Lagrangian + differentiation (detailed derivation in §6.1):</p>
654
+ <p>$$\pi^*(y|x) = \frac{1}{Z(x)} \pi_\text{ref}(y|x) \exp\!\left(\frac{r(x, y)}{\beta}\right)$$</p>
655
+ <p>Inversion:</p>
656
+ <p>$$r(x, y) = \beta \log\frac{\pi^*(y|x)}{\pi_\text{ref}(y|x)} + \beta \log Z(x)$$</p>
657
+ <p>Plugging into Bradley-Terry $P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))$, the $\beta \log Z$ cancels, giving DPO:</p>
658
+ <p>$$\boxed{\;\mathcal{L}_\text{DPO}(\theta) = -\mathbb{E}_{(x,y_w,y_l)}\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_\text{ref}(y_l|x)}\right)\;}$$</p>
659
+ <h4 id="512-kl-interpretation-of-dpos-gradient">5.1.2　KL interpretation of DPO's gradient</h4>
660
+ <p>$$\nabla_\theta \mathcal{L}_\text{DPO} = -\beta \mathbb{E}\!\Big[\sigma(\hat{r}_l - \hat{r}_w)\big(\nabla_\theta \log\pi_\theta(y_w|x) - \nabla_\theta \log\pi_\theta(y_l|x)\big)\Big]$$</p>
661
+ <p><strong>Interpretation</strong>:</p>
662
+ <ul><li>$\sigma(\hat{r}_l - \hat{r}_w)$ is "the current model's confidence in mis-ranking the preference order".</li><li>$\nabla \log\pi(y_w) - \nabla \log\pi(y_l)$ is "raise probability of $y_w$ + lower $y_l$".</li><li>$\beta$ appears twice: once inside the implicit reward $\hat{r}$ (determining the sigmoid's interior), once explicitly as a gradient scale. So <strong>β controls both KL strength and gradient magnitude in DPO</strong> — unlike in RLHF where β controls only one thing.</li></ul>
663
+ <div class="callout callout-warn"><div class="callout-title">DPO&#x27;s β hyperparameter pitfall</div><p>Tuning β in DPO is not semantically equivalent to tuning β in RLHF. In PPO, β only affects KL penalty strength; in DPO, β simultaneously affects: (1) implicit-reward scale, (2) gradient magnitude (outer β), (3) sigmoid saturation position (inner β). <strong>Empirically $\beta \in [0.05, 0.5]$ but the optimal differs per task</strong>.</p></div>
664
+ <h3 id="52-grpo-k3-kl-in-loss">5.2　GRPO: k3 KL in loss</h3>
665
+ <p>DeepSeekMath GRPO uses k3, with KL as a separate loss term:</p>
666
+ <p>$$L^\text{GRPO}(\theta) = \mathbb{E}\!\left[\frac{1}{G}\sum_{i=1}^G \frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\!\Big(\min(\rho_{i,t} \hat{A}_{i,t}, \text{clip}(\rho_{i,t}, 1{-}\epsilon, 1{+}\epsilon)\hat{A}_{i,t}) - \beta\,\widehat{\text{KL}}_3^{i,t}\Big)\right]$$</p>
667
+ <p>where $\widehat{\text{KL}}_3^{i,t} = e^{-\log r_{i,t}} + \log r_{i,t} - 1$ (per-token k3) and $\log r_{i,t} = \log\pi_\theta(y_{i,t}|\cdot) - \log\pi_\text{ref}(y_{i,t}|\cdot)$.</p>
668
+ <p><strong>Why did GRPO historically pick k3</strong> (note: these are all <strong>value-estimator</strong> properties; they do not directly guarantee gradient correctness — see §3.6):</p>
669
+ <ol><li><strong>Non-negative</strong>: the loss-value display is intuitive (k1 values can be negative, making optimizer monitoring curves look odd during debugging — but k1 in reward is harmless).</li><li><strong>Value estimator unbiased</strong>: $\mathbb{E}_{\pi_\theta}[\widehat{\text{KL}}_3] = \text{KL}$ (<strong>unbiased only as a numerical KL estimate; when backpropagated w.r.t. θ, the gradient is a first-order Taylor approximation with $O(\Delta^2)$ bias</strong>).</li><li><strong>Value estimator low variance</strong>: smaller variance than k1's value (smoother monitoring curves).</li><li><strong>Doesn't enter advantage</strong>: relatively decouples from In-reward placement, easier to diagnose (KL loss and PG loss monitored separately).</li></ol>
670
+ <div class="callout callout-warn"><div class="callout-title">GRPO&#x27;s k3-as-loss is not the theoretically principled best choice</div><p>The Rethinking KL paper (arXiv 2510.01555) systematically analyzes that on-policy, (P1) k1-in-reward or (P2) k2-as-loss gives the strict reverse-KL gradient; GRPO/DAPO's actual engineering with tight β and small $\Delta$ makes the first-order approximation error limited — but <strong>in an interview, one should distinguish "value-estimator advantage" from "loss-gradient correctness"</strong>.</p></div>
671
+ <h3 id="53-simpo-drops-reference-no-kl-anchor">5.3　SimPO: drops reference, no KL anchor</h3>
672
+ <p>SimPO redefines the implicit reward as length-normalized log-prob:</p>
673
+ <p>$$r_\text{SimPO}(x, y) = \frac{\beta}{|y|} \log \pi_\theta(y|x)$$</p>
674
+ <p><strong>Key difference</strong>: <strong>no $\pi_\text{ref}$</strong>, hence <strong>no KL term</strong>. Loss:</p>
675
+ <p>$$\mathcal{L}_\text{SimPO} = -\mathbb{E}\log\sigma\!\left(\frac{\beta}{|y_w|}\log\pi(y_w) - \frac{\beta}{|y_l|}\log\pi(y_l) - \gamma\right)$$</p>
676
+ <p><strong>Consequences</strong>:</p>
677
+ <ul><li>✅ Saves one reference policy at training time (memory halved).</li><li>❌ No KL anchor, so the distance from SFT is <strong>completely uncontrolled</strong>.</li><li>Empirically, SimPO outperforms DPO on certain benchmarks (AlpacaEval-2 / Arena-Hard); but generation quality on OOD prompts is <strong>less stable than DPO</strong>. SimPO's design is "length-norm + margin replaces KL anchor" — it assumes "for short prompt-response tasks, length-norm + margin is enough to constrain the policy", which <strong>doesn't hold universally</strong>.</li></ul>
678
+ <div class="callout callout-info"><div class="callout-title">Engineering impact of SimPO&#x27;s missing KL</div><p>In practice, SimPO-trained models exhibit <strong>worse robustness on repetition patterns, generation length, and prompt-perturbation than DPO</strong>. That's why many production RLHF deployments still use DPO + small β rather than SimPO — the KL anchor has a cost, but it has value.</p></div>
679
+ <h3 id="54-kl-handling-in-kto--ipo--orpo">5.4　KL handling in KTO / IPO / ORPO</h3>
680
+ <table><thead><tr><th>Algorithm</th><th>KL form</th><th>Notes</th></tr></thead><tbody><tr><td><strong>KTO</strong> (Ethayarajh 2024 ICML)</td><td>Through reference point $z_0 = \mathbb{E}[\beta\cdot\text{KL}]$ (estimated from batch mismatched pairs, detached)</td><td>KL acts implicitly as an "anchor": how far implicit reward deviates from $z_0$</td></tr><tr><td><strong>IPO</strong> (Azar 2024 AISTATS)</td><td>Same as DPO ($\log(\pi/\pi_\text{ref})$), but loss changes from sigmoid to squared</td><td>Prevents unbounded growth of $\hat{r}$ under deterministic preferences</td></tr><tr><td><strong>ORPO</strong> (Hong 2024 EMNLP)</td><td><strong>No reference model</strong> (similar to SimPO), uses odds-ratio instead</td><td>Single-stage SFT + preference learning; KL anchor implicitly in SFT loss</td></tr></tbody></table>
681
+ <h3 id="55-recent-papers-2024-2026">5.5　Recent papers (2024-2026)</h3>
682
+ <table><thead><tr><th>Paper</th><th>Claim</th><th>KL angle</th></tr></thead><tbody><tr><td><strong>DeepSeekMath GRPO</strong> (Shao et al. 2024 arXiv 2402.03300)</td><td>Group-relative normalization + k3 KL in loss</td><td>First default use of k3 KL in LLM RL</td></tr><tr><td><strong>DPO Implicit Reward Models</strong> (Rafailov 2023 NeurIPS)</td><td>DPO's implicit reward equals the KL log-ratio</td><td>DPO is fundamentally the inverse of KL-regularized optimization</td></tr><tr><td><strong>Reward Model Overoptimization Scaling Laws</strong> (Gao, Schulman, Hilton 2023 ICML)</td><td>Gold reward shows an inverted-U over KL distance</td><td>KL is the natural x-axis for overoptimization</td></tr><tr><td><strong>DAPO</strong> (Yu et al. 2025 ByteDance arXiv 2503.14476)</td><td>clip-higher + dynamic sampling + token-level loss</td><td>KL term often set small or off, with prompt-cleaning dynamic sampling</td></tr><tr><td><strong>Cohere DRO / OPO</strong> (various 2024-2025 works)</td><td>offline IS-correction RL with KL</td><td>Combines IS-correction with KL anchor</td></tr><tr><td><strong>"Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization"</strong> (Kezhao Liu et al. 2025-10, arXiv 2510.01555)</td><td>Systematically distinguishes KL value estimation from gradient optimization: k3-as-loss is a biased first-order approximation; recommends (1) k1 in reward (strict reverse-KL score-function gradient) or (2) <strong>k2 as loss</strong> (value-estimator biased but on-policy loss gradient strictly equivalent to k1-in-reward / reverse-KL gradient); off-policy needs IS correction</td><td>Historical GRPO's "k3 as loss" is engineering convention, not theoretically principled</td></tr><tr><td><strong>"A Comedy of Estimators: On KL Regularization in RL Training of LLMs"</strong> (Vedant Shah et al. 2025-12, arXiv 2512.21852, v3 2026-03)</td><td>Systematic comparison across many RL algorithms + estimator-placement combinations of k1/k2/k3 estimator bias and gradient bias; analyzes placement-effect</td><td>No universally best estimator exists; reward-shaping vs loss-term have different sweet spots</td></tr></tbody></table>
683
+ <h2 id="6-theoretical-optimal-policy-of-kl-regularized-rl--reward-overoptimization">§6 Theoretical: optimal policy of KL-regularized RL + reward overoptimization</h2>
684
+ <h3 id="61-closed-form-solution-of-kl-regularized-rl-the-mathematical-basis-for-dpo">6.1　Closed-form solution of KL-regularized RL (the mathematical basis for DPO)</h3>
685
+ <p><strong>Theorem</strong> (closed-form solution of KL-regularized policy optimization):</p>
686
+ <p>Consider:</p>
687
+ <p>$$\max_\pi J(\pi) = \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)] - \beta\, \text{KL}\!\big(\pi(\cdot|x) \| \pi_\text{ref}(\cdot|x)\big)$$</p>
688
+ <p>where $\pi$ is a distribution over any $x$, and $\pi_\text{ref}$ is strictly positive ($\pi_\text{ref}(y|x) > 0$ for all $y$).</p>
689
+ <p><strong>Unique optimal solution</strong>:</p>
690
+ <p>$$\boxed{\;\pi^*(y|x) = \frac{1}{Z(x)}\, \pi_\text{ref}(y|x)\, \exp\!\left(\frac{r(x, y)}{\beta}\right),\quad Z(x) = \sum_{y'} \pi_\text{ref}(y'|x) \exp\!\left(\frac{r(x, y')}{\beta}\right)\;}$$</p>
691
+ <p><strong>Proof</strong> (Lagrangian):</p>
692
+ <p>Fix $x$ and write the objective (general continuous/discrete; discrete shown):</p>
693
+ <p>$$\mathcal{L}_x[\pi] = \sum_y \pi(y|x) r(x, y) - \beta \sum_y \pi(y|x) \log\frac{\pi(y|x)}{\pi_\text{ref}(y|x)} - \mu\!\left(\sum_y \pi(y|x) - 1\right)$$</p>
694
+ <p>where $\mu$ is the Lagrange multiplier for normalization (sign note: when deriving KKT from max, writing $\sum_y\pi = 1$ as $-\mu(\sum - 1)$ keeps the interior-optimum condition concise).</p>
695
+ <p>Differentiating w.r.t. $\pi(y|x)$:</p>
696
+ <p>$$\frac{\partial \mathcal{L}_x}{\partial \pi(y|x)} = r(x, y) - \beta \log\frac{\pi(y|x)}{\pi_\text{ref}(y|x)} - \beta - \mu = 0$$</p>
697
+ <p>Rearranging:</p>
698
+ <p>$$\log\frac{\pi(y|x)}{\pi_\text{ref}(y|x)} = \frac{r(x, y) - \mu - \beta}{\beta}$$</p>
699
+ <p>Exponentiating:</p>
700
+ <p>$$\pi(y|x) = \pi_\text{ref}(y|x) \exp\!\left(\frac{r(x, y) - \mu - \beta}{\beta}\right) = \pi_\text{ref}(y|x) \exp\!\left(\frac{r(x, y)}{\beta}\right) \cdot e^{-(\mu+\beta)/\beta}$$</p>
701
+ <p>Plugging into the normalization constraint $\sum_y \pi(y|x) = 1$:</p>
702
+ <p>$$e^{(\mu+\beta)/\beta} = \sum_y \pi_\text{ref}(y|x) \exp\!\left(\frac{r(x, y)}{\beta}\right) = Z(x)$$</p>
703
+ <p>So $e^{-(\mu+\beta)/\beta} = 1/Z(x)$, giving:</p>
704
+ <p>$$\pi^*(y|x) = \frac{1}{Z(x)} \pi_\text{ref}(y|x) \exp\!\left(\frac{r(x, y)}{\beta}\right) \quad \blacksquare$$</p>
705
+ <p><strong>Uniqueness</strong>: because $J$ is <strong>strictly concave</strong> in $\pi$ (KL is convex, negating gives strictly concave; the reward term is linear), concave optimization has a unique optimum.</p>
706
+ <h3 id="62-inverting-the-optimal-policy-to-extract-implicit-reward">6.2　Inverting the optimal policy to extract implicit reward</h3>
707
+ <p>Taking the log of $\pi^*$ from §6.1:</p>
708
+ <p>$$\log\pi^*(y|x) = \log\pi_\text{ref}(y|x) + \frac{r(x, y)}{\beta} - \log Z(x)$$</p>
709
+ <p>Inverting for $r$:</p>
710
+ <p>$$\boxed{\;r(x, y) = \beta\log\frac{\pi^*(y|x)}{\pi_\text{ref}(y|x)} + \beta\log Z(x)\;}$$</p>
711
+ <p><strong>This is DPO's implicit reward</strong>. $\beta \log Z(x)$ is the partition function, which <strong>depends only on $x$, not on $y$</strong> — so it <strong>cancels</strong> in the Bradley-Terry difference $r(x, y_w) - r(x, y_l)$.</p>
712
+ <h3 id="63-rl-under-bradley-terry-preference-is-equivalent-to-dpo">6.3　RL under Bradley-Terry preference is equivalent to DPO</h3>
713
+ <p>BT model: $P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))$.</p>
714
+ <p>Plug §6.2 in:</p>
715
+ <p>$$r(x, y_w) - r(x, y_l) = \beta \log\frac{\pi^*(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta\log\frac{\pi^*(y_l|x)}{\pi_\text{ref}(y_l|x)}$$</p>
716
+ <p>Replace $\pi^*$ with learnable $\pi_\theta$, and take NLL over preference data:</p>
717
+ <p>$$\mathcal{L}_\text{DPO}(\theta) = -\mathbb{E}\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_\text{ref}(y_l|x)}\right)$$</p>
718
+ <p><strong>This is the DPO formula in §5.1</strong>. So <strong>DPO ≡ Bradley-Terry preference + KL-regularized RL closed form + preference-data NLL</strong>, three pieces unified.</p>
719
+ <h3 id="64-reward-overoptimization-gao-schulman-hilton-2023-icml">6.4　Reward Overoptimization (Gao, Schulman, Hilton 2023 ICML)</h3>
720
+ <h4 id="641-problem-statement">6.4.1　Problem statement</h4>
721
+ <p>Let gold reward $r_g$ (true human-perceived quality) and proxy reward $r_p$ (RM estimate). The RM is trained on finite preference data, so $r_p \ne r_g$.</p>
722
+ <p>As RL training pushes KL distance $d = \text{KL}(\pi_\theta \| \pi_\text{ref})$ up:</p>
723
+ <ul><li>$\mathbb{E}_{\pi_\theta}[r_p]$ rises monotonically (the policy learns the RM's preference).</li><li>$\mathbb{E}_{\pi_\theta}[r_g]$ rises then falls (<strong>inverted-U</strong>) — this is reward overoptimization.</li></ul>
724
+ <h4 id="642-gao-2023s-fitted-form">6.4.2　Gao 2023's fitted form</h4>
725
+ <p>Gao 2023 ran large-scale experiments over RM size + KL distance, giving a fitted form for gold reward vs KL distance $d$ (using $\sqrt{d}$ on the x-axis):</p>
726
+ <p>$$R_g(d) = d \cdot (\alpha_g - \gamma_g \cdot d) \quad \text{(BoN)}$$</p>
727
+ <p>$$R_g(d) = d \cdot (\alpha_g - \gamma_g \cdot d) - \delta_g\, d^{3/2}\quad \text{(PPO, an extra higher-order term)}$$</p>
728
+ <p>where $\alpha_g, \gamma_g, \delta_g$ are coefficients dependent on RM size; larger RM → smaller "higher-order / quadratic" weight → slower overoptimization.</p>
729
+ <h4 id="643-kl-is-overoptimizations-natural-axis">6.4.3　KL is overoptimization's "natural axis"</h4>
730
+ <p>Whether BoN, PPO, or DPO (via accumulated implicit reward), one can plot training curves as "<strong>KL distance vs gold reward</strong>". Gao 2023 found:</p>
731
+ <ul><li><strong>Different algorithms (BoN / PPO) exhibit similar over-optimization behavior at the same KL</strong> (the gold curve shape under the same RM is consistent).</li><li><strong>Larger RM → slower over-optimization</strong> (gold-curve peak shifts right).</li><li><strong>KL is a one-dimensional progress indicator</strong> — better than step count / reward count for explaining over-optimization.</li></ul>
732
+ <p><strong>Interview takeaway</strong>: in RLHF monitoring, plot <code>(measured_KL, gold_reward)</code> to see whether it has entered the descending region — that is the most direct "stop early" signal.</p>
733
+ <h3 id="65-relation-between-kl-and-chi2-gap">6.5　Relation between KL and $\chi^2$ gap</h3>
734
+ <p>KL and $\chi^2$ are related via the Pinsker family of inequalities:</p>
735
+ <p>$$\text{TV} \le \sqrt{\text{KL}/2}\qquad \text{(Pinsker)}$$</p>
736
+ <p>$$\text{KL} \le \log(1 + \chi^2)$$</p>
737
+ <p>In RLHF, $\chi^2$ is occasionally used as a "sensitive" upper bound on KL: <strong>when $\chi^2$ is large but KL is small, $\pi_\theta$ has heavy-tail deviation from $\pi_\text{ref}$</strong> (high variance but low mean). Some works (Yu et al. 2024 / others) plot KL and $\chi^2$ together when monitoring reward hacking, to track tail behavior.</p>
738
+ <h3 id="66-sequence-level-vs-token-level-kl">6.6　Sequence-level vs token-level KL</h3>
739
+ <p>By the <strong>chain rule</strong> in §1.1:</p>
740
+ <p>$$\text{KL}(\pi_\theta(\cdot|x) \| \pi_\text{ref}(\cdot|x)) = \mathbb{E}_{y \sim \pi_\theta}\!\left[\sum_t \log\frac{\pi_\theta(y_t | x, y_{<t})}{\pi_\text{ref}(y_t | x, y_{<t})}\right]$$</p>
741
+ <p>i.e. <strong>sequence-level KL = expectation of the sum of token-level KLs</strong> (autoregressive chain rule). Note: directly treating $\sum_t \log\pi_\theta(y_t)/\pi_\text{ref}(y_t)$ from a single rollout as KL is the <strong>k1 estimator</strong> (unbiased but high per-rollout variance); the true token-level KL $D(\pi_\theta(\cdot|s_t)\|\pi_\text{ref}(\cdot|s_t))$ would still require a full-vocab forward sum. <strong>The two have the same expectation</strong>; engineering uses the sampled log-ratio because vocab summation is expensive.</p>
742
+ <p>But two <strong>implementation tricks</strong> relate to token-level:</p>
743
+ <ol><li><strong>Per-token clipping</strong>: occasionally a single token's KL can be very large (rare tokens, long tails); per-token KL clipping prevents one token from dragging the batch total KL away.</li><li><strong>Mask on assistant tokens only</strong>: in chat / agent settings, prompt tokens should not enter KL computation (the prompt is identical, policy and ref agree on it exactly, KL = 0; but floating-point noise pollutes it). So the KL mask coincides with the PPO action_mask — <strong>compute KL only on assistant generation tokens</strong>.</li></ol>
744
+ <pre><code class="language-python"># Per-token KL with action mask (chat / agentic RL setting)
745
+ def per_token_kl_with_mask(logp_theta, logp_ref, action_mask, estimator=&quot;k3&quot;):
746
+ &quot;&quot;&quot;
747
+ logp_theta, logp_ref: [B, T] per-token log-prob
748
+ action_mask: [B, T] 1 = assistant token, 0 = prompt/system/pad
749
+ estimator: &quot;k1&quot; | &quot;k2&quot; | &quot;k3&quot;
750
+ Returns: mean KL over assistant tokens, scalar.
751
+ &quot;&quot;&quot;
752
+ log_r = logp_theta - logp_ref
753
+ if estimator == &quot;k1&quot;:
754
+ kl_per_tok = log_r
755
+ elif estimator == &quot;k2&quot;:
756
+ kl_per_tok = 0.5 * log_r ** 2
757
+ elif estimator == &quot;k3&quot;:
758
+ delta = -log_r # log(π_ref / π_θ)
759
+ kl_per_tok = torch.exp(delta) - delta - 1.0
760
+ else:
761
+ raise ValueError(f&quot;Unknown estimator: {estimator}&quot;)
762
+ masked = (kl_per_tok * action_mask).sum()
763
+ n = action_mask.sum().clamp_min(1.0)
764
+ return masked / n</code></pre>
765
+ <h2 id="7-practice--code">§7 Practice + code</h2>
766
+ <h3 id="71-ppo-style-in-reward-shaping-implementation">7.1　PPO style: In-reward shaping implementation</h3>
767
+ <pre><code class="language-python">import torch
768
+ import torch.nn.functional as F
769
+
770
+ def ppo_reward_with_kl(rewards_terminal, rollout_logp_theta, logp_ref,
771
+ action_mask, beta=0.02):
772
+ &quot;&quot;&quot;
773
+ PPO / InstructGPT style: KL penalty in reward (k1 estimator).
774
+ rewards_terminal: [B] only the last assistant token gets the RM reward
775
+ rollout_logp_theta: [B, T] log π_θ_old(y_t | ...), recorded during rollout, DETACHED
776
+ logp_ref: [B, T] log π_ref(y_t | ...), reference policy frozen
777
+ action_mask: [B, T] 1 = assistant token
778
+ Returns: shaped_reward [B, T] per-token reward with KL penalty baked in.
779
+
780
+ ⚠️ Crucial: rollout_logp_theta and logp_ref must be detached scalars (recorded at
781
+ rollout time, or via no_grad forward). The shaped reward is the reward tensor passed
782
+ through PPO / score-function backprop and **must not carry gradient**; otherwise the
783
+ KL term and the PG surrogate will double-backprop and break the score-function
784
+ semantics. Production code typically stores logp_old as fixed data in the rollout
785
+ buffer, then re-forwards at update time to get new_logp.
786
+ &quot;&quot;&quot;
787
+ B, T = rollout_logp_theta.shape
788
+ # k1 KL per token (signed; mean is unbiased). Both inputs assumed detached.
789
+ kl_per_tok = (rollout_logp_theta.detach() - logp_ref.detach()) * action_mask # [B, T]
790
+ # spread terminal reward to last assistant token
791
+ last_token_idx = action_mask.cumsum(dim=-1).argmax(dim=-1) # [B]
792
+ R_per_tok = torch.zeros_like(kl_per_tok)
793
+ R_per_tok[torch.arange(B), last_token_idx] = rewards_terminal
794
+ # combine
795
+ shaped = R_per_tok - beta * kl_per_tok
796
+ return shaped # the whole tensor is detached / gradient-free, serving as data input to PPO advantage computation</code></pre>
797
+ <h3 id="72-grpo-style-in-loss-regularization-implementation-k3">7.2　GRPO style: In-loss regularization implementation (k3)</h3>
798
+ <pre><code class="language-python">def grpo_loss_with_k3_kl(policy, ref_policy, batch, eps_clip=0.2, beta=0.04):
799
+ &quot;&quot;&quot;
800
+ GRPO / DeepSeekMath style: KL penalty in loss (k3 estimator).
801
+ batch:
802
+ input_ids: [N, L] N = sum_b G_b samples
803
+ action_mask: [N, L]
804
+ old_log_probs: [N, L]
805
+ rewards: [N] sequence-level reward
806
+ group_id: [N] which prompt
807
+ &quot;&quot;&quot;
808
+ rewards = batch[&quot;rewards&quot;]
809
+ gid = batch[&quot;group_id&quot;].long()
810
+ num_groups = int(gid.max().item()) + 1
811
+
812
+ # Group-relative advantage (z-score within group)
813
+ counts = torch.zeros(num_groups, device=rewards.device).scatter_add_(
814
+ 0, gid, torch.ones_like(rewards))
815
+ sums = torch.zeros(num_groups, device=rewards.device).scatter_add_(
816
+ 0, gid, rewards)
817
+ group_mean = sums / counts.clamp_min(1.0)
818
+ diff_sq = (rewards - group_mean[gid]) ** 2
819
+ sq_sums = torch.zeros(num_groups, device=rewards.device).scatter_add_(
820
+ 0, gid, diff_sq)
821
+ group_std = (sq_sums / counts.clamp_min(1.0)).sqrt()
822
+ A = (rewards - group_mean[gid]) / (group_std[gid] + 1e-8)
823
+ A = A.unsqueeze(-1) # [N, 1] shared per token
824
+
825
+ # Forward pass policy + ref
826
+ logits = policy(batch[&quot;input_ids&quot;]).logits[:, :-1]
827
+ log_probs = F.log_softmax(logits, dim=-1)
828
+ tgt = batch[&quot;input_ids&quot;][:, 1:].unsqueeze(-1)
829
+ new_log_probs = log_probs.gather(-1, tgt).squeeze(-1)
830
+ new_log_probs = F.pad(new_log_probs, (1, 0)) # [N, L]
831
+ mask = batch[&quot;action_mask&quot;].float()
832
+
833
+ with torch.no_grad():
834
+ ref_logits = ref_policy(batch[&quot;input_ids&quot;]).logits[:, :-1]
835
+ ref_log_probs = F.log_softmax(ref_logits, dim=-1)
836
+ ref_token_lp = ref_log_probs.gather(-1, tgt).squeeze(-1)
837
+ ref_token_lp = F.pad(ref_token_lp, (1, 0))
838
+
839
+ # PPO-Clip surrogate (advantage shared per sample)
840
+ ratio = torch.exp(new_log_probs - batch[&quot;old_log_probs&quot;])
841
+ surr1 = ratio * A
842
+ surr2 = torch.clamp(ratio, 1 - eps_clip, 1 + eps_clip) * A
843
+ pg_per_tok = torch.min(surr1, surr2) # [N, L]
844
+
845
+ # k3 KL per token: exp(Δ) - Δ - 1, Δ = log(π_ref / π_θ)
846
+ delta = ref_token_lp - new_log_probs # log(π_ref / π_θ)
847
+ kl_per_tok_k3 = torch.exp(delta) - delta - 1.0 # ≥ 0
848
+
849
+ # Combine: maximize PG - β·KL → minimize -PG + β·KL
850
+ token_obj = pg_per_tok - beta * kl_per_tok_k3
851
+ seq_len = mask.sum(dim=-1).clamp_min(1.0)
852
+ per_seq = (token_obj * mask).sum(dim=-1) / seq_len
853
+ loss = -per_seq.mean()
854
+
855
+ with torch.no_grad():
856
+ kl_mean = (kl_per_tok_k3 * mask).sum() / mask.sum().clamp_min(1.0)
857
+ return loss, {&quot;kl_k3&quot;: kl_mean.item(), &quot;advantage_std&quot;: A.std().item()}</code></pre>
858
+ <h3 id="73-dpo-closed-form-loss--implicit-reward-monitoring">7.3　DPO closed-form loss + implicit-reward monitoring</h3>
859
+ <pre><code class="language-python">def dpo_loss_with_implicit_reward_monitor(policy, ref_policy, batch, beta=0.1):
860
+ &quot;&quot;&quot;
861
+ DPO loss + implicit reward margin (β × pointwise sequence log-density ratio on
862
+ preference data; this is NOT a KL estimator on preference data — only equals
863
+ β·KL when expectation is taken under y~π_θ, which doesn&#x27;t hold for fixed preference pairs).
864
+ &quot;&quot;&quot;
865
+ def log_prob_sum(model, ids, mask):
866
+ logits = model(ids).logits[:, :-1]
867
+ logp = F.log_softmax(logits, dim=-1)
868
+ tgt = ids[:, 1:].unsqueeze(-1)
869
+ token_logp = logp.gather(-1, tgt).squeeze(-1)
870
+ token_mask = mask[:, 1:]
871
+ return (token_logp * token_mask).sum(dim=-1)
872
+
873
+ pi_w = log_prob_sum(policy, batch[&quot;chosen_ids&quot;], batch[&quot;chosen_mask&quot;])
874
+ pi_l = log_prob_sum(policy, batch[&quot;rejected_ids&quot;], batch[&quot;rejected_mask&quot;])
875
+ with torch.no_grad():
876
+ ref_w = log_prob_sum(ref_policy, batch[&quot;chosen_ids&quot;], batch[&quot;chosen_mask&quot;])
877
+ ref_l = log_prob_sum(ref_policy, batch[&quot;rejected_ids&quot;], batch[&quot;rejected_mask&quot;])
878
+
879
+ # sequence-level log-density ratios on preference data
880
+ log_ratio_w = pi_w - ref_w # Σ_t log(π_θ/π_ref) on y_w
881
+ log_ratio_l = pi_l - ref_l
882
+
883
+ diff = beta * (log_ratio_w - log_ratio_l)
884
+ loss = -F.logsigmoid(diff).mean()
885
+
886
+ # implicit rewards (DPO definition: β × pointwise sequence log-ratio, detached for logging)
887
+ chosen_reward = beta * log_ratio_w.detach()
888
+ rejected_reward = beta * log_ratio_l.detach()
889
+ margin = (chosen_reward - rejected_reward).mean()
890
+
891
+ # ⚠️ Note: the average log-ratio on preference data is **not** a KL estimator — KL requires y ~ π_θ samples.
892
+ # Here we can only monitor &quot;how much the model deviates from ref on preference data&quot; as a DPO training-health indicator, not as KL.
893
+ avg_pref_log_ratio = ((log_ratio_w.detach() + log_ratio_l.detach()) / 2).mean()
894
+ return loss, {&quot;margin&quot;: margin.item(),
895
+ &quot;avg_pref_log_ratio&quot;: avg_pref_log_ratio.item(),
896
+ &quot;chosen_reward&quot;: chosen_reward.mean().item(),
897
+ &quot;rejected_reward&quot;: rejected_reward.mean().item()}</code></pre>
898
+ <h3 id="74-reward-overoptimization-monitoring">7.4　Reward overoptimization monitoring</h3>
899
+ <pre><code class="language-python">def overoptimization_monitor(policy, ref_policy, gold_reward_fn, prompts,
900
+ checkpoints_kl, gold_at_kl):
901
+ &quot;&quot;&quot;
902
+ Track gold reward as a function of measured KL distance during training.
903
+ Call periodically; plot (measured_KL, gold_reward) to visualize the inverted-U.
904
+
905
+ Args:
906
+ policy: current π_θ
907
+ ref_policy: frozen π_ref
908
+ gold_reward_fn: callable (text -&gt; float), uses gold RM or human eval
909
+ prompts: list of held-out prompts (small, e.g. 64)
910
+ checkpoints_kl: running list of KL values across training
911
+ gold_at_kl: running list of gold reward values
912
+ &quot;&quot;&quot;
913
+ measured_kl_total = 0.0
914
+ gold_reward_total = 0.0
915
+ for prompt in prompts:
916
+ # Generate from policy and ref; compute per-token k3 KL on policy generation
917
+ out_policy = policy.generate(prompt, do_sample=True)
918
+ # ... (use generate to get logits, mask, compute k3 KL) ...
919
+ # For simplicity here, just track the average sequence-level KL
920
+ kl_seq = compute_seq_kl_k3(policy, ref_policy, out_policy)
921
+ gold = gold_reward_fn(out_policy)
922
+ measured_kl_total += kl_seq
923
+ gold_reward_total += gold
924
+ avg_kl = measured_kl_total / len(prompts)
925
+ avg_gold = gold_reward_total / len(prompts)
926
+ checkpoints_kl.append(avg_kl)
927
+ gold_at_kl.append(avg_gold)
928
+ # Detection heuristic: if last 5 gold values are decreasing while KL is rising,
929
+ # we are likely in the over-optimization regime.
930
+ if len(gold_at_kl) &gt;= 5:
931
+ recent_gold = gold_at_kl[-5:]
932
+ recent_kl = checkpoints_kl[-5:]
933
+ if all(recent_gold[i] &gt;= recent_gold[i+1] for i in range(4)) \
934
+ and all(recent_kl[i] &lt;= recent_kl[i+1] for i in range(4)):
935
+ print(f&quot;⚠️ Possible reward over-optimization: KL ↑, gold ↓ over 5 checkpoints&quot;)
936
+ return avg_kl, avg_gold</code></pre>
937
+ <h3 id="75-engineering-checklist-debug-checklist">7.5　Engineering checklist (debug checklist)</h3>
938
+ <div class="callout callout-warn"><div class="callout-title">Top-8 KL-related bugs</div></div>
939
+ <ol><li><strong>Wrong mask for KL</strong>: KL on prompt tokens should be 0 (same prompt input), but floating-point noise pollutes it — so <strong>mask only assistant tokens</strong>.</li><li><strong>k1 shows negative</strong>: a single-sample log-ratio can be negative, but <strong>the expectation is non-negative</strong>. Not a bug — it's an estimator property. Switch to k3 / k2 for more intuitive monitoring.</li><li><strong>Wrong β unit</strong>: when reward scale = O(1), β = 0.02; when reward scale = O(100), β must be scaled accordingly. Otherwise the KL anchor fails.</li><li><strong>Adaptive β oscillates</strong>: target_kl too strict / mult too large. Clip mult to [0.8, 1.2], reduce update frequency to every 100 steps.</li><li><strong>Reference policy not frozen</strong>: forgetting <code>ref_policy.eval()</code> + <code>torch.no_grad()</code>, ref trains along with policy, KL turns into self-distillation.</li><li><strong>Per-token vs sequence-level KL confused</strong>: monitoring sometimes reports per-token, sometimes sequence-level — misreading leads to mis-tuning β. Unify units.</li><li><strong>KL term shrinks reward in GRPO</strong>: when reward is binary {0, 1} (math problems) and β = 0.04 with KL ≈ 0.5, the KL penalty already exceeds the mean reward — loss is dominated by KL. <strong>Drop β to 1e-3 or smaller</strong>.</li><li><strong>Float overflow in $\pi_\text{ref}/\pi_\theta$</strong>: when the policy diverges from ref by a lot, $\pi_\text{ref}/\pi_\theta$ can be very large and $e^\Delta$ overflows. <strong>Compute k3 in log-space</strong>: <code>exp(delta) - delta - 1</code> may still overflow for large $\Delta$, so add <code>torch.clamp(delta, max=10)</code> or use a numerically stable form.</li></ol>
940
+ <h2 id="8-failure-modes-kl-collapse--runaway--reward-hacking">§8 Failure modes: KL Collapse / Runaway / Reward Hacking</h2>
941
+ <h3 id="81-kl-collapse--too-large-or-entropy-too-low">8.1　KL Collapse (β too large or entropy too low)</h3>
942
+ <p><strong>Symptoms</strong>:</p>
943
+ <ul><li>KL curve $\approx 0$, policy equals reference.</li><li>Reward doesn't rise.</li><li>Generation identical to SFT.</li></ul>
944
+ <p><strong>Causes</strong>:</p>
945
+ <ul><li>β too large, KL penalty dominates loss, PG signal is suppressed.</li><li>Policy entropy too low (SFT was already deterministic) → further updates are difficult.</li></ul>
946
+ <p><strong>Fixes</strong>:</p>
947
+ <ul><li>Lower β (adaptive controller should handle this automatically).</li><li>Add entropy bonus ($+ c_e \cdot H[\pi_\theta]$).</li><li>Check PG-vs-KL-loss magnitude ratio: ideal $|\text{PG}| / (\beta \cdot |\text{KL}|) \in [1, 10]$.</li></ul>
948
+ <h3 id="82-kl-runaway--too-small-or-reward-signal-too-strong">8.2　KL Runaway (β too small or reward signal too strong)</h3>
949
+ <p><strong>Symptoms</strong>:</p>
950
+ <ul><li>KL keeps rising, policy drifts further from ref.</li><li>Generation lengthens, repetition patterns appear, style drifts.</li><li>Gold reward on validation starts dropping (reward overoptimization).</li></ul>
951
+ <p><strong>Causes</strong>:</p>
952
+ <ul><li>β too small, KL anchor fails.</li><li>Reward signal too strong (neural RM produces huge gradient).</li><li>PPO importance ratio frequently outside clip range, KL signal masked by clip (Option A specific).</li></ul>
953
+ <p><strong>Fixes</strong>:</p>
954
+ <ul><li>Raise β (adaptive).</li><li>Add max_kl_budget early stop (when measured KL &gt; target, stop training).</li><li>Use RM ensemble + conservative aggregation (min / mean - std).</li><li>For Option A, consider switching to Option B (KL not masked by PPO clip).</li></ul>
955
+ <h3 id="83-reward-overoptimization-the-inverted-u-over-kl-distance">8.3　Reward Overoptimization (the inverted-U over KL distance)</h3>
956
+ <p><strong>Symptoms</strong>:</p>
957
+ <ul><li>Proxy reward keeps rising.</li><li>Gold reward rises then falls (inverted-U).</li><li>Human eval: model looks good on in-distribution data, but crashes on OOD prompts.</li></ul>
958
+ <p><strong>Causes</strong>: see §6.4 — the RM and gold reward are fundamentally different.</p>
959
+ <p><strong>Fixes</strong>:</p>
960
+ <ol><li><strong>KL budget early stop</strong>: cap measured KL before the inverted-U peak (requires gold reward monitoring).</li><li><strong>RM ensemble</strong>: multiple seeds of RM, take min or mean - $k\cdot\text{std}$.</li><li><strong>Mix RL + DPO</strong>: DPO for 70%, then small-β PPO for the last mile.</li><li><strong>Rule-based reward replacing neural RM</strong>: the fundamental fix for math/code tasks.</li><li><strong>PRM replacing ORM</strong>: dense reward → bounded per-step over-optimization.</li></ol>
961
+ <h3 id="84-length-bias-dpo-specific-also-a-form-of-insufficient-kl-anchor">8.4　Length bias (DPO-specific, also a form of insufficient KL anchor)</h3>
962
+ <p><strong>Symptoms</strong>:</p>
963
+ <ul><li>DPO-trained outputs visibly longer.</li><li>AlpacaEval score high but users find them verbose.</li></ul>
964
+ <p><strong>Cause</strong>: DPO loss is sequence-level log-ratio difference. $y_w$ is typically longer (humans prefer more detailed answers); longer $y_w$ → more negative $\log\pi(y_w)$ → larger log-ratio difference → smaller loss. But this is RM scale, not reasoning quality.</p>
965
+ <p><strong>Fixes</strong>:</p>
966
+ <ul><li>SimPO's length-normalization ($r = (\beta/|y|)\log\pi$).</li><li>Add a length penalty in reward shaping.</li><li>Data curation: make $y_w$ and $y_l$ have similar length distributions.</li></ul>
967
+ <h3 id="85-reference-policy-wrong-checkpoint-failure">8.5　Reference policy "wrong checkpoint" failure</h3>
968
+ <p><strong>Symptoms</strong>: training looks normal but downstream eval is awful.</p>
969
+ <p><strong>Cause</strong>: wrong ref checkpoint (e.g. pretrain base instead of SFT). KL anchors to "language model" rather than "instruction model".</p>
970
+ <p><strong>Fix</strong>: reference must be the immediate-previous-stage SFT checkpoint — never skip levels.</p>
971
+ <h2 id="9-25-high-frequency-interview-questions">§9 25 high-frequency interview questions</h2>
972
+ <p>Split by difficulty into 3 tiers: L1 = any LLM engineering role; L2 = research / alignment teams; L3 = top-lab / DeepSeek-grade hardcore. Each question has an answer summary + common pitfalls. Click to expand.</p>
973
+ <h3 id="l1-must-know-10-questions">L1 must-know (10 questions)</h3>
974
+ <details>
975
+
976
+ <summary>Q1. KL divergence's definition and 3 core properties?</summary>
977
+ <ul><li>Definition: $\text{KL}(p\|q) = \mathbb{E}_{x\sim p}[\log(p(x)/q(x))]$</li><li>Non-negative (Jensen); equality ⟺ $p = q$ a.e.</li><li>Asymmetric ($\text{KL}(p\|q) \ne \text{KL}(q\|p)$), not a metric</li><li>Jointly convex, invariant under reparameterization (reparam invariant)</li></ul>
978
+ <p>Saying KL is a "distance" — wrong (no triangle inequality); not knowing convexity.</p>
979
+ </details>
980
+ <details>
981
+
982
+ <summary>Q2. Why does RLHF need a KL penalty?</summary>
983
+ <ul><li>Prevent reward hacking: the policy finds RM blind spots for high scores but worse human-perceived quality</li><li>Prevent linguistic fluency collapse: without KL, the model may emit "gibberish that RM scores high"</li><li>Prevent distribution shift: when the policy drifts far from SFT, even the RM can't score accurately (OOD)</li><li>Provides a closed-form optimal policy: $\pi^* \propto \pi_\text{ref}\exp(r/\beta)$</li></ul>
984
+ <p>Only saying "prevents overfitting" — not specific enough; not knowing about reward hacking.</p>
985
+ </details>
986
+ <details>
987
+
988
+ <summary>Q3. What is the k1 estimator? Why can a single sample be negative?</summary>
989
+ <ul><li>k1: $\widehat{\text{KL}}_1 = \log(\pi_\theta(y)/\pi_\text{ref}(y))$ (direct log-ratio)</li><li><strong>Expectation</strong> $\mathbb{E}_{y\sim\pi_\theta}[\log r] = \text{KL}$ (unbiased), but <strong>a single sample</strong> $\log r$ can be positive or negative</li><li>Reason: at a single sample $y$, $\pi_\theta$ may be larger or smaller than $\pi_\text{ref}$, the log-ratio has no "non-negative" constraint</li><li>Engineering impact: don't panic when k1 monitoring shows negative — that's the estimator's nature</li></ul>
990
+ <p>Confusing single estimator with expectation; not knowing expectation non-negative ≠ pointwise non-negative.</p>
991
+ </details>
992
+ <details>
993
+
994
+ <summary>Q4. What is the k3 estimator (Schulman 2020) formula? What are its three properties?</summary>
995
+ <ul><li>Formula: $\widehat{\text{KL}}_3 = e^\Delta - \Delta - 1$, $\Delta = \log(\pi_\text{ref}/\pi_\theta) = -\log r$</li><li>Equivalent form: $\widehat{\text{KL}}_3 = (\pi_\text{ref}/\pi_\theta) - \log(\pi_\text{ref}/\pi_\theta) - 1$</li><li><strong>Unbiased</strong> ($\mathbb{E}_{\pi_\theta} = \text{KL}$, using $\sum_y\pi_\text{ref} = 1$)</li><li><strong>Non-negative</strong> ($f(x) = e^x - x - 1 \ge 0$)</li><li><strong>Low variance</strong> (with the expectation-0 control variate $\pi_\text{ref}/\pi_\theta - 1$)</li></ul>
996
+ <p>Only memorizing the formula without knowing the derivation; or not knowing it is non-negative.</p>
997
+ </details>
998
+ <details>
999
+
1000
+ <summary>Q5. What are the two KL placements in RLHF? What's the difference?</summary>
1001
+ <ul><li><strong>In-reward shaping (Option A)</strong>: $\tilde{r}_t = r_t - \beta \cdot \text{KL}_t$ (uses k1), KL enters advantage / GAE. InstructGPT standard.</li><li><strong>In-loss regularization (Option B)</strong>: $\mathcal{L} = \mathcal{L}_\text{PG} + \beta \cdot \mathbb{E}[\text{KL}]$. GRPO / DAPO historically pick k3 estimator.</li><li>Same objective, but <strong>the gradient path differs</strong>. <strong>Principled gradient</strong> choices: (P1) k1 in reward; (P2) k2 as loss — on-policy these are gradient-equivalent, both being the strict reverse-KL gradient. <strong>GRPO's k3-as-loss</strong> is a first-order Taylor approximation of the reverse-KL gradient, with $O(\Delta^2)$ bias (see §3.6 / Rethinking KL arXiv 2510.01555).</li><li>PPO clip in Option A "masks" out-of-clip KL (advantage's embedded KL gets clipped together); Option B isn't affected.</li><li>Practice: math/code RL historically picks B (k3) because β is small and $\Delta$ is small — bias is negligible; helpfulness / safety mostly uses A (k1); for strictly principled on-policy, P2 (k2-as-loss) is the simplest principled choice.</li></ul>
1002
+ <p>Saying the two are fully equivalent — same objective but not gradient-equivalent; saying k3-as-loss is principled — it is a first-order approximation, theoretically inexact.</p>
1003
+ </details>
1004
+ <details>
1005
+
1006
+ <summary>Q6. How do you tune β? What are the symptoms when it's too large / too small?</summary>
1007
+ <ul><li>β too large: KL ≈ 0, policy stuck near ref, reward doesn't rise, model essentially equals SFT</li><li>β too small: KL runaway, policy drifts, generation lengthens / repeats / panders, reward hacking</li><li>Starting points: InstructGPT uses β ≈ 0.02 (per-token); GRPO math task β ≈ 0.04 or smaller; DAPO often β ≈ 0</li><li>Methods: fixed / adaptive (pull β based on measured KL vs target) / annealing schedule</li></ul>
1008
+ <p>Only saying "tune by feel"; not knowing the engineering order of magnitude for β.</p>
1009
+ </details>
1010
+ <details>
1011
+
1012
+ <summary>Q7. Which one does RLHF use — Forward KL or Reverse KL? Why?</summary>
1013
+ <ul><li>RLHF almost always uses $\text{KL}(\pi_\theta \| \pi_\text{ref})$; under the standard VI/RLHF convention (used by DPO/RLOO/Rethinking KL etc.) this is <strong>reverse KL</strong> ($\pi_\theta$ is the variational side), mode-seeking</li><li>Because training samples from $\pi_\theta$ (rollout), and $\mathbb{E}_{\pi_\theta}[\log r]$ is directly estimable from samples</li><li><strong>Forward KL</strong> $\text{KL}(\pi_\text{ref} \| \pi_\theta)$ requires sampling from $\pi_\text{ref}$, which is engineering-meaningless (we are training $\pi_\theta$, not $\pi_\text{ref}$), and mass-covering doesn't fit the RLHF goal</li><li>Reverse KL's mode-seeking matches the rule perfectly: pick the reward-highest mode inside the high-density region of $\pi_\text{ref}$</li></ul>
1014
+ <p>Not knowing the naming convention (a few early RL textbooks name by sampling distribution and label reverse as forward; in an interview, giving the formula is safer); or saying forward is better.</p>
1015
+ </details>
1016
+ <details>
1017
+
1018
+ <summary>Q8. How does DPO's implicit reward relate to KL?</summary>
1019
+ <ul><li>DPO implicit reward $\hat{r}_\theta(x, y) = \beta\log(\pi_\theta(y|x)/\pi_\text{ref}(y|x))$, i.e. <strong>β × pointwise sequence log-density ratio</strong></li><li>⚠️ <strong>It is not KL itself</strong>: only when taking expectation over $y\sim\pi_\theta$ does $\mathbb{E}_{y\sim\pi_\theta}[\hat r_\theta(x,y)] = \beta\cdot\text{KL}(\pi_\theta(\cdot|x)\,\|\,\pi_\text{ref}(\cdot|x))$. <strong>In DPO training, $y_w, y_l$ come from fixed preference data</strong>, so the $\hat r$ here is not a KL estimator nor a KL margin</li><li>During training, maximizing $\hat{r}(y_w) - \hat{r}(y_l)$ ≡ pairwise log-ratio margin (relative confidence on the preference pair), <strong>not</strong> a "KL margin"</li><li>The KL anchor is <strong>implicit</strong>: $\pi_\text{ref}$ appearing in the denominator of $\hat r$ provides anchoring, but no explicit KL term is in the loss</li><li>Dual RL view: DPO is derived from the closed-form optimal policy $\pi^* \propto \pi_\text{ref}\exp(r/\beta)$ of KL-regularized RL; the implicit objective is the same RLHF objective, but the estimator path differs</li></ul>
1020
+ <p>Saying DPO has no KL (wrong — it's an implicit anchor); reading $\hat r$ directly as KL / KL margin (wrong — requires the y~π_θ expectation condition).</p>
1021
+ </details>
1022
+ <details>
1023
+
1024
+ <summary>Q9. What is reward overoptimization? Why does it happen?</summary>
1025
+ <ul><li>Proxy reward (RM) rises monotonically with KL, but gold reward (human eval) rises then falls (<strong>inverted-U</strong>)</li><li>Gao, Schulman, Hilton 2023 ICML gives the fitted form for KL vs gold-reward</li><li>Cause: RM and gold reward differ (RM is a finite-data learned approximation), policy learns RM's preference but not necessarily true quality</li><li>Mitigations: KL budget early stop / RM ensemble / PRM / mix DPO + PPO</li></ul>
1026
+ <p>Only saying "overfitting" — too vague; not knowing KL is the axis for over-optimization.</p>
1027
+ </details>
1028
+ <details>
1029
+
1030
+ <summary>Q10. Why did GRPO historically use k3 instead of k1?</summary>
1031
+ <ul><li><strong>Engineering motivation</strong>: GRPO puts KL in the loss, and k3's <strong>value-estimator</strong> properties (non-negative + value-side unbiased + low variance) make the KL monitoring curve intuitive</li><li><strong>k1-as-loss is infeasible</strong>: backpropagating $\log r = \log\pi_\theta - \log\pi_\text{ref}$ directly gives $\nabla = \nabla\log\pi_\theta$, which has expectation 0 on-policy — <strong>not</strong> the reverse-KL gradient. k1 must <strong>enter reward / advantage</strong> (as a detached scalar via the score-function trick) to give the strict reverse-KL gradient. Putting k1 as a loss term is bad both because the value can be negative (visually unappealing) and because the gradient is wrong</li><li><strong>k3-as-loss's true cost</strong>: backpropagating w.r.t. θ gives a <strong>first-order Taylor approximation</strong> of the reverse-KL gradient, $1 - e^{-\Delta} = \Delta - \tfrac12\Delta^2 + O(\Delta^3)$, with $O(\Delta^2)$ bias (see §3.6 + Rethinking KL arXiv 2510.01555)</li><li><strong>Principled alternative</strong>: on-policy, (P1) k1 in reward or (P2) <strong>k2 as loss</strong> ($\tfrac12\Delta^2$, $\nabla = \Delta\cdot\nabla\log\pi_\theta$, gradient-equivalent to P1) are both the strict reverse-KL gradient</li><li><strong>Historical DeepSeekMath / DAPO / verl default to k3</strong> as engineering convention; when β is tight and $\Delta$ is small, bias is negligible — but this is empirical rationalization, not theoretical principle</li></ul>
1032
+ <p>Only saying k3 "satisfies all three" without the gradient-bias caveat; not knowing that (P2) k2-as-loss is the gradient-equivalent alternative to P1.</p>
1033
+ </details>
1034
+ <h3 id="l2-advanced-10-questions">L2 advanced (10 questions)</h3>
1035
+ <details>
1036
+
1037
+ <summary>Q11. Derive the unbiasedness of k3.</summary>
1038
+ <p>We need to show $\mathbb{E}_{y\sim\pi_\theta}[(\pi_\text{ref}/\pi_\theta) - \log(\pi_\text{ref}/\pi_\theta) - 1] = \text{KL}(\pi_\theta\|\pi_\text{ref})$.</p>
1039
+ <ol><li>$\mathbb{E}_{\pi_\theta}[\pi_\text{ref}/\pi_\theta] = \sum_y \pi_\theta(y)\cdot\pi_\text{ref}(y)/\pi_\theta(y) = \sum_y\pi_\text{ref}(y) = 1$</li><li>$\mathbb{E}_{\pi_\theta}[-\log(\pi_\text{ref}/\pi_\theta)] = \mathbb{E}_{\pi_\theta}[\log(\pi_\theta/\pi_\text{ref})] = \text{KL}(\pi_\theta\|\pi_\text{ref})$</li><li>Combining: $\mathbb{E}[\widehat{\text{KL}}_3] = 1 + \text{KL} - 1 = \text{KL}$ ✓</li></ol>
1040
+ <p>Key: use $\sum_y\pi_\text{ref} = 1$. If you forget the $\pi_\text{ref}/\pi_\theta$ term is a control variate with expectation 1, you can't prove unbiasedness.</p>
1041
+ </details>
1042
+ <details>
1043
+
1044
+ <summary>Q12. Derive the k3 estimator from $f(x) = e^x - x - 1$.</summary>
1045
+ <p>Constructive proof:</p>
1046
+ <ol><li>$f(x) = e^x - x - 1$ is non-negative on $\mathbb{R}$ (the convex $e^x$ lies above its tangent line $1 + x$ at $x = 0$).</li><li><p>Set $x = \log(\pi_\text{ref}(y)/\pi_\theta(y)) = -\log r$:</p>
1047
+ <ul><li>$e^{-\log r} = 1/r = \pi_\text{ref}/\pi_\theta$</li><li>$-(-\log r) = \log r = \log(\pi_\theta/\pi_\text{ref})$</li><li>$f(-\log r) = \pi_\text{ref}/\pi_\theta - \log(\pi_\text{ref}/\pi_\theta) - 1 = e^\Delta - \Delta - 1$, $\Delta = -\log r$</li></ul></li><li>Expectation: $\mathbb{E}_{\pi_\theta}[f(-\log r)] = \text{KL}$ (proved in Q11).</li></ol>
1048
+ <p>So k3 = special form of $f(\log p/q)$, which happens to satisfy all three properties "non-negative + unbiased + with control variate".</p>
1049
+ <p>Just writing the formula without explaining the motivation of $f$; not knowing the convex tangent → non-negative.</p>
1050
+ </details>
1051
+ <details>
1052
+
1053
+ <summary>Q13. Derive the closed-form optimal policy of KL-regularized RL under BT preference.</summary>
1054
+ <p>Objective: $\max_\pi \mathbb{E}_{y\sim\pi}[r(x,y)] - \beta\,\text{KL}(\pi\|\pi_\text{ref})$.</p>
1055
+ <ol><li>Lagrangian: $\sum_y \pi r - \beta\sum_y\pi\log(\pi/\pi_\text{ref}) - \mu(\sum_y\pi - 1)$</li><li>Differentiate w.r.t. $\pi(y) = 0$: $r - \beta(\log(\pi/\pi_\text{ref}) + 1) - \mu = 0$</li><li>Rearrange: $\log(\pi/\pi_\text{ref}) = (r - \mu - \beta)/\beta$</li><li>Exponentiate: $\pi(y) = \pi_\text{ref}(y)\exp((r - \mu - \beta)/\beta)$</li><li>Use $\sum_y\pi = 1$ to solve for $\mu$: $e^{(\mu+\beta)/\beta} = \sum_y\pi_\text{ref}\exp(r/\beta) = Z$</li><li>Get: $\pi^*(y|x) = \pi_\text{ref}(y|x)\exp(r/\beta)/Z(x)$</li></ol>
1056
+ <p>Note: $J$ is strictly concave in $\pi$, so the optimum is unique.</p>
1057
+ <p>Missing the normalization constraint; missing the strict concavity argument; sign flip.</p>
1058
+ </details>
1059
+ <details>
1060
+
1061
+ <summary>Q14. Derive the DPO loss starting from the closed-form $\pi^*$ in §6.1.</summary>
1062
+ <ol><li>§6.1 gives $\pi^*(y|x) = \pi_\text{ref}\exp(r/\beta)/Z(x)$</li><li>Inversion: $r(x,y) = \beta\log(\pi^*/\pi_\text{ref}) + \beta\log Z(x)$</li><li>Bradley-Terry: $P(y_w \succ y_l) = \sigma(r_w - r_l)$</li><li>Key observation: $\beta\log Z(x)$ doesn't depend on $y$, so <strong>it cancels in $r_w - r_l$</strong>: $r_w - r_l = \beta\log(\pi^*(y_w)/\pi_\text{ref}(y_w)) - \beta\log(\pi^*(y_l)/\pi_\text{ref}(y_l))$</li><li>Replace $\pi^*$ with learnable $\pi_\theta$, NLL over preference data: $\mathcal{L}_\text{DPO} = -\mathbb{E}\log\sigma(\beta\log(\pi_\theta(y_w)/\pi_\text{ref}(y_w)) - \beta\log(\pi_\theta(y_l)/\pi_\text{ref}(y_l)))$</li></ol>
1063
+ <p>Key trick: <strong>$\log Z$ doesn't depend on $y$</strong>, so it cancels in the pairwise difference (this is why DPO doesn't need sampling).</p>
1064
+ <p>Not explaining why $\log Z$ cancels; not knowing about BT's sigmoid.</p>
1065
+ </details>
1066
+ <details>
1067
+
1068
+ <summary>Q15. Derive the DPO gradient.</summary>
1069
+ <p>Let $h_\theta = \beta\log(\pi_\theta(y_w)/\pi_\text{ref}(y_w)) - \beta\log(\pi_\theta(y_l)/\pi_\text{ref}(y_l))$ (implicit reward margin).</p>
1070
+ <p>$\mathcal{L}_\text{DPO} = -\mathbb{E}\log\sigma(h_\theta)$, $\nabla = -\mathbb{E}\sigma'(h_\theta)/\sigma(h_\theta) \cdot \nabla h_\theta$.</p>
1071
+ <p>Using $\sigma'(x) = \sigma(x)\sigma(-x)$, $\sigma'(h)/\sigma(h) = \sigma(-h) = \sigma(\hat{r}_l - \hat{r}_w)$ (i.e. "misranking confidence").</p>
1072
+ <p>$\nabla h_\theta = \beta(\nabla\log\pi_\theta(y_w) - \nabla\log\pi_\theta(y_l))$ ($\pi_\text{ref}$ is constant, derivative w.r.t. $\theta$ is 0).</p>
1073
+ <p>Combining:</p>
1074
+ <p>$$\nabla\mathcal{L}_\text{DPO} = -\beta\mathbb{E}[\sigma(\hat{r}_l - \hat{r}_w)(\nabla\log\pi_\theta(y_w) - \nabla\log\pi_\theta(y_l))]$$</p>
1075
+ <p>Interpretation:</p>
1076
+ <ul><li>The coefficient $\sigma(\hat{r}_l - \hat{r}_w)$ is "how much the current model misranks $y_l$ above $y_w$" → hard-example mining</li><li>The rest is "raise $y_w$ probability + lower $y_l$ probability"</li></ul>
1077
+ <p>Missing a step in the sigmoid' derivation; not knowing $\sigma'(x) = \sigma(x)\sigma(-x)$.</p>
1078
+ </details>
1079
+ <details>
1080
+
1081
+ <summary>Q16. Why is the k2 estimator biased? In the small-KL limit, by how much?</summary>
1082
+ <p>Let $r = \pi_\theta/\pi_\text{ref}$, $y \sim \pi_\theta$, $\text{KL}(\pi_\theta\|\pi_\text{ref}) = \mathbb{E}_{\pi_\theta}[\log r]$.</p>
1083
+ <ul><li><strong>k2</strong> is defined as $\mathbb{E}_{\pi_\theta}[\tfrac12(\log r)^2]$, generally $\ne \mathbb{E}[\log r]$ — so <strong>biased</strong>.</li><li><strong>The correct Taylor expansion</strong> should use $s = \pi_\text{ref}/\pi_\theta = 1/r$ (this is the random variable with $\mathbb{E}_{\pi_\theta}[s] = 1$, so we can do a mean-1 expansion). $\log r = -\log s$, $\log s = (s-1) - \tfrac12 (s-1)^2 + O((s-1)^3)$.</li><li>$\mathbb{E}_{\pi_\theta}[\log r] = -\mathbb{E}[\log s] = -\mathbb{E}[s-1] + \tfrac12\mathbb{E}[(s-1)^2] + O(\cdot) = 0 + \tfrac12\mathrm{Var}(s) + O$. This matches the Fisher second-order expansion.</li><li>$\mathbb{E}_{\pi_\theta}[\tfrac12 (\log r)^2] = \tfrac12\mathbb{E}[(\log s)^2] = \tfrac12\mathbb{E}[(s-1)^2] + O((s-1)^3) = \tfrac12\mathrm{Var}(s) + O$.</li><li>So <strong>in the small-KL limit (i.e. $r\approx 1$)</strong>: $\mathbb{E}[k_2] \approx \mathbb{E}[\log r] = \text{KL}$, second-order equivalent.</li><li>For large KL, k2 systematically deviates from KL (higher-order terms can't be ignored, direction depends on higher moments).</li></ul>
1084
+ <p><strong>Key correction</strong>: a previous version wrote "$\mathbb{E}[r-1] = 0$" — wrong, because under $y\sim\pi_\theta$, $\mathbb{E}[r] = \mathbb{E}_{\pi_\theta}[\pi_\theta/\pi_\text{ref}]$ is generally $\ne 1$. $\mathbb{E}[s] = 1$ is the correct identity (this is the standard importance-sampling result).</p>
1085
+ <p>Only saying "second-order approximation"; not expanding Taylor, or mistakenly using $r$ instead of $s$ for the mean-1 expansion.</p>
1086
+ </details>
1087
+ <details>
1088
+
1089
+ <summary>Q17. Analogy between adaptive β and PID controllers?</summary>
1090
+ <ul><li>Adaptive β (PPO Schulman 2017) only uses the P term: if KL &gt; target → β ↑; if KL &lt; target → β ↓</li><li>Comparison to a PID controller: $u(t) = K_p e + K_i\int e + K_d \dot{e}$</li><li>P term = current-error proportional response</li><li>I term = accumulated error (prevents steady-state error), but KL is stochastic; adding I oscillates</li><li>D term = error derivative (damping), but KL estimates have high variance; D amplifies noise</li><li>Practice: <strong>only the P term</strong> is most robust (TRL default); some frameworks add a small I for long-term convergence</li></ul>
1091
+ <p>Only saying "P-controller-like" without elaboration; or adding D term without knowing it has problems.</p>
1092
+ </details>
1093
+ <details>
1094
+
1095
+ <summary>Q18. How to tune β in GRPO with k3 KL when reward is binary (0/1)?</summary>
1096
+ <ul><li>Reward = {0, 1}, advantage scale ≈ O(1)</li><li>KL_k3 per-token starts at ≈ 0 (policy equals ref), can reach 0.1 ~ 0.5 during training</li><li>If β = 0.04, KL penalty per-token ≈ 0.04 × 0.3 ≈ 0.012, reasonable vs advantage</li><li>If β = 1, KL penalty per-token ≈ 0.3, <strong>much larger than</strong> advantage, loss dominated by KL</li><li>Experience: math task starts at β = 0.01 ~ 0.04, adjusted by KL curve; DAPO often β = 1e-3 or 0</li><li>Compare RLHF helpfulness: reward scale is large (continuous [-5, 5]) → β can be larger (0.1 ~ 0.5)</li></ul>
1097
+ <p>Not knowing the coupling between β and reward scale; or mechanically applying β = 0.04.</p>
1098
+ </details>
1099
+ <details>
1100
+
1101
+ <summary>Q19. SimPO has no reference model — why does it still work? Without the KL anchor, doesn't it reward-hack?</summary>
1102
+ <ul><li>SimPO uses <strong>length-normalization</strong> $r = (\beta/|y|)\log\pi(y)$ + <strong>target reward margin</strong> $\gamma$</li><li>No KL anchor, but length-norm prevents the "$y_w$ longer = better" degeneration</li><li>Margin $\gamma$ makes the loss saturate at $r_w - r_l > \gamma$, avoiding unbounded implicit reward</li><li>Empirically SimPO outperforms DPO on some benchmarks (AlpacaEval-2 / Arena-Hard)</li><li>But <strong>the cost of no KL anchor</strong>: poor OOD robustness, sensitive to $\beta, \gamma$ tuning, generation length may still be too large</li><li>In production, DPO + small β is still mainstream; SimPO is a "stronger on some benchmarks but with different tradeoffs" choice</li></ul>
1103
+ <p>Saying SimPO is strictly better; not knowing length-norm + margin is the KL substitute.</p>
1104
+ </details>
1105
+ <details>
1106
+
1107
+ <summary>Q20. How are per-token KL and sequence-level KL related?</summary>
1108
+ <ul><li>By KL's <strong>chain rule</strong> (conditional decomposition): $\text{KL}(\pi_\theta(\cdot|x) \| \pi_\text{ref}(\cdot|x)) = \mathbb{E}_{y\sim\pi_\theta}[\sum_t \log(\pi_\theta(y_t|x,y_{<t})/\pi_\text{ref}(y_t|x,y_{<t}))]$</li><li>I.e. <strong>sequence-level KL = expectation of the sum of token-level KLs</strong></li><li>Engineering must compute at the token level (vocab summation is fine, sequence summation explodes combinatorially)</li><li>Actual RLHF loss: per-token KL → mask assistant tokens → sum → average per sequence</li><li>Note: on prompt tokens, policy and ref have identical input, so KL should = 0, but floating-point noise pollutes → mask required</li></ul>
1109
+ <p>Confusing the two; or not knowing why masking only assistant tokens is necessary.</p>
1110
+ </details>
1111
+ <h3 id="l3-top-lab-questions-5-questions">L3 top-lab questions (5 questions)</h3>
1112
+ <details>
1113
+
1114
+ <summary>Q21. Prove $f(x) = e^x - x - 1 \ge 0$ for all $x \in \mathbb{R}$. Starting from this inequality, derive the non-negativity and unbiasedness of k3. Which property is more critical for "KL as a loss term", and why?</summary>
1115
+ <p><strong>Non-negativity proof</strong>:</p>
1116
+ <p>$f(x) = e^x - x - 1$. $f'(x) = e^x - 1$, $f'(x) = 0$ ⟺ $x = 0$. $f''(x) = e^x > 0$, so $f$ is strictly convex; $x = 0$ is the global minimum, $f(0) = 0$. So $f(x) \ge 0$ for all $x$. ✓</p>
1117
+ <p><strong>k3 non-negativity</strong>: Setting $x = \log(\pi_\text{ref}/\pi_\theta) = -\log r$, $\widehat{\text{KL}}_3 = f(-\log r) \ge 0$ always. ✓</p>
1118
+ <p><strong>k3 unbiasedness</strong>:</p>
1119
+ <p>$\mathbb{E}_{\pi_\theta}[\widehat{\text{KL}}_3] = \mathbb{E}_{\pi_\theta}[e^{-\log r}] + \mathbb{E}_{\pi_\theta}[\log r] - 1 = 1 + \text{KL} - 1 = \text{KL}$ ✓</p>
1120
+ <p><strong>Which property is more critical for "KL as a loss term"?</strong></p>
1121
+ <ul><li><strong>Non-negative + value-side unbiased</strong>: these are good <strong>monitoring/numerical-display</strong> properties — the KL loss curve doesn't show negatives or huge noise. But <strong>the key issue with k3-as-loss is not in the numerical layer</strong>, but in the gradient layer.</li><li>⚠️ <strong>The gradient when k3 is backpropagated as a loss is a first-order Taylor approximation of the reverse-KL gradient</strong>: $\nabla(e^{-\Delta} + \Delta - 1) = (1-e^{-\Delta})\nabla\Delta = (\Delta - \tfrac12\Delta^2 + O(\Delta^3))\,\nabla\log\pi_\theta$, with $O(\Delta^2)$ bias (see §3.6 + Rethinking KL arXiv 2510.01555).</li><li><strong>The truly principled loss choice is (P2) k2-as-loss</strong>: $\nabla(\tfrac12\Delta^2) = \Delta\,\nabla\log\pi_\theta$ equals the strict reverse-KL gradient in expectation on-policy. k2's value is not calibrated but its gradient is correct — gradient-equivalent to k1-in-reward.</li><li><strong>If "non-negative + unbiased + easy to monitor" is required</strong> ⇒ k3 remains the best <strong>value estimator</strong>; but distinguishing "value-side k3 monitor + loss-side k2/k1-in-reward gradient" is a more principled combination than "k3-as-loss".</li></ul>
1122
+ <p>So k3 is most convenient at the <strong>value-monitoring layer</strong> (non-negative + value-unbiased + low variance). But <strong>for the principled-ness of "as a loss addend", the key is not non-negativity but whether the loss-gradient matches the true reverse-KL gradient</strong> — k3-as-loss's gradient $(1 - e^{-\Delta})\nabla\log\pi_\theta$ is only a first-order Taylor approximation with $O(\Delta^2)$ bias; on-policy, the strictly principled losses are (P2) k2-as-loss or (P1) k1-in-reward. GRPO/DAPO historically used k3 mostly as engineering convention + because the bias is negligible at small β.</p>
1123
+ <p><strong>Common mistake</strong>: treating non-negativity as the "key for loss-addend" — that's only the numerical display advantage; what truly determines principled-ness is gradient correctness. The ideal combination is "value-side k3 monitor + loss-side k2/k1-in-reward gradient".</p>
1124
+ </details>
1125
+ <details>
1126
+
1127
+ <summary>Q22. Derive the "dual equivalence" between DPO's implicit reward and the policy gradient of KL-regularized RL.</summary>
1128
+ <p>Setup: $\max_\pi \mathbb{E}_\pi[r] - \beta\,\text{KL}(\pi\|\pi_\text{ref})$, closed-form $\pi^*(y|x) = \pi_\text{ref}\exp(r/\beta)/Z$.</p>
1129
+ <p><strong>RL view</strong>: when reward $r$ is given, the optimal policy moves towards $\pi^*$ via importance-weighted updates. Score-function policy gradient:</p>
1130
+ <p>$\nabla_\theta J = \mathbb{E}_{\pi_\theta}[\nabla\log\pi_\theta \cdot (r - \beta\log(\pi_\theta/\pi_\text{ref}))]$</p>
1131
+ <p><strong>DPO view</strong>: treat $\pi$ as the learnable $\pi_\theta$, invert BT to get implicit reward $\hat{r} = \beta\log(\pi_\theta/\pi_\text{ref}) + \beta\log Z$, do NLL over preference pairs.</p>
1132
+ <p><strong>Equivalence</strong>:</p>
1133
+ <ol><li>The closed-form gives $r = \beta\log(\pi^*/\pi_\text{ref}) + \beta\log Z$.</li><li>If $\pi_\theta = \pi^*$ (i.e. RL has converged), then the implicit reward recovers the true $r$ (up to additive $\beta\log Z$, which cancels in BT pairwise).</li><li>RL gradient $\propto r - \beta\log(\pi_\theta/\pi_\text{ref})$ at convergence $= \beta\log Z(x)$ (constant in $y$) → gradient is 0 (PG stops at $\pi^*$).</li><li>DPO gradient $\propto \sigma(\hat{r}_l - \hat{r}_w)(\nabla\log\pi(y_w) - \nabla\log\pi(y_l))$, at $\pi_\theta = \pi^*$ corresponds to "BT model fits perfectly" → gradient also vanishes.</li><li>The fixed points of both views are $\pi^*$, so <strong>RL and DPO are optimizing the same objective from two angles</strong>: RL is forward optimization (directly max $J$), DPO is inverse optimization (fit BT model to preference data).</li></ol>
1134
+ <p><strong>Key</strong>: DPO is not a replacement for RL — it is <strong>a different reduction of the same objective</strong>. RL via sampling-based PG, DPO via closed-form + supervised learning. "DPO has no RL" is a misreading.</p>
1135
+ <p>Only saying "DPO derives from RL closed form" without expanding gradient equivalence; or not knowing both views share the same fixed point.</p>
1136
+ </details>
1137
+ <details>
1138
+
1139
+ <summary>Q23. From $\sqrt{\text{KL}}$ and the inverted-U gold curve, derive a "safe KL budget" for reward overoptimization.</summary>
1140
+ <p>Gao 2023's BoN gold reward fit: $R_g(d) = d(\alpha_g - \gamma_g d)$, $d = \sqrt{\text{KL}}$.</p>
1141
+ <p>Find peak: $dR_g/dd = \alpha_g - 2\gamma_g d = 0$ → $d_\text{peak} = \alpha_g / (2\gamma_g)$.</p>
1142
+ <p>Corresponding KL distance: $\text{KL}_\text{peak} = d_\text{peak}^2 = \alpha_g^2 / (4\gamma_g^2)$.</p>
1143
+ <p><strong>Safe budget</strong>: stop before peak. Typically pick $\text{KL}_\text{stop} = 0.5 \cdot \text{KL}_\text{peak}$ (50% safety margin).</p>
1144
+ <p><strong>How to estimate $\alpha_g, \gamma_g$?</strong> Need gold reward signal (human / strong RM ensemble). Run a small pilot at a few KL points and fit $R_g(d)$.</p>
1145
+ <p><strong>Larger RM, peak further right</strong>: Gao 2023's scaling law shows $\alpha_g, \gamma_g$ depend on RM size (fit coefficients themselves depend on RM scale and data); larger RM has $\text{KL}_\text{peak}$ further right, with a higher $R_g(\text{peak})$ as well. This is why a larger RM both "rewards the budget" and reaches higher gold.</p>
1146
+ <p><strong>PPO slightly more complex than BoN</strong>: $R_g(d) = d(\alpha_g - \gamma_g d) - \delta_g d^{3/2}$; the third-order term shifts the peak slightly left.</p>
1147
+ <p>Application: in practice, cap measured KL at $\text{KL}_\text{peak} \cdot 0.5$ for early stop, i.e. "$\sqrt{\text{KL}}$ &lt; $d_\text{peak}/2$".</p>
1148
+ <p>Missing the derivative; or not knowing the RM-size-vs-peak scaling.</p>
1149
+ </details>
1150
+ <details>
1151
+
1152
+ <summary>Q24. If you had to design an RLHF algorithm combining reverse + forward KL, how would you use them?</summary>
1153
+ <p><strong>Motivation</strong> (under the standard convention: reverse = $\text{KL}(q\|p)$ mode-seeking; forward = $\text{KL}(p\|q)$ mass-covering):</p>
1154
+ <ul><li><strong>Reverse KL</strong> $\text{KL}(\pi_\theta\|\pi_\text{ref})$ = mode-seeking on $\pi_\theta$ → pick reward-high modes inside high-density regions of ref (matches the rule). <strong>RLHF's default</strong>.</li><li><strong>Forward KL</strong> $\text{KL}(\pi_\text{ref}\|\pi_\theta)$ = mass-covering on $\pi_\theta$ → make $\pi_\theta$ cover all of ref's possible outputs (prevents mode collapse).</li></ul>
1155
+ <p><strong>Problem</strong>: forward KL requires sampling from $\pi_\text{ref}$, which is engineering-meaningless (we are training $\pi_\theta$, not $\pi_\text{ref}$).</p>
1156
+ <p><strong>Workaround proposals (theoretical)</strong>:</p>
1157
+ <ol><li><strong>JSD-style combo</strong>: use $\text{JS}(\pi_\theta\|\pi_\text{ref}) = \tfrac{1}{2}\text{KL}(\pi_\theta\|m) + \tfrac{1}{2}\text{KL}(\pi_\text{ref}\|m)$, $m = (\pi_\theta + \pi_\text{ref})/2$. Symmetric, bounded, captures both mode-seeking and mass-covering. Problem: $m$ has no closed form (mixture distribution), gradient computation is complex.</li><li><strong>Importance sampling</strong>: sample from $\pi_\theta$ but reweight to $\pi_\text{ref}$ expectation: $\mathbb{E}_{\pi_\theta}[(\pi_\text{ref}/\pi_\theta)\log(\pi_\text{ref}/\pi_\theta)] = \text{KL}(\pi_\text{ref}\|\pi_\theta)$. Problem: tails have very large $\pi_\text{ref}/\pi_\theta$, variance explodes.</li><li><strong>Symmetric KL</strong>: $\text{KL}_\text{sym} = \tfrac{1}{2}(\text{KL}(\pi_\theta\|\pi_\text{ref}) + \text{KL}(\pi_\text{ref}\|\pi_\theta))$, symmetric but still needs the forward-side estimate.</li><li><strong>Hybrid penalty</strong>: early training mostly reverse KL (drive policy towards modes), later add small forward-KL term estimated via importance sampling to prevent mode collapse.</li></ol>
1158
+ <p><strong>Practical industrial approach (simpler)</strong>:</p>
1159
+ <ul><li>Use entropy bonus to replace forward KL's "mass coverage" goal — entropy doesn't need ref sampling.</li><li>Train an ensemble of policies, each mode-seeks a different mode, preserving diversity overall.</li><li>Combine BoN inference + DPO/PPO training: mode-seek during training, source diversity from BoN sampling at inference.</li></ul>
1160
+ <p><strong>Summary</strong>: forward KL is "theoretically attractive but engineering-difficult" in RLHF; mainstream approaches use entropy / multiple policies as substitutes.</p>
1161
+ <p>Just saying "add both" — too naive, must give engineering scheme; not knowing forward KL's sampling barrier.</p>
1162
+ </details>
1163
+ <details>
1164
+
1165
+ <summary>Q25. How might next-generation RLHF algorithms improve KL regularization? Give 3 directions + tradeoffs.</summary>
1166
+ <p><strong>Direction 1: Adaptive KL estimator per token</strong></p>
1167
+ <ul><li>Pick k1 vs k3 per token: small-KL tokens use k1 (unbiased + simple), large-KL tokens use k3 (prevent variance explosion).</li><li>Tradeoff: implementation complexity + hard-to-determine per-token threshold.</li></ul>
1168
+ <p><strong>Direction 2: Per-task β controller</strong></p>
1169
+ <ul><li>Different sub-tasks (math / code / dialogue / safety) use different β, dispatched online by a task classifier.</li><li>Tradeoff: requires accurate task labels; cross-task β interference needs study.</li></ul>
1170
+ <p><strong>Direction 3: Adversarial KL</strong></p>
1171
+ <ul><li>Don't fix $\pi_\text{ref}$ — train it online (e.g. self-rewarding LM, iterative DPO), but freeze $\pi_\text{ref}$ within each time window.</li><li>Tradeoff: potentially unstable (GAN-like); needs fresh human labels to prevent drift.</li></ul>
1172
+ <p><strong>Direction 4: Distributional reward + W2 KL</strong></p>
1173
+ <ul><li>Model reward as a distribution $p(r|x,y)$, replace KL with the Wasserstein-2 metric (geometric).</li><li>Tradeoff: W2 computation is complex, needs sliced approximation; theoretical convergence unclear.</li></ul>
1174
+ <p><strong>Direction 5: Hierarchical KL</strong></p>
1175
+ <ul><li>Sentence-level KL + token-level KL combined: sentence-level controls total KL budget, token-level provides fine-grained anchoring.</li><li>Analogy: PRM (process reward) vs ORM (outcome reward).</li><li>Tradeoff: sentence boundaries are not explicit during generation, requires extra annotation or heuristics.</li></ul>
1176
+ <p><strong>Direction 6: KL-free but trust-region substitute</strong></p>
1177
+ <ul><li>Drop KL entirely, use PPO clip's trust region for constraint ("clip is enough").</li><li>Already explored in IRL / SimPO.</li><li>Tradeoff: loses the closed-form RL mathematical foundation (no $\pi^* \propto \pi_\text{ref}\exp(r/\beta)$).</li></ul>
1178
+ <p>Only listing without tradeoffs; not knowing SimPO / IPO have already partly explored the KL-free route.</p>
1179
+ </details>
1180
+ <h2 id="a-appendix-reference-list">§A Appendix: reference list</h2>
1181
+ <p>Grouped by chapter. This section is a draft; <strong>arXiv IDs and exact venues are not all verified online</strong> — proofreading should re-check via <code>/arxiv</code> or codex web_search.</p>
1182
+ <p><strong>KL fundamentals and estimators</strong></p>
1183
+ <ul><li>Schulman 2020 blog <em>Approximating KL Divergence</em> <code>http://joschu.net/blog/kl-approx.html</code> (the canonical source for k1 / k2 / k3 estimators)</li><li>Endres &amp; Schindelin 2003 IEEE TIT 49(7) <em>A New Metric for Probability Distributions</em> (proof that $\sqrt{\text{JS}}$ is a metric)</li><li>Pinsker 1964 (original Pinsker inequality $\text{TV} \le \sqrt{\text{KL}/2}$)</li></ul>
1184
+ <p><strong>RLHF / PPO</strong></p>
1185
+ <ul><li>Schulman et al. 2017 arXiv 1707.06347 <em>Proximal Policy Optimization Algorithms</em> (PPO-Clip + PPO-Penalty adaptive β)</li><li>Ouyang et al. 2022 NeurIPS <em>Training Language Models to Follow Instructions with Human Feedback</em> (InstructGPT, per-token KL in reward)</li><li>Bai et al. 2022 Anthropic arXiv 2204.05862 <em>Training a Helpful and Harmless Assistant with RLHF</em> (adaptive β controller)</li></ul>
1186
+ <p><strong>DPO family</strong></p>
1187
+ <ul><li>Rafailov et al. 2023 NeurIPS <em>Direct Preference Optimization: Your Language Model is Secretly a Reward Model</em> (closed form + inversion + BT)</li><li>Azar et al. 2024 AISTATS <em>A General Theoretical Paradigm to Understand Learning from Human Preferences</em> (IPO, prevents unbounded reward under deterministic preference)</li><li>Ethayarajh et al. 2024 ICML <em>KTO: Model Alignment as Prospect Theoretic Optimization</em> (reference point replacing KL)</li><li>Meng et al. 2024 NeurIPS <em>SimPO: Simple Preference Optimization with a Reference-Free Reward</em> (drops ref, length-norm + margin)</li><li>Hong et al. 2024 EMNLP <em>ORPO: Monolithic Preference Optimization without Reference Model</em> (odds-ratio single-stage)</li></ul>
1188
+ <p><strong>GRPO family / RL with k3 KL</strong></p>
1189
+ <ul><li>Shao et al. 2024 arXiv 2402.03300 <em>DeepSeekMath: Pushing the Limits of Mathematical Reasoning</em> (first systematic use of GRPO with k3 + group-relative advantage)</li><li>DeepSeek-AI 2025 arXiv 2501.12948 <em>DeepSeek-R1</em> (GRPO + rule-based reward + emergent CoT)</li><li>Yu et al. 2025 ByteDance arXiv 2503.14476 <em>DAPO: An Open-Source LLM Reinforcement Learning System at Scale</em> (clip-higher + dynamic sampling + token-level loss + small β)</li></ul>
1190
+ <p><strong>Reward Overoptimization</strong></p>
1191
+ <ul><li>Gao, Schulman, Hilton 2023 ICML <em>Scaling Laws for Reward Model Overoptimization</em> (KL distance vs gold reward inverted-U + fitted form)</li><li>Coste et al. 2024 ICLR <em>Reward Model Ensembles Help Mitigate Overoptimization</em></li><li>Eisenstein et al. 2024 COLM (arXiv 2312.09244, 2023) <em>Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking</em></li></ul>
1192
+ <p><strong>Critic-free RL (related to KL placement)</strong></p>
1193
+ <ul><li>Ahmadian et al. 2024 ACL <em>Back to Basics: Revisiting REINFORCE Style Optimization</em> (RLOO, leave-one-out baseline, KL in reward)</li><li>Li et al. 2024 ICML <em>ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method</em></li></ul>
1194
+ <p><strong>Two 2024-2026 systematic analyses of KL-in-RLHF (arXiv metadata verified)</strong></p>
1195
+ <ul><li>Kezhao Liu et al., <em>Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization</em>, arXiv 2510.01555 (2025-10-02). Systematically distinguishes KL value estimation vs gradient optimization; shows k3-as-loss is a biased first-order approximation; recommends (P1) k1 in reward or (P2) k2 as loss; off-policy needs IS correction. §3.6 of this tutorial is based on this paper.</li><li>Vedant Shah et al., <em>A Comedy of Estimators: On KL Regularization in RL Training of LLMs</em>, arXiv 2512.21852 (2025-12-26, v3 2026-03-18). Empirical comparison of k1/k2/k3 across RL algorithms + placement combinations, focusing on estimator bias / gradient bias / placement-effect; conclusion: no universally best estimator.</li></ul>
1196
+ <p><strong>Zhihu Chinese resources</strong></p>
1197
+ <ul><li><code>https://zhuanlan.zhihu.com/p/1979720260128118305</code> — <em>KL deep dive: Forward KL, Reverse KL, KL estimation and applications</em> (verified, topic consistent with RLHF KL estimators)</li><li><code>https://zhuanlan.zhihu.com/p/1892008158626546312</code> — [needs-verify URL accessibility] topic appears to cover k2-loss vs k3-loss / GRPO off-policy / clip_std; replace if the link is broken at verification time.</li></ul>
1198
+ <p><strong>Closely related internal tutorials</strong></p>
1199
+ <ul><li><code>docs/tutorials/rlhf_dpo_grpo_ppo_tutorial.md</code> — RLHF / DPO / GRPO / PPO overview (includes BT + closed form + DPO derivation)</li><li><code>docs/tutorials/reasoning_models_tutorial.md</code> — RL training details of reasoning models</li><li><code>docs/tutorials/agentic_rl_tutorial.md</code> — token mask + KL discussion in the agentic setting</li></ul>
1200
+
1201
+ <footer class="aris-footer">
1202
+ Generated by <a href="https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep/blob/main/skills/render-html/SKILL.md">ARIS <code>/render-html</code></a> ·
1203
+ source path <code>docs/tutorials/kl_divergence_rlhf_tutorial_en.md</code> ·
1204
+ SHA256 <code>4477782e0f50</code> ·
1205
+ generated at 2026-05-19 19:29 UTC.
1206
+ This is a generated view — edit the source Markdown, then re-render.
1207
+ </footer>
1208
+ </main>
1209
+
1210
+ </div>
1211
+ </body>
1212
+ </html>