pwn 0.5.670 → 0.5.673

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: df46a72d0f582458534c1bcfcbdc0fcbf121cf002828ff5da313a5cf5eddea37
-  data.tar.gz: c4bf47cbbe4e7ad22d52104f4fa9af499c06176cae9b93500652956428abe4a2
+  metadata.gz: 9640ee1c83ff54dc1e10246d519a448d648806e30b35b579284b23708b5b9924
+  data.tar.gz: 2cbd3b2296d32ad6d068d276c3badbfa599065c668bbec7ac58fa2baf84651d7
 SHA512:
-  metadata.gz: c51242fc7542f675245804852cb313d75a8779675032148585cfde8afab392934d9da03eef08765d1339bca64732c7c3d7350a6ff76a366801c36e5b9e315df5
-  data.tar.gz: 983ff27f8a551664fbc42a5d72357389fee1aaeeb67da5c7b82e4704fa54c06d469aa4589585369c48eca7441193644fd837a08049ee3a560a6bb99f424ba007
+  metadata.gz: fa11ac68afcc6cde52bc207526a50cf53283ac59dd89340f25f303329562e7d34c7a02cbd286ca6fe34fffcaa307a9b0e3f9c03c03c89dca3788f9150f702f0a
+  data.tar.gz: c9bd66494f36503ba864a7c97af7d7a77b2d9502a502ed04c897ce08e2e693982efe79a43abc4c12d95e1d4b4b97ea3a1ec6fba61b0711599e025bcc2b19618d

data/.rubocop.yml

@@ -9,7 +9,7 @@ Lint/UselessRescue:
 Metrics/AbcSize:
   Max: 720
 Metrics/BlockLength:
-  Max: 292
+  Max: 320
 Metrics/BlockNesting:
   Max: 6
 Metrics/ClassLength:

data/README.md

@@ -35,7 +35,7 @@ and automate instead of gluing together a pile of separate CLIs.
 
 **In numbers:** 66 `PWN::Plugins` · 48 `PWN::SAST` rules · 90 `PWN::AWS`
 service wrappers · 21 `PWN::WWW` site drivers · 53 `bin/pwn_*` CLI drivers ·
-6 LLM engines · 12 agent toolsets · 82 LLM-callable tools.
+6 LLM engines · 13 agent toolsets · 85 LLM-callable tools.
 
 Full page: [What is PWN](documentation/What-is-PWN.md)
 
@@ -66,7 +66,7 @@ My take: pwn-ai implements a **closed-loop, dual-horizon, self-and-world-aware r
 
 - fast aversive conditioning (Mistakes),
 - value estimates for actions (Metrics),
-- episode scoring and replay (Learning/Reward),
+- episode scoring and replay (Learning/Reward, cheap LLM ORM with overlap last),
 - deliberate practice (Curriculum),
 - and an external reality check (Extrospection),
 - with a slower supervised/DPO hatch only when the diet and gates look sane.
@@ -96,12 +96,13 @@ small and easy to swap:
 ![PWN Overall Architecture](documentation/diagrams/overall-pwn-architecture.svg)
 
 On every turn the AI layer runs a **feedback loop**. It checks inward
-(Metrics, Learning, and **Mistakes**: what failed last time) and outward
+(Metrics, Learning, **Mistakes**, and **Policy**: what failed last time, and
+which tool the live Q / REINFORCE table currently prefers) and outward
 (Snapshot, Drift, Intel, RF, and **Web**: did the host or network change?).
 Live checks use browser-backed **`extro_verify`** / **`extro_watch`** and RF
 **`extro_rf_tune`**. `extro_correlate` joins those views so the agent can tell
 *"I messed up"* from *"the world moved"*, and **does not repeat the same
-mistake**:
+mistake**. Policy is advice only. Planning still owns the task list:
 
 ![pwn-ai Feedback Learning Loop](documentation/diagrams/pwn-ai-feedback-learning-loop.svg)
 
@@ -139,9 +140,10 @@ The complete wiki lives in this repo at **[`documentation/Home.md`](documentatio
 | [Why PWN](documentation/Why-PWN.md) | [`pwn-ai` Agent](documentation/pwn-ai-Agent.md) | [Agent Tool Registry](documentation/Agent-Tool-Registry.md) | [SAST (48)](documentation/SAST.md) |
 | [How PWN Works](documentation/How-PWN-Works.md) | [CLI Drivers (53)](documentation/CLI-Drivers.md) | [Memory · Skills · Learning](documentation/Skills-Memory-Learning.md) | [AWS (90)](documentation/AWS.md) |
 | [Installation](documentation/Installation.md) | [Build a Driver](documentation/Drivers.md) | [Mistakes (neg-feedback)](documentation/Mistakes.md) | [WWW (21)](documentation/WWW.md) |
-| [General Usage](documentation/General-PWN-Usage.md) | | [Extrospection](documentation/Extrospection.md) | [SDR / Radio](documentation/SDR.md) |
-| [Configuration](documentation/Configuration.md) | | [Swarm (multi-agent)](documentation/Swarm.md) | [Hardware](documentation/Hardware.md) |
-| [`~/.pwn/` Persistence](documentation/Persistence.md) | | [Sessions](documentation/Sessions.md) · [Cron](documentation/Cron.md) | [Reports](documentation/Reporting.md) |
+| [General Usage](documentation/General-PWN-Usage.md) | | [Reinforcement Learning](documentation/Reinforcement-Learning.md) | [SDR / Radio](documentation/SDR.md) |
+| [Configuration](documentation/Configuration.md) | | [Extrospection](documentation/Extrospection.md) | [Hardware](documentation/Hardware.md) |
+| [`~/.pwn/` Persistence](documentation/Persistence.md) | | [Swarm (multi-agent)](documentation/Swarm.md) | [Reports](documentation/Reporting.md) |
+| | | [Sessions](documentation/Sessions.md) · [Cron](documentation/Cron.md) | |
 | **[All Diagrams](documentation/Diagrams.md)** (29) | | | [BurpSuite](documentation/BurpSuite.md) · [NmapIt](documentation/NmapIt.md) |
 | [Troubleshooting](documentation/Troubleshooting.md) | | | [Metasploit](documentation/Metasploit.md) · [Fuzzing](documentation/Fuzzing.md) |
 | [Contributing](documentation/Contributing.md) | | | [Blockchain](documentation/Blockchain.md) · [Bounty](documentation/Bounty.md) |

data/documentation/Agent-Tool-Registry.md

@@ -6,12 +6,12 @@ toolsets; the JSON-Schema for each tool is what the model actually sees.
 
 ![Tool registry](diagrams/agent-tool-registry.svg)
 
-## Toolsets to tools  (12 toolsets · 82 tools)
+## Toolsets to tools  (13 toolsets · 85 tools)
 
 | Toolset | Tools | Backed by |
 |---|---|---|
-| `terminal` | `shell` | `Open3.capture3` on the host |
-| `pwn` | `pwn_eval` | `TOPLEVEL_BINDING.eval` in the live REPL process |
+| `terminal` | `shell` | `Open3.capture3` on the host, after `PWN::AI::Agent::ToolGuard` |
+| `pwn` | `pwn_eval` | `TOPLEVEL_BINDING.eval` in the live REPL process, after `ToolGuard` |
 | `memory` | `memory_remember` · `memory_recall` · `memory_forget` · `memory_clear` · **`memory_lean`** | `PWN::Memory` → `~/.pwn/memory.json` |
 | `skills` | `skill_list` · `skill_view` · `skill_create` · `skill_add_reference` · `skill_delete` · `skill_migrate_legacy` | `~/.pwn/skills/<name>/SKILL.md` (**[agentskills.io](https://agentskills.io) spec**; legacy flat `*.md` auto-migrated) |
 | `sessions` | `sessions_list` · `sessions_view` · `sessions_current` · `sessions_delete` · `sessions_stats` · **`sessions_lean`** | `PWN::Sessions` → `~/.pwn/sessions/` |
@@ -19,6 +19,7 @@ toolsets; the JSON-Schema for each tool is what the model actually sees.
 | `reward` | **`reward_generator_mix`** | `PWN::AI::Agent::Reward.generator_mix` → online preference source-mix controller (`preferences.jsonl`) |
 | `curriculum` | **`curriculum_practice_kpi`** | `PWN::AI::Agent::Curriculum.practice_kpi` → `~/.pwn/curriculum_kpi.jsonl` |
 | `metrics` | `metrics_summary` · `metrics_reset` | `PWN::AI::Agent::Metrics` → `~/.pwn/metrics.json` |
+| `policy` | **`policy_stats`** · **`policy_evaluate`** · **`policy_recommend`** | `PWN::AI::Agent::Policy` → `~/.pwn/policy.json` + `~/.pwn/policy_traj.jsonl` |
 | `extrospection` | `extro_snapshot` · `extro_drift` · `extro_observe` · `extro_observations` · `extro_intel` · **`extro_watch`** · **`extro_verify`** · **`extro_rf_tune`** · **`extro_osint`** · **`extro_serial`** · **`extro_telecomm`** · **`extro_packet`** · **`extro_vision`** · **`extro_voice`** · `extro_correlate` · `extro_stats` · `extro_reset` · `extro_auto_toggle` | `PWN::AI::Agent::Extrospection` (+ Serial/Packet/OCR/Voice/BareSIP/TransparentBrowser/GQRX) → `~/.pwn/extrospection.json` |
 | `cron` | `cron_list` · `cron_create` · `cron_run` · `cron_enable` · `cron_disable` · `cron_remove` | `PWN::Cron` → `~/.pwn/cron/jobs.yml` |
 | `swarm` | `agent_list` · `agent_spawn` · `agent_ask` · `agent_debate` · `agent_broadcast` · `swarm_bus` · `swarm_list` | `PWN::AI::Agent::Swarm` → `~/.pwn/agents.yml` + `~/.pwn/swarm/` |
@@ -30,6 +31,17 @@ the same self-improvement loop - see [Reinforcement Learning](Reinforcement-Lear
 The thin `reward` and `curriculum` toolsets expose controller and KPI surfaces
 (`reward_generator_mix`, `curriculum_practice_kpi`) so personas can grant just
 those without the full learning surface.
+`shell` and `pwn_eval` share `PWN::AI::Agent::ToolGuard`
+(`lib/pwn/ai/agent/tool_guard.rb`) before they run. The guard remaps common
+wrong keys, rejects ellipsis placeholders, refuses bash-only syntax unless
+`ai.agent.shell_bash` is true (default runner is `/bin/sh`), and blocks live
+host-discovery unless the request is in-scope or `ai.agent.recon_authorized`
+is true.
+
+The `policy` toolset is inspect-only. `policy_stats`, `policy_evaluate`, and
+`policy_recommend` read the live Q / REINFORCE table. Reset is Ruby-only, so a
+tool call cannot wipe the weights. `Registry.rank` may add a small Q-advantage
+after a pair has been visited at least twice. Planning still owns the work.
 
 **Store hygiene tools** (`memory_lean`, `sessions_lean`, `mistakes_lean`,
 `learning_gc_stores`) trim ephemeral or oversized state without dropping
@@ -54,8 +66,8 @@ CORE_TOOLS  = shell · pwn_eval · memory_remember · memory_recall
 ```ruby
 PWN::AI::Agent::Registry.definitions(relevance: 'nmap sweep 10.0.0.0/8', top_k: 10)
 PWN::AI::Agent::Registry.rank(query: 'run a shell command')   # inspect ranking
-PWN::AI::Agent::Registry.toolsets                              # → the 12 names above
-PWN::AI::Agent::Registry.all.count                             # → 82
+PWN::AI::Agent::Registry.toolsets                              # → the 13 names above
+PWN::AI::Agent::Registry.all.count                             # → 85
 ```
 
 Frontier engines leave `tool_router` off and receive the full set.

data/documentation/Configuration.md

@@ -109,10 +109,12 @@ ai:
     auto_introspect: true          # Run Learning.auto_introspect (outcome logging + lesson mining) after every final answer.
     auto_extrospect: false         # Optional ambient baseline (host/repo/env ONLY - never launches burpsuite/zaproxy/msf/gqrx). Sense tools (intel/verify/watch/rf_tune/observe) stay on-demand.
     recon_authorized: false        # Allow live subnet sweeps / raw-socket discovery tools this session
+    shell_bash: false              # true → run shell via bash -lc (PIPESTATUS, [[ ]], process substitution). Default is /bin/sh.
     plan_first: ~                  # Plan-then-act pre-pass: force the model to externalize a numbered tool plan BEFORE its first dispatch. nil = auto (true when ai.active == ollama).
     tool_router: false             # Dynamic tool-set slimming: ship only Registry::CORE_TOOLS + top-K keyword-relevant schemas per turn (helps small models route correctly).
     escalation_persona: ~          # Swarm persona name to ask for a 3-line corrective hint once a local model burns ≥ Loop::ESCALATE_AFTER_FAILS in-turn failures. nil = disabled.
-    toolsets: ~                    # Allow-list of toolsets exposed to the agent. nil = all. Valid: cron, curriculum, extrospection, learning, memory, metrics, pwn, reward, sessions, skills, swarm, terminal.
+    policy: true                   # Live tabular Q / REINFORCE. Advisory only; never replaces TaskSummarizer / plan_first. false disables.
+    toolsets: ~                    # Allow-list of toolsets exposed to the agent. nil = all. Valid: cron, curriculum, extrospection, learning, memory, metrics, policy, pwn, reward, sessions, skills, swarm, terminal.
     extrospection:
       web:
         anchors:                   # URLs the headless browser fingerprints on extro_snapshot(sections:[:web]). Alias: web_anchors.
@@ -275,16 +277,21 @@ PWN::Config.refresh_env
 | `ai.agent.max_depth` | Integer | `3` | `PWN::AI::Agent::Swarm` | Recursion guard for `agent_ask` / `agent_debate` sub-agents spawning sub-agents. |
 | `ai.agent.auto_introspect` | Boolean | `true` | `PWN::AI::Agent::Learning.auto_introspect` | Run outcome logging + lesson mining after every final answer. Toggle live via `learning_auto_introspect_toggle`. |
 | `ai.agent.auto_extrospect` | Boolean | `false` | `PWN::AI::Agent::Extrospection.auto_extrospect` | Optional ambient baseline after every final answer (`AUTO_SECTIONS` = host/repo/env only; never spawns GUI/JVM tools). Prefer on-demand sense tools (`intel`/`verify`/`watch`/`rf_tune`/`observe`). Toggle live via `extro_auto_toggle`. |
-| `ai.agent.toolsets` | Array\<String\> \| `nil` | `nil` (all) | `bin/pwn`, `PWN::Plugins::REPL`, `PWN::AI::Agent::Registry` | Allow-list of toolsets exposed to the agent. Valid: `cron`, `curriculum`, `extrospection`, `learning`, `memory`, `metrics`, `pwn`, `reward`, `sessions`, `skills`, `swarm`, `terminal`. |
-| `ai.agent.recon_authorized` | Boolean | `false` | `PWN::AI::Agent::Loop.recon_authorized?` / shell guard | When true (or the user request contains in-scope / engagement language), live host-discovery tools may run. Default refuses unauthorized sweeps. |
+| `ai.agent.toolsets` | Array\<String\> \| `nil` | `nil` (all) | `bin/pwn`, `PWN::Plugins::REPL`, `PWN::AI::Agent::Registry` | Allow-list of toolsets exposed to the agent. Valid: `cron`, `curriculum`, `extrospection`, `learning`, `memory`, `metrics`, `policy`, `pwn`, `reward`, `sessions`, `skills`, `swarm`, `terminal`. |
+| `ai.agent.recon_authorized` | Boolean | `false` | `PWN::AI::Agent::ToolGuard.recon_authorized?` / `Loop.recon_authorized?` | When true (or the user request contains in-scope / engagement language), live host-discovery tools may run. Default refuses unauthorized sweeps. |
+| `ai.agent.shell_bash` | Boolean | `false` | `PWN::AI::Agent::ToolGuard.shell_bash?` | When true, `shell` runs via `bash -lc` so bash-only syntax is allowed. Default is POSIX `/bin/sh` and bashisms are rejected with a rewrite hint. |
 | `ai.agent.plan_first` | Boolean \| `nil` | `nil` (auto: `true` when `ai.active == ollama`) | `PWN::AI::Agent::Loop.plan_first` | Plan-then-act pre-pass: the model must emit a numbered tool plan (as an assistant message) *before* it may dispatch anything. Cheap chain-of-thought scaffolding for local models. |
 | `ai.agent.tool_router` | Boolean | `false` | `PWN::AI::Agent::Registry.definitions` | Dynamic tool-set slimming: expose only `Registry::CORE_TOOLS` + the top-K keyword-relevant schemas for *this* request. Ties break on historical `Metrics` success rate so the router itself is a learned component. |
 | `ai.agent.escalation_persona` | String \| `nil` | `nil` | `PWN::AI::Agent::Loop.escalate` → `Swarm.ask` | Circuit-breaker: once a local model accumulates ≥ `Loop::ESCALATE_AFTER_FAILS` in-turn failures, ask this Swarm persona for a 3-line corrective hint (injected as a synthetic tool result). The local model still authors the final answer so Learning/Metrics stay attributed. |
-| `ai.agent.critic` | Boolean | `false` | `PWN::AI::Agent::Curriculum.critic` (S3) | Tool-armed constitutional self-critic reviews (and may `shell`/`extro_verify`) every final answer before it is returned. |
-| `ai.agent.red_team_plan` | Boolean | `false` | `PWN::AI::Agent::Curriculum.red_team_plan` (S4) | Adversarial review of the `plan_first` numbered plan, grounded in Metrics/Mistakes/`extro_drift` telemetry, before the first dispatch. |
-| `ai.agent.counterfactual` | Boolean | `false` | `PWN::AI::Agent::Curriculum.counterfactual` (S2) | On `[REPEATING]`, fork an alt-persona branch, judge both, and record the `(loser, winner)` DPO preference pair. |
-| `ai.agent.hindsight` | Boolean | `true` | `PWN::AI::Agent::Curriculum.hindsight` (C3) | Hindsight Experience Replay - relabel a failed trajectory as `success:true` for whatever it *did* accomplish. Free positive samples from failures. |
-| `ai.agent.verify_as_reward` | Boolean | `false` | `PWN::AI::Agent::Reward.verify_as_reward` (E3) | Ground the LLM judge score by browser-verifying any checkable claim in the final via `extro_verify`; verdict caps/floors `Reward.judge`. |
+| `ai.agent.critic` | Boolean | `false` | `PWN::AI::Agent::Curriculum.critic` | Tool-armed constitutional self-critic reviews (and may `shell`/`extro_verify`) every final answer before it is returned. |
+| `ai.agent.red_team_plan` | Boolean | `false` | `PWN::AI::Agent::Curriculum.red_team_plan` | Adversarial review of the `plan_first` numbered plan, grounded in Metrics/Mistakes/`extro_drift` telemetry, before the first dispatch. |
+| `ai.agent.counterfactual` | Boolean | `false` | `PWN::AI::Agent::Curriculum.counterfactual` | On `[REPEATING]`, fork an alt-persona branch, judge both, and record the `(loser, winner)` DPO preference pair. |
+| `ai.agent.hindsight` | Boolean | `true` | `PWN::AI::Agent::Curriculum.hindsight` | Hindsight Experience Replay - relabel a failed trajectory as `success:true` for whatever it *did* accomplish. Free positive samples from failures. |
+| `ai.agent.policy` | Boolean | `true` | `PWN::AI::Agent::Policy` | Live tabular Q-learning + REINFORCE. Records `(s,a,r,s')` per tool step, trains on `Reward.judge` at episode end, and adds a small Q-advantage term to `Registry.rank`. Advisory only: never replaces TaskSummarizer or plan_first. |
+| `ai.agent.reward_llm` | Boolean \| `nil` | `nil` (auto: on for remote, off for ollama) | `PWN::AI::Agent::Reward.judge` / `.prm` | Use a cheap LLM teacher for outcome/process judges even when `module_reflection` is false. Local ollama stays heuristic unless this is `true`. |
+| `ai.agent.reward_model` | String \| `nil` | `nil` | `Reward.judge` / `.prm` | Optional cheaper model id for the ORM/PRM chat. Falls back to `ai.reflect_model`, then the active engine default. |
+| `ai.agent.reward_llm_timeout` | Integer | `12` | `Reward.judge` / `.prm` | Seconds for the cheap ORM chat (clamped 2..30). Fail fast to the overlap heuristic rather than a 900s hang. |
+| `ai.agent.verify_as_reward` | Boolean | `false` | `PWN::AI::Agent::Reward.verify_as_reward` | Ground the LLM judge score by browser-verifying any checkable claim in the final via `extro_verify`; verdict caps/floors `Reward.judge`. |
 | `ai.agent.extrospection.web.anchors` | Array\<String\> | `DEFAULT_WEB_ANCHORS` | `PWN::AI::Agent::Extrospection.probe_web` | URLs the headless browser fingerprints on `extro_snapshot(sections:[:web])`. Alias: `web_anchors`. |
 | `ai.agent.extrospection.web.proxy` | String | - | `Extrospection.probe_web` / `.verify` / `.watch` | Upstream proxy for `PWN::Plugins::TransparentBrowser` (e.g. `tor`, `http://127.0.0.1:8080`). |
 | `ai.agent.extrospection.web.max_anchors` | Integer | `8` | `Extrospection.probe_web` | Cap on anchors rendered per snapshot. |

data/documentation/Diagrams.md

@@ -60,7 +60,7 @@ groups) so lines never criss-cross.
 [source](diagrams/dot/mistakes-negative-feedback.dot) · doc: [Mistakes](Mistakes.md)
 ![mistakes-negative-feedback](diagrams/mistakes-negative-feedback.svg)
 
-### Reinforcement Learning - Reward + Curriculum
+### Reinforcement Learning - Reward + Curriculum + Policy
 [source](diagrams/dot/reinforcement-learning.dot) · doc: [Reinforcement Learning](Reinforcement-Learning.md)
 ![reinforcement-learning](diagrams/reinforcement-learning.svg)
 
@@ -68,7 +68,7 @@ groups) so lines never criss-cross.
 [source](diagrams/dot/ai-integration-tool-calling.dot) · doc: [AI Integration](AI-Integration.md)
 ![ai-integration-tool-calling](diagrams/ai-integration-tool-calling.svg)
 
-### Agent Tool Registry (12 toolsets · 82 tools)
+### Agent Tool Registry (13 toolsets · 85 tools)
 [source](diagrams/dot/agent-tool-registry.dot) · doc: [Agent Tool Registry](Agent-Tool-Registry.md)
 ![agent-tool-registry](diagrams/agent-tool-registry.svg)
 

data/documentation/Home.md

@@ -36,10 +36,10 @@
 | | |
 |---|---|
 | [AI / LLM Integration](AI-Integration.md) | OpenAI · Anthropic · Grok (OAuth) · Gemini · Ollama · Open WebUI |
-| [Agent Tool Registry](Agent-Tool-Registry.md) | 12 toolsets · **82** LLM-callable tools |
+| [Agent Tool Registry](Agent-Tool-Registry.md) | 13 toolsets · **85** LLM-callable tools |
 | [Memory · Skills · Learning](Skills-Memory-Learning.md) | Introspection - the self-improvement loop |
 | [Mistakes](Mistakes.md) | **Negative feedback** - fingerprint failures · do-NOT-repeat · `[REPEATING]`/`[REGRESSED]` · inline self-correction |
-| [Reinforcement Learning](Reinforcement-Learning.md) | **`Reward` + `Curriculum`** - outcome/process judges · preference ledger · self-play practice · export-ready LoRA gate |
+| [Reinforcement Learning](Reinforcement-Learning.md) | **`Reward` + `Curriculum` + `Policy`** - outcome/process judges · preference ledger · self-play practice · live Q / REINFORCE (advisory) · export-ready LoRA gate |
 | [Extrospection](Extrospection.md) | World-awareness - snapshot · drift · intel · **watch** · **verify** · **rf_tune** · **osint** · serial · telecomm · packet · vision · voice · correlate |
 | [Swarm (Multi-Agent)](Swarm.md) | Personas · ask · debate · broadcast · shared bus |
 | [Sessions](Sessions.md) | Transcript persistence + reflection |

data/documentation/How-PWN-Works.md

@@ -29,19 +29,19 @@ hardware).
 |---|---|
 | `Loop` | plan → **TaskSummarizer** briefs → dispatch tool_calls → observe → repeat until final answer; tightens runway when recent turns exhausted the budget |
 | **`TaskSummarizer`** | Executive UX: LLM/heuristic `request_kind` gates plans (only autonomous goals, including host-evidence Qs) - English tasks primary - `emit_plan!` · `about_to` as `task k/n` + via tools · Loop `plan_context` injection · clearer `plan_idx` advance |
-| `Registry` | JSON-Schema function definitions grouped into 12 **toolsets** · **82 tools** |
+| `Registry` | JSON-Schema function definitions grouped into 13 **toolsets** · **85 tools** |
 | `Dispatch` / `Result` | execute a tool, capture stdout/value/error/duration |
-| `PromptBuilder` | inject MEMORY / SKILLS / LEARNING / **KNOWN MISTAKES + FIXES** / METRICS / EXTROSPECTION blocks |
-| `Metrics` · `Learning` · `Reflect` | **introspection** - how well am I doing? |
+| `PromptBuilder` | inject MEMORY / SKILLS / LEARNING / **KNOWN MISTAKES + FIXES** / METRICS / **POLICY** / EXTROSPECTION / RECENT TURNS |
+| `Metrics` · `Learning` · `Reflect` · **`Policy`** | **introspection** - how well am I doing? (Policy is live Q / REINFORCE, advisory rank only) |
 | `Mistakes` | **negative feedback** - fingerprint failures, do NOT repeat, `[REPEATING]`/`[REGRESSED]`, inline `correction_hint` |
-| **`Reward`** | outcome `judge` · per-step process credit · `sentinel` (proxy vs judge drift) · `semantic_ok` · DPO `preferences.jsonl` |
+| **`Reward`** | cheap LLM outcome `judge` (heuristic overlap last) · per-step process credit · `sentinel` (proxy vs ORM-weighted judge) · `semantic_ok` · DPO `preferences.jsonl` |
 | **`Curriculum`** | mistake-driven self-play `practice` · `counterfactual` A/B · tool-armed `critic` · `red_team_plan` · `hindsight` (HER) · `train_and_gate` regression-gated LoRA |
 | `Extrospection` | **extrospection** - on-demand world sensing (`intel` · **`verify`** · **`watch`** · **`rf_tune`** · **`osint`** · `serial` · `telecomm` · `packet` · `vision` · `voice`) + ambient baseline (host · net · toolchain · repo · env · **rf** · **web**) joined to introspection via `correlate` |
 | `Swarm` | multi-agent personas over a shared JSONL bus |
 
 See [Agent Tool Registry](Agent-Tool-Registry.md) for every tool the LLM can
 call, and [Reinforcement Learning](Reinforcement-Learning.md) for how
-`Reward` + `Curriculum` close the weight-level loop.
+`Reward` + `Curriculum` + `Policy` close the learning loop.
 
 ## L3 - Capability namespaces (`lib/pwn/*`)
 
@@ -69,10 +69,12 @@ lessons, **failures become fingerprinted mistakes with fixes**, **world state
 is sensed on demand** (`extro_verify` / `extro_watch` / `extro_rf_tune` /
 `extro_osint` / `extro_serial` / `extro_telecomm` / `extro_packet` /
 `extro_vision` / `extro_voice` / `extro_intel`) and correlated against those
-failures, an **LLM judge scores the final answer** and a **process reward
-model tags each tool step**, and **all six prompt blocks** (MEMORY · SKILLS ·
-LEARNING · KNOWN MISTAKES/FIXES · TOOL EFFECTIVENESS · EXTROSPECTION) are
-re-injected into the next system prompt.
+failures, a **cheap LLM judge scores the final answer** (token overlap only if the engine is unavailable) and a **process reward
+model tags each tool step**, **Policy records the live MDP step** and updates
+Q / REINFORCE when the judge scores the turn, and **the prompt blocks**
+(MEMORY · SKILLS · LEARNING · KNOWN MISTAKES/FIXES · TOOL EFFECTIVENESS ·
+POLICY · EXTROSPECTION · RECENT TURNS) are re-injected into the next system
+prompt.
 Nightly cron practices the top unresolved Mistakes; weekly cron builds a
 LoRA and only promotes it if it beats the previous adapter on that same
 mistake set:

data/documentation/Installation.md

@@ -187,7 +187,9 @@ predates the running gem (`PWN::Migrate.needed?`).
 
 Schema `v1` also seeds `PWN::Cron.install_defaults` - the nightly
 `curriculum_practice` and weekly `curriculum_train` self-improvement jobs
-(see [Reinforcement Learning](Reinforcement-Learning.md)).
+(see [Reinforcement Learning](Reinforcement-Learning.md)). Live Policy
+files (`policy.json`, `policy_traj.jsonl`) are created on first agent turn
+when `ai.agent.policy` is on.
 
 From a checkout:
 

data/documentation/Mistakes.md

@@ -38,10 +38,13 @@ Loop.run --(same sig, count≥3)----> guard_repeated_failure  (uses PERSISTENT c
 Loop.run --(failure w/ known fix)-> inline correction_hint  ("seen 5×, sig=..., KNOWN FIX: ..."
                                                              → self-corrects NEXT iteration)
 Loop.run --(user says "wrong")----> check_user_correction   (flip last outcome + record)
+Loop.run --(start of turn)--------> extinguish_parked!      (close inbox scars that already have a known recipe)
 PromptBuilder <-------------------- Mistakes.to_context     (KNOWN MISTAKES + KNOWN FIXES)
 model --(tool call)---------------> mistakes_record / mistakes_resolve
 ```
 
+Repeating shell failures with a known recipe (`command is required`, missing path, not-found junk, syntax, raw-socket deny) are resolved in place instead of parked as "needs human". Only scars the loop cannot close stay in the operator inbox.
+
 ## Five ingest paths - nothing slips through
 
 | Source | Trigger | What is recorded |

data/documentation/Persistence.md

@@ -16,6 +16,8 @@ Every byte PWN remembers between processes lives here.
 | **`preferences.jsonl`** | **`PWN::AI::Agent::Reward`** | JSON-per-line `{prompt,rejected,chosen,source}` | `rm` | **DPO/KTO/ORPO preference-pair ledger - user_correction · mistakes_resolve · counterfactual · critic · curriculum** |
 | **`mistakes.json`** | **`PWN::AI::Agent::Mistakes`** | **JSON `{sig → entry}`** | **`mistakes_reset`** | **failure fingerprints · cross-session count · fix · `[REPEATING]` · `[REGRESSED]`** |
 | `metrics.json` | `PWN::AI::Agent::Metrics` | JSON | `metrics_reset` | per-tool calls · success · avg_duration · last_error · **per-engine** sub-buckets · calibration |
+| **`policy.json`** | **`PWN::AI::Agent::Policy`** | JSON `{q,h,visits,returns}` | `PWN::AI::Agent::Policy.reset` | Live Q table + REINFORCE logits. Advisory rank only. |
+| **`policy_traj.jsonl`** | **`PWN::AI::Agent::Policy`** | JSON-per-line episodes | `PWN::AI::Agent::Policy.reset` | MDP trajectories `(s,a,r,s')` from each Loop turn |
 | `reward_sentinel.json` | `PWN::AI::Agent::Reward` | JSON | `rm` | proxy vs judge vs user-correction gap history |
 | `extrospection.json` | `PWN::AI::Agent::Extrospection` | JSON | `extro_reset` | host/net/toolchain/repo/env/**rf**/**web**/osint/serial/telecomm/packet/vision/voice snapshot + previous baseline + observations[] |
 | `extrospection/web/*.png` | `Extrospection` | PNG | `rm -rf` | headless-browser screenshots from `probe_web` / `extro_watch` (opt-in) |
@@ -76,6 +78,7 @@ extro_reset(confirm: true)     # host snapshot + observations
 mistakes_reset(confirm: true)  # failure fingerprints (host-specific errors)
 learning_reset(confirm: true)  # task outcomes (optional)
 metrics_reset(confirm: true)   # tool telemetry (optional)
+PWN::AI::Agent::Policy.reset   # live Q table + trajectory log (optional)
 ```
 
 [← Home](Home.md) · [Configuration](Configuration.md) · [Installation](Installation.md)
@@ -90,7 +93,7 @@ protected operator preferences:
 | `memory_lean` | expired `session_*` keys and overlong values (`VALUE_MAX_CHARS`) |
 | `sessions_lean` | stub/aged transcripts not pinned by gold outcomes or open mistakes |
 | `mistakes_lean` | compact fields; age out resolved-once signatures whose fix already lives in Memory |
-| `learning_gc_stores` | coordinated lean pass across memory, learning, mistakes, and sessions |
+| `learning_gc_stores` | coordinated lean pass across memory, learning, mistakes, sessions, and Policy trajectories |
 
 All four support `dry_run: true` (plan only). Protected `operator_pref_*`,
 `process_sop_*`, and `mistake_fix_*` memory keys are never removed by lean.

data/documentation/Reinforcement-Learning.md

@@ -1,51 +1,77 @@
 # Reinforcement Learning in pwn-ai
 
-pwn-ai runs an in-context learning loop that can also export training data for
-a local model. On hosts **with a trainer and GPU**, the path
+pwn-ai learns while you work. Most of that learning stays **in context**: the
+agent writes what happened to disk and puts the useful bits back into the next
+prompt. On a host **with a trainer and a GPU**, the same data can also train a
+local adapter.
 
 `Curriculum.practice` → `Reward.export_dpo` → `Curriculum.train_and_gate`
 
-can promote a new LoRA. Without a trainer the same path is **export-ready**
-(datasets plus a manual CLI). Live improvement still happens in-context on
-every turn.
+That path can promote a new LoRA when the candidate beats the current one.
+Without a trainer it still **exports** the datasets and a manual CLI. Live
+improvement does not wait on weights.
 
 ![Reinforcement-learning loop](diagrams/reinforcement-learning.svg)
 
 ```
                        +------------------------------------------------+
         request -----> | Loop.run                                       |
-                       |  plan_first -> Curriculum.red_team_plan  (S4)  |
-                       |  Dispatch   -> Reward.semantic_ok        (R4)  |
-                       |             -> Mistakes.record(cause:)         |
-                       |  guard      -> Curriculum.counterfactual (S2)  |--> preference ledger (W1)
-                       |  final      -> Curriculum.critic         (S3)  |
-                       |             -> Reward.judge (outcome)    (R1)  |--> verify_as_reward (E3)
-                       |             -> Reward.prm (process)      (R2)  |--> Sessions[step_reward] (C4)
-                       |             -> Curriculum.hindsight      (C3)  |
-                       |             -> Curriculum.calibrate      (W3)  |--> Metrics.calibration
-                       |             -> Reward.sentinel           (R3)  |--> Mistakes(reward_signal)
+                       |  plan_first -> Curriculum.red_team_plan   (S4) |
+                       |  Dispatch   -> Reward.semantic_ok         (R4) |
+                       |             -> Mistakes.record(cause:)    (E1) |
+                       |  guard      -> Curriculum.counterfactual  (S2) |--> preference ledger (W1)
+                       |  final      -> Curriculum.critic          (S3) |
+                       |             -> Reward.judge (outcome)     (R1) |--> verify_as_reward (E3)
+                       |             -> Reward.prm (process)       (R2) |--> Sessions[step_reward] (C4)
+                       |             -> Curriculum.hindsight       (C3) |
+                       |             -> Curriculum.calibrate       (W3) |--> Metrics.calibration
+                       |             -> Reward.sentinel            (R3) |--> Mistakes(reward_signal)
                        +------------------------------------------------+
                                            |
-                    Learning.consolidate (M1 semantic merge, M3 importance eviction)
+                    Learning.consolidate (M1 merge, M3 importance-evict)
                     MemoryIndex.recall_semantic (M2 similarity x recency x importance)
-                    Registry.rank (C1 keyword fit + advantage + UCB)
+                    Registry.rank (C1 keyword + UCB + Q-advantage)
+                    Policy (R5 live Q / REINFORCE on a judge-scored MDP)
                     Learning.exemplars_for (C2 prioritized replay, C4 minimal trace)
                                            |
-              nightly cron --> Curriculum.practice (S1) --> Mistakes.resolve --> preference ledger (W1)
+              nightly cron --> Curriculum.practice (S1) --> Mistakes.resolve --> preference (W1)
               weekly  cron --> Curriculum.train_and_gate (W2) --> optional LoRA --> A/B gate --> promote
                                            |
-                    Extrospection.correlate (E2 world vs self join)
-                    Metrics.changepoints (E1) --> Mistakes(cause: :env_drift)
+                    Extrospection.correlate (E2 world vs self)
+                    Metrics.changepoints (E1 CUSUM) --> Mistakes(cause: :env_drift)
 ```
 
+## Live Policy on every turn (R5 · `PWN::AI::Agent::Policy`)
+
+This is the live numeric controller. It does not replace planning.
+
+| Piece | What it is |
+|---|---|
+| State | request kind, task family, plan quality, answer completeness, usable-result, last action, fail bin, and engine |
+| Action | tool name, or `final` |
+| Step reward | `Reward.semantic_ok` hygiene (`+0.05` / `-0.20`) |
+| Terminal reward | `Reward.judge` score (skipped when the cheap proxy is untrusted and there is no judge) |
+| Updates | Q-learning (`alpha=0.15`, `gamma=0.85`) and REINFORCE (`alpha=0.05`). Stored trajectories replay twice on warmup so a short table is not empty advice. |
+| Budget | Eight finished episodes (live or warmup-credited) unlock greedy suggestions. Until then the prompt omits them. |
+| Steer | Q-advantage in `Registry.rank` once the episode budget is met; keyword fit and CORE_TOOLS still come first |
+| Files | `~/.pwn/policy.json`, `~/.pwn/policy_traj.jsonl` |
+| Tools | `policy_stats` · `policy_evaluate` · `policy_recommend` (inspect only) |
+| Off switch | `ai.agent.policy: false` |
+
+Loop calls `begin_episode` before the first rank, `observe_step` after each
+tool, and `finish` from `Learning.auto_introspect` (or Loop if introspect is
+skipped). `Learning.gc_stores` can trim old trajectories without dropping
+high-return / high-score episodes.
+
 ## Reward signal (`PWN::AI::Agent::Reward`)
 
 | ID | Method | What it does |
 |----|--------|--------------|
-| **R1** | `.judge` | Outcome score on `(request, final)` → `{score:0..1, verdict:, rationale:, key_step:}`. Replaces brittle success regexes. |
+| **R1** | `.judge` | Cheap LLM outcome score on `(request, final)` → `{score:0..1, verdict:, rationale:, key_step:, source:}`. Calls the active engine `.chat` with a short timeout (default 12s). `Reflect.on` is used only when `module_reflection` is on. Fallback scores completeness, plan cover, claims, and tool-trace echo. Token overlap is only a small on-topic gate. |
 | **R2** | `.prm` | Process reward - per-tool-step `+1/0/-1` written into `Sessions[:step_reward]`. |
 | **R3** | `.sentinel` | Compares proxy success rate vs judge mean vs user-correction rate. A large gap fingerprints `reward_signal` so the agent distrusts a lying proxy. |
-| **R4** | `.semantic_ok` | Treats informational non-zero exits (e.g. `grep`/`rg` with no match) as benign. Metrics count them as OK; Mistakes only see true dispatch failures. |
+| **R4** | `.semantic_ok` | Treats informational non-zero exits (for example `grep` / `rg` with no match) as benign. Metrics count them as OK; Mistakes only see true dispatch failures. |
+| **R5** | `Policy` (live MDP) | Tabular Q-learning + REINFORCE on real Loop turns. Q-advantage is an advisory `Registry.rank` term and never replaces TaskSummarizer or plan_first. |
 | - | `.warm_sentinel` | Backfills the sentinel window from scored Learning outcomes so local hosts can engage proxy distrust without waiting for live remote introspect. |
 | **W1** | `.record_preference` / `.export_dpo` | Preference ledger (`~/.pwn/preferences.jsonl`) from user corrections, resolve, counterfactual, critic, and practice. Caps per source; keeps trajectory-shaped pairs (winning traces / revised answers), not fix commentary. |
 | - | `.scrub_preferences` / `.preference_balance` / `.generator_mix` | Ledger hygiene and source-mix health so one channel cannot flood preference export. |
@@ -54,9 +80,9 @@ every turn.
 
 | ID | Where | What |
 |----|-------|------|
-| **C1** | `Registry.rank` + `Metrics.{ucb,thompson,advantage,prm_advantage}` | Live tool choice blends keyword fit, historical advantage, exploration bonus, and process-reward signal. |
+| **C1** | `Registry.rank` + `Metrics.{ucb,thompson,advantage,prm_advantage}` + `Policy.advantage` | Live tool choice blends keyword fit, historical advantage, exploration bonus, process-reward signal, and (once the episode budget is met) Q(s,a)-V(s). Neighbor states fill cold (s,a) pairs. |
 | **C2** | `Learning.exemplars_for` | Prior successful traces ranked by judge score, recency, and keyword fit. Low-score "proxy success" rows are dropped when judge distrust is high. |
-| **C3** | `Curriculum.hindsight` | On a failed goal, relabel what the trajectory *did* achieve (`success: 'soft'`). Soft rows stay out of hard SFT and are down-weighted in exemplars. |
+| **C3** | `Curriculum.hindsight` | On a failed goal, relabel what the trajectory *did* achieve (`success: 'soft'`). Soft rows stay out of hard supervised export and are down-weighted in exemplars. |
 | **C4** | `Learning.compress_exemplar` / skill build | Keep steps with positive `step_reward` so few-shot traces stay short. |
 
 ## Memory that stays high-signal
@@ -77,9 +103,9 @@ every turn.
 | **S3** | `.critic` | Constitutional critic of the final answer (can use tools). Under budget pressure, runs text-only so it cannot burn the remaining iterations. |
 | **S4** | `.red_team_plan` | Adversarial review of the plan-first outline using Metrics / Mistakes / drift. |
 | - | `.offline_judge` | Score recent sessions with outcome + process judges, warm the sentinel, optional ledger scrub. Meant for nightly cron on local hosts that only introspect failures live. |
-| **C3** | `.hindsight` | HER relabel described above. |
+| **C3** | `.hindsight` | Relabel described above. |
 | **W3** | `.calibrate` | Plan `p(success)=` vs actual outcome → per-engine Brier / overconfidence. Overconfidence can force plan_first + critic and tighten `max_iters`. |
-| **W2** | `.train_and_gate` | Export SFT + DPO → optional LoRA train → promote only if the candidate wins on resolved margin, mean judge, smoke set, and a healthy preference diet. Without a trainer: `weight_loop: :export_ready`. |
+| **W2** | `.train_and_gate` | Export supervised + preference data → optional LoRA train → promote only if the candidate wins on resolved margin, mean judge, smoke set, and a healthy preference diet. Without a trainer: `weight_loop: :export_ready`. |
 | - | `.practice_kpi` | Week-over-week trend of repeating mistakes (outer curriculum health). |
 
 ## Budget pressure (iteration ceiling)
@@ -88,7 +114,7 @@ When unresolved `agent_loop` / `assistant_answer` budget-exhaustion fingerprints
 dominate, the loop marks the budget path hot and tightens the live turn:
 
 - lower effective `max_iters` (stricter on local/ollama engines than remote)
-- **Last-iter force-final**: tools=nil on the final iteration so a text answer is required
+- last-iter force-final: tools=nil on the final iteration so a text answer is required
 - skip counterfactual / red-team forks that would spend more tool rounds
 - still flush TaskSummarizer state and Learning on the exhaust path
 - end-of-turn critic runs text-only under the same pressure
@@ -96,27 +122,50 @@ dominate, the loop marks the budget path hot and tightens the live turn:
 Practice prioritizes those scars with short-horizon "finish the task" prompts.
 Raising `ai.agent.max_iters` or resolving the scar returns normal runway.
 
+## Design-priority STATUS (post P14-P25)
+
+This table is the flag authority. Stop minting new P-numbers for chores already
+covered. Track these outcomes instead of hunting comments in the source.
+
+| Pri | ID | Control | Module(s) | Success criterion |
+|-----|----|---------|-----------|-------------------|
+| **P0** | W1 generator diversity | `Reward::TARGET_SOURCE_MIX` + `generator_mix` + mix-urgent force on critic/counterfactual | `reward.rb`, `curriculum.rb` | `generator_mix.healthy` OR `recommendation` not stuck on `suppress:mistakes_resolve`; trajectory_fraction ≥ 0.5 |
+| **P0** | Introspect budget | `Learning::INTROSPECT_SOFT_MS` / `HARD_MS`; stage skip under soft/hard / `budget_exhaustion_hot?` | `learning.rb` | `auto_introspect` returns `stages_skipped` when over soft; post-answer path cannot re-thrash tool critic |
+| **P1** | Local judge calibration | Heuristic score shrinkage + `confidence`; `Metrics.effective_rate` scales distrust by `judge_confidence` | `reward.rb`, `metrics.rb` | distrust × heuristic no longer fully replaces proxy; local no-trace highs capped |
+| **ops** | Cheap LLM ORM | Direct engine `.chat` for `Reward.judge` / `.prm`; optional `reward_model` / `reward_llm_timeout`; evidence-prior fallback last | `reward.rb` | remote turns grade with ORM even when `module_reflection` is off |
+| **ops** | Outcome-signal haircut | `judge_sample_weight` (ORM 1.0, heuristic 0.25) in sentinel / `Learning.stats` / `Metrics.judge_rate` | `reward.rb`, `learning.rb`, `metrics.rb` | proxy distrust blends toward ORM scores, not bag-of-words overlap |
+| **P1** | Practice outer KPI | `Curriculum.practice_kpi` / `repeating_trend` → `~/.pwn/curriculum_kpi.jsonl` | `curriculum.rb` | week-over-week `delta_repeating` ≤ 0 on budget fingerprints after practice nights |
+| **P2** | PRM sample efficiency | `PRM_MIN_N=5`, shrinkage to `PRM_FULL_N=20`, fleet coverage gate in `Registry.rank` | `metrics.rb`, `registry.rb` | `prm_advantage=0` until n≥5; rank delta=0 until ≥3 tools ready |
+| **P2** | STATUS over flag archaeology | This table | docs | New work cites Pri/ID here, not fresh P26+ comments for the same theme |
+| **ops** | Nightly diet close | `offline_judge` → `scrub_preferences` + `generator_mix` + `practice_kpi` | `curriculum.rb` | Cron path returns `scrub`/`generator_mix`/`practice_kpi`; raw resolve prose does not survive the night |
+| **ops** | Shape backfill | `Reward.infer_shape` + scrub rewrite | `reward.rb` | Legacy shapeless rows get `winning_trace`/`revised_answer` when content warrants; traj_f measurable |
+| **ops** | Mix in prompt | `Metrics.to_context` emits `W1 MIX:` when unhealthy | `metrics.rb` | Unhealthy diet visible every turn without a tool call |
+| **P0** | Budget exhaust deepen | Last-iter force-final (tools=nil); skip CF when `budget_exhaustion_hot?`; tighter caps 24 local / 75 remote; exhaust path `append_session`+`auto_introspect` | `loop.rb` | Exhaust returns a judged final, not a bare string; CF cannot re-enter under hot; last iter cannot tool-call |
+
 ## Intro and extro join
 
-| Where | What |
-|-------|------|
-| `Metrics.changepoints` + `Loop.attribute_cause` (**E1**) | Env-drift-attributed failures get `cause: :env_drift` and do not inflate `[REPEATING]`. |
-| `Extrospection.correlate` (**E2**) | Lead-lag style joins ("tool X started failing after toolchain Y changed"). |
-| `Reward.verify_as_reward` (**E3**) | Browser-backed claim checks can floor/cap the outcome score. |
+| ID | Where | What |
+|----|-------|------|
+| **E1** | `Metrics.changepoints` + `Loop.attribute_cause` | Env-drift-attributed failures get `cause: :env_drift` and do not inflate `[REPEATING]`. |
+| **E2** | `Extrospection.correlate` | Lead-lag style joins ("tool X started failing after toolchain Y changed"). |
+| **E3** | `Reward.verify_as_reward` | Browser-backed claim checks can floor/cap the outcome score. |
 
 ## Config (`PWN::Env[:ai][:agent]`)
 
 ```yaml
 :ai:
-  :module_reflection: false  # gates Reflect lesson writing (not ORM alone)
+  :module_reflection: false  # gates Reflect lesson writing (not the judge alone)
   :agent:
     :critic: null            # S3 - nil = ON for remote engines, OFF for ollama
-    :red_team_plan: null     # S4 - same auto policy
-    :counterfactual: null    # S2 - same auto policy
+    :red_team_plan: null     # S4 - same auto rule
+    :counterfactual: null    # S2 - same auto rule
     :hindsight: true         # C3 - hindsight relabel on failed turns (default true)
+    :policy: true            # R5 - live tabular Q / REINFORCE (advisory rank only)
     :verify_as_reward: null  # E3 - nil = auto sample on claim-shaped answers
     :reward_llm: null        # nil = outcome/process judges use LLM teacher on remote
-    :local_introspect: :failure_only   # ollama cost policy; remote always introspects
+    :reward_model: null      # optional cheaper model id for ORM/PRM (nil = engine default)
+    :reward_llm_timeout: 12  # cheap ORM chat timeout seconds (clamped 2..30)
+    :local_introspect: :failure_only   # ollama cost rule; remote always introspects
     :introspect_every_n: 3
     :max_iters: 25           # hard cap; budget pressure may lower effective value
 ```
@@ -141,17 +190,18 @@ PWN::Cron.install_defaults
 `reward_preferences` · `reward_scrub_preferences` · `reward_preference_balance` ·
 `reward_export_dpo` · `reward_generator_mix` · `curriculum_practice` ·
 `curriculum_train` · `curriculum_hindsight` · `curriculum_offline_judge` ·
-`curriculum_preference_balance` · `curriculum_practice_kpi` ·
-`learning_purge_noise`
+`curriculum_preference_balance` · `curriculum_practice_kpi` · `policy_stats` ·
+`policy_evaluate` · `policy_recommend` · `learning_purge_noise`
 
-## Design claims
+## What this stack actually does
 
 1. Process reward on real security tool traces, not only math demos (**R2**).
-2. Automatic blame attribution: self vs environment drift (**E1** + **E2**).
+2. Automatic blame: self vs environment drift (**E1** + **E2**).
 3. Reward-hacking self-detection when proxy success diverges from the judge (**R3**).
 4. Mistake-driven curriculum with regression-gated LoRA promotion when a trainer exists (**S1** + **W2**).
 5. Preference pairs from normal agent work (corrections, resolve, critic, practice) with no separate human labelling queue (**W1**).
 6. Export and promote only when the preference diet is diverse and trajectory-shaped.
+7. Live tabular Q-learning + REINFORCE on real Loop turns, used only as advice next to planning (**R5**).
 
 ## Preference signal quality
 
@@ -166,39 +216,6 @@ keep it honest by:
 5. Requiring smoke checks and mean judge improvement before LoRA promote.
 6. Treating budget exhaustion as a first-class practice target so the agent learns to finish.
 
-## Design-priority STATUS
-
-Living checklist for the reinforced feedback loop. Cite the Pri/ID here instead
-of inventing new milestone labels for the same theme.
-
-| Pri | ID | Control | Module(s) | Success criterion |
-|-----|----|---------|-----------|-------------------|
-| **P0** | W1 generator diversity | `Reward::TARGET_SOURCE_MIX` + `generator_mix` + mix-urgent force on critic/counterfactual | `reward.rb`, `curriculum.rb` | `generator_mix.healthy` OR `recommendation` not stuck on `suppress:mistakes_resolve`; trajectory_fraction ≥ 0.5 |
-| **P0** | Introspect budget | `Learning::INTROSPECT_SOFT_MS` / `HARD_MS`; stage skip under soft/hard / budget-pressure mode | `learning.rb` | `auto_introspect` returns `stages_skipped` when over soft; post-answer path cannot re-thrash tool critic |
-| **P0** | Local judge calibration | Heuristic score shrinkage + `confidence`; `Metrics.effective_rate` scales distrust by `judge_confidence` | `reward.rb`, `metrics.rb` | distrust×heuristic no longer fully replaces proxy; local no-trace highs capped |
-| **P0** | Practice outer KPI | `Curriculum.practice_kpi` / `repeating_trend` → `~/.pwn/curriculum_kpi.jsonl` | `curriculum.rb` | week-over-week `delta_repeating` ≤ 0 on budget fingerprints after practice nights |
-| **P0** | PRM sample efficiency | `PRM_MIN_N=5`, shrinkage to `PRM_FULL_N=20`, fleet coverage gate in `Registry.rank` | `metrics.rb`, `registry.rb` | `prm_advantage=0` until n≥5; rank delta=0 until ≥3 tools ready |
-| **P0** | STATUS over flag archaeology | This table | docs | New work cites Pri/ID here, not fresh comments for the same theme |
-| **ops** | Nightly diet close | `offline_judge` → `scrub_preferences` + `generator_mix` + `practice_kpi` | `curriculum.rb` | Cron path returns `scrub`/`generator_mix`/`practice_kpi`; raw resolve prose does not survive the night |
-| **ops** | Shape backfill | `Reward.infer_shape` + scrub rewrite | `reward.rb` | Legacy shapeless rows get `winning_trace`/`revised_answer` when content warrants; traj_f measurable |
-| **ops** | Mix in prompt | `Metrics.to_context` emits `W1 MIX:` when unhealthy | `metrics.rb` | Unhealthy diet visible every turn without a tool call |
-| **P0** | Budget exhaust deepen | Last-iter force-final (tools=nil); skip CF under budget-pressure mode; hot caps 24 ollama / 75 remote; exhaust path `append_session`+`auto_introspect` | `loop.rb` | Exhaust returns a judged final, not a bare string; CF cannot re-enter under hot; last iter cannot tool-call |
-
-### Config additions
-
-```yaml
-:ai:
-  :agent:
-    # Introspect budget (ms wall-clock inside auto_introspect)
-    # INTROSPECT_SOFT_MS / HARD_MS are constants; override only via code/reload today.
-    :critic: null            # also force-on when generator_mix.urgent includes critic
-    :counterfactual: null    # also force-on when generator_mix.urgent includes counterfactual
-```
-
-### Tools (design-priority)
-
-`reward_generator_mix` · `curriculum_practice_kpi` (plus existing reward/curriculum set)
-
 ---
 
 **See also:** [pwn-ai Agent](pwn-ai-Agent.md) · [Mistakes](Mistakes.md) ·