pwn 0.5.669 → 0.5.673

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 746a770eb9c9d4d410cd3fff4deb631ac67476a8d4f01ba7cf17c8414d4930d9
-  data.tar.gz: 0aa9628ff2b8f80ed5b06fe682d8c252d4609c3545c96b68dd3890897de3d402
+  metadata.gz: 9640ee1c83ff54dc1e10246d519a448d648806e30b35b579284b23708b5b9924
+  data.tar.gz: 2cbd3b2296d32ad6d068d276c3badbfa599065c668bbec7ac58fa2baf84651d7
 SHA512:
-  metadata.gz: adee5114bbd4a83820b12f591d334aeec378f5bf47d7b51d10e3ee9d14497e08207ee532eb2eab7881cb8bb2a1f8910ef6a83712b8b3cb1f7ed7d25cc14f0312
-  data.tar.gz: 5343e693c473baf4112b47b1c8a932a22b6ec46ddf1998c34e828aa79483d7c350bce500ed502f015944960b8e8ec867028f4261f11ddd57360853286778a829
+  metadata.gz: fa11ac68afcc6cde52bc207526a50cf53283ac59dd89340f25f303329562e7d34c7a02cbd286ca6fe34fffcaa307a9b0e3f9c03c03c89dca3788f9150f702f0a
+  data.tar.gz: c9bd66494f36503ba864a7c97af7d7a77b2d9502a502ed04c897ce08e2e693982efe79a43abc4c12d95e1d4b4b97ea3a1ec6fba61b0711599e025bcc2b19618d

data/.rubocop.yml

@@ -9,7 +9,7 @@ Lint/UselessRescue:
 Metrics/AbcSize:
   Max: 720
 Metrics/BlockLength:
-  Max: 292
+  Max: 320
 Metrics/BlockNesting:
   Max: 6
 Metrics/ClassLength:

data/Gemfile

@@ -18,7 +18,7 @@ gem 'aws-sdk', '3.3.0'
 gem 'barby', '0.7.0'
 gem 'base32', '0.3.4'
 gem 'bitcoin-ruby', '0.0.20'
-gem 'brakeman', '8.0.5'
+gem 'brakeman', '8.0.6'
 gem 'bson', '5.2.0'
 gem 'bundler', '>=4.0.18'
 gem 'bundler-audit', '>=0.9.3'

data/README.md

@@ -35,7 +35,7 @@ and automate instead of gluing together a pile of separate CLIs.
 
 **In numbers:** 66 `PWN::Plugins` · 48 `PWN::SAST` rules · 90 `PWN::AWS`
 service wrappers · 21 `PWN::WWW` site drivers · 53 `bin/pwn_*` CLI drivers ·
-6 LLM engines · 12 agent toolsets · 82 LLM-callable tools.
+6 LLM engines · 13 agent toolsets · 85 LLM-callable tools.
 
 Full page: [What is PWN](documentation/What-is-PWN.md)
 
@@ -66,7 +66,7 @@ My take: pwn-ai implements a **closed-loop, dual-horizon, self-and-world-aware r
 
 - fast aversive conditioning (Mistakes),
 - value estimates for actions (Metrics),
-- episode scoring and replay (Learning/Reward),
+- episode scoring and replay (Learning/Reward, cheap LLM ORM with overlap last),
 - deliberate practice (Curriculum),
 - and an external reality check (Extrospection),
 - with a slower supervised/DPO hatch only when the diet and gates look sane.
@@ -96,12 +96,13 @@ small and easy to swap:
 ![PWN Overall Architecture](documentation/diagrams/overall-pwn-architecture.svg)
 
 On every turn the AI layer runs a **feedback loop**. It checks inward
-(Metrics, Learning, and **Mistakes**: what failed last time) and outward
+(Metrics, Learning, **Mistakes**, and **Policy**: what failed last time, and
+which tool the live Q / REINFORCE table currently prefers) and outward
 (Snapshot, Drift, Intel, RF, and **Web**: did the host or network change?).
 Live checks use browser-backed **`extro_verify`** / **`extro_watch`** and RF
 **`extro_rf_tune`**. `extro_correlate` joins those views so the agent can tell
 *"I messed up"* from *"the world moved"*, and **does not repeat the same
-mistake**:
+mistake**. Policy is advice only. Planning still owns the task list:
 
 ![pwn-ai Feedback Learning Loop](documentation/diagrams/pwn-ai-feedback-learning-loop.svg)
 
@@ -118,8 +119,10 @@ message bus:
 ![Swarm Multi-Agent](documentation/diagrams/swarm-multi-agent.svg)
 
 Long-running turns also show **executive task briefs** (not raw commands) via
-`TaskSummarizer`: one full plan on submit (`emit_plan!`), then per-batch
-`about_to` lines keyed by `tool_counts_phrase` + `intent_phrase` with
+`TaskSummarizer`. Every request is first classified as a **general statement**,
+a **question**, or an **autonomous goal**. Only autonomous goals get a multi-step
+breakdown on submit (`emit_plan!`); statements and questions stay single-turn.
+Per-batch `about_to` lines use `tool_counts_phrase` + `intent_phrase` with
 `last_brief_fp` duplicate suppression. When recent turns keep hitting the iteration ceiling, the Loop tightens the remaining runway (lower `max_iters` on local engines, text-only tail, no counterfactual fork) so the agent still finishes instead of thrashing.
 
 Full pages: [How PWN Works](documentation/How-PWN-Works.md) ·
@@ -137,9 +140,10 @@ The complete wiki lives in this repo at **[`documentation/Home.md`](documentatio
 | [Why PWN](documentation/Why-PWN.md) | [`pwn-ai` Agent](documentation/pwn-ai-Agent.md) | [Agent Tool Registry](documentation/Agent-Tool-Registry.md) | [SAST (48)](documentation/SAST.md) |
 | [How PWN Works](documentation/How-PWN-Works.md) | [CLI Drivers (53)](documentation/CLI-Drivers.md) | [Memory · Skills · Learning](documentation/Skills-Memory-Learning.md) | [AWS (90)](documentation/AWS.md) |
 | [Installation](documentation/Installation.md) | [Build a Driver](documentation/Drivers.md) | [Mistakes (neg-feedback)](documentation/Mistakes.md) | [WWW (21)](documentation/WWW.md) |
-| [General Usage](documentation/General-PWN-Usage.md) | | [Extrospection](documentation/Extrospection.md) | [SDR / Radio](documentation/SDR.md) |
-| [Configuration](documentation/Configuration.md) | | [Swarm (multi-agent)](documentation/Swarm.md) | [Hardware](documentation/Hardware.md) |
-| [`~/.pwn/` Persistence](documentation/Persistence.md) | | [Sessions](documentation/Sessions.md) · [Cron](documentation/Cron.md) | [Reports](documentation/Reporting.md) |
+| [General Usage](documentation/General-PWN-Usage.md) | | [Reinforcement Learning](documentation/Reinforcement-Learning.md) | [SDR / Radio](documentation/SDR.md) |
+| [Configuration](documentation/Configuration.md) | | [Extrospection](documentation/Extrospection.md) | [Hardware](documentation/Hardware.md) |
+| [`~/.pwn/` Persistence](documentation/Persistence.md) | | [Swarm (multi-agent)](documentation/Swarm.md) | [Reports](documentation/Reporting.md) |
+| | | [Sessions](documentation/Sessions.md) · [Cron](documentation/Cron.md) | |
 | **[All Diagrams](documentation/Diagrams.md)** (29) | | | [BurpSuite](documentation/BurpSuite.md) · [NmapIt](documentation/NmapIt.md) |
 | [Troubleshooting](documentation/Troubleshooting.md) | | | [Metasploit](documentation/Metasploit.md) · [Fuzzing](documentation/Fuzzing.md) |
 | [Contributing](documentation/Contributing.md) | | | [Blockchain](documentation/Blockchain.md) · [Bounty](documentation/Bounty.md) |

data/documentation/AI-Integration.md

@@ -48,7 +48,7 @@ The harness adapts to the *class* of engine, not the model name:
 | **MEMORY ranking** | relevance-ranked when a local Ollama `embed_model` is reachable, else newest-first | relevance-ranked via `PWN::MemoryIndex` (`~/.pwn/memory.idx`) |
 | **Tool schemas shipped** | all toolsets | `CORE_TOOLS` + top-K keyword matches when `ai.agent.tool_router` is on |
 | **Pre-pass** | none | `plan_first` numbered tool plan before first dispatch |
-| **Intent route** | always | `request_intent` short-circuits how-to (text only) and unauthorized recon (refuse) on all engines; critical for ollama/openwebui |
+| **Intent route** | always | `request_intent` + LLM/heuristic `request_kind` (statement \| question \| autonomous_goal). Short-circuits how-to/questions (text only), greetings/statements (fixed ack), pure recall, and unauthorized recon on all engines; host-evidence Qs (hostname/cwd/whoami) and only true autonomous goals get multi-step TaskSummarizer plans. Critical for ollama/openwebui |
 | **Few-shot** | none | `Learning.exemplars_for(request)` splices a prior successful trace |
 | **Dispatch parsing** | strict | tolerant - Levenshtein tool-name repair + JSON5-ish arg cleanup, each repair fingerprinted into `Mistakes` |
 | **Post-answer** | `auto_introspect` | `auto_introspect` **+** `fact_check_local_final` (auto `extro_verify` on CVE/version-shaped claims) |

data/documentation/Agent-Tool-Registry.md

@@ -6,12 +6,12 @@ toolsets; the JSON-Schema for each tool is what the model actually sees.
 
 ![Tool registry](diagrams/agent-tool-registry.svg)
 
-## Toolsets to tools  (12 toolsets · 82 tools)
+## Toolsets to tools  (13 toolsets · 85 tools)
 
 | Toolset | Tools | Backed by |
 |---|---|---|
-| `terminal` | `shell` | `Open3.capture3` on the host |
-| `pwn` | `pwn_eval` | `TOPLEVEL_BINDING.eval` in the live REPL process |
+| `terminal` | `shell` | `Open3.capture3` on the host, after `PWN::AI::Agent::ToolGuard` |
+| `pwn` | `pwn_eval` | `TOPLEVEL_BINDING.eval` in the live REPL process, after `ToolGuard` |
 | `memory` | `memory_remember` · `memory_recall` · `memory_forget` · `memory_clear` · **`memory_lean`** | `PWN::Memory` → `~/.pwn/memory.json` |
 | `skills` | `skill_list` · `skill_view` · `skill_create` · `skill_add_reference` · `skill_delete` · `skill_migrate_legacy` | `~/.pwn/skills/<name>/SKILL.md` (**[agentskills.io](https://agentskills.io) spec**; legacy flat `*.md` auto-migrated) |
 | `sessions` | `sessions_list` · `sessions_view` · `sessions_current` · `sessions_delete` · `sessions_stats` · **`sessions_lean`** | `PWN::Sessions` → `~/.pwn/sessions/` |
@@ -19,6 +19,7 @@ toolsets; the JSON-Schema for each tool is what the model actually sees.
 | `reward` | **`reward_generator_mix`** | `PWN::AI::Agent::Reward.generator_mix` → online preference source-mix controller (`preferences.jsonl`) |
 | `curriculum` | **`curriculum_practice_kpi`** | `PWN::AI::Agent::Curriculum.practice_kpi` → `~/.pwn/curriculum_kpi.jsonl` |
 | `metrics` | `metrics_summary` · `metrics_reset` | `PWN::AI::Agent::Metrics` → `~/.pwn/metrics.json` |
+| `policy` | **`policy_stats`** · **`policy_evaluate`** · **`policy_recommend`** | `PWN::AI::Agent::Policy` → `~/.pwn/policy.json` + `~/.pwn/policy_traj.jsonl` |
 | `extrospection` | `extro_snapshot` · `extro_drift` · `extro_observe` · `extro_observations` · `extro_intel` · **`extro_watch`** · **`extro_verify`** · **`extro_rf_tune`** · **`extro_osint`** · **`extro_serial`** · **`extro_telecomm`** · **`extro_packet`** · **`extro_vision`** · **`extro_voice`** · `extro_correlate` · `extro_stats` · `extro_reset` · `extro_auto_toggle` | `PWN::AI::Agent::Extrospection` (+ Serial/Packet/OCR/Voice/BareSIP/TransparentBrowser/GQRX) → `~/.pwn/extrospection.json` |
 | `cron` | `cron_list` · `cron_create` · `cron_run` · `cron_enable` · `cron_disable` · `cron_remove` | `PWN::Cron` → `~/.pwn/cron/jobs.yml` |
 | `swarm` | `agent_list` · `agent_spawn` · `agent_ask` · `agent_debate` · `agent_broadcast` · `swarm_bus` · `swarm_list` | `PWN::AI::Agent::Swarm` → `~/.pwn/agents.yml` + `~/.pwn/swarm/` |
@@ -30,6 +31,17 @@ the same self-improvement loop - see [Reinforcement Learning](Reinforcement-Lear
 The thin `reward` and `curriculum` toolsets expose controller and KPI surfaces
 (`reward_generator_mix`, `curriculum_practice_kpi`) so personas can grant just
 those without the full learning surface.
+`shell` and `pwn_eval` share `PWN::AI::Agent::ToolGuard`
+(`lib/pwn/ai/agent/tool_guard.rb`) before they run. The guard remaps common
+wrong keys, rejects ellipsis placeholders, refuses bash-only syntax unless
+`ai.agent.shell_bash` is true (default runner is `/bin/sh`), and blocks live
+host-discovery unless the request is in-scope or `ai.agent.recon_authorized`
+is true.
+
+The `policy` toolset is inspect-only. `policy_stats`, `policy_evaluate`, and
+`policy_recommend` read the live Q / REINFORCE table. Reset is Ruby-only, so a
+tool call cannot wipe the weights. `Registry.rank` may add a small Q-advantage
+after a pair has been visited at least twice. Planning still owns the work.
 
 **Store hygiene tools** (`memory_lean`, `sessions_lean`, `mistakes_lean`,
 `learning_gc_stores`) trim ephemeral or oversized state without dropping
@@ -54,8 +66,8 @@ CORE_TOOLS  = shell · pwn_eval · memory_remember · memory_recall
 ```ruby
 PWN::AI::Agent::Registry.definitions(relevance: 'nmap sweep 10.0.0.0/8', top_k: 10)
 PWN::AI::Agent::Registry.rank(query: 'run a shell command')   # inspect ranking
-PWN::AI::Agent::Registry.toolsets                              # → the 12 names above
-PWN::AI::Agent::Registry.all.count                             # → 82
+PWN::AI::Agent::Registry.toolsets                              # → the 13 names above
+PWN::AI::Agent::Registry.all.count                             # → 85
 ```
 
 Frontier engines leave `tool_router` off and receive the full set.

data/documentation/Configuration.md

@@ -103,14 +103,18 @@ ai:
     task_summary_every: 5          # When task_summary_verbose: emit Progress every N completed tools.
     task_summary_interval_s: 8.0   # When verbose: also emit when this many seconds elapsed.
     task_summary_verbose: false    # Mid-flight Progress/Finished lines (default: only plan + about_to).
+    task_summary_llm: true         # LLM tangible-task decompose for autonomous goals (default on). false = offline fallback.
+    request_kind_llm: ~            # LLM request_kind classifier (statement|question|autonomous_goal). nil = follow task_summary_llm.
     max_depth: 3                   # Recursion guard: how many levels deep agent_ask/agent_debate sub-agents may spawn sub-agents.
     auto_introspect: true          # Run Learning.auto_introspect (outcome logging + lesson mining) after every final answer.
     auto_extrospect: false         # Optional ambient baseline (host/repo/env ONLY - never launches burpsuite/zaproxy/msf/gqrx). Sense tools (intel/verify/watch/rf_tune/observe) stay on-demand.
     recon_authorized: false        # Allow live subnet sweeps / raw-socket discovery tools this session
+    shell_bash: false              # true → run shell via bash -lc (PIPESTATUS, [[ ]], process substitution). Default is /bin/sh.
     plan_first: ~                  # Plan-then-act pre-pass: force the model to externalize a numbered tool plan BEFORE its first dispatch. nil = auto (true when ai.active == ollama).
     tool_router: false             # Dynamic tool-set slimming: ship only Registry::CORE_TOOLS + top-K keyword-relevant schemas per turn (helps small models route correctly).
     escalation_persona: ~          # Swarm persona name to ask for a 3-line corrective hint once a local model burns ≥ Loop::ESCALATE_AFTER_FAILS in-turn failures. nil = disabled.
-    toolsets: ~                    # Allow-list of toolsets exposed to the agent. nil = all. Valid: cron, curriculum, extrospection, learning, memory, metrics, pwn, reward, sessions, skills, swarm, terminal.
+    policy: true                   # Live tabular Q / REINFORCE. Advisory only; never replaces TaskSummarizer / plan_first. false disables.
+    toolsets: ~                    # Allow-list of toolsets exposed to the agent. nil = all. Valid: cron, curriculum, extrospection, learning, memory, metrics, policy, pwn, reward, sessions, skills, swarm, terminal.
     extrospection:
       web:
         anchors:                   # URLs the headless browser fingerprints on extro_snapshot(sections:[:web]). Alias: web_anchors.
@@ -268,19 +272,26 @@ PWN::Config.refresh_env
 | `ai.agent.task_summary_every` | Integer | `5` | `TaskSummarizer.every_n` | Verbose progress cadence (tools). |
 | `ai.agent.task_summary_interval_s` | Float | `8.0` | `TaskSummarizer.interval_s` | Verbose progress cadence (seconds). |
 | `ai.agent.task_summary_verbose` | Boolean | `false` | `TaskSummarizer.verbose?` | Emit mid-flight `Progress:` / `Finished:` lines; default keeps only plan + about_to. |
+| `ai.agent.task_summary_llm` | Boolean \| `nil` | `nil` (on) | `TaskSummarizer.llm_plan_enabled?` | LLM tangible-task decomposition for autonomous goals. `false` forces offline generic fallback (tests / air-gap). |
+| `ai.agent.request_kind_llm` | Boolean \| `nil` | `nil` (follow `task_summary_llm`) | `TaskSummarizer.llm_kind_enabled?` / `request_kind` | LLM classifier for `statement` \| `question` \| `autonomous_goal`. Cheap intents and host-evidence heuristics still win first; `false` is heuristic-only. |
 | `ai.agent.max_depth` | Integer | `3` | `PWN::AI::Agent::Swarm` | Recursion guard for `agent_ask` / `agent_debate` sub-agents spawning sub-agents. |
 | `ai.agent.auto_introspect` | Boolean | `true` | `PWN::AI::Agent::Learning.auto_introspect` | Run outcome logging + lesson mining after every final answer. Toggle live via `learning_auto_introspect_toggle`. |
 | `ai.agent.auto_extrospect` | Boolean | `false` | `PWN::AI::Agent::Extrospection.auto_extrospect` | Optional ambient baseline after every final answer (`AUTO_SECTIONS` = host/repo/env only; never spawns GUI/JVM tools). Prefer on-demand sense tools (`intel`/`verify`/`watch`/`rf_tune`/`observe`). Toggle live via `extro_auto_toggle`. |
-| `ai.agent.toolsets` | Array\<String\> \| `nil` | `nil` (all) | `bin/pwn`, `PWN::Plugins::REPL`, `PWN::AI::Agent::Registry` | Allow-list of toolsets exposed to the agent. Valid: `cron`, `curriculum`, `extrospection`, `learning`, `memory`, `metrics`, `pwn`, `reward`, `sessions`, `skills`, `swarm`, `terminal`. |
-| `ai.agent.recon_authorized` | Boolean | `false` | `PWN::AI::Agent::Loop.recon_authorized?` / shell guard | When true (or the user request contains in-scope / engagement language), live host-discovery tools may run. Default refuses unauthorized sweeps. |
+| `ai.agent.toolsets` | Array\<String\> \| `nil` | `nil` (all) | `bin/pwn`, `PWN::Plugins::REPL`, `PWN::AI::Agent::Registry` | Allow-list of toolsets exposed to the agent. Valid: `cron`, `curriculum`, `extrospection`, `learning`, `memory`, `metrics`, `policy`, `pwn`, `reward`, `sessions`, `skills`, `swarm`, `terminal`. |
+| `ai.agent.recon_authorized` | Boolean | `false` | `PWN::AI::Agent::ToolGuard.recon_authorized?` / `Loop.recon_authorized?` | When true (or the user request contains in-scope / engagement language), live host-discovery tools may run. Default refuses unauthorized sweeps. |
+| `ai.agent.shell_bash` | Boolean | `false` | `PWN::AI::Agent::ToolGuard.shell_bash?` | When true, `shell` runs via `bash -lc` so bash-only syntax is allowed. Default is POSIX `/bin/sh` and bashisms are rejected with a rewrite hint. |
 | `ai.agent.plan_first` | Boolean \| `nil` | `nil` (auto: `true` when `ai.active == ollama`) | `PWN::AI::Agent::Loop.plan_first` | Plan-then-act pre-pass: the model must emit a numbered tool plan (as an assistant message) *before* it may dispatch anything. Cheap chain-of-thought scaffolding for local models. |
 | `ai.agent.tool_router` | Boolean | `false` | `PWN::AI::Agent::Registry.definitions` | Dynamic tool-set slimming: expose only `Registry::CORE_TOOLS` + the top-K keyword-relevant schemas for *this* request. Ties break on historical `Metrics` success rate so the router itself is a learned component. |
 | `ai.agent.escalation_persona` | String \| `nil` | `nil` | `PWN::AI::Agent::Loop.escalate` → `Swarm.ask` | Circuit-breaker: once a local model accumulates ≥ `Loop::ESCALATE_AFTER_FAILS` in-turn failures, ask this Swarm persona for a 3-line corrective hint (injected as a synthetic tool result). The local model still authors the final answer so Learning/Metrics stay attributed. |
-| `ai.agent.critic` | Boolean | `false` | `PWN::AI::Agent::Curriculum.critic` (S3) | Tool-armed constitutional self-critic reviews (and may `shell`/`extro_verify`) every final answer before it is returned. |
-| `ai.agent.red_team_plan` | Boolean | `false` | `PWN::AI::Agent::Curriculum.red_team_plan` (S4) | Adversarial review of the `plan_first` numbered plan, grounded in Metrics/Mistakes/`extro_drift` telemetry, before the first dispatch. |
-| `ai.agent.counterfactual` | Boolean | `false` | `PWN::AI::Agent::Curriculum.counterfactual` (S2) | On `[REPEATING]`, fork an alt-persona branch, judge both, and record the `(loser, winner)` DPO preference pair. |
-| `ai.agent.hindsight` | Boolean | `true` | `PWN::AI::Agent::Curriculum.hindsight` (C3) | Hindsight Experience Replay - relabel a failed trajectory as `success:true` for whatever it *did* accomplish. Free positive samples from failures. |
-| `ai.agent.verify_as_reward` | Boolean | `false` | `PWN::AI::Agent::Reward.verify_as_reward` (E3) | Ground the LLM judge score by browser-verifying any checkable claim in the final via `extro_verify`; verdict caps/floors `Reward.judge`. |
+| `ai.agent.critic` | Boolean | `false` | `PWN::AI::Agent::Curriculum.critic` | Tool-armed constitutional self-critic reviews (and may `shell`/`extro_verify`) every final answer before it is returned. |
+| `ai.agent.red_team_plan` | Boolean | `false` | `PWN::AI::Agent::Curriculum.red_team_plan` | Adversarial review of the `plan_first` numbered plan, grounded in Metrics/Mistakes/`extro_drift` telemetry, before the first dispatch. |
+| `ai.agent.counterfactual` | Boolean | `false` | `PWN::AI::Agent::Curriculum.counterfactual` | On `[REPEATING]`, fork an alt-persona branch, judge both, and record the `(loser, winner)` DPO preference pair. |
+| `ai.agent.hindsight` | Boolean | `true` | `PWN::AI::Agent::Curriculum.hindsight` | Hindsight Experience Replay - relabel a failed trajectory as `success:true` for whatever it *did* accomplish. Free positive samples from failures. |
+| `ai.agent.policy` | Boolean | `true` | `PWN::AI::Agent::Policy` | Live tabular Q-learning + REINFORCE. Records `(s,a,r,s')` per tool step, trains on `Reward.judge` at episode end, and adds a small Q-advantage term to `Registry.rank`. Advisory only: never replaces TaskSummarizer or plan_first. |
+| `ai.agent.reward_llm` | Boolean \| `nil` | `nil` (auto: on for remote, off for ollama) | `PWN::AI::Agent::Reward.judge` / `.prm` | Use a cheap LLM teacher for outcome/process judges even when `module_reflection` is false. Local ollama stays heuristic unless this is `true`. |
+| `ai.agent.reward_model` | String \| `nil` | `nil` | `Reward.judge` / `.prm` | Optional cheaper model id for the ORM/PRM chat. Falls back to `ai.reflect_model`, then the active engine default. |
+| `ai.agent.reward_llm_timeout` | Integer | `12` | `Reward.judge` / `.prm` | Seconds for the cheap ORM chat (clamped 2..30). Fail fast to the overlap heuristic rather than a 900s hang. |
+| `ai.agent.verify_as_reward` | Boolean | `false` | `PWN::AI::Agent::Reward.verify_as_reward` | Ground the LLM judge score by browser-verifying any checkable claim in the final via `extro_verify`; verdict caps/floors `Reward.judge`. |
 | `ai.agent.extrospection.web.anchors` | Array\<String\> | `DEFAULT_WEB_ANCHORS` | `PWN::AI::Agent::Extrospection.probe_web` | URLs the headless browser fingerprints on `extro_snapshot(sections:[:web])`. Alias: `web_anchors`. |
 | `ai.agent.extrospection.web.proxy` | String | - | `Extrospection.probe_web` / `.verify` / `.watch` | Upstream proxy for `PWN::Plugins::TransparentBrowser` (e.g. `tor`, `http://127.0.0.1:8080`). |
 | `ai.agent.extrospection.web.max_anchors` | Integer | `8` | `Extrospection.probe_web` | Cap on anchors rendered per snapshot. |

data/documentation/Diagrams.md

@@ -60,7 +60,7 @@ groups) so lines never criss-cross.
 [source](diagrams/dot/mistakes-negative-feedback.dot) · doc: [Mistakes](Mistakes.md)
 ![mistakes-negative-feedback](diagrams/mistakes-negative-feedback.svg)
 
-### Reinforcement Learning - Reward + Curriculum
+### Reinforcement Learning - Reward + Curriculum + Policy
 [source](diagrams/dot/reinforcement-learning.dot) · doc: [Reinforcement Learning](Reinforcement-Learning.md)
 ![reinforcement-learning](diagrams/reinforcement-learning.svg)
 
@@ -68,7 +68,7 @@ groups) so lines never criss-cross.
 [source](diagrams/dot/ai-integration-tool-calling.dot) · doc: [AI Integration](AI-Integration.md)
 ![ai-integration-tool-calling](diagrams/ai-integration-tool-calling.svg)
 
-### Agent Tool Registry (12 toolsets · 82 tools)
+### Agent Tool Registry (13 toolsets · 85 tools)
 [source](diagrams/dot/agent-tool-registry.dot) · doc: [Agent Tool Registry](Agent-Tool-Registry.md)
 ![agent-tool-registry](diagrams/agent-tool-registry.svg)
 

data/documentation/Home.md

@@ -36,10 +36,10 @@
 | | |
 |---|---|
 | [AI / LLM Integration](AI-Integration.md) | OpenAI · Anthropic · Grok (OAuth) · Gemini · Ollama · Open WebUI |
-| [Agent Tool Registry](Agent-Tool-Registry.md) | 12 toolsets · **82** LLM-callable tools |
+| [Agent Tool Registry](Agent-Tool-Registry.md) | 13 toolsets · **85** LLM-callable tools |
 | [Memory · Skills · Learning](Skills-Memory-Learning.md) | Introspection - the self-improvement loop |
 | [Mistakes](Mistakes.md) | **Negative feedback** - fingerprint failures · do-NOT-repeat · `[REPEATING]`/`[REGRESSED]` · inline self-correction |
-| [Reinforcement Learning](Reinforcement-Learning.md) | **`Reward` + `Curriculum`** - outcome/process judges · preference ledger · self-play practice · export-ready LoRA gate |
+| [Reinforcement Learning](Reinforcement-Learning.md) | **`Reward` + `Curriculum` + `Policy`** - outcome/process judges · preference ledger · self-play practice · live Q / REINFORCE (advisory) · export-ready LoRA gate |
 | [Extrospection](Extrospection.md) | World-awareness - snapshot · drift · intel · **watch** · **verify** · **rf_tune** · **osint** · serial · telecomm · packet · vision · voice · correlate |
 | [Swarm (Multi-Agent)](Swarm.md) | Personas · ask · debate · broadcast · shared bus |
 | [Sessions](Sessions.md) | Transcript persistence + reflection |

data/documentation/How-PWN-Works.md

@@ -28,20 +28,20 @@ hardware).
 | Module | Role |
 |---|---|
 | `Loop` | plan → **TaskSummarizer** briefs → dispatch tool_calls → observe → repeat until final answer; tightens runway when recent turns exhausted the budget |
-| **`TaskSummarizer`** | Executive UX: English tasks primary - `emit_plan!` · `about_to` as `task k/n` + via tools · Loop `plan_context` injection · clearer `plan_idx` advance |
-| `Registry` | JSON-Schema function definitions grouped into 12 **toolsets** · **82 tools** |
+| **`TaskSummarizer`** | Executive UX: LLM/heuristic `request_kind` gates plans (only autonomous goals, including host-evidence Qs) - English tasks primary - `emit_plan!` · `about_to` as `task k/n` + via tools · Loop `plan_context` injection · clearer `plan_idx` advance |
+| `Registry` | JSON-Schema function definitions grouped into 13 **toolsets** · **85 tools** |
 | `Dispatch` / `Result` | execute a tool, capture stdout/value/error/duration |
-| `PromptBuilder` | inject MEMORY / SKILLS / LEARNING / **KNOWN MISTAKES + FIXES** / METRICS / EXTROSPECTION blocks |
-| `Metrics` · `Learning` · `Reflect` | **introspection** - how well am I doing? |
+| `PromptBuilder` | inject MEMORY / SKILLS / LEARNING / **KNOWN MISTAKES + FIXES** / METRICS / **POLICY** / EXTROSPECTION / RECENT TURNS |
+| `Metrics` · `Learning` · `Reflect` · **`Policy`** | **introspection** - how well am I doing? (Policy is live Q / REINFORCE, advisory rank only) |
 | `Mistakes` | **negative feedback** - fingerprint failures, do NOT repeat, `[REPEATING]`/`[REGRESSED]`, inline `correction_hint` |
-| **`Reward`** | outcome `judge` · per-step process credit · `sentinel` (proxy vs judge drift) · `semantic_ok` · DPO `preferences.jsonl` |
+| **`Reward`** | cheap LLM outcome `judge` (heuristic overlap last) · per-step process credit · `sentinel` (proxy vs ORM-weighted judge) · `semantic_ok` · DPO `preferences.jsonl` |
 | **`Curriculum`** | mistake-driven self-play `practice` · `counterfactual` A/B · tool-armed `critic` · `red_team_plan` · `hindsight` (HER) · `train_and_gate` regression-gated LoRA |
 | `Extrospection` | **extrospection** - on-demand world sensing (`intel` · **`verify`** · **`watch`** · **`rf_tune`** · **`osint`** · `serial` · `telecomm` · `packet` · `vision` · `voice`) + ambient baseline (host · net · toolchain · repo · env · **rf** · **web**) joined to introspection via `correlate` |
 | `Swarm` | multi-agent personas over a shared JSONL bus |
 
 See [Agent Tool Registry](Agent-Tool-Registry.md) for every tool the LLM can
 call, and [Reinforcement Learning](Reinforcement-Learning.md) for how
-`Reward` + `Curriculum` close the weight-level loop.
+`Reward` + `Curriculum` + `Policy` close the learning loop.
 
 ## L3 - Capability namespaces (`lib/pwn/*`)
 
@@ -69,10 +69,12 @@ lessons, **failures become fingerprinted mistakes with fixes**, **world state
 is sensed on demand** (`extro_verify` / `extro_watch` / `extro_rf_tune` /
 `extro_osint` / `extro_serial` / `extro_telecomm` / `extro_packet` /
 `extro_vision` / `extro_voice` / `extro_intel`) and correlated against those
-failures, an **LLM judge scores the final answer** and a **process reward
-model tags each tool step**, and **all six prompt blocks** (MEMORY · SKILLS ·
-LEARNING · KNOWN MISTAKES/FIXES · TOOL EFFECTIVENESS · EXTROSPECTION) are
-re-injected into the next system prompt.
+failures, a **cheap LLM judge scores the final answer** (token overlap only if the engine is unavailable) and a **process reward
+model tags each tool step**, **Policy records the live MDP step** and updates
+Q / REINFORCE when the judge scores the turn, and **the prompt blocks**
+(MEMORY · SKILLS · LEARNING · KNOWN MISTAKES/FIXES · TOOL EFFECTIVENESS ·
+POLICY · EXTROSPECTION · RECENT TURNS) are re-injected into the next system
+prompt.
 Nightly cron practices the top unresolved Mistakes; weekly cron builds a
 LoRA and only promotes it if it beats the previous adapter on that same
 mistake set:

data/documentation/Installation.md

@@ -187,7 +187,9 @@ predates the running gem (`PWN::Migrate.needed?`).
 
 Schema `v1` also seeds `PWN::Cron.install_defaults` - the nightly
 `curriculum_practice` and weekly `curriculum_train` self-improvement jobs
-(see [Reinforcement Learning](Reinforcement-Learning.md)).
+(see [Reinforcement Learning](Reinforcement-Learning.md)). Live Policy
+files (`policy.json`, `policy_traj.jsonl`) are created on first agent turn
+when `ai.agent.policy` is on.
 
 From a checkout:
 

data/documentation/Mistakes.md

@@ -38,10 +38,13 @@ Loop.run --(same sig, count≥3)----> guard_repeated_failure  (uses PERSISTENT c
 Loop.run --(failure w/ known fix)-> inline correction_hint  ("seen 5×, sig=..., KNOWN FIX: ..."
                                                              → self-corrects NEXT iteration)
 Loop.run --(user says "wrong")----> check_user_correction   (flip last outcome + record)
+Loop.run --(start of turn)--------> extinguish_parked!      (close inbox scars that already have a known recipe)
 PromptBuilder <-------------------- Mistakes.to_context     (KNOWN MISTAKES + KNOWN FIXES)
 model --(tool call)---------------> mistakes_record / mistakes_resolve
 ```
 
+Repeating shell failures with a known recipe (`command is required`, missing path, not-found junk, syntax, raw-socket deny) are resolved in place instead of parked as "needs human". Only scars the loop cannot close stay in the operator inbox.
+
 ## Five ingest paths - nothing slips through
 
 | Source | Trigger | What is recorded |

data/documentation/Persistence.md

@@ -16,6 +16,8 @@ Every byte PWN remembers between processes lives here.
 | **`preferences.jsonl`** | **`PWN::AI::Agent::Reward`** | JSON-per-line `{prompt,rejected,chosen,source}` | `rm` | **DPO/KTO/ORPO preference-pair ledger - user_correction · mistakes_resolve · counterfactual · critic · curriculum** |
 | **`mistakes.json`** | **`PWN::AI::Agent::Mistakes`** | **JSON `{sig → entry}`** | **`mistakes_reset`** | **failure fingerprints · cross-session count · fix · `[REPEATING]` · `[REGRESSED]`** |
 | `metrics.json` | `PWN::AI::Agent::Metrics` | JSON | `metrics_reset` | per-tool calls · success · avg_duration · last_error · **per-engine** sub-buckets · calibration |
+| **`policy.json`** | **`PWN::AI::Agent::Policy`** | JSON `{q,h,visits,returns}` | `PWN::AI::Agent::Policy.reset` | Live Q table + REINFORCE logits. Advisory rank only. |
+| **`policy_traj.jsonl`** | **`PWN::AI::Agent::Policy`** | JSON-per-line episodes | `PWN::AI::Agent::Policy.reset` | MDP trajectories `(s,a,r,s')` from each Loop turn |
 | `reward_sentinel.json` | `PWN::AI::Agent::Reward` | JSON | `rm` | proxy vs judge vs user-correction gap history |
 | `extrospection.json` | `PWN::AI::Agent::Extrospection` | JSON | `extro_reset` | host/net/toolchain/repo/env/**rf**/**web**/osint/serial/telecomm/packet/vision/voice snapshot + previous baseline + observations[] |
 | `extrospection/web/*.png` | `Extrospection` | PNG | `rm -rf` | headless-browser screenshots from `probe_web` / `extro_watch` (opt-in) |
@@ -76,6 +78,7 @@ extro_reset(confirm: true)     # host snapshot + observations
 mistakes_reset(confirm: true)  # failure fingerprints (host-specific errors)
 learning_reset(confirm: true)  # task outcomes (optional)
 metrics_reset(confirm: true)   # tool telemetry (optional)
+PWN::AI::Agent::Policy.reset   # live Q table + trajectory log (optional)
 ```
 
 [← Home](Home.md) · [Configuration](Configuration.md) · [Installation](Installation.md)
@@ -90,7 +93,7 @@ protected operator preferences:
 | `memory_lean` | expired `session_*` keys and overlong values (`VALUE_MAX_CHARS`) |
 | `sessions_lean` | stub/aged transcripts not pinned by gold outcomes or open mistakes |
 | `mistakes_lean` | compact fields; age out resolved-once signatures whose fix already lives in Memory |
-| `learning_gc_stores` | coordinated lean pass across memory, learning, mistakes, and sessions |
+| `learning_gc_stores` | coordinated lean pass across memory, learning, mistakes, sessions, and Policy trajectories |
 
 All four support `dry_run: true` (plan only). Protected `operator_pref_*`,
 `process_sop_*`, and `mistake_fix_*` memory keys are never removed by lean.