pwn 0.5.669 → 0.5.673

data/documentation/Reinforcement-Learning.md

@@ -1,51 +1,77 @@
 # Reinforcement Learning in pwn-ai
 
-pwn-ai runs an in-context learning loop that can also export training data for
-a local model. On hosts **with a trainer and GPU**, the path
+pwn-ai learns while you work. Most of that learning stays **in context**: the
+agent writes what happened to disk and puts the useful bits back into the next
+prompt. On a host **with a trainer and a GPU**, the same data can also train a
+local adapter.
 
 `Curriculum.practice` → `Reward.export_dpo` → `Curriculum.train_and_gate`
 
-can promote a new LoRA. Without a trainer the same path is **export-ready**
-(datasets plus a manual CLI). Live improvement still happens in-context on
-every turn.
+That path can promote a new LoRA when the candidate beats the current one.
+Without a trainer it still **exports** the datasets and a manual CLI. Live
+improvement does not wait on weights.
 
 ![Reinforcement-learning loop](diagrams/reinforcement-learning.svg)
 
 ```
                        +------------------------------------------------+
         request -----> | Loop.run                                       |
-                       |  plan_first -> Curriculum.red_team_plan  (S4)  |
-                       |  Dispatch   -> Reward.semantic_ok        (R4)  |
-                       |             -> Mistakes.record(cause:)         |
-                       |  guard      -> Curriculum.counterfactual (S2)  |--> preference ledger (W1)
-                       |  final      -> Curriculum.critic         (S3)  |
-                       |             -> Reward.judge (outcome)    (R1)  |--> verify_as_reward (E3)
-                       |             -> Reward.prm (process)      (R2)  |--> Sessions[step_reward] (C4)
-                       |             -> Curriculum.hindsight      (C3)  |
-                       |             -> Curriculum.calibrate      (W3)  |--> Metrics.calibration
-                       |             -> Reward.sentinel           (R3)  |--> Mistakes(reward_signal)
+                       |  plan_first -> Curriculum.red_team_plan   (S4) |
+                       |  Dispatch   -> Reward.semantic_ok         (R4) |
+                       |             -> Mistakes.record(cause:)    (E1) |
+                       |  guard      -> Curriculum.counterfactual  (S2) |--> preference ledger (W1)
+                       |  final      -> Curriculum.critic          (S3) |
+                       |             -> Reward.judge (outcome)     (R1) |--> verify_as_reward (E3)
+                       |             -> Reward.prm (process)       (R2) |--> Sessions[step_reward] (C4)
+                       |             -> Curriculum.hindsight       (C3) |
+                       |             -> Curriculum.calibrate       (W3) |--> Metrics.calibration
+                       |             -> Reward.sentinel            (R3) |--> Mistakes(reward_signal)
                        +------------------------------------------------+
                                            |
-                    Learning.consolidate (M1 semantic merge, M3 importance eviction)
+                    Learning.consolidate (M1 merge, M3 importance-evict)
                     MemoryIndex.recall_semantic (M2 similarity x recency x importance)
-                    Registry.rank (C1 keyword fit + advantage + UCB)
+                    Registry.rank (C1 keyword + UCB + Q-advantage)
+                    Policy (R5 live Q / REINFORCE on a judge-scored MDP)
                     Learning.exemplars_for (C2 prioritized replay, C4 minimal trace)
                                            |
-              nightly cron --> Curriculum.practice (S1) --> Mistakes.resolve --> preference ledger (W1)
+              nightly cron --> Curriculum.practice (S1) --> Mistakes.resolve --> preference (W1)
               weekly  cron --> Curriculum.train_and_gate (W2) --> optional LoRA --> A/B gate --> promote
                                            |
-                    Extrospection.correlate (E2 world vs self join)
-                    Metrics.changepoints (E1) --> Mistakes(cause: :env_drift)
+                    Extrospection.correlate (E2 world vs self)
+                    Metrics.changepoints (E1 CUSUM) --> Mistakes(cause: :env_drift)
 ```
 
+## Live Policy on every turn (R5 · `PWN::AI::Agent::Policy`)
+
+This is the live numeric controller. It does not replace planning.
+
+| Piece | What it is |
+|---|---|
+| State | request kind, task family, plan quality, answer completeness, usable-result, last action, fail bin, and engine |
+| Action | tool name, or `final` |
+| Step reward | `Reward.semantic_ok` hygiene (`+0.05` / `-0.20`) |
+| Terminal reward | `Reward.judge` score (skipped when the cheap proxy is untrusted and there is no judge) |
+| Updates | Q-learning (`alpha=0.15`, `gamma=0.85`) and REINFORCE (`alpha=0.05`). Stored trajectories replay twice on warmup so a short table is not empty advice. |
+| Budget | Eight finished episodes (live or warmup-credited) unlock greedy suggestions. Until then the prompt omits them. |
+| Steer | Q-advantage in `Registry.rank` once the episode budget is met; keyword fit and CORE_TOOLS still come first |
+| Files | `~/.pwn/policy.json`, `~/.pwn/policy_traj.jsonl` |
+| Tools | `policy_stats` · `policy_evaluate` · `policy_recommend` (inspect only) |
+| Off switch | `ai.agent.policy: false` |
+
+Loop calls `begin_episode` before the first rank, `observe_step` after each
+tool, and `finish` from `Learning.auto_introspect` (or Loop if introspect is
+skipped). `Learning.gc_stores` can trim old trajectories without dropping
+high-return / high-score episodes.
+
 ## Reward signal (`PWN::AI::Agent::Reward`)
 
 | ID | Method | What it does |
 |----|--------|--------------|
-| **R1** | `.judge` | Outcome score on `(request, final)` → `{score:0..1, verdict:, rationale:, key_step:}`. Replaces brittle success regexes. |
+| **R1** | `.judge` | Cheap LLM outcome score on `(request, final)` → `{score:0..1, verdict:, rationale:, key_step:, source:}`. Calls the active engine `.chat` with a short timeout (default 12s). `Reflect.on` is used only when `module_reflection` is on. Fallback scores completeness, plan cover, claims, and tool-trace echo. Token overlap is only a small on-topic gate. |
 | **R2** | `.prm` | Process reward - per-tool-step `+1/0/-1` written into `Sessions[:step_reward]`. |
 | **R3** | `.sentinel` | Compares proxy success rate vs judge mean vs user-correction rate. A large gap fingerprints `reward_signal` so the agent distrusts a lying proxy. |
-| **R4** | `.semantic_ok` | Treats informational non-zero exits (e.g. `grep`/`rg` with no match) as benign. Metrics count them as OK; Mistakes only see true dispatch failures. |
+| **R4** | `.semantic_ok` | Treats informational non-zero exits (for example `grep` / `rg` with no match) as benign. Metrics count them as OK; Mistakes only see true dispatch failures. |
+| **R5** | `Policy` (live MDP) | Tabular Q-learning + REINFORCE on real Loop turns. Q-advantage is an advisory `Registry.rank` term and never replaces TaskSummarizer or plan_first. |
 | - | `.warm_sentinel` | Backfills the sentinel window from scored Learning outcomes so local hosts can engage proxy distrust without waiting for live remote introspect. |
 | **W1** | `.record_preference` / `.export_dpo` | Preference ledger (`~/.pwn/preferences.jsonl`) from user corrections, resolve, counterfactual, critic, and practice. Caps per source; keeps trajectory-shaped pairs (winning traces / revised answers), not fix commentary. |
 | - | `.scrub_preferences` / `.preference_balance` / `.generator_mix` | Ledger hygiene and source-mix health so one channel cannot flood preference export. |
@@ -54,9 +80,9 @@ every turn.
 
 | ID | Where | What |
 |----|-------|------|
-| **C1** | `Registry.rank` + `Metrics.{ucb,thompson,advantage,prm_advantage}` | Live tool choice blends keyword fit, historical advantage, exploration bonus, and process-reward signal. |
+| **C1** | `Registry.rank` + `Metrics.{ucb,thompson,advantage,prm_advantage}` + `Policy.advantage` | Live tool choice blends keyword fit, historical advantage, exploration bonus, process-reward signal, and (once the episode budget is met) Q(s,a)-V(s). Neighbor states fill cold (s,a) pairs. |
 | **C2** | `Learning.exemplars_for` | Prior successful traces ranked by judge score, recency, and keyword fit. Low-score "proxy success" rows are dropped when judge distrust is high. |
-| **C3** | `Curriculum.hindsight` | On a failed goal, relabel what the trajectory *did* achieve (`success: 'soft'`). Soft rows stay out of hard SFT and are down-weighted in exemplars. |
+| **C3** | `Curriculum.hindsight` | On a failed goal, relabel what the trajectory *did* achieve (`success: 'soft'`). Soft rows stay out of hard supervised export and are down-weighted in exemplars. |
 | **C4** | `Learning.compress_exemplar` / skill build | Keep steps with positive `step_reward` so few-shot traces stay short. |
 
 ## Memory that stays high-signal
@@ -77,9 +103,9 @@ every turn.
 | **S3** | `.critic` | Constitutional critic of the final answer (can use tools). Under budget pressure, runs text-only so it cannot burn the remaining iterations. |
 | **S4** | `.red_team_plan` | Adversarial review of the plan-first outline using Metrics / Mistakes / drift. |
 | - | `.offline_judge` | Score recent sessions with outcome + process judges, warm the sentinel, optional ledger scrub. Meant for nightly cron on local hosts that only introspect failures live. |
-| **C3** | `.hindsight` | HER relabel described above. |
+| **C3** | `.hindsight` | Relabel described above. |
 | **W3** | `.calibrate` | Plan `p(success)=` vs actual outcome → per-engine Brier / overconfidence. Overconfidence can force plan_first + critic and tighten `max_iters`. |
-| **W2** | `.train_and_gate` | Export SFT + DPO → optional LoRA train → promote only if the candidate wins on resolved margin, mean judge, smoke set, and a healthy preference diet. Without a trainer: `weight_loop: :export_ready`. |
+| **W2** | `.train_and_gate` | Export supervised + preference data → optional LoRA train → promote only if the candidate wins on resolved margin, mean judge, smoke set, and a healthy preference diet. Without a trainer: `weight_loop: :export_ready`. |
 | - | `.practice_kpi` | Week-over-week trend of repeating mistakes (outer curriculum health). |
 
 ## Budget pressure (iteration ceiling)
@@ -88,7 +114,7 @@ When unresolved `agent_loop` / `assistant_answer` budget-exhaustion fingerprints
 dominate, the loop marks the budget path hot and tightens the live turn:
 
 - lower effective `max_iters` (stricter on local/ollama engines than remote)
-- **Last-iter force-final**: tools=nil on the final iteration so a text answer is required
+- last-iter force-final: tools=nil on the final iteration so a text answer is required
 - skip counterfactual / red-team forks that would spend more tool rounds
 - still flush TaskSummarizer state and Learning on the exhaust path
 - end-of-turn critic runs text-only under the same pressure
@@ -96,27 +122,50 @@ dominate, the loop marks the budget path hot and tightens the live turn:
 Practice prioritizes those scars with short-horizon "finish the task" prompts.
 Raising `ai.agent.max_iters` or resolving the scar returns normal runway.
 
+## Design-priority STATUS (post P14-P25)
+
+This table is the flag authority. Stop minting new P-numbers for chores already
+covered. Track these outcomes instead of hunting comments in the source.
+
+| Pri | ID | Control | Module(s) | Success criterion |
+|-----|----|---------|-----------|-------------------|
+| **P0** | W1 generator diversity | `Reward::TARGET_SOURCE_MIX` + `generator_mix` + mix-urgent force on critic/counterfactual | `reward.rb`, `curriculum.rb` | `generator_mix.healthy` OR `recommendation` not stuck on `suppress:mistakes_resolve`; trajectory_fraction ≥ 0.5 |
+| **P0** | Introspect budget | `Learning::INTROSPECT_SOFT_MS` / `HARD_MS`; stage skip under soft/hard / `budget_exhaustion_hot?` | `learning.rb` | `auto_introspect` returns `stages_skipped` when over soft; post-answer path cannot re-thrash tool critic |
+| **P1** | Local judge calibration | Heuristic score shrinkage + `confidence`; `Metrics.effective_rate` scales distrust by `judge_confidence` | `reward.rb`, `metrics.rb` | distrust × heuristic no longer fully replaces proxy; local no-trace highs capped |
+| **ops** | Cheap LLM ORM | Direct engine `.chat` for `Reward.judge` / `.prm`; optional `reward_model` / `reward_llm_timeout`; evidence-prior fallback last | `reward.rb` | remote turns grade with ORM even when `module_reflection` is off |
+| **ops** | Outcome-signal haircut | `judge_sample_weight` (ORM 1.0, heuristic 0.25) in sentinel / `Learning.stats` / `Metrics.judge_rate` | `reward.rb`, `learning.rb`, `metrics.rb` | proxy distrust blends toward ORM scores, not bag-of-words overlap |
+| **P1** | Practice outer KPI | `Curriculum.practice_kpi` / `repeating_trend` → `~/.pwn/curriculum_kpi.jsonl` | `curriculum.rb` | week-over-week `delta_repeating` ≤ 0 on budget fingerprints after practice nights |
+| **P2** | PRM sample efficiency | `PRM_MIN_N=5`, shrinkage to `PRM_FULL_N=20`, fleet coverage gate in `Registry.rank` | `metrics.rb`, `registry.rb` | `prm_advantage=0` until n≥5; rank delta=0 until ≥3 tools ready |
+| **P2** | STATUS over flag archaeology | This table | docs | New work cites Pri/ID here, not fresh P26+ comments for the same theme |
+| **ops** | Nightly diet close | `offline_judge` → `scrub_preferences` + `generator_mix` + `practice_kpi` | `curriculum.rb` | Cron path returns `scrub`/`generator_mix`/`practice_kpi`; raw resolve prose does not survive the night |
+| **ops** | Shape backfill | `Reward.infer_shape` + scrub rewrite | `reward.rb` | Legacy shapeless rows get `winning_trace`/`revised_answer` when content warrants; traj_f measurable |
+| **ops** | Mix in prompt | `Metrics.to_context` emits `W1 MIX:` when unhealthy | `metrics.rb` | Unhealthy diet visible every turn without a tool call |
+| **P0** | Budget exhaust deepen | Last-iter force-final (tools=nil); skip CF when `budget_exhaustion_hot?`; tighter caps 24 local / 75 remote; exhaust path `append_session`+`auto_introspect` | `loop.rb` | Exhaust returns a judged final, not a bare string; CF cannot re-enter under hot; last iter cannot tool-call |
+
 ## Intro and extro join
 
-| Where | What |
-|-------|------|
-| `Metrics.changepoints` + `Loop.attribute_cause` (**E1**) | Env-drift-attributed failures get `cause: :env_drift` and do not inflate `[REPEATING]`. |
-| `Extrospection.correlate` (**E2**) | Lead-lag style joins ("tool X started failing after toolchain Y changed"). |
-| `Reward.verify_as_reward` (**E3**) | Browser-backed claim checks can floor/cap the outcome score. |
+| ID | Where | What |
+|----|-------|------|
+| **E1** | `Metrics.changepoints` + `Loop.attribute_cause` | Env-drift-attributed failures get `cause: :env_drift` and do not inflate `[REPEATING]`. |
+| **E2** | `Extrospection.correlate` | Lead-lag style joins ("tool X started failing after toolchain Y changed"). |
+| **E3** | `Reward.verify_as_reward` | Browser-backed claim checks can floor/cap the outcome score. |
 
 ## Config (`PWN::Env[:ai][:agent]`)
 
 ```yaml
 :ai:
-  :module_reflection: false  # gates Reflect lesson writing (not ORM alone)
+  :module_reflection: false  # gates Reflect lesson writing (not the judge alone)
   :agent:
     :critic: null            # S3 - nil = ON for remote engines, OFF for ollama
-    :red_team_plan: null     # S4 - same auto policy
-    :counterfactual: null    # S2 - same auto policy
+    :red_team_plan: null     # S4 - same auto rule
+    :counterfactual: null    # S2 - same auto rule
     :hindsight: true         # C3 - hindsight relabel on failed turns (default true)
+    :policy: true            # R5 - live tabular Q / REINFORCE (advisory rank only)
     :verify_as_reward: null  # E3 - nil = auto sample on claim-shaped answers
     :reward_llm: null        # nil = outcome/process judges use LLM teacher on remote
-    :local_introspect: :failure_only   # ollama cost policy; remote always introspects
+    :reward_model: null      # optional cheaper model id for ORM/PRM (nil = engine default)
+    :reward_llm_timeout: 12  # cheap ORM chat timeout seconds (clamped 2..30)
+    :local_introspect: :failure_only   # ollama cost rule; remote always introspects
     :introspect_every_n: 3
     :max_iters: 25           # hard cap; budget pressure may lower effective value
 ```
@@ -141,17 +190,18 @@ PWN::Cron.install_defaults
 `reward_preferences` · `reward_scrub_preferences` · `reward_preference_balance` ·
 `reward_export_dpo` · `reward_generator_mix` · `curriculum_practice` ·
 `curriculum_train` · `curriculum_hindsight` · `curriculum_offline_judge` ·
-`curriculum_preference_balance` · `curriculum_practice_kpi` ·
-`learning_purge_noise`
+`curriculum_preference_balance` · `curriculum_practice_kpi` · `policy_stats` ·
+`policy_evaluate` · `policy_recommend` · `learning_purge_noise`
 
-## Design claims
+## What this stack actually does
 
 1. Process reward on real security tool traces, not only math demos (**R2**).
-2. Automatic blame attribution: self vs environment drift (**E1** + **E2**).
+2. Automatic blame: self vs environment drift (**E1** + **E2**).
 3. Reward-hacking self-detection when proxy success diverges from the judge (**R3**).
 4. Mistake-driven curriculum with regression-gated LoRA promotion when a trainer exists (**S1** + **W2**).
 5. Preference pairs from normal agent work (corrections, resolve, critic, practice) with no separate human labelling queue (**W1**).
 6. Export and promote only when the preference diet is diverse and trajectory-shaped.
+7. Live tabular Q-learning + REINFORCE on real Loop turns, used only as advice next to planning (**R5**).
 
 ## Preference signal quality
 
@@ -166,39 +216,6 @@ keep it honest by:
 5. Requiring smoke checks and mean judge improvement before LoRA promote.
 6. Treating budget exhaustion as a first-class practice target so the agent learns to finish.
 
-## Design-priority STATUS
-
-Living checklist for the reinforced feedback loop. Cite the Pri/ID here instead
-of inventing new milestone labels for the same theme.
-
-| Pri | ID | Control | Module(s) | Success criterion |
-|-----|----|---------|-----------|-------------------|
-| **P0** | W1 generator diversity | `Reward::TARGET_SOURCE_MIX` + `generator_mix` + mix-urgent force on critic/counterfactual | `reward.rb`, `curriculum.rb` | `generator_mix.healthy` OR `recommendation` not stuck on `suppress:mistakes_resolve`; trajectory_fraction ≥ 0.5 |
-| **P0** | Introspect budget | `Learning::INTROSPECT_SOFT_MS` / `HARD_MS`; stage skip under soft/hard / budget-pressure mode | `learning.rb` | `auto_introspect` returns `stages_skipped` when over soft; post-answer path cannot re-thrash tool critic |
-| **P0** | Local judge calibration | Heuristic score shrinkage + `confidence`; `Metrics.effective_rate` scales distrust by `judge_confidence` | `reward.rb`, `metrics.rb` | distrust×heuristic no longer fully replaces proxy; local no-trace highs capped |
-| **P0** | Practice outer KPI | `Curriculum.practice_kpi` / `repeating_trend` → `~/.pwn/curriculum_kpi.jsonl` | `curriculum.rb` | week-over-week `delta_repeating` ≤ 0 on budget fingerprints after practice nights |
-| **P0** | PRM sample efficiency | `PRM_MIN_N=5`, shrinkage to `PRM_FULL_N=20`, fleet coverage gate in `Registry.rank` | `metrics.rb`, `registry.rb` | `prm_advantage=0` until n≥5; rank delta=0 until ≥3 tools ready |
-| **P0** | STATUS over flag archaeology | This table | docs | New work cites Pri/ID here, not fresh comments for the same theme |
-| **ops** | Nightly diet close | `offline_judge` → `scrub_preferences` + `generator_mix` + `practice_kpi` | `curriculum.rb` | Cron path returns `scrub`/`generator_mix`/`practice_kpi`; raw resolve prose does not survive the night |
-| **ops** | Shape backfill | `Reward.infer_shape` + scrub rewrite | `reward.rb` | Legacy shapeless rows get `winning_trace`/`revised_answer` when content warrants; traj_f measurable |
-| **ops** | Mix in prompt | `Metrics.to_context` emits `W1 MIX:` when unhealthy | `metrics.rb` | Unhealthy diet visible every turn without a tool call |
-| **P0** | Budget exhaust deepen | Last-iter force-final (tools=nil); skip CF under budget-pressure mode; hot caps 24 ollama / 75 remote; exhaust path `append_session`+`auto_introspect` | `loop.rb` | Exhaust returns a judged final, not a bare string; CF cannot re-enter under hot; last iter cannot tool-call |
-
-### Config additions
-
-```yaml
-:ai:
-  :agent:
-    # Introspect budget (ms wall-clock inside auto_introspect)
-    # INTROSPECT_SOFT_MS / HARD_MS are constants; override only via code/reload today.
-    :critic: null            # also force-on when generator_mix.urgent includes critic
-    :counterfactual: null    # also force-on when generator_mix.urgent includes counterfactual
-```
-
-### Tools (design-priority)
-
-`reward_generator_mix` · `curriculum_practice_kpi` (plus existing reward/curriculum set)
-
 ---
 
 **See also:** [pwn-ai Agent](pwn-ai-Agent.md) · [Mistakes](Mistakes.md) ·

data/documentation/Skills-Memory-Learning.md

@@ -1,4 +1,4 @@
-# Memory · Skills · Learning · Mistakes · Metrics - Introspection
+# Memory · Skills · Learning · Mistakes · Metrics · Policy - Introspection
 
 The **inward-facing** half of the pwn-ai feedback loop: how the agent measures
 its own performance, turns wins into permanent capability, and - critically -
@@ -6,12 +6,13 @@ its own performance, turns wins into permanent capability, and - critically -
 
 ![Memory / Skills detail](diagrams/memory-skills-detailed.svg)
 
-## The five stores
+## The six stores
 
 | Store | File | Write tool | Read tool | Injected as |
 |---|---|---|---|---|
 | **Memory** | `memory.json` (+ `memory.idx`) | `memory_remember` | `memory_recall` · `PWN::MemoryIndex.recall_semantic` | `MEMORY` block - durable facts / prefs / lessons / env. **Relevance-ranked** for the current request via a local embedding index when `ai.ollama.embed_model` is available; falls back to newest-first otherwise. |
 | **Skills** | `skills/<name>/SKILL.md` | `skill_create` · `skill_migrate_legacy` · `learning_distill_skill` | `skill_list` · `skill_view` | `SKILLS` list - reusable procedures + `references:` (CWE/CVE/ATT&CK/NIST/URL) |
+| **Policy** | `policy.json` + `policy_traj.jsonl` | Loop `begin_episode` / `observe_step` / `finish` | `policy_stats` · `policy_evaluate` · `policy_recommend` | `POLICY` block - live tabular Q / REINFORCE. Advisory rank only. Disable with `ai.agent.policy: false`. |
 | **Learning** | `learning.jsonl` | `learning_note_outcome` · `learning_reflect` | `learning_outcomes` · `learning_stats` · `Learning.exemplars_for` | `LEARNING` block - recent outcomes + success_rate. Prior *successful* traces are also spliced in as **few-shot exemplars** for local models. |
 | **Mistakes** | `mistakes.json` | `mistakes_record` · `mistakes_resolve` · *auto on failure* | `mistakes_list` | `KNOWN MISTAKES` + `KNOWN FIXES` blocks - do-NOT-repeat + do-THIS-instead |
 | **Metrics** | `metrics.json` | *automatic* (every Dispatch) | `metrics_summary` | `TOOL EFFECTIVENESS` block - steer tool choice. **Segmented per engine** (`engine=...`) so a local model's telemetry never blends with a frontier model's. |
@@ -33,6 +34,7 @@ wiping durable facts, preferences, and lessons. Clearing is still available via
      abstract lessons for a small model.
 3. (local model) Loop.plan_first forces a numbered tool plan BEFORE dispatch.
 4. Dispatch runs a tool              → Metrics.record(tool, ok?, ms, engine:)
+   ↳ Policy.observe_step              → hygiene reward (semantic_ok) into the live MDP episode
    ↳ tool FAILED?                    → Mistakes.record(tool, error)  (count++, cross-session)
    ↳ same sig ≥3×?                   → guard_repeated_failure + inline correction_hint
    ↳ (local) ≥ ESCALATE_AFTER_FAILS  → Swarm.ask(escalation_persona) → 3-line frontier hint
@@ -40,6 +42,7 @@ wiping durable facts, preferences, and lessons. Clearing is still available via
    ↳ extro_verify → :refuted         → Mistakes.record(tool:'assumption', ...)  # proactive
    ↳ extro_verify → :confirmed       → observe(:intel, ttl:30d)
 6. Final answer produced             → Learning.auto_introspect(session_id)
+   ↳ Reward.judge (cheap LLM ORM)    → Policy.finish (terminal reward; Q + REINFORCE update)
    ↳ (local) fact_check_local_final  → auto extro_verify every CVE/version claim in the answer
    ↳ if auto_extrospect enabled      → Extrospection.auto_extrospect  # AUTO_SECTIONS only
 7. Reflect.on(engine: reflect_engine)→ Memory.remember(lesson_xxxx, ...)   # teacher-student: a
@@ -51,7 +54,7 @@ wiping durable facts, preferences, and lessons. Clearing is still available via
      model via Curriculum.train_and_gate (resolved margin + mean judge + smoke) - the ONLY step that
      changes weights, not just the scaffold. Without a trainer this stays export-ready.
 11. Next launch: PromptBuilder injects the budgeted blocks → the model already knows:
-     MEMORY · SKILLS · LEARNING · KNOWN MISTAKES/FIXES · TOOL EFFECTIVENESS · EXTROSPECTION
+     MEMORY · SKILLS · LEARNING · KNOWN MISTAKES/FIXES · TOOL EFFECTIVENESS · POLICY · EXTROSPECTION · RECENT TURNS
 ```
 
 `extro_correlate` is the **join** - it tells the agent whether a failure was
@@ -190,6 +193,7 @@ when deciding which side of the loop to exercise.
 | "Reflect on this session / extract lessons" | `learning_reflect` / `sessions_current` |
 | "Don't do that again / that was wrong / resolve..." | `mistakes_record` / `mistakes_resolve` / `mistakes_list` |
 | "Which tools are unhealthy / avg duration" | `metrics_summary` |
+| "What did the live policy learn / suggest a tool" | `policy_stats` / `policy_evaluate` / `policy_recommend` |
 | "What did we run in session X / active session" | `sessions_view` / `sessions_current` |
 | "Disable reflection while we fuzz" | `learning_auto_introspect_toggle` |
 
@@ -203,8 +207,28 @@ and which procedures to promote permanently.
 [Sessions](Sessions.md) · [Persistence](Persistence.md)
 
 
+## Live Policy (advisory Q / REINFORCE)
+
+When `ai.agent.policy` is on (the default), every Loop turn is one learning
+episode:
+
+1. `begin_episode` opens before the first tool rank.
+2. `observe_step` writes a hygiene reward after each tool (`semantic_ok`).
+3. `finish` (from `Learning.auto_introspect`, or Loop if introspect is skipped)
+   adds the `Reward.judge` terminal score and updates Q and REINFORCE.
+
+State is a short key: request kind, a hash of the active English task, last
+action, fail count bin, and engine. Files:
+
+- `~/.pwn/policy.json` - Q table, REINFORCE logits, visit counts, recent returns
+- `~/.pwn/policy_traj.jsonl` - episode log
+
+Tools `policy_stats`, `policy_evaluate`, and `policy_recommend` are read-only. Reset is Ruby-only (`PWN::AI::Agent::Policy.reset`) so a tool call cannot wipe the table. `Registry.rank` may add a small Q-advantage after a pair has been visited at least twice. Planning still owns the task list.
+
+Turn it off with `ai.agent.policy: false` in `~/.pwn/pwn.yaml`.
+
 ## Task briefs vs learning
 
-`TaskSummarizer` is **UX**, not a persistence layer. Plan/`about_to` lines are ephemeral TUI briefs (deduped by `last_brief_fp`). Durable learning still flows through Learning · Mistakes · Reward · sessions only. See [pwn-ai Agent § Task summaries](pwn-ai-Agent.md#task-summaries-long-autonomous-turns).
+`TaskSummarizer` is **UX**, not a persistence layer. Plan/`about_to` lines are ephemeral TUI briefs (deduped by `last_brief_fp`). Durable learning still flows through Learning · Mistakes · Reward · Policy · sessions. See [pwn-ai Agent § Task summaries](pwn-ai-Agent.md#task-summaries-long-autonomous-turns).
 
 [← Home](Home.md)

data/documentation/What-is-PWN.md

@@ -20,7 +20,7 @@ with a **tool-calling AI agent** on top that can run the same methods.
 | `PWN::FFI::*` | **8** | Native DSP/RF backends: Volk · Liquid · FFTW · RTLSdr · HackRF · AdalmPluto · SoapySDR · Stdio |
 | `PWN::AI::*` | **6** engines | OpenAI, Anthropic, Grok (OAuth device-flow), Gemini, Ollama, Open WebUI |
 | `bin/pwn_*` | **53** | Headless CLI drivers for CI/CD |
-| Agent toolsets | **12** · **82 tools** | terminal · pwn · memory · skills · sessions · learning · metrics · extrospection · cron · swarm · reward · curriculum |
+| Agent toolsets | **13** · **85 tools** | terminal · pwn · memory · skills · sessions · learning · metrics · policy · extrospection · cron · swarm · reward · curriculum |
 
 ## Three ways to use it
 
@@ -36,12 +36,13 @@ with a **tool-calling AI agent** on top that can run the same methods.
 - **Everything is Ruby, everything is a method.** No YAML DSLs, no black-box
   plugins. If you can call it in the REPL, the AI agent can call it, a driver
   can call it, and a cron job can call it.
-- **Closed feedback loop.** Metrics + Learning + **Reward** (ORM/PRM judge,
-  sentinel) + **Curriculum** (mistake-driven self-play, HER, regression-gated
-  LoRA) on the introspection side; Snapshot + Drift + Intel + Verify on the
-  extrospection side; joined by `extro_correlate`, which tells the agent
-  whether a failure was *its* fault or *the world* changed - and writes the
-  lesson back into the next prompt.
+- **Closed feedback loop.** Metrics + Learning + **Reward** (outcome and process
+  judges, sentinel) + **Curriculum** (mistake-driven self-play, hindsight relabel,
+  export-ready LoRA gate) + **Policy** (live tabular Q-learning and REINFORCE on
+  each turn, advisory rank only) on the introspection side; Snapshot + Drift +
+  Intel + Verify on the extrospection side; joined by `extro_correlate`, which
+  tells the agent whether a failure was *its* fault or *the world* changed - and
+  writes the lesson back into the next prompt.
 - **Native multi-agent.** `PWN::AI::Agent::Swarm` runs personas (each a full
   tool-calling agent, optionally on a *different* LLM engine) that debate,
   broadcast, and share an append-only bus - no IRC daemon, no external service.

data/documentation/Why-PWN.md

@@ -39,8 +39,9 @@ critical when the caller is an autonomous agent.
 
 A pentest framework that does not learn repeats the same dead-end scans
 forever. PWN records **per-tool success rate**, **per-task outcome**, **host
-drift**, and **external CVE intel**, then correlates them so tomorrow's run
-can start where today's left off. See
+drift**, **external CVE intel**, and a **live Policy** table (Q / REINFORCE
+advice next to planning), then correlates them so tomorrow's run can start
+where today's left off. See
 [Skills, Memory & Learning](Skills-Memory-Learning.md),
 [Reinforcement Learning](Reinforcement-Learning.md), and
 [Extrospection](Extrospection.md).