pwn 0.5.683 → 0.5.685

data/documentation/Reinforcement-Learning.md

@@ -5,10 +5,10 @@ agent writes what happened to disk and puts the useful bits back into the next
 prompt. On a host **with a trainer and a GPU**, the same data can also train a
 local adapter.
 
-`Curriculum.practice` → `Reward.export_dpo` → `Curriculum.train_and_gate`
+`Curriculum.practice` -> `Reward.export_dpo` -> `Curriculum.train_and_gate`
 
-That path can promote a new LoRA when the candidate beats the current one.
-Without a trainer it still **exports** the datasets and a manual CLI. Live
+That path can promote a new local adapter when the candidate beats the current
+one. Without a trainer it still **exports** the datasets and a manual CLI. Live
 improvement does not wait on weights.
 
 ![Reinforcement-learning loop](diagrams/reinforcement-learning.svg)
@@ -16,32 +16,32 @@ improvement does not wait on weights.
 ```
                        +------------------------------------------------+
         request -----> | Loop.run                                       |
-                       |  plan_first -> Curriculum.red_team_plan   (S4) |
-                       |  Dispatch   -> Reward.semantic_ok         (R4) |
-                       |             -> Mistakes.record(cause:)    (E1) |
-                       |  guard      -> Curriculum.counterfactual  (S2) |--> preference ledger (W1)
-                       |  final      -> Curriculum.critic          (S3) |
-                       |             -> Reward.judge (outcome)     (R1) |--> verify_as_reward (E3)
-                       |             -> Reward.prm (process)       (R2) |--> Sessions[step_reward] (C4)
-                       |             -> Curriculum.hindsight       (C3) |
-                       |             -> Curriculum.calibrate       (W3) |--> Metrics.calibration
-                       |             -> Reward.sentinel            (R3) |--> Mistakes(reward_signal)
+                       |  plan_first -> Curriculum.red_team_plan        |
+                       |  Dispatch   -> Reward.semantic_ok              |
+                       |             -> Mistakes.record(cause:)         |
+                       |  guard      -> Curriculum.counterfactual       |--> preference ledger
+                       |  final      -> Curriculum.critic               |
+                       |             -> Reward.judge (outcome)          |--> verify_as_reward
+                       |             -> Reward.prm (process)            |--> Sessions[step_reward]
+                       |             -> Curriculum.hindsight            |
+                       |             -> Curriculum.calibrate            |--> Metrics.calibration
+                       |             -> Reward.sentinel                 |--> Mistakes(reward_signal)
                        +------------------------------------------------+
                                            |
-                    Learning.consolidate (M1 merge, M3 importance-evict)
-                    MemoryIndex.recall_semantic (M2 similarity x recency x importance)
-                    Registry.rank (C1 keyword + UCB + Q-advantage + tool_preference)
-                    Policy (R5 live Q / REINFORCE on a judge-scored MDP)
-                    Learning.exemplars_for (C2 prioritized replay, C4 minimal trace)
+                    Learning.consolidate (merge + importance-evict)
+                    MemoryIndex.recall_semantic (similarity x recency x importance)
+                    Registry.rank (keyword + UCB + Q-advantage + tool_preference)
+                    Policy (live Q / REINFORCE on a judge-scored MDP)
+                    Learning.exemplars_for (prioritized replay, short traces)
                                            |
-              nightly cron --> Curriculum.practice (S1) --> Mistakes.resolve --> preference (W1)
-              weekly  cron --> Curriculum.train_and_gate (W2) --> optional LoRA --> A/B gate --> promote
+              nightly cron --> Curriculum.practice --> Mistakes.resolve --> preference
+              weekly  cron --> Curriculum.train_and_gate --> optional adapter --> A/B gate --> promote
                                            |
-                    Extrospection.correlate (E2 world vs self)
-                    Metrics.changepoints (E1 CUSUM) --> Mistakes(cause: :env_drift)
+                    Extrospection.correlate (world vs self)
+                    Metrics.changepoints (CUSUM) --> Mistakes(cause: :env_drift)
 ```
 
-## Live Policy on every turn (R5 · `PWN::AI::Agent::Policy`)
+## Live Policy on every turn (`PWN::AI::Agent::Policy`)
 
 This is the live numeric controller. It does not replace planning.
 
@@ -65,89 +65,91 @@ high-return / high-score episodes.
 
 ## Reward signal (`PWN::AI::Agent::Reward`)
 
-| ID | Method | What it does |
-|----|--------|--------------|
-| **R1** | `.judge` | Cheap LLM outcome score on `(request, final)` → `{score:0..1, verdict:, rationale:, key_step:, source:}`. Calls the active engine `.chat` with a short timeout (default 12s). `Reflect.on` is used only when `module_reflection` is on. Fallback scores completeness, plan cover, claims, and tool-trace echo. Token overlap is only a small on-topic gate. |
-| **R2** | `.prm` | Process reward - per-tool-step `+1/0/-1` written into `Sessions[:step_reward]`. |
-| **R3** | `.sentinel` | Compares proxy success rate vs judge mean vs user-correction rate. A large gap fingerprints `reward_signal` so the agent distrusts a lying proxy. |
-| **R4** | `.semantic_ok` | Treats informational non-zero exits (for example `grep` / `rg` with no match) as benign. Metrics count them as OK; Mistakes only see true dispatch failures. |
-| **R5** | `Policy` (live MDP) | Tabular Q-learning + REINFORCE on real Loop turns. Q-advantage is an advisory `Registry.rank` term and never replaces TaskSummarizer or plan_first. |
-| - | `.warm_sentinel` | Backfills the sentinel window from scored Learning outcomes so local hosts can engage proxy distrust without waiting for live remote introspect. |
-| **W1** | `.record_preference` / `.export_dpo` | Preference ledger (`~/.pwn/preferences.jsonl`) from user corrections, resolve, counterfactual, critic, and practice. Caps per source; keeps trajectory-shaped pairs (winning traces / revised answers), not fix commentary. |
-| - | `.scrub_preferences` / `.preference_balance` / `.generator_mix` | Ledger hygiene and source-mix health so one channel cannot flood preference export. |
+| Method | What it does |
+|--------|--------------|
+| **R1** `.judge` | Cheap LLM outcome score on `(request, final)` -> `{score:0..1, verdict:, rationale:, key_step:, source:}`. Calls the active engine `.chat` with a short timeout (default 12s). `Reflect.on` is used only when `module_reflection` is on. Fallback scores completeness, plan cover, claims, and tool-trace echo. Token overlap is only a small on-topic gate. |
+| **R2** `.prm` | Process reward - per-tool-step `+1/0/-1` written into `Sessions[:step_reward]`. |
+| **R3** `.sentinel` | Compares proxy success rate vs judge mean vs user-correction rate. A large gap fingerprints `reward_signal` so the agent distrusts a lying proxy. |
+| **R4** `.semantic_ok` | Treats informational non-zero exits (for example `grep` / `rg` with no match) as benign. Metrics count them as OK; Mistakes only see true dispatch failures. |
+| **R5** `Policy` (live MDP) | Tabular Q-learning + REINFORCE on real Loop turns. Q-advantage is an advisory `Registry.rank` term and never replaces TaskSummarizer or plan_first. |
+| `.warm_sentinel` | Backfills the sentinel window from scored Learning outcomes so local hosts can engage proxy distrust without waiting for live remote introspect. |
+| `.record_preference` / `.export_dpo` | Preference ledger (`~/.pwn/preferences.jsonl`) from user corrections, resolve, counterfactual, critic, and practice. Caps per source; keeps trajectory-shaped pairs (winning traces / revised answers), not fix commentary. |
+| `.scrub_preferences` / `.preference_balance` / `.generator_mix` | Ledger hygiene and source-mix health so one channel cannot flood preference export. |
 
 ## Credit assignment and replay
 
-| ID | Where | What |
-|----|-------|------|
-| **C1** | `Registry.rank` + `Metrics.{ucb,thompson,advantage,prm_advantage}` + `Policy.advantage` | Live tool choice blends keyword fit, historical advantage, exploration bonus, process-reward signal, and (once the episode budget is met) Q(s,a)-V(s). Neighbor states fill cold (s,a) pairs. |
-| **C2** | `Learning.exemplars_for` | Prior successful traces ranked by judge score, recency, and keyword fit. Low-score "proxy success" rows are dropped when judge distrust is high. |
-| **C3** | `Curriculum.hindsight` | On a failed goal, relabel what the trajectory *did* achieve (`success: 'soft'`). Soft rows stay out of hard supervised export and are down-weighted in exemplars. |
-| **C4** | `Learning.compress_exemplar` / skill build | Keep steps with positive `step_reward` so few-shot traces stay short. |
+| Where | What |
+|-------|------|
+| **C1** `Registry.rank` + `Metrics.{ucb,thompson,advantage,prm_advantage}` + `Policy.advantage` | Live tool choice blends keyword fit, historical advantage, exploration bonus, process-reward signal, and (once the episode budget is met) Q(s,a)-V(s). Neighbor states fill cold (s,a) pairs. |
+| **C2** `Learning.exemplars_for` | Prior successful traces ranked by judge score, recency, and keyword fit. Low-score "proxy success" rows are dropped when judge distrust is high. |
+| **C3** `Curriculum.hindsight` | On a failed goal, relabel what the trajectory *did* achieve (`success: 'soft'`). Soft rows stay out of hard supervised export and are down-weighted in exemplars. |
+| **C4** `Learning.compress_exemplar` / skill build | Keep steps with positive `step_reward` so few-shot traces stay short. |
 
 ## Memory that stays high-signal
 
-| ID | Where | What |
-|----|-------|------|
-| **M1** | `Learning.consolidate` | Semantic merge of near-duplicate lessons; importance-weighted eviction. |
-| **M2** | `MemoryIndex.recall_semantic` | Rank MEMORY by similarity × recency × importance when embeddings are available. |
-| **M3** | `Memory.remember` | Supports source, confidence, importance, and TTL so garbage self-evicts. |
-| **M4** | `Learning.note_outcome` | Task outcomes go to `learning.jsonl` only. Memory `:lesson` is reserved for reflect, resolve, and human notes. `purge_noise` cleans older noisy lesson shapes. |
+| Where | What |
+|-------|------|
+| **M1** `Learning.consolidate` | Semantic merge of near-duplicate lessons; importance-weighted eviction. |
+| **M2** `MemoryIndex.recall_semantic` | Rank MEMORY by similarity x recency x importance when embeddings are available. |
+| **M3** `Memory.remember` | Supports source, confidence, importance, and TTL so garbage self-evicts. |
+| **M4** `Learning.note_outcome` | Task outcomes go to `learning.jsonl` only. Memory `:lesson` is reserved for reflect, resolve, and human notes. `purge_noise` cleans older noisy lesson shapes. |
 
 ## Curriculum and self-play (`PWN::AI::Agent::Curriculum`)
 
-| ID | Method | What |
-|----|--------|------|
-| **S1** | `.practice` | Mine top unresolved Mistakes → natural reproducers → self-play under the judge → auto-resolve on strong holdouts. Budget-exhaustion scars get "finish under N steps" prompts. |
-| **S2** | `.counterfactual` | On repeated failure, fork an alt-persona branch, judge both, emit a preference pair. Skipped when the iteration budget is already under pressure. |
-| **S3** | `.critic` | Constitutional critic of the final answer (can use tools). Under budget pressure, runs text-only so it cannot burn the remaining iterations. |
-| **S4** | `.red_team_plan` | Adversarial review of the plan-first outline using Metrics / Mistakes / drift. |
-| - | `.offline_judge` | Score recent sessions with outcome + process judges, warm the sentinel, optional ledger scrub. Meant for nightly cron on local hosts that only introspect failures live. |
-| **C3** | `.hindsight` | Relabel described above. |
-| **W3** | `.calibrate` | Plan `p(success)=` vs actual outcome → per-engine Brier / overconfidence. Overconfidence can force plan_first + critic and tighten `max_iters`. |
-| **W2** | `.train_and_gate` | Export supervised + preference data → optional LoRA train → promote only if the candidate wins on resolved margin, mean judge, smoke set, and a healthy preference diet. Without a trainer: `weight_loop: :export_ready`. |
-| - | `.practice_kpi` | Week-over-week trend of repeating mistakes (outer curriculum health). |
+| Method | What |
+|--------|------|
+| **S1** `.practice` | Mine top unresolved Mistakes -> natural reproducers -> self-play under the judge -> auto-resolve on strong holdouts. Budget-exhaustion scars get "finish under N steps" prompts. |
+| **S2** `.counterfactual` | On repeated failure, fork an alt-persona branch, judge both, emit a preference pair. Skipped when the iteration budget is already under pressure. |
+| **S3** `.critic` | Constitutional critic of the final answer (can use tools). Under budget pressure, runs text-only so it cannot burn the remaining iterations. |
+| **S4** `.red_team_plan` | Adversarial review of the plan-first outline using Metrics / Mistakes / drift. |
+| `.offline_judge` | Score recent sessions with outcome + process judges, warm the sentinel, optional ledger scrub. Meant for nightly cron on local hosts that only introspect failures live. |
+| `.hindsight` | Relabel described above. |
+| **W3** `.calibrate` | Plan `p(success)=` vs actual outcome -> per-engine Brier / overconfidence. Overconfidence can force plan_first + critic. It does **not** shrink this request's `max_iters`. |
+| **W2** `.train_and_gate` | Export supervised + preference data -> optional local adapter train -> promote only if the candidate wins on resolved margin, mean judge, smoke set, and a healthy preference diet. Without a trainer: `weight_loop: :export_ready`. |
+| **W1** `.record_preference` | Preference ledger from corrections, resolve, counterfactual, critic, and practice. |
+| `.practice_kpi` | Week-over-week trend of repeating mistakes (outer curriculum health). |
 
 ## Budget pressure (iteration ceiling)
 
 When unresolved `agent_loop` / `assistant_answer` budget-exhaustion fingerprints
-dominate, the loop marks the budget path hot and tightens the live turn:
+dominate, the loop marks the budget path hot and tightens *side work* only:
 
-- lower effective `max_iters` (stricter on local/ollama engines than remote)
-- last-iter force-final: tools=nil on the final iteration so a text answer is required
 - skip counterfactual / red-team forks that would spend more tool rounds
+- last-iter strips tools only when the original request is already satisfied
 - still flush TaskSummarizer state and Learning on the exhaust path
 - end-of-turn critic runs text-only under the same pressure
+- **do not shrink `max_iters`** - yesterday's scars must not abort this request
 
 Practice prioritizes those scars with short-horizon "finish the task" prompts.
-Raising `ai.agent.max_iters` or resolving the scar returns normal runway.
+Raising `ai.agent.max_iters` still sets the hard ceiling.
 
 ## Design-priority STATUS
 
-This table is the live control list. Track the outcomes, not source comments.
-
-| Pri | ID | Control | Module(s) | Success criterion |
-|-----|----|---------|-----------|-------------------|
-| **P0** | W1 generator diversity | `Reward::TARGET_SOURCE_MIX` + `generator_mix` + mix-urgent force on critic/counterfactual | `reward.rb`, `curriculum.rb` | `generator_mix.healthy` OR `recommendation` not stuck on `suppress:mistakes_resolve`; trajectory_fraction ≥ 0.5 |
-| **P0** | Introspect budget | `Learning::INTROSPECT_SOFT_MS` / `HARD_MS`; stage skip under soft/hard / `budget_exhaustion_hot?` | `learning.rb` | `auto_introspect` returns `stages_skipped` when over soft; post-answer path cannot re-thrash tool critic |
-| **P1** | Local judge calibration | Heuristic score shrinkage + `confidence`; `Metrics.effective_rate` scales distrust by `judge_confidence` | `reward.rb`, `metrics.rb` | distrust × heuristic no longer fully replaces proxy; local no-trace highs capped |
-| **ops** | Cheap LLM ORM | Direct engine `.chat` for `Reward.judge` / `.prm`; optional `reward_model` / `reward_llm_timeout`; evidence-prior fallback last | `reward.rb` | remote turns grade with ORM even when `module_reflection` is off |
-| **ops** | Outcome-signal haircut | `judge_sample_weight` (ORM 1.0, heuristic 0.25) in sentinel / `Learning.stats` / `Metrics.judge_rate` | `reward.rb`, `learning.rb`, `metrics.rb` | proxy distrust blends toward ORM scores, not bag-of-words overlap |
-| **P1** | Practice outer KPI | `Curriculum.practice_kpi` / `repeating_trend` → `~/.pwn/curriculum_kpi.jsonl` | `curriculum.rb` | week-over-week `delta_repeating` ≤ 0 on budget fingerprints after practice nights |
-| **P2** | PRM sample efficiency | `PRM_MIN_N=5`, shrinkage to `PRM_FULL_N=20`, fleet coverage gate in `Registry.rank` | `metrics.rb`, `registry.rb` | `prm_advantage=0` until n≥5; rank delta=0 until ≥3 tools ready |
-| **P2** | STATUS over flag archaeology | This table | docs | New work cites Pri/ID here, not fresh P26+ comments for the same theme |
-| **ops** | Nightly diet close | `offline_judge` → `scrub_preferences` + `generator_mix` + `practice_kpi` | `curriculum.rb` | Cron path returns `scrub`/`generator_mix`/`practice_kpi`; raw resolve prose does not survive the night |
-| **ops** | Shape backfill | `Reward.infer_shape` + scrub rewrite | `reward.rb` | Legacy shapeless rows get `winning_trace`/`revised_answer` when content warrants; traj_f measurable |
-| **ops** | Mix in prompt | `Metrics.to_context` emits `W1 MIX:` when unhealthy | `metrics.rb` | Unhealthy diet visible every turn without a tool call |
-| **P0** | Budget exhaust deepen | Last-iter force-final (tools=nil); skip CF when `budget_exhaustion_hot?`; tighter caps 24 local / 75 remote; exhaust path `append_session`+`auto_introspect` | `loop.rb` | Exhaust returns a judged final, not a bare string; CF cannot re-enter under hot; last iter cannot tool-call |
+This is the live control list. Track the outcomes, not source comments.
+
+`PRM_MIN_N` / `PRM_FULL_N` gate process-reward rank until enough samples exist.
+
+| Priority | Control | Success look |
+|----------|---------|--------------|
+| must | Preference-source diversity (`Reward::TARGET_SOURCE_MIX` + `generator_mix`) | Mix is healthy, or the recommendation is not stuck suppressing `mistakes_resolve`; trajectory-shaped pairs stay at least half the ledger |
+| must | Introspect budget (`Learning::INTROSPECT_SOFT_MS` / `HARD_MS`) | `auto_introspect` skips extra stages when over the soft cap so the post-answer path cannot re-thrash a tool critic |
+| high | Local judge calibration | Heuristic scores shrink by confidence; a local no-trace high does not fully replace the proxy |
+| ops | Cheap LLM outcome judge | Remote turns grade with the engine chat even when `module_reflection` is off; token overlap is last-resort |
+| ops | Outcome-signal haircut (`judge_sample_weight`) | Proxy distrust blends toward engine scores, not bag-of-words overlap |
+| high | Practice outer KPI (`Curriculum.practice_kpi`) | Week-over-week repeating-mistake trend on budget fingerprints is not rising after practice nights |
+| next | Process-reward sample efficiency | `prm_advantage` stays 0 until enough samples exist; rank does not swing on a tiny fleet |
+| ops | Nightly diet close | `offline_judge` then scrub + mix + KPI so raw resolve prose does not survive the night |
+| ops | Shape backfill (`Reward.infer_shape`) | Legacy shapeless rows become `winning_trace` / `revised_answer` when the content warrants |
+| ops | Mix in prompt | `Metrics.to_context` emits `MIX:` when preference sources are unhealthy |
+| must | Budget exhaust | Last-iter force-final only when the original request is already satisfied; skip extra forks when the budget path is hot; **do not shrink max_iters**; still flush the session + introspect |
 
 ## Intro and extro join
 
-| ID | Where | What |
-|----|-------|------|
-| **E1** | `Metrics.changepoints` + `Loop.attribute_cause` | Env-drift-attributed failures get `cause: :env_drift` and do not inflate `[REPEATING]`. |
-| **E2** | `Extrospection.correlate` | Lead-lag style joins ("tool X started failing after toolchain Y changed"). |
-| **E3** | `Reward.verify_as_reward` | Browser-backed claim checks can floor/cap the outcome score. |
+| Where | What |
+|-------|------|
+| **E1** `Metrics.changepoints` + `Loop.attribute_cause` | Env-drift-attributed failures get `cause: :env_drift` and do not inflate `[REPEATING]`. |
+| **E2** `Extrospection.correlate` | Lead-lag style joins ("tool X started failing after toolchain Y changed"). |
+| **E3** `Reward.verify_as_reward` | Browser-backed claim checks can floor/cap the outcome score. |
 
 ## Config (`PWN::Env[:ai][:agent]`)
 
@@ -155,18 +157,18 @@ This table is the live control list. Track the outcomes, not source comments.
 :ai:
   :module_reflection: false  # gates Reflect lesson writing (not the judge alone)
   :agent:
-    :critic: null            # S3 - nil = ON for remote engines, OFF for ollama
-    :red_team_plan: null     # S4 - same auto rule
-    :counterfactual: null    # S2 - same auto rule
-    :hindsight: true         # C3 - hindsight relabel on failed turns (default true)
-    :policy: true            # R5 - live tabular Q / REINFORCE (advisory rank only)
-    :verify_as_reward: null  # E3 - nil = auto sample on claim-shaped answers
+    :critic: null            # nil = ON for remote engines, OFF for ollama
+    :red_team_plan: null     # same auto rule
+    :counterfactual: null    # same auto rule
+    :hindsight: true         # hindsight relabel on failed turns (default true)
+    :policy: true            # live tabular Q / REINFORCE (advisory rank only)
+    :verify_as_reward: null  # nil = auto sample on claim-shaped answers
     :reward_llm: null        # nil = outcome/process judges use LLM teacher on remote
     :reward_model: null      # optional cheaper model id for ORM/PRM (nil = engine default)
     :reward_llm_timeout: 12  # cheap ORM chat timeout seconds (clamped 2..30)
     :local_introspect: :failure_only   # ollama cost rule; remote always introspects
     :introspect_every_n: 3
-    :max_iters: 75           # hard cap; budget pressure may lower effective value
+    :max_iters: 777          # hard cap for this request; scars do not lower it
     :defer_introspect: true  # post-answer Learning after the user-visible reply
     :prompt_cache: true      # engine-native prefix cache (not ollama / openwebui)
     :tool_preference: [memory_recall, session_recall, skills_recall, pwn_eval, shell, mistakes_record, mistakes_resolve, learning_note_outcome, memory_remember]
@@ -177,10 +179,10 @@ This table is the live control list. Track the outcomes, not source comments.
 ```ruby
 # Seeded idempotently by PWN::Cron.install_defaults (pwn setup --migrate):
 PWN::Cron.install_defaults
-# → curriculum_practice_nightly   0 3  * * *  Curriculum.practice(limit: 3)
-# → curriculum_offline_judge     30 3  * * *  Curriculum.offline_judge(since_hours: 24, limit: 40)
-# → curriculum_train_weekly       0 4  * * 0  Curriculum.train_and_gate(dry_run: true)  # false only with trainer+GPU
-# → learning_consolidate_nightly  0 5  * * *  Learning.consolidate
+# -> curriculum_practice_nightly   0 3  * * *  Curriculum.practice(limit: 3)
+# -> curriculum_offline_judge     30 3  * * *  Curriculum.offline_judge(since_hours: 24, limit: 40)
+# -> curriculum_train_weekly       0 4  * * 0  Curriculum.train_and_gate(dry_run: true)  # false only with trainer+GPU
+# -> learning_consolidate_nightly  0 5  * * *  Learning.consolidate
 ```
 
 `install_defaults` treats the legacy name `offline_judge_nightly` as an alias of
@@ -199,13 +201,13 @@ without a per-job crontab line.
 
 ## What this stack actually does
 
-1. Process reward on real security tool traces, not only math demos (**R2**).
-2. Automatic blame: self vs environment drift (**E1** + **E2**).
-3. Reward-hacking self-detection when proxy success diverges from the judge (**R3**).
-4. Mistake-driven curriculum with regression-gated LoRA promotion when a trainer exists (**S1** + **W2**).
-5. Preference pairs from normal agent work (corrections, resolve, critic, practice) with no separate human labelling queue (**W1**).
+1. Process reward on real security tool traces, not only math demos.
+2. Automatic blame: self vs environment drift.
+3. Reward-hacking self-detection when proxy success diverges from the judge.
+4. Mistake-driven curriculum with regression-gated adapter promotion when a trainer exists.
+5. Preference pairs from normal agent work (corrections, resolve, critic, practice) with no separate human labelling queue.
 6. Export and promote only when the preference diet is diverse and trajectory-shaped.
-7. Live tabular Q-learning + REINFORCE on real Loop turns, used only as advice next to planning (**R5**).
+7. Live tabular Q-learning + REINFORCE on real Loop turns, used only as advice next to planning.
 
 ## Preference signal quality
 
@@ -215,9 +217,9 @@ keep it honest by:
 
 1. Capping how much any one preference source can dominate the ledger.
 2. Preferring winning tool traces and revised full answers over fix commentary.
-3. Scrubbing historical prose-only pairs before DPO export.
+3. Scrubbing historical prose-only pairs before preference export.
 4. Warming the sentinel from offline scores so local hosts are not stuck cold.
-5. Requiring smoke checks and mean judge improvement before LoRA promote.
+5. Requiring smoke checks and mean judge improvement before adapter promote.
 6. Treating budget exhaustion as a first-class practice target so the agent learns to finish.
 
 ---

data/documentation/SAST.md

@@ -1,6 +1,6 @@
 # `PWN::SAST` - Static Application Security Testing
 
-48 language-aware rule modules + a `TestCaseEngine` + a `Factory` loader.
+48 modules in `PWN::SAST`: 45 language-aware scan rules plus `Factory`, `TestCaseEngine`, and `Version`.
 Source: `lib/pwn/sast/*.rb`. CLI: `bin/pwn_sast`.
 
 ![SAST pipeline](diagrams/code-scanning-sast.svg)

data/documentation/Session-Workflow.md

@@ -0,0 +1,132 @@
+# Session Workflow
+
+How a `pwn-ai` line is routed, when a goal is considered unfinished, and
+which phrases flip those switches. Exact matching lives in
+`PWN::AI::Agent::Loop` and `PWN::AI::Agent::OpenGoal`. This page is the
+operator cheat sheet.
+
+**See also:** [Sessions](Sessions.md) · [pwn-ai Agent](pwn-ai-Agent.md) ·
+[Persistence](Persistence.md)
+
+---
+
+## Life of a line
+
+1. Entering `pwn-ai` always creates a **new** `~/.pwn/sessions/<timestamp>_<hex>.jsonl`.
+2. Cheap intents (greeting, how-to, recall) never open a host-work goal.
+3. Everything else is an autonomous goal. `Loop.run` keeps CORE_TOOLS until
+   the original request is done or truly blocked.
+4. An unfinished host-work request is written to `~/.pwn/open_goal.json`.
+5. An accepted final answer deletes that file. Budget exhaust leaves it.
+
+```
+you type a line
+        |
+        +-- greeting / how-to / recall -----> short answer, no tools
+        |
+        +-- continue / resume / keep going --> reload open_goal.json
+        |
+        +-- last / previous session ---------> prior JSONL, not this line
+        |
+        +-- everything else -----------------> one Loop.run
+                                              write? then read it back
+                                              listing is not done
+```
+
+---
+
+## Unfinished goals
+
+File: `~/.pwn/open_goal.json` (`PWN::AI::Agent::OpenGoal`).
+
+| You type (whole line) | Effect |
+|---|---|
+| `continue` | Reload the saved request and keep working |
+| `resume` | same |
+| `keep going` | same |
+| `pick up` | same |
+| `carry on` | same |
+| `continue please` | same |
+| `resume the goal` / `resume the task` / `resume work` | same |
+
+The line must be **only** that phrase (optional `please` / `the goal|task|work`
+and a trailing `.` / `!`). A new sentence is a new goal and **replaces** the
+saved request.
+
+`continue scanning the lab` is a new ask, not a resume.
+
+Delete `~/.pwn/open_goal.json` to drop a stuck checkpoint.
+
+---
+
+## This session vs last session
+
+Entering `pwn-ai` is a new transcript. "Last session" is the newest other
+`~/.pwn/sessions/*.jsonl`.
+
+| Phrase | Where it reads |
+|---|---|
+| `what did I just say?` | **This** session |
+| `what did I just ask?` / `what was my last request?` | this session |
+| `how did you respond?` / `what did you just say?` | this session |
+| `what did I just say in the last session?` | **Previous** JSONL |
+| `in the last session` / `previous session` / `prior session` | previous JSONL |
+| `session_recall` (tool) | older transcripts, skips the current id |
+
+If there is no previous file: `I do not have a previous session transcript yet.`
+
+---
+
+## Cheap short-circuits (no tools)
+
+Whole-line greetings only, so `hi, scan this host` stays a goal:
+
+| Trigger | Examples |
+|---|---|
+| Greeting | `hi` `hello` `hey` `howdy` `yo` `good morning` |
+| How-to | `how to` `how do I` `how can I` `syntax for` `usage of` `man page` |
+| Recall | table above |
+
+---
+
+## Host work stays open until the artefact is checked
+
+| Need | Done only when |
+|---|---|
+| Write / update / regenerate / docs / fix | A **write** effect, then a later **read** (`cat`, `ruby -c`, eval read) |
+| Browser / navigate / `TransparentBrowser` | A **browse** effect (goto / dump_links / close) |
+| Hostname / uname / cwd / whoami | Any live **read** |
+| Other long goals (bounty, scrape, recon) | **write**, **browse**, or **eval** (`ls` alone is not enough) |
+
+These do **not** count as finishing a write: `memory_remember`,
+`learning_note_outcome`, `mistakes_*`, rspec/rubocop green, a README listing,
+or "I will do that next time."
+
+After you mutate a file, read it back before a final.
+
+---
+
+## Phrases that are **not** a final answer
+
+Loop keeps calling tools if the model emits any of:
+
+- `shall I` / `should I` / `want me to` / `proceed?` / `continue?`
+- `# Remaining block` / heading-only outlines
+- `were not applied` / `not written to disk` / `next time`
+- narrated next tool (`Wait, let's try...`) with no `tool_calls`
+
+Type `continue` yourself only to resume a **saved** open goal after the REPL
+died. Do not use it as a mid-turn "ok, go on" - the loop should already be
+going.
+
+---
+
+## Files
+
+| Path | Role |
+|---|---|
+| `~/.pwn/sessions/<id>.jsonl` | This activation's transcript |
+| `~/.pwn/open_goal.json` | Unfinished host-work request |
+| `~/.pwn/memory.json` | Durable facts (not "last line") |
+
+[← Home](Home.md)

data/documentation/Sessions.md

@@ -38,7 +38,8 @@ appended as JSON-per-line to `~/.pwn/sessions/<id>.jsonl`.
 {"role":"tool","ts":"...","name":"shell","content":"...","step_reward":1}
 ```
 
-**See also:** [Skills, Memory & Learning](Skills-Memory-Learning.md) ·
+**See also:** [Session Workflow](Session-Workflow.md) ·
+[Skills, Memory & Learning](Skills-Memory-Learning.md) ·
 [Swarm](Swarm.md) · [Persistence](Persistence.md)
 
 [← Home](Home.md)

data/documentation/Skills-Memory-Learning.md

@@ -103,6 +103,26 @@ them in-place to the spec-conformant `<name>/SKILL.md` layout with
 back-filled `name`/`description` front-matter. `skill_create` always writes
 the new format.
 
+A fresh install (first `pwn` launch or `pwn setup --migrate`) copies these
+bundled skills into `~/.pwn/skills/` when the name is missing:
+
+| Skill | For |
+|---|---|
+| `vulnerability-research-fundamentals` | First-pass research methodology with PWN plugins |
+| `deep-exploitation` | Crash / primitive to reliable PoC |
+| `bug-bounty-hunting` | Program scope, Burp, authz replay, report |
+| `sast-code-scans` | `PWN::SAST::Factory` / `pwn_sast` + reports |
+| `reverse-engineering-binaries` | checksec, disasm, `PWN::Plugins::Assembly` |
+| `penetration-testing` | PTES / NIST / OSSTMM / ISSAF engagement loop |
+| `web-application-penetration-testing` | OWASP WSTG / ASVS / API Top 10 |
+| `red-teaming` | MITRE ATT&CK, TIBER-EU, Kill Chain |
+| `hardware-and-firmware-testing` | OWASP FSTM / ISTG, serial, firmware |
+| `social-engineering` | OSSTMM human channel, ATT&CK phishing |
+| `osint` | `extro_osint` kinds, feeds, keys, pivots |
+
+Source: `etc/default_skills/` in the gem. Edits in `~/.pwn/skills` are never
+overwritten.
+
 ## Housekeeping
 
 | Tool | When |

data/documentation/Troubleshooting.md

@@ -138,7 +138,7 @@ list, then install equivalents by hand.
 
 **Cause:** older dual-emit paths or `about_to` restating the full goal every batch.
 
-**Current behavior (v0.5.660+):**
+**Current behavior (v0.5.684+):**
 
 1. `Loop.task_summary_about_to!` is the only about_to entry.
 2. `TaskSummarizer.about_to` builds distinct lines with `tool_counts_phrase` + `intent_phrase`.
@@ -149,6 +149,6 @@ If you still see doubles, confirm you are on a build with `lib/pwn/ai/agent/task
 
 ## Agent stops early / "iteration budget exhausted"
 
-When unresolved budget-exhaustion mistakes dominate, the Loop tightens effective `max_iters` (stricter on local/ollama than remote), and the last iterations are text-only so a final answer is forced without starving multi-step autonomy. Check `mistakes_list`, resolve fixed signatures, or raise `ai.agent.max_iters` after the scar cools. Exhaust paths still run Learning + task_summary flush.
+When unresolved budget-exhaustion mistakes dominate, the Loop does **not** shrink `max_iters` (default 777). It only trims side work: skip extra counterfactual forks, and strip tools on the last iteration only if the original request is already satisfied. Check `mistakes_list`, resolve fixed signatures, or raise `ai.agent.max_iters` if a long task still hits the cap. Exhaust paths still run Learning and the task-summary flush.
 
 [← Home](Home.md)

data/documentation/What-is-PWN.md

@@ -12,15 +12,15 @@ with a **tool-calling AI agent** on top that can run the same methods.
 
 | Namespace | Count | What it is |
 |---|---|---|
-| `PWN::Plugins::*` | **66** | Wrappers for external and native tooling (Burp, Nmap, Metasploit, Shodan, browsers, serial, ...) |
+| `PWN::Plugins::*` | **67** | Wrappers for external and native tooling (Burp, Nmap, Metasploit, Shodan, browsers, serial, ...) |
 | `PWN::SAST::*` | **48** | Static-analysis rules across C/Java/Go/Python/Ruby/Scala/PHP/TS |
 | `PWN::AWS::*` | **90** | One module per AWS service for cloud enumeration |
 | `PWN::WWW::*` | **22** | Site-specific browser automations (HackerOne, BugCrowd, GitHub, Google, LinkedIn, ...) |
 | `PWN::SDR::*` | **6** (+ **20** protocol decoders + Base/DSP) | GQRX, FlipperZero, RFIDler, SonMicro, band tables, `Decoder::{ADSB,POCSAG,RDS,LoRa,...}` |
 | `PWN::FFI::*` | **8** | Native DSP/RF backends: Volk · Liquid · FFTW · RTLSdr · HackRF · AdalmPluto · SoapySDR · Stdio |
 | `PWN::AI::*` | **6** engines | OpenAI, Anthropic, Grok (OAuth device-flow), Gemini, Ollama, Open WebUI |
-| `bin/pwn_*` | **53** | Headless CLI drivers for CI/CD |
-| Agent toolsets | **13** · **85 tools** | terminal · pwn · memory · skills · sessions · learning · metrics · policy · extrospection · cron · swarm · reward · curriculum |
+| `bin/pwn_*` + `pwn` | **54** | Headless CLI executables for CI/CD |
+| Agent toolsets | **13** · **87 tools** | terminal · pwn · memory · skills · sessions · learning · metrics · policy · extrospection · cron · swarm · reward · curriculum |
 
 ## Three ways to use it
 

data/documentation/Why-PWN.md

@@ -33,7 +33,7 @@ critical when the caller is an autonomous agent.
 | "Trust our scanner" | `cat lib/pwn/sast/sql.rb` - read the regex yourself |
 | Per-seat license for the glue | MIT-licensed glue; bring your own Burp Pro / Nessus key |
 | Agent output is a PDF | Agent output is a `PWN::Reports` object *and* a distilled skill *and* a memory entry |
-| One vendor's model | Five interchangeable engines; swarm can pit them against each other |
+| One vendor's model | Six interchangeable engines; swarm can pit them against each other |
 
 ## Why the feedback loop matters