pwn 0.5.638 → 0.5.642

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c889fade648cffc3bfe73e8a168b4e5353f3eb1fe652fa1b1839f9733c630bc8
-  data.tar.gz: c02957f5bc40cb4d8a3dc18d5c078540f5395118ade25df55379c6399c264d03
+  metadata.gz: 191435688dbf322cbae6cc09116253b08831fce203ebdded672e6b19021a30ff
+  data.tar.gz: 8ecb562e1d87bf8ad036e8ced596bdd37ab76dc24464a3a2dbb6133011b263c9
 SHA512:
-  metadata.gz: 97e57a81e839aaafcd83921d258ab2a95d9b7671120ef9c72a0df742c2ddfd4a84bb2d92a1b5fc426979910aea82e1b192226e1dfaddac218a057791ca892012
-  data.tar.gz: 1d89d4153ec4cd0b1604f2a8e4a735d00606d45111f9ad1db92b302968a510fbedfd6a588022ab1120fac4ed9413e7aac3e0694b6cd4200e65cfcf162ed2375e
+  metadata.gz: 27b7a67355852e7a017016acdc3e10a6929c7d679d37aff6a7d0b183e347480b340ddd9fceb39cc70f8d1c77e13eeb6d49685213a9678576cfd4f921828d9cfc
+  data.tar.gz: 846879b068546b96f8b40e54a5b454b7234823612403a4eb4bfef3be42c1682e1ef219a85097c72f6edbdf8db23905cfef38ccf8fb615302cd6ecdc489a93b83

data/.github/workflows/install-matrix.yml

@@ -92,6 +92,13 @@ jobs:
               { sed -i '/--user-install/d' /etc/gemrc || true; }
     env:
       DEBIAN_FRONTEND: noninteractive
+      # Minimal debian/ubuntu/fedora containers ship with no LANG →
+      # Encoding.default_external == US-ASCII → any regex over the
+      # UTF-8 doc/bin files raises `invalid byte sequence in US-ASCII`
+      # (kali/arch already default to UTF-8, which is why only 6/12
+      # legs failed). C.UTF-8 exists on every glibc ≥ 2.35 image.
+      LANG: C.UTF-8
+      LC_ALL: C.UTF-8
       PWN_FRESH_INSTALL: '1'
       PWN_EXPECTED_PM: ${{ matrix.pm }}
       PWN_PROFILE: ${{ matrix.profile }}
@@ -183,6 +190,13 @@ jobs:
       image: kalilinux/kali-rolling
     env:
       DEBIAN_FRONTEND: noninteractive
+      # Minimal debian/ubuntu/fedora containers ship with no LANG →
+      # Encoding.default_external == US-ASCII → any regex over the
+      # UTF-8 doc/bin files raises `invalid byte sequence in US-ASCII`
+      # (kali/arch already default to UTF-8, which is why only 6/12
+      # legs failed). C.UTF-8 exists on every glibc ≥ 2.35 image.
+      LANG: C.UTF-8
+      LC_ALL: C.UTF-8
       PWN_FRESH_INSTALL: '1'
       PWN_EXPECTED_PM: apt
       PWN_PROFILE: core

data/Gemfile

@@ -20,7 +20,7 @@ gem 'base32', '0.3.4'
 gem 'bitcoin-ruby', '0.0.20'
 gem 'brakeman', '8.0.5'
 gem 'bson', '5.2.0'
-gem 'bundler', '>=4.0.16'
+gem 'bundler', '>=4.0.17'
 gem 'bundler-audit', '>=0.9.3'
 gem 'bunny', '3.1.0'
 gem 'colorize', '1.1.0'
@@ -49,10 +49,10 @@ gem 'jwt', '3.2.0'
 gem 'libusb', '0.7.2'
 gem 'luhn', '3.0.0'
 gem 'mail', '2.9.1'
-gem 'mcp', '0.24.0'
+gem 'mcp', '0.25.0'
 gem 'meshtastic', '0.0.172'
 gem 'metasm', '1.0.6'
-gem 'mongo', '2.24.1'
+gem 'mongo', '2.25.0'
 gem 'msfrpc-client', '1.1.2'
 gem 'netaddr', '2.0.6'
 gem 'net-ldap', '0.20.0'

data/documentation/Cron.md

@@ -26,20 +26,22 @@ invokes `PWN::Cron.run(<id>)` on schedule.
 
 ## Seeded self-improvement jobs (`PWN::Cron.install_defaults`)
 
-`pwn setup --migrate` (schema `v1`) seeds two jobs into every fresh
-`~/.pwn/cron/jobs.yml`:
+`pwn setup --migrate` (schema `v1`) seeds four jobs into every fresh
+`~/.pwn/cron/jobs.yml` via `PWN::Cron.install_defaults`:
 
 | Name | Schedule | Ruby |
 |---|---|---|
 | `curriculum_practice_nightly` | `0 3 * * *` | `PWN::AI::Agent::Curriculum.practice(limit: 3)` |
+| `curriculum_offline_judge` | `30 3 * * *` | `PWN::AI::Agent::Curriculum.offline_judge(since_hours: 24, limit: 40)` |
 | `curriculum_train_weekly` | `0 4 * * 0` | `PWN::AI::Agent::Curriculum.train_and_gate(dry_run: true)` |
+| `learning_consolidate_nightly` | `0 5 * * *` | `PWN::AI::Agent::Learning.consolidate` |
 
-The first practises the top unresolved `Mistakes` under `Reward.judge` and
-auto-`resolve`s any it now solves; the second exports SFT + DPO datasets to
-`~/.pwn/finetune/`, LoRA-trains a `pwn-vN+1` local model, replays the
-`Mistakes.top` set on both, and only promotes the new adapter when it wins.
-Set `dry_run: false` (via `pwn-ai-cron` or `cron_create`) once a local
-trainer is installed. See [Reinforcement Learning](Reinforcement-Learning.md).
+- **practice** — top unresolved `Mistakes` under `Reward.judge`; auto-`resolve` with ≥2 holdouts
+- **offline_judge** — P3 backfill of ORM/PRM labels + W3 calibration from PLAN `p(success)=` so `:failure_only` local introspect does not starve the corpus
+- **train** — export SFT + balanced DPO, LoRA-train `pwn-vN+1`, replay Mistakes.top, promote only on win. `dry_run: false` only with a trainer+GPU
+- **consolidate** — M1/M3 memory GC so the injected MEMORY block stays high-signal
+
+See [Reinforcement Learning](Reinforcement-Learning.md).
 
 `cron_disable(id:)` turns either off; `install_defaults` is idempotent and
 never overwrites a job you already have with the same name.

data/documentation/Reinforcement-Learning.md

@@ -1,10 +1,12 @@
 # Reinforcement Learning in pwn-ai
 
-pwn-ai implements a **six-tier** in-context → weight-level RL loop that
-surpasses every published LLM-agent harness (Reflexion, Voyager, ExpeL,
-LATS, Generative Agents, PRM, RLAIF, DPO, Self-Refine, Toolformer, DSPy)
-on at least one axis, and on **five axes simultaneously** for the full
-`Curriculum.practice → Reward.export_dpo → Curriculum.train_and_gate` path.
+pwn-ai implements a **six-tier** in-context → weight-export RL loop that
+combines ORM/PRM, preference ledger, mistake curriculum, env-drift blame,
+balanced DPO export, and regression-gated LoRA promotion. On hosts **with a
+trainer + GPU**, the full
+`Curriculum.practice → Reward.export_dpo → Curriculum.train_and_gate`
+path closes the weight loop; without a trainer the path is **export-ready**
+(datasets + manual CLI) and the live learning is in-context only.
 
 ![Reinforcement-learning loop](diagrams/reinforcement-learning.svg)
 
@@ -50,7 +52,7 @@ on at least one axis, and on **five axes simultaneously** for the full
 |----|-------|------|
 | **C1** | `Registry.rank` + `Metrics.{ucb,thompson,advantage}` | score = α·keyword_sim + β·advantage + γ·UCB1. Untried tools get exploration bonus. |
 | **C2** | `Learning.exemplars_for` | priority = judge_score × e^(−Δt/30d) × keyword_sim. |
-| **C3** | `Curriculum.hindsight` | HER — relabel failed trajectory with achieved-goal as `success:true`. |
+| **C3** | `Curriculum.hindsight` | HER — relabel failed trajectory with achieved-goal as `success: 'soft'` + tags `hindsight/her/soft`. Soft rows are excluded from SFT and 0.35×-weighted in C2 exemplars. |
 | **C4** | `Learning.{compress_exemplar,build_skill_from_session}` | keep only `step_reward > 0` — minimal sufficient trace. |
 
 ## Tier 3 — Memory that stays high-signal
@@ -75,8 +77,8 @@ on at least one axis, and on **five axes simultaneously** for the full
 
 | ID | Where | What |
 |----|-------|------|
-| **W1** | `Reward.{record_preference,export_dpo}` | 5 free preference sources: user_correction, mistakes_resolve, counterfactual, curriculum, critic. |
-| **W2** | `Curriculum.train_and_gate` | SFT+DPO → unsloth/axolotl LoRA → `ollama create pwn-vN+1` → replay `Mistakes.top` on vN vs vN+1 → promote iff `resolved(N+1) > resolved(N)`. |
+| **W1** | `Reward.{record_preference,export_dpo}` | 5 free preference sources: user_correction, mistakes_resolve, counterfactual, curriculum, critic. `export_dpo` enforces ≤40% per-source (`DPO_SOURCE_CAP`); pass `balance: false` for a raw dump. |
+| **W2** | `Curriculum.train_and_gate` | SFT+DPO → unsloth/axolotl LoRA → `ollama create pwn-vN+1` → replay `Mistakes.top` on vN vs vN+1 → promote iff `resolved(N+1) > resolved(N)`. **Without a trainer: export-only** (`weight_loop: :export_ready`). |
 | **W3** | `Curriculum.calibrate` + `Metrics.{record_calibration,calibration}` | plan_first `p(success)` vs actual → per-engine Brier/overconfidence. |
 
 ## Tier 6 — Deepen the intro↔extro join
@@ -91,32 +93,37 @@ on at least one axis, and on **five axes simultaneously** for the full
 
 ```yaml
 :ai:
+  :module_reflection: false  # gates Reflect lesson writing (not ORM alone)
   :agent:
-    :critic: true            # S3
-    :red_team_plan: true     # S4
-    :counterfactual: true    # S2
-    :hindsight: true         # C3 (default true)
-    :verify_as_reward: true  # E3
+    :critic: null            # S3 — nil = ON for remote engines, OFF for ollama
+    :red_team_plan: null     # S4 — same auto policy
+    :counterfactual: null    # S2 — same auto policy
+    :hindsight: true         # C3 (default true; soft-success, 0.35× in C2)
+    :verify_as_reward: null  # E3 — nil = auto (~10% local / always remote on CLAIM_RX)
+    :reward_llm: null        # nil = ORM/PRM use LLM teacher on remote even if module_reflection is false
+    :local_introspect: :failure_only   # ollama cost policy; remote always introspects
+    :introspect_every_n: 3
 ```
 
 ## Cron self-improvement
 
 ```ruby
-PWN::Cron.create(name: 'self_play',   schedule: '0 3 * * *',
-  ruby: 'PWN::AI::Agent::Curriculum.practice(limit: 5)')
-PWN::Cron.create(name: 'weight_loop', schedule: '0 4 * * 0',
-  ruby: 'PWN::AI::Agent::Curriculum.train_and_gate(dry_run: false)')
-PWN::Cron.create(name: 'mem_gc',      schedule: '0 5 * * *',
-  ruby: 'PWN::AI::Agent::Learning.consolidate')
+# Seeded idempotently by PWN::Cron.install_defaults (pwn setup --migrate):
+PWN::Cron.install_defaults
+# → curriculum_practice_nightly   0 3  * * *  Curriculum.practice(limit: 3)
+# → curriculum_offline_judge     30 3  * * *  Curriculum.offline_judge(since_hours: 24, limit: 40)
+# → curriculum_train_weekly       0 4  * * 0  Curriculum.train_and_gate(dry_run: true)  # false only with trainer+GPU
+# → learning_consolidate_nightly  0 5  * * *  Learning.consolidate
 ```
 
 ## Tools exposed to the model
 
 `reward_judge` · `reward_prm` · `reward_sentinel` · `reward_preferences` ·
 `reward_export_dpo` · `curriculum_practice` · `curriculum_train` ·
-`curriculum_hindsight` · `learning_purge_noise`
+`curriculum_hindsight` · `curriculum_offline_judge` ·
+`curriculum_preference_balance` · `learning_purge_noise`
 
-## What no other harness does simultaneously
+## Design claims (architecture — weight promotion requires a trainer)
 
 1. **Process reward on real security tool traces** (R2)
 2. **Automatic blame attribution** self vs env-drift via CUSUM×correlate (E1+E2)
@@ -124,6 +131,21 @@ PWN::Cron.create(name: 'mem_gc',      schedule: '0 5 * * *',
 4. **Mistake-driven curriculum with regression-gated LoRA promotion** (S1+W2)
 5. **Five naturally-generated DPO sources** with zero human labelling (W1)
 
+
+## Operational controls (priority fixes)
+
+| ID | Control | What |
+|----|---------|------|
+| **P1** | `Curriculum.practice` cooldown + natural prompts | Hard-skips `reward_signal` / parked / `needs_code_change`; N-night zero-score cooldown parks thrash; reproducers are natural user tasks, never signature dumps. |
+| **P2** | R4 `semantic_ok` + structured resolve | `31f1871b8a15`-class exit≠0 phantoms stay closed via structured holdouts. |
+| **P3** | `Curriculum.offline_judge` | Scores last-24h sessions under ORM/PRM so local `:failure_only` introspect does not starve labels. Cron nightly. |
+| **P4** | `Reward.proxy_distrust` | When sentinel fires, Metrics.to_context / Registry.rank haircut proxy rates — actionable, not just another Mistakes row. |
+| **R3** | `Reward.sentinel` ring buffer | Fixed-N (`SENTINEL_WINDOW=40`) `{judge,proxy}` window replaces decaying `proxy_sum`/`proxy_n`. Means are always ∈[0,1]; `set_proxy_distrust` refuses proxy∉[0,1]; `reset_sentinel` wipes corrupt state without touching prefs. Legacy decay×`to_i` files auto-clear stuck distrust on load. |
+| **P5** | `Curriculum.preference_balance` + `export_dpo` source-cap | Surfaces W1 monoculture **and enforces** ≤40% per source at export (`DPO_SOURCE_CAP`); critic/counterfactual auto-ON for remote engines so the diet rebalances online. |
+| **P6** | W2 honesty | Docs + `train_and_gate` return `weight_loop: :export_ready` when `trainer: null`. |
+| **P7** | W3 as controller | Engine Brier > 0.35 or overconfidence > 0.25 (n≥8) → force plan_first + critic, cap max_iters at 12. `offline_judge` also records calibration from PLAN `p(success)=` so the controller can fire under `:failure_only`. |
+| **P8** | Remote reward teacher | `agent.reward_llm` nil → ORM/PRM use the LLM teacher on remote engines even when `module_reflection` is false. Local ollama stays heuristic unless explicitly enabled. PRM prompts carry R4 tags so benign recon exits score 0 not −1. |
+
 **See also:** [Skills, Memory & Learning](Skills-Memory-Learning.md) ·
 [Mistakes](Mistakes.md) · [Cron](Cron.md) · [pwn-ai Agent](pwn-ai-Agent.md)