pwn 0.5.661 → 0.5.662
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +26 -2
- data/documentation/How-PWN-Works.md +1 -1
- data/documentation/pwn-ai-Agent.md +7 -5
- data/lib/pwn/ai/agent/learning.rb +30 -1
- data/lib/pwn/ai/agent/loop.rb +75 -17
- data/lib/pwn/ai/agent/reflect.rb +57 -9
- data/lib/pwn/ai/agent/reward.rb +79 -0
- data/lib/pwn/ai/agent/task_summarizer.rb +957 -182
- data/lib/pwn/version.rb +1 -1
- data/spec/conventions_spec.rb +120 -0
- data/spec/lib/pwn/ai/agent/loop_spec.rb +59 -0
- data/spec/lib/pwn/ai/agent/reflect_spec.rb +25 -0
- data/spec/lib/pwn/ai/agent/reward_spec.rb +26 -0
- data/spec/lib/pwn/ai/agent/task_summarizer_spec.rb +436 -28
- data/third_party/pwn_rdoc.jsonl +34 -11
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4df999d2170ed79823d65f8ec78c2f2b7e54f818d965f1758e9c51370723a70d
|
|
4
|
+
data.tar.gz: 701ecebc0de769e0e299a10969002e9dd64581e0e033224ffc93be7137c2db08
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: cd2d20216c848affe6ab50549054c589d2805d152da9d446768dfafb501c762ee23abdffbab22ddce38bfd11950de1101b051ae4266da6a47c37226655f8795a
|
|
7
|
+
data.tar.gz: 6c03147a77c31db940b5d14b55bd6848a4cfa7bd549dc622326d614e94b439baceaab106ba37588b8931567245c42eaaf0aba28dcbc899093b0b9151b457520b
|
data/README.md
CHANGED
|
@@ -41,9 +41,33 @@ Full page: [What is PWN](documentation/What-is-PWN.md)
|
|
|
41
41
|
|
|
42
42
|
#### **Why PWN** ####
|
|
43
43
|
|
|
44
|
+
Here's what `grok-4.5` has to say about pwn (more specifically `pwn-ai`):
|
|
45
|
+
```
|
|
46
|
+
pwn.ai:grok/grok-4.5/0:500K[CURRENT_VERSION]:002 >>> Briefly describe your take on the reinforced learning feedback loop implemented within `pwn-ai` in clear, concise English so that a human understands your response.
|
|
47
|
+
|
|
48
|
+
[ <TIMESTAMP> → pwn-ai → task ] Goal: Briefly describe your take on the reinforced learning feedback loop implemented within `pwn-ai` in clear, concise English so that a human understands your response.
|
|
49
|
+
Tangible tasks (6) — each task may leverage one or more tools to complete its objective(s):
|
|
50
|
+
task 1/6: Review available knowledge of pwn-ai architecture and any reinforced learning elements
|
|
51
|
+
task 2/6: Identify the key stages of the feedback loop used for reinforcement
|
|
52
|
+
task 3/6: Summarize how actions, rewards, and model updates connect in that loop
|
|
53
|
+
task 4/6: Draft a brief plain-English explanation of the overall take on the loop
|
|
54
|
+
task 5/6: Refine the draft for clarity and conciseness suitable for a human reader
|
|
55
|
+
task 6/6: Present the final short description
|
|
56
|
+
|
|
57
|
+
... performs tasks ...
|
|
58
|
+
|
|
59
|
+
**My take, in plain English:**
|
|
60
|
+
|
|
61
|
+
pwn-ai’s reinforcement loop is mostly **learning in context**, not only weight training.
|
|
62
|
+
|
|
63
|
+
On every turn the agent acts with tools. Each result is scored in a few ways: quick “did this tool call really fail?” checks, longer-term tool success stats, and—when a final answer lands—a judge that scores whether the *goal* was actually met, plus per-step credit for which tool calls helped or hurt. Failures become fingerprints (mistakes) with fixes; wins become outcomes, exemplars, memory lessons, and sometimes skills. All of that is written back into the next prompt as “what worked,” “what not to repeat,” and “which tools pay off,” so behavior improves on the following run without waiting for a model train.
|
|
64
|
+
|
|
65
|
+
Around that core sit self-checks: a sentinel that distrusts inflated success rates when the judge disagrees, night jobs that practice the worst repeating mistakes, and optional export of preference pairs for a gated LoRA update when a trainer and GPU exist. So the loop is: **act → measure honestly → remember (and fix) → bias the next act**—with real weight updates as an optional outer ring, not the main daily path.
|
|
66
|
+
```
|
|
67
|
+
|
|
44
68
|
Offensive work is hard because the *tools* do not fit together. PWN's fix is
|
|
45
|
-
simple: every capability is a
|
|
46
|
-
|
|
69
|
+
simple: every capability is a ruby module that can be used with other modules to produce
|
|
70
|
+
a diverse set of security "drivers". That one idea means the *same* code runs:
|
|
47
71
|
|
|
48
72
|
- live in the REPL
|
|
49
73
|
- from an LLM agent in a tool loop
|
|
@@ -28,7 +28,7 @@ hardware).
|
|
|
28
28
|
| Module | Role |
|
|
29
29
|
|---|---|
|
|
30
30
|
| `Loop` | plan → **TaskSummarizer** briefs → dispatch tool_calls → observe → repeat until final answer; tightens runway when recent turns exhausted the budget |
|
|
31
|
-
| **`TaskSummarizer`** | Executive UX: `emit_plan!`
|
|
31
|
+
| **`TaskSummarizer`** | Executive UX: English tasks primary - `emit_plan!` · `about_to` as `task k/n` + via tools · Loop `plan_context` injection · clearer `plan_idx` advance |
|
|
32
32
|
| `Registry` | JSON-Schema function definitions grouped into 12 **toolsets** · **78 tools** |
|
|
33
33
|
| `Dispatch` / `Result` | execute a tool, capture stdout/value/error/duration |
|
|
34
34
|
| `PromptBuilder` | inject MEMORY / SKILLS / LEARNING / **KNOWN MISTAKES + FIXES** / METRICS / EXTROSPECTION blocks |
|
|
@@ -117,16 +117,18 @@ full `Loop.run` under a persona overlay) that share a JSONL bus. See
|
|
|
117
117
|
|
|
118
118
|
| Surface | When | Content |
|
|
119
119
|
|---|---|---|
|
|
120
|
-
| `emit_plan!` | User submit | **Full** goal + ordered tangible tasks (each
|
|
121
|
-
| `about_to` | Before each tool batch | `
|
|
122
|
-
| `
|
|
123
|
-
| `
|
|
120
|
+
| `emit_plan!` | User submit | **Full** goal + ordered plain-English tangible tasks (each may need many tools) |
|
|
121
|
+
| `about_to` | Before each tool batch | **Primary:** `task k/n: <english>` - **secondary:** `via shell×2 (search)` (not raw argv) |
|
|
122
|
+
| `plan_context` / `active_task_prompt` | Into Loop messages | Same English tasks steer tool choice (not TUI-only) |
|
|
123
|
+
| `record!` | After each tool | Advances `plan_idx`; emits English advancement brief when the index moves; verbose progress only if `task_summary_verbose` |
|
|
124
|
+
| `flush!` | End of turn | Optional closing brief with active `task k/n` |
|
|
124
125
|
|
|
125
126
|
**Dedup rules (operational):**
|
|
126
127
|
|
|
127
128
|
- Fingerprint = whitespace-normalized brief; `last_brief_fp` match → return `nil` (no emit).
|
|
128
129
|
- Intent verbs distinguish batches that share tools (`shell` search ≠ `shell` edit).
|
|
129
|
-
- Goal string only on the plan line when a plan exists (`why_bit(with_goal:
|
|
130
|
+
- Goal string only on the plan line when a plan exists (`why_bit(with_goal:)` when no plan bit).
|
|
131
|
+
- Advancement needs a PRM +1 streak or a clear phase shift after tools on the active task (not a blind every-3-tools hop).
|
|
130
132
|
- REPL contract: `on_tool.call('task', full_summary_text, '')` - result empty, no truncation.
|
|
131
133
|
|
|
132
134
|
**Budget pressure:** when unresolved `agent_loop` / `assistant_answer` budget-exhaustion fingerprints dominate, `Loop.budget_exhaustion_hot?` tightens the live turn (stricter `max_iters` on local engines than remote), forces a text-only tail, skips counterfactual forks, and still flushes task state + Learning on the exhaust path.
|
|
@@ -485,14 +485,43 @@ module PWN
|
|
|
485
485
|
Thread.current[:pwn_pending_pref] = nil
|
|
486
486
|
end
|
|
487
487
|
|
|
488
|
+
# Soft plan-quality feature (W3) — tag only; not full DPO.
|
|
489
|
+
plan_cov = nil
|
|
490
|
+
if defined?(Reward) && Reward.respond_to?(:plan_coverage)
|
|
491
|
+
begin
|
|
492
|
+
plan_for_cov = opts[:plan]
|
|
493
|
+
plan_for_cov = opts[:ts_state][:plan] if plan_for_cov.nil? && opts[:ts_state].is_a?(Hash)
|
|
494
|
+
if plan_for_cov.nil? && defined?(TaskSummarizer)
|
|
495
|
+
# Recover numbered tasks from the final/request only when caller
|
|
496
|
+
# did not pass a plan — still keeps TaskSummarizer out of the
|
|
497
|
+
# credit path (parse is pure text).
|
|
498
|
+
plan_for_cov = nil
|
|
499
|
+
end
|
|
500
|
+
if !plan_for_cov.nil? || opts[:final].to_s.length.positive?
|
|
501
|
+
plan_cov = Reward.plan_coverage(
|
|
502
|
+
plan: plan_for_cov || [],
|
|
503
|
+
final: opts[:final],
|
|
504
|
+
request: opts[:request],
|
|
505
|
+
session_id: session_id
|
|
506
|
+
)
|
|
507
|
+
stages_run << :plan_coverage if plan_cov && plan_cov[:total].to_i.positive?
|
|
508
|
+
end
|
|
509
|
+
rescue StandardError => e
|
|
510
|
+
warn "[pwn-ai/learning] plan_coverage swallowed: #{e.class}: #{e.message}"
|
|
511
|
+
end
|
|
512
|
+
end
|
|
513
|
+
|
|
488
514
|
stages_run << :note_outcome
|
|
515
|
+
outcome_tags = ['auto', 'loop', v[:verdict].to_s]
|
|
516
|
+
outcome_tags << plan_cov[:tag] if plan_cov && plan_cov[:tag]
|
|
517
|
+
outcome_tags << "plan_cover=#{plan_cov[:score]}" if plan_cov && plan_cov[:total].to_i.positive?
|
|
489
518
|
note_outcome(
|
|
490
519
|
task: opts[:request].to_s[0, 120],
|
|
491
520
|
success: ok,
|
|
492
521
|
score: v[:score],
|
|
493
522
|
details: "#{v[:verdict]}(#{v[:score].round(2)}) #{v[:rationale]} | #{opts[:final].to_s[0, 200]}",
|
|
494
523
|
session_id: session_id,
|
|
495
|
-
tags:
|
|
524
|
+
tags: outcome_tags
|
|
496
525
|
)
|
|
497
526
|
|
|
498
527
|
stages_run << :fold_judge
|
data/lib/pwn/ai/agent/loop.rb
CHANGED
|
@@ -453,10 +453,26 @@ module PWN
|
|
|
453
453
|
# S4 — adversarial plan review grounded in THIS host's telemetry.
|
|
454
454
|
# P17 — never fork red_team when budget fingerprints dominate: it is
|
|
455
455
|
# another mini agent loop and compounds iteration-budget exhaustion.
|
|
456
|
+
rt = nil
|
|
456
457
|
if defined?(Curriculum) && !hot
|
|
457
458
|
rt = Curriculum.red_team_plan(request: opts[:request], plan: plan)
|
|
458
459
|
messages << { role: 'user', content: rt } if rt
|
|
459
460
|
end
|
|
461
|
+
# P2 — unify TaskSummarizer plan object with surviving outline so the
|
|
462
|
+
# task line and adversarial/plan_first plan are one thing. Index-only;
|
|
463
|
+
# credit stays in Reward. Optional: only when ts_state is live.
|
|
464
|
+
if defined?(TaskSummarizer) && opts[:ts_state].is_a?(Hash)
|
|
465
|
+
outline = [plan.to_s, rt.to_s].reject { |s| s.to_s.strip.empty? }.join("\n")
|
|
466
|
+
begin
|
|
467
|
+
TaskSummarizer.unify_plan!(
|
|
468
|
+
state: opts[:ts_state],
|
|
469
|
+
outline: outline,
|
|
470
|
+
source: rt ? :red_team : :plan_first
|
|
471
|
+
)
|
|
472
|
+
rescue StandardError => e
|
|
473
|
+
warn "[pwn-ai/loop] unify_plan! swallowed: #{e.class}: #{e.message}"
|
|
474
|
+
end
|
|
475
|
+
end
|
|
460
476
|
# W3/P22 — extract predicted p(success) for calibration tracking.
|
|
461
477
|
# Accept p(success)=0.7 | p(success) = .7 | confidence=0.7 on last lines.
|
|
462
478
|
predicted = plan[/p\(\s*success\s*\)\s*=\s*([01]?(?:\.\d+)?)/i, 1]&.to_f
|
|
@@ -706,13 +722,33 @@ module PWN
|
|
|
706
722
|
nil
|
|
707
723
|
end
|
|
708
724
|
|
|
725
|
+
# Push active English task into model messages when plan_idx changes.
|
|
726
|
+
# Uses TaskSummarizer.active_task_prompt (full plan_context on first
|
|
727
|
+
# force, compact focus thereafter). No-ops when already injected.
|
|
728
|
+
private_class_method def self.inject_task_focus!(opts = {})
|
|
729
|
+
state = opts[:state]
|
|
730
|
+
messages = opts[:messages]
|
|
731
|
+
return nil unless state.is_a?(Hash) && messages.is_a?(Array)
|
|
732
|
+
return nil unless defined?(TaskSummarizer) && TaskSummarizer.enabled?
|
|
733
|
+
|
|
734
|
+
text =
|
|
735
|
+
(TaskSummarizer.active_task_prompt(state: state, force: opts[:force]) if TaskSummarizer.respond_to?(:active_task_prompt))
|
|
736
|
+
return nil if text.to_s.strip.empty?
|
|
737
|
+
|
|
738
|
+
messages << { role: 'user', content: text }
|
|
739
|
+
text
|
|
740
|
+
rescue StandardError => e
|
|
741
|
+
warn "[pwn-ai/loop] inject_task_focus! swallowed: #{e.class}: #{e.message}"
|
|
742
|
+
nil
|
|
743
|
+
end
|
|
744
|
+
|
|
709
745
|
# On user-request submit: break the goal into tangible tasks and
|
|
710
746
|
# emit the FULL plan on the pwn-ai task line (no truncation).
|
|
711
747
|
private_class_method def self.task_summary_plan!(opts = {})
|
|
712
748
|
state = opts[:state]
|
|
713
749
|
return nil unless state && defined?(TaskSummarizer) && TaskSummarizer.enabled?
|
|
714
750
|
|
|
715
|
-
line = TaskSummarizer.emit_plan!(state, request: opts[:request])
|
|
751
|
+
line = TaskSummarizer.emit_plan!(state: state, request: opts[:request])
|
|
716
752
|
emit_task_summary(line: line, on_tool: opts[:on_tool]) if line
|
|
717
753
|
line
|
|
718
754
|
rescue StandardError
|
|
@@ -727,8 +763,8 @@ module PWN
|
|
|
727
763
|
return nil unless state && defined?(TaskSummarizer) && TaskSummarizer.enabled?
|
|
728
764
|
|
|
729
765
|
line = TaskSummarizer.about_to(
|
|
730
|
-
opts[:name],
|
|
731
|
-
opts[:args],
|
|
766
|
+
name: opts[:name],
|
|
767
|
+
args: opts[:args],
|
|
732
768
|
state: state,
|
|
733
769
|
request: opts[:request],
|
|
734
770
|
tools: opts[:tools]
|
|
@@ -746,7 +782,12 @@ module PWN
|
|
|
746
782
|
state = opts[:state]
|
|
747
783
|
return nil unless state && defined?(TaskSummarizer)
|
|
748
784
|
|
|
749
|
-
line = TaskSummarizer.record!(
|
|
785
|
+
line = TaskSummarizer.record!(
|
|
786
|
+
state: state,
|
|
787
|
+
name: opts[:name],
|
|
788
|
+
args: opts[:args],
|
|
789
|
+
result: opts[:result]
|
|
790
|
+
)
|
|
750
791
|
# record! is silent by default; only verbose progress returns a line
|
|
751
792
|
emit_task_summary(line: line, on_tool: opts[:on_tool]) if line
|
|
752
793
|
line
|
|
@@ -758,7 +799,7 @@ module PWN
|
|
|
758
799
|
state = opts[:state]
|
|
759
800
|
return nil unless state && defined?(TaskSummarizer)
|
|
760
801
|
|
|
761
|
-
line = TaskSummarizer.flush!(state)
|
|
802
|
+
line = TaskSummarizer.flush!(state: state)
|
|
762
803
|
# Closing brief only — still no result payload on the task row
|
|
763
804
|
emit_task_summary(line: line, on_tool: opts[:on_tool]) if line
|
|
764
805
|
line
|
|
@@ -780,7 +821,7 @@ module PWN
|
|
|
780
821
|
session_id = opts[:session_id]
|
|
781
822
|
on_tool = opts[:on_tool]
|
|
782
823
|
# Live coalesced "what am I doing" lines for the TUI (not a model tool).
|
|
783
|
-
ts_state = (TaskSummarizer.fresh(request: request) if defined?(TaskSummarizer) && TaskSummarizer.enabled?)
|
|
824
|
+
ts_state = (TaskSummarizer.fresh(request: request) if defined?(TaskSummarizer) && TaskSummarizer.enabled? && Thread.current[:pwn_reflect_depth].to_i.zero?)
|
|
784
825
|
engine = active_engine
|
|
785
826
|
local = engine == :ollama
|
|
786
827
|
system_role_content = opts[:system_role_content] ||= PWN::AI::Agent::PromptBuilder.build(session_id: session_id, request: request)
|
|
@@ -789,6 +830,10 @@ module PWN
|
|
|
789
830
|
expose_current_session(session_id: session_id)
|
|
790
831
|
Mistakes.check_user_correction(request: request, session_id: session_id) if defined?(Mistakes)
|
|
791
832
|
|
|
833
|
+
# Initial tool pool from the user request (bootstrap only). After
|
|
834
|
+
# TaskSummarizer.emit_plan! we re-rank using English tangible tasks
|
|
835
|
+
# so generated tasks — not the bare request — drive which tools
|
|
836
|
+
# the model may call.
|
|
792
837
|
tools = Registry.definitions(enabled: opts[:enabled_toolsets], relevance: request)
|
|
793
838
|
messages = [{ role: 'system', content: system_role_content }]
|
|
794
839
|
messages.concat(Learning.exemplars_for(request: request)) if local && defined?(Learning) && Learning.respond_to?(:exemplars_for)
|
|
@@ -797,15 +842,32 @@ module PWN
|
|
|
797
842
|
|
|
798
843
|
# Show full tangible-task breakdown as soon as the user submits.
|
|
799
844
|
task_summary_plan!(state: ts_state, request: request, on_tool: on_tool)
|
|
845
|
+
# Re-bind tools from English plan so task list is the sole driver of
|
|
846
|
+
# tool exposure/ranking (Registry keyword router + CORE).
|
|
847
|
+
if ts_state.is_a?(Hash) && defined?(TaskSummarizer) && TaskSummarizer.respond_to?(:relevance_query)
|
|
848
|
+
rq = TaskSummarizer.relevance_query(state: ts_state, request: request)
|
|
849
|
+
tools = Registry.definitions(enabled: opts[:enabled_toolsets], relevance: rq) unless rq.to_s.strip.empty?
|
|
850
|
+
end
|
|
851
|
+
# English-task-as-primary: inject the same tangible tasks into model
|
|
852
|
+
# context so tool selection follows the plan, not only the TUI banner.
|
|
853
|
+
inject_task_focus!(messages: messages, state: ts_state, force: true)
|
|
800
854
|
|
|
801
855
|
predicted = nil
|
|
802
856
|
Thread.current[:pwn_plan_predicted] = nil
|
|
803
857
|
cal_state = calibration_state
|
|
804
858
|
force_plan = cal_state[:force_plan]
|
|
805
859
|
if (force_plan || agent_flag(key: :plan_first, default: local) || budget_exhaustion_hot?) && !Array(tools).empty?
|
|
806
|
-
predicted = plan_first(messages: messages, request: request)
|
|
860
|
+
predicted = plan_first(messages: messages, request: request, ts_state: ts_state)
|
|
807
861
|
# P22 — prefer explicit return; fall back to thread stash
|
|
808
862
|
predicted = Thread.current[:pwn_plan_predicted] if predicted.nil?
|
|
863
|
+
# unify_plan! may have rewritten English tasks — force refresh focus.
|
|
864
|
+
# Re-rank tools from (possibly unified) English plan; never from
|
|
865
|
+
# PLAN: tool-call scaffold jargon (unify_plan! refuses that).
|
|
866
|
+
if ts_state.is_a?(Hash) && defined?(TaskSummarizer) && TaskSummarizer.respond_to?(:relevance_query)
|
|
867
|
+
rq = TaskSummarizer.relevance_query(state: ts_state, request: request)
|
|
868
|
+
tools = Registry.definitions(enabled: opts[:enabled_toolsets], relevance: rq) unless rq.to_s.strip.empty?
|
|
869
|
+
end
|
|
870
|
+
inject_task_focus!(messages: messages, state: ts_state, force: true)
|
|
809
871
|
end
|
|
810
872
|
if budget_exhaustion_hot?
|
|
811
873
|
hot_hint = if active_engine == :ollama
|
|
@@ -837,6 +899,9 @@ module PWN
|
|
|
837
899
|
max_iters.times do |i|
|
|
838
900
|
# 3.1 — compact history on local so tool dumps don't fill num_ctx
|
|
839
901
|
compact_history!(messages: messages) if local
|
|
902
|
+
# English-task-as-primary: when plan_idx advanced, tell the model
|
|
903
|
+
# which plain-English task is active before the next tool batch.
|
|
904
|
+
inject_task_focus!(messages: messages, state: ts_state)
|
|
840
905
|
|
|
841
906
|
# P17 — on the final iteration, strip tools and demand a plain-text
|
|
842
907
|
# answer. Without this the model happily emits one more tool_calls
|
|
@@ -934,7 +999,7 @@ module PWN
|
|
|
934
999
|
next
|
|
935
1000
|
end
|
|
936
1001
|
append_session(session_id: session_id, role: 'assistant', content: text)
|
|
937
|
-
Learning.auto_introspect(session_id: session_id, request: request, final: text, predicted: predicted) if defined?(Learning) && should_auto_introspect?(local: local, turn_fails: turn_fails, iter: i)
|
|
1002
|
+
Learning.auto_introspect(session_id: session_id, request: request, final: text, predicted: predicted, plan: ts_state && ts_state[:plan], ts_state: ts_state) if defined?(Learning) && should_auto_introspect?(local: local, turn_fails: turn_fails, iter: i)
|
|
938
1003
|
task_summary_flush!(state: ts_state, on_tool: on_tool)
|
|
939
1004
|
return text
|
|
940
1005
|
end
|
|
@@ -1039,7 +1104,7 @@ module PWN
|
|
|
1039
1104
|
)
|
|
1040
1105
|
end
|
|
1041
1106
|
append_session(session_id: session_id, role: 'assistant', content: msg)
|
|
1042
|
-
Learning.auto_introspect(session_id: session_id, request: request, final: msg, predicted: predicted) if defined?(Learning) && should_auto_introspect?(local: local, turn_fails: turn_fails, iter: i)
|
|
1107
|
+
Learning.auto_introspect(session_id: session_id, request: request, final: msg, predicted: predicted, plan: ts_state && ts_state[:plan], ts_state: ts_state) if defined?(Learning) && should_auto_introspect?(local: local, turn_fails: turn_fails, iter: i)
|
|
1043
1108
|
task_summary_flush!(state: ts_state, on_tool: on_tool)
|
|
1044
1109
|
return msg
|
|
1045
1110
|
end
|
|
@@ -1068,14 +1133,7 @@ module PWN
|
|
|
1068
1133
|
)
|
|
1069
1134
|
end
|
|
1070
1135
|
append_session(session_id: session_id, role: 'assistant', content: final_msg)
|
|
1071
|
-
if defined?(Learning) && should_auto_introspect?(local: local, turn_fails: turn_fails, iter: max_iters)
|
|
1072
|
-
Learning.auto_introspect(
|
|
1073
|
-
session_id: session_id,
|
|
1074
|
-
request: request,
|
|
1075
|
-
final: final_msg,
|
|
1076
|
-
predicted: predicted
|
|
1077
|
-
)
|
|
1078
|
-
end
|
|
1136
|
+
Learning.auto_introspect(session_id: session_id, request: request, final: final_msg, predicted: predicted, plan: ts_state && ts_state[:plan], ts_state: ts_state) if defined?(Learning) && should_auto_introspect?(local: local, turn_fails: turn_fails, iter: max_iters)
|
|
1079
1137
|
task_summary_flush!(state: ts_state, on_tool: on_tool)
|
|
1080
1138
|
final_msg
|
|
1081
1139
|
end
|
data/lib/pwn/ai/agent/reflect.rb
CHANGED
|
@@ -30,7 +30,24 @@ module PWN
|
|
|
30
30
|
# local Ollama model EXECUTE the task while a frontier model WRITES
|
|
31
31
|
# the durable lessons about it — the local model then reads back
|
|
32
32
|
# distilled reasoning it could never have produced itself.
|
|
33
|
+
#
|
|
34
|
+
# IMPLEMENTATION NOTE
|
|
35
|
+
# -------------------
|
|
36
|
+
# Reflect.on MUST call the engine's text .chat API directly — never
|
|
37
|
+
# Loop.run. Nesting Loop.run re-enters TaskSummarizer/PromptBuilder/
|
|
38
|
+
# auto_introspect and produces SystemStackError at the Pry after_read
|
|
39
|
+
# boundary whenever module_reflection is enabled. A thread-local
|
|
40
|
+
# depth counter still gates re-entrant Reflect.on (e.g. chat_for_plan
|
|
41
|
+
# inside an outer agent turn that also judges/reflects).
|
|
33
42
|
module Reflect
|
|
43
|
+
ENGINE_MODS = {
|
|
44
|
+
openai: 'PWN::AI::OpenAI',
|
|
45
|
+
grok: 'PWN::AI::Grok',
|
|
46
|
+
ollama: 'PWN::AI::Ollama',
|
|
47
|
+
anthropic: 'PWN::AI::Anthropic',
|
|
48
|
+
gemini: 'PWN::AI::Gemini'
|
|
49
|
+
}.freeze
|
|
50
|
+
|
|
34
51
|
# Supported Method Parameters::
|
|
35
52
|
# response = PWN::AI::Agent::Reflect.on(
|
|
36
53
|
# request: 'required - String - What you want the AI to reflect on',
|
|
@@ -55,21 +72,36 @@ module PWN
|
|
|
55
72
|
|
|
56
73
|
ai_module_reflection = PWN::Env[:ai][:module_reflection]
|
|
57
74
|
|
|
75
|
+
# Re-entrancy guard: nested Reflect.on (TaskSummarizer inside a
|
|
76
|
+
# Reflect call, Reward.judge during auto_introspect, etc.) returns
|
|
77
|
+
# nil so the outer caller can fall back. Never Loop.run from here.
|
|
78
|
+
return nil if Thread.current[:pwn_reflect_depth].to_i.positive?
|
|
79
|
+
|
|
58
80
|
if ai_module_reflection && request.length.positive?
|
|
59
81
|
override = opts[:engine] || PWN::Env.dig(:ai, :reflect_engine)
|
|
60
82
|
model = opts[:model] || PWN::Env.dig(:ai, :reflect_model)
|
|
61
83
|
engine = (override || PWN::Env[:ai][:active]).to_s.downcase.to_sym
|
|
62
|
-
valid_ai_engines =
|
|
84
|
+
valid_ai_engines = ENGINE_MODS.keys
|
|
63
85
|
raise "ERROR: Unsupported AI engine. Supported engines are: #{valid_ai_engines}" unless valid_ai_engines.include?(engine)
|
|
64
86
|
|
|
65
87
|
warn "AI Reflection is enabled. Ensure #{engine} has been authorized for use and/or requests are sanitized properly." unless suppress_pii_warning
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
88
|
+
Thread.current[:pwn_reflect_depth] = Thread.current[:pwn_reflect_depth].to_i + 1
|
|
89
|
+
begin
|
|
90
|
+
response = with_engine(engine: override, model: model) do
|
|
91
|
+
engine_chat(
|
|
92
|
+
engine: engine,
|
|
93
|
+
request: request.chomp,
|
|
94
|
+
system_role_content: system_role_content,
|
|
95
|
+
spinner: spinner
|
|
96
|
+
)
|
|
97
|
+
end
|
|
98
|
+
ensure
|
|
99
|
+
d = Thread.current[:pwn_reflect_depth].to_i - 1
|
|
100
|
+
if d.positive?
|
|
101
|
+
Thread.current[:pwn_reflect_depth] = d
|
|
102
|
+
else
|
|
103
|
+
Thread.current[:pwn_reflect_depth] = nil
|
|
104
|
+
end
|
|
73
105
|
end
|
|
74
106
|
end
|
|
75
107
|
|
|
@@ -78,7 +110,23 @@ module PWN
|
|
|
78
110
|
raise e
|
|
79
111
|
end
|
|
80
112
|
|
|
81
|
-
#
|
|
113
|
+
# Direct provider .chat — never Loop.run (avoids after_read stack blow).
|
|
114
|
+
private_class_method def self.engine_chat(opts = {})
|
|
115
|
+
engine = opts[:engine].to_s.downcase.to_sym
|
|
116
|
+
mod_name = ENGINE_MODS[engine]
|
|
117
|
+
raise "ERROR: Unsupported AI engine: #{engine}" unless mod_name
|
|
118
|
+
|
|
119
|
+
mod = Object.const_get(mod_name)
|
|
120
|
+
raise "ERROR: #{mod_name} does not implement .chat" unless mod.respond_to?(:chat)
|
|
121
|
+
|
|
122
|
+
mod.chat(
|
|
123
|
+
request: opts[:request],
|
|
124
|
+
system_role_content: opts[:system_role_content],
|
|
125
|
+
spinner: opts[:spinner]
|
|
126
|
+
)
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
# Temporarily override PWN::Env[:ai][:active] so engine_chat routes to
|
|
82
130
|
# the teacher engine, restoring afterwards even on raise. No-op when
|
|
83
131
|
# engine is nil/blank or PWN::Env[:ai] is frozen.
|
|
84
132
|
private_class_method def self.with_engine(opts = {})
|
data/lib/pwn/ai/agent/reward.rb
CHANGED
|
@@ -210,6 +210,84 @@ module PWN
|
|
|
210
210
|
[]
|
|
211
211
|
end
|
|
212
212
|
|
|
213
|
+
# ----------------------------------------------------------------
|
|
214
|
+
# Plan-quality soft signal (W3 feature / Learning tag)
|
|
215
|
+
# ----------------------------------------------------------------
|
|
216
|
+
# Cheap heuristic: did the final (+ optional tool trace) cover the
|
|
217
|
+
# tangible plan tasks? Not full DPO — trajectory-shaped pairs come
|
|
218
|
+
# later. Score is a soft feature for calibration / tagging only.
|
|
219
|
+
#
|
|
220
|
+
# Supported Method Parameters::
|
|
221
|
+
# r = PWN::AI::Agent::Reward.plan_coverage(
|
|
222
|
+
# plan: 'required - Array of task strings or outline text',
|
|
223
|
+
# final: 'required - assistant final answer',
|
|
224
|
+
# request: 'optional - original user request',
|
|
225
|
+
# trace: 'optional - Array of tool-result strings',
|
|
226
|
+
# session_id: 'optional - load trace from session when trace empty'
|
|
227
|
+
# )
|
|
228
|
+
# => { score: 0.0..1.0, covered: N, total: M, missing: [...], tag: 'plan_cover_high|mid|low' }
|
|
229
|
+
public_class_method def self.plan_coverage(opts = {})
|
|
230
|
+
plan = opts[:plan]
|
|
231
|
+
tasks =
|
|
232
|
+
case plan
|
|
233
|
+
when Array then plan.map(&:to_s)
|
|
234
|
+
when String
|
|
235
|
+
if defined?(TaskSummarizer) && TaskSummarizer.respond_to?(:parse_outline_tasks)
|
|
236
|
+
TaskSummarizer.parse_outline_tasks(outline: plan)
|
|
237
|
+
else
|
|
238
|
+
plan.to_s.split(/\n+/).map { |l| l.sub(/\A(?:\d+[.):]|[-*•])\s+/, '').strip }
|
|
239
|
+
end
|
|
240
|
+
else
|
|
241
|
+
Array(plan).map(&:to_s)
|
|
242
|
+
end
|
|
243
|
+
tasks = tasks.map { |t| t.to_s.gsub(/\s+/, ' ').strip }.reject(&:empty?)
|
|
244
|
+
return { score: 0.0, covered: 0, total: 0, missing: [], tag: 'plan_cover_none' } if tasks.empty?
|
|
245
|
+
|
|
246
|
+
final = opts[:final].to_s
|
|
247
|
+
request = opts[:request].to_s
|
|
248
|
+
trace = Array(opts[:trace])
|
|
249
|
+
trace = load_trace(session_id: opts[:session_id]) if trace.empty? && opts[:session_id]
|
|
250
|
+
blob = "#{final}\n#{request}\n#{trace.join("\n")}".downcase
|
|
251
|
+
|
|
252
|
+
covered = []
|
|
253
|
+
missing = []
|
|
254
|
+
tasks.each do |task|
|
|
255
|
+
stems = task.downcase.scan(/[a-z0-9]{4,}/).uniq
|
|
256
|
+
# Drop ultra-generic plan fillers that would false-positive everything.
|
|
257
|
+
stems.reject! { |s| %w[result results report verify complete completion present carry work task step this that with from into].include?(s) }
|
|
258
|
+
if stems.empty?
|
|
259
|
+
covered << task
|
|
260
|
+
next
|
|
261
|
+
end
|
|
262
|
+
# A task is covered when >= half of its distinctive stems appear
|
|
263
|
+
# in final+trace (soft — not DPO-grade evidence).
|
|
264
|
+
hits = stems.count { |s| blob.include?(s) }
|
|
265
|
+
need = [1, (stems.length / 2.0).ceil].max
|
|
266
|
+
if hits >= need
|
|
267
|
+
covered << task
|
|
268
|
+
else
|
|
269
|
+
missing << task
|
|
270
|
+
end
|
|
271
|
+
end
|
|
272
|
+
|
|
273
|
+
total = tasks.length
|
|
274
|
+
score = (covered.length.to_f / total).round(3).clamp(0.0, 1.0)
|
|
275
|
+
tag =
|
|
276
|
+
if score >= 0.75 then 'plan_cover_high'
|
|
277
|
+
elsif score >= 0.4 then 'plan_cover_mid'
|
|
278
|
+
else 'plan_cover_low'
|
|
279
|
+
end
|
|
280
|
+
{
|
|
281
|
+
score: score,
|
|
282
|
+
covered: covered.length,
|
|
283
|
+
total: total,
|
|
284
|
+
missing: missing.first(6),
|
|
285
|
+
tag: tag
|
|
286
|
+
}
|
|
287
|
+
rescue StandardError
|
|
288
|
+
{ score: 0.0, covered: 0, total: 0, missing: [], tag: 'plan_cover_error' }
|
|
289
|
+
end
|
|
290
|
+
|
|
213
291
|
# ----------------------------------------------------------------
|
|
214
292
|
# R3 — Reward-hacking sentinel
|
|
215
293
|
# ----------------------------------------------------------------
|
|
@@ -1306,6 +1384,7 @@ module PWN
|
|
|
1306
1384
|
# Tier 1 — reward signal
|
|
1307
1385
|
PWN::AI::Agent::Reward.judge(request: req, final: text, session_id: sid) # R1 ORM → {score:, verdict:, rationale:}
|
|
1308
1386
|
PWN::AI::Agent::Reward.prm(request: req, session_id: sid) # R2 PRM → per-step credit
|
|
1387
|
+
PWN::AI::Agent::Reward.plan_coverage(plan: tasks, final: text, session_id: sid) # soft plan-quality feature
|
|
1309
1388
|
PWN::AI::Agent::Reward.sentinel # R3 reward-hacking detector
|
|
1310
1389
|
PWN::AI::Agent::Reward.reset_sentinel # wipe corrupt window + distrust
|
|
1311
1390
|
PWN::AI::Agent::Reward.warm_sentinel # P10 fill R3 window from Learning outcomes
|