pwn 0.5.744 → 0.5.745
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/documentation/CLI-Drivers.md +10 -0
- data/documentation/Installation.md +1 -0
- data/documentation/Policy-Benchmark.md +76 -0
- data/documentation/Reinforcement-Learning.md +7 -0
- data/documentation/Skills-Memory-Learning.md +2 -0
- data/etc/default_skills/pwn/ai/agent/skill_review/SKILL.md +49 -0
- data/lib/pwn/ai/agent/learning.rb +8 -1
- data/lib/pwn/ai/agent/skill_review.rb +223 -0
- data/lib/pwn/ai/agent.rb +1 -0
- data/lib/pwn/ai/cli.rb +54 -0
- data/lib/pwn/config.rb +2 -0
- data/lib/pwn/migrate.rb +5 -1
- data/lib/pwn/plugins/repl/mesh.rb +1 -1
- data/lib/pwn/version.rb +1 -1
- data/spec/lib/pwn/ai/agent/rsi_metrics_spec.rb +5 -0
- data/spec/lib/pwn/ai/agent/skill_review_spec.rb +127 -0
- data/spec/lib/pwn/ai/cli_spec.rb +108 -0
- data/spec/lib/pwn/migrate_spec.rb +29 -2
- data/third_party/pwn_rdoc.jsonl +18 -0
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f3d60fd604a668b531459517b97dfee6448f9485679bd4c25bfe3ec12090cee9
|
|
4
|
+
data.tar.gz: 64a4dd65248017933e877c03db222ce61741e60044a03a1c9c9a1297ca7a4fde
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1792ff5aea6a7c406248ab0fd6647732c490e56450afdf768840f340b4d7b4847a2d2cd996ff1349b3ca1a1bbfc05be40310225345e675937cc16022aaad5ec3
|
|
7
|
+
data.tar.gz: 7155e5d3cfca02bda195d0b0f926b85d6cd95b306b570d5956484ae237d360b36628a979e2d98a2c5ce3008343c7cecc1d14f12d0a986dbc577923a9f7905930
|
|
@@ -59,6 +59,16 @@ pwn_setup --migrate --fix # ~/.pwn state doctor + autofix (PWN::Migrate)
|
|
|
59
59
|
See [Installation](Installation.md) for the full profile table, the
|
|
60
60
|
`PWN::Setup` API and the `PWN::Migrate` state-file registry.
|
|
61
61
|
|
|
62
|
+
## Offline policy evaluation with `pwn-ai`
|
|
63
|
+
|
|
64
|
+
`pwn-ai --policy evaluate --baseline PATH --candidate PATH` prints two frozen
|
|
65
|
+
held-out reports without opening a vault or starting an AI session. The separate
|
|
66
|
+
`--policy promote` and `--policy rollback` actions require an explicit target and
|
|
67
|
+
both `--approve-policy-change` and `--policy-writers-stopped`. They never start
|
|
68
|
+
network tasks or automatically promote on success. See
|
|
69
|
+
[Policy-Benchmark](Policy-Benchmark.md#offline-operator-cli) for runnable local
|
|
70
|
+
commands, receipt preservation, fresh replay gates, and limitations.
|
|
71
|
+
|
|
62
72
|
## Typical CI usage
|
|
63
73
|
|
|
64
74
|
```yaml
|
|
@@ -187,6 +187,7 @@ What `--migrate` does:
|
|
|
187
187
|
directory layout) → **deep-merge** any keys the current
|
|
188
188
|
`PWN::Config.env_template` added into your encrypted `~/.pwn/pwn.yaml`
|
|
189
189
|
**without overwriting your values** (re-encrypted with the same key/IV).
|
|
190
|
+
Schema 5 adds `ai.agent.skill_review` (`recommend` when the key is absent).
|
|
190
191
|
|
|
191
192
|
Everything is idempotent and dry-run capable. The plain `pwn` launcher also
|
|
192
193
|
prints a one-line drift warning on startup whenever `~/.pwn/.schema`
|
|
@@ -220,6 +220,82 @@ fields except `elapsed_seconds`. They contain no wall-clock timestamps, random
|
|
|
220
220
|
IDs or temporary paths. The original training report still contains the timing
|
|
221
221
|
and metadata variability described above.
|
|
222
222
|
|
|
223
|
+
## Offline operator CLI
|
|
224
|
+
|
|
225
|
+
`pwn-ai --policy evaluate|promote|rollback` reuses `PolicyEvaluation`; it never
|
|
226
|
+
starts a session, loads the encrypted configuration, runs `Learning.rsi_tick`,
|
|
227
|
+
or invokes the agent loop. It cannot be combined with `--ai`, replay, mission,
|
|
228
|
+
or other session options. Evaluation accepts existing explicit frozen snapshots,
|
|
229
|
+
runs suites 0 and 1, and prints a JSON **array** suitable for `--reports`. It does
|
|
230
|
+
not train or write the live policy. The Ruby API supports other indices 0..7.
|
|
231
|
+
|
|
232
|
+
From a source checkout, this disposable demonstration stays entirely under a
|
|
233
|
+
new `/tmp` directory (use `pwn-ai` instead of `ruby -Ilib bin/pwn-ai` for an
|
|
234
|
+
installed executable):
|
|
235
|
+
|
|
236
|
+
```sh
|
|
237
|
+
umask 077
|
|
238
|
+
work=$(mktemp -d /tmp/pwn-offline-policy.XXXXXX)
|
|
239
|
+
ruby scripts/benchmark_policy.rb --heldout \
|
|
240
|
+
--snapshot-dir "$work/snapshots" --output "$work/benchmark.json"
|
|
241
|
+
ruby -Ilib bin/pwn-ai --policy evaluate \
|
|
242
|
+
--baseline "$work/snapshots/off.json" --candidate "$work/snapshots/on.json" \
|
|
243
|
+
> "$work/reports.json"
|
|
244
|
+
cp "$work/snapshots/off.json" "$work/demo-live.json"
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Review `reports.json` before deciding whether to proceed. For a real target,
|
|
248
|
+
first stop **all** agent processes and other policy writers, take the baseline
|
|
249
|
+
copy only after stopping them, and keep writers stopped through promotion and
|
|
250
|
+
readback. The acknowledgement does not stop processes or acquire a writer lock.
|
|
251
|
+
Concurrent promotions are also unsupported. Do not replace a real policy with
|
|
252
|
+
these demonstration snapshots or infer live gains from this fixture experiment.
|
|
253
|
+
|
|
254
|
+
Only if approving the change to the **disposable demo target**, run:
|
|
255
|
+
|
|
256
|
+
```sh
|
|
257
|
+
ruby -Ilib bin/pwn-ai --policy promote \
|
|
258
|
+
--baseline "$work/snapshots/off.json" --candidate "$work/snapshots/on.json" \
|
|
259
|
+
--reports "$work/reports.json" --live-policy "$work/demo-live.json" \
|
|
260
|
+
--approve-policy-change --policy-writers-stopped > "$work/promotion.json"
|
|
261
|
+
cmp "$work/demo-live.json" "$work/snapshots/on.json"
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
The command re-executes both reports before considering replacement. It returns
|
|
265
|
+
nonzero on refusal or malformed input; check the exit status and JSON, not just
|
|
266
|
+
whether a redirected file exists. Keep the successful `promotion.json` receipt
|
|
267
|
+
and digest-named backup. Use distinct, new output paths: shell redirection opens
|
|
268
|
+
files before the CLI starts, so never redirect onto snapshots, the live target,
|
|
269
|
+
or an existing receipt. A failed retry must not overwrite the successful receipt.
|
|
270
|
+
|
|
271
|
+
Rollback is a separate explicit operator decision with the same stopped-writer
|
|
272
|
+
requirement; it refuses intervening changes to the target:
|
|
273
|
+
|
|
274
|
+
```sh
|
|
275
|
+
ruby -Ilib bin/pwn-ai --policy rollback \
|
|
276
|
+
--receipt "$work/promotion.json" --live-policy "$work/demo-live.json" \
|
|
277
|
+
--approve-policy-change --policy-writers-stopped > "$work/rollback.json"
|
|
278
|
+
cmp "$work/demo-live.json" "$work/snapshots/off.json"
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
Neither approval flag alone is sufficient. There is no default live path,
|
|
282
|
+
automatic promotion, target discovery, network task, or model training in this
|
|
283
|
+
CLI path. Reports and receipts are operator-owned local JSON, not executable
|
|
284
|
+
configuration; neither can supply approval flags. Existing `Learning.rsi_tick`
|
|
285
|
+
only snapshots measured rates and records a regression lesson. It does not call
|
|
286
|
+
this gate, generate candidates, schedule practice, or approve changes. The
|
|
287
|
+
broader online learning and curriculum paths remain unchanged and separate.
|
|
288
|
+
|
|
289
|
+
Focused verification commands (no hardware, providers, or live models):
|
|
290
|
+
|
|
291
|
+
```sh
|
|
292
|
+
bundle exec rspec spec/lib/pwn/ai/cli_spec.rb \
|
|
293
|
+
spec/lib/pwn/ai/agent/policy_evaluation_spec.rb \
|
|
294
|
+
spec/lib/pwn/ai/agent/learning_spec.rb spec/lib/pwn/ai/agent/rsi_metrics_spec.rb
|
|
295
|
+
bundle exec rubocop lib/pwn/ai/cli.rb spec/lib/pwn/ai/cli_spec.rb \
|
|
296
|
+
spec/lib/pwn/ai/agent/rsi_metrics_spec.rb
|
|
297
|
+
```
|
|
298
|
+
|
|
223
299
|
## Explicit promotion and rollback
|
|
224
300
|
|
|
225
301
|
This is an **operator-invoked local eligibility gate**, not automatic online
|
|
@@ -12,6 +12,13 @@ one. Without a trainer it still **exports** the datasets and a manual CLI. Live
|
|
|
12
12
|
|
|
13
13
|
ESR and ASR are the rates RSI actually compares. ESR is `verified_exploit_tools / vulnerable_tools`. ASR is `successful_attacks / total_attack_attempts`. `Learning.rsi_tick` stores the pair and writes an `rsi` lesson when ESR falls. Tool telemetry and the judge score are separate. A passing suite is not either rate.
|
|
14
14
|
|
|
15
|
+
RSI's rate tick does not generate or promote policy candidates. For the separate
|
|
16
|
+
**offline operator evaluation** path, use `pwn-ai --policy evaluate` and review
|
|
17
|
+
the fixed local held-out reports. Promotion and rollback each require explicit
|
|
18
|
+
`--approve-policy-change --policy-writers-stopped` plus a named live policy path;
|
|
19
|
+
neither runs automatically from RSI. See the executable commands, stopped-writer
|
|
20
|
+
requirements, and benchmark limits in [Policy-Benchmark](Policy-Benchmark.md#offline-operator-cli).
|
|
21
|
+
|
|
15
22
|

|
|
16
23
|
|
|
17
24
|
```
|
|
@@ -132,6 +132,8 @@ bundled skills into `~/.pwn/skills/` when the name is missing:
|
|
|
132
132
|
| `att&ck` | Exhaustive test procedure per ATT&CK technique (`references/T1059.001.md`) |
|
|
133
133
|
| `humanizer` | Strip AI writing patterns from prose. Keep meaning and identifiers. |
|
|
134
134
|
|
|
135
|
+
`PWN::AI::Agent::SkillReview` runs after introspection. A routine success does not create a skill. A tested procedure, a resolved recurring mistake, or an explicit correction can recommend an update to the closest skill under `~/.pwn/skills`. The default mode is `recommend`. `auto-safe` writes only a small addition that an execution fixture passed in three sessions, and it keeps a backup. It does not create skills, rewrite generated `pwn/` module skills, or save secrets and raw tool output. Set `ai.agent.skill_review` to `off`, `recommend`, or `auto-safe`.
|
|
136
|
+
|
|
135
137
|
Source: `etc/default_skills/` in the gem. SOP edits in `~/.pwn/skills/<name>/SKILL.md`
|
|
136
138
|
are never overwritten. Generated `~/.pwn/skills/pwn/**/SKILL.md` module skills
|
|
137
139
|
are updated on migrate when `lib/pwn` changes.
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: pwn-ai-agent-skillreview
|
|
3
|
+
description: Drive PWN::AI::Agent::SkillReview from pwn_eval.
|
|
4
|
+
license: MIT
|
|
5
|
+
allowed-tools: [pwn, pwn_eval]
|
|
6
|
+
metadata:
|
|
7
|
+
bundled: true
|
|
8
|
+
generated: true
|
|
9
|
+
module: PWN::AI::Agent::SkillReview
|
|
10
|
+
source: pwn/ai/agent/skill_review.rb
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
# PWN::AI::Agent::SkillReview
|
|
14
|
+
|
|
15
|
+
Decide whether a completed task should update or create a skill. recommend is the default. auto-safe writes only a small verified addition.
|
|
16
|
+
|
|
17
|
+
## When to use
|
|
18
|
+
|
|
19
|
+
Call `PWN::AI::Agent::SkillReview` from `pwn_eval` when the task needs this module.
|
|
20
|
+
Do not reimplement it in shell.
|
|
21
|
+
|
|
22
|
+
## Methodologies
|
|
23
|
+
|
|
24
|
+
Generated from `pwn/ai/agent/skill_review.rb`. Prefer the public class methods below.
|
|
25
|
+
Class methods take `(opts = {})` and read `opts`.
|
|
26
|
+
|
|
27
|
+
## How to call
|
|
28
|
+
|
|
29
|
+
```ruby
|
|
30
|
+
PWN::AI::Agent::SkillReview.help
|
|
31
|
+
PWN::AI::Agent::SkillReview.review(opts)
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Public methods
|
|
35
|
+
|
|
36
|
+
- `review`
|
|
37
|
+
- `review_turn`
|
|
38
|
+
- `note_reuse`
|
|
39
|
+
- `authors`
|
|
40
|
+
- `help`
|
|
41
|
+
|
|
42
|
+
## Source
|
|
43
|
+
|
|
44
|
+
`pwn/ai/agent/skill_review.rb`
|
|
45
|
+
|
|
46
|
+
## Verification
|
|
47
|
+
|
|
48
|
+
`PWN::AI::Agent::SkillReview.respond_to?(:review)` after the
|
|
49
|
+
module is loaded. Read the source for parameter names.
|
|
@@ -822,13 +822,20 @@ module PWN
|
|
|
822
822
|
rsi_tick(request: opts[:request])
|
|
823
823
|
end
|
|
824
824
|
|
|
825
|
+
skill_review = nil
|
|
826
|
+
if defined?(SkillReview)
|
|
827
|
+
stages_run << :skill_review
|
|
828
|
+
skill_review = SkillReview.review_turn(request: opts[:request], final: opts[:final], success: opts[:success], session_id: opts[:session_id])
|
|
829
|
+
end
|
|
830
|
+
|
|
825
831
|
{
|
|
826
832
|
ok: ok,
|
|
827
833
|
score: v[:score],
|
|
828
834
|
elapsed_ms: elapsed_ms.call,
|
|
829
835
|
budget_hot: budget_hot,
|
|
830
836
|
stages_run: stages_run,
|
|
831
|
-
stages_skipped: stages_skipped
|
|
837
|
+
stages_skipped: stages_skipped,
|
|
838
|
+
skill_review: skill_review
|
|
832
839
|
}
|
|
833
840
|
rescue StandardError => e
|
|
834
841
|
warn "[pwn-ai/learning] auto_introspect swallowed: #{e.class}: #{e.message}"
|
|
@@ -0,0 +1,223 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'digest'
|
|
4
|
+
require 'fileutils'
|
|
5
|
+
require 'json'
|
|
6
|
+
|
|
7
|
+
module PWN
|
|
8
|
+
module AI
|
|
9
|
+
module Agent
|
|
10
|
+
# Decide whether a completed task should update or create a skill.
|
|
11
|
+
# recommend is the default. auto-safe writes only a small verified addition.
|
|
12
|
+
module SkillReview
|
|
13
|
+
LEDGER = File.join(Dir.home, '.pwn', 'skill_review.jsonl').freeze
|
|
14
|
+
MODES = %w[off recommend auto-safe].freeze
|
|
15
|
+
|
|
16
|
+
public_class_method def self.review(opts = {})
|
|
17
|
+
mode = mode_from(mode: opts[:mode])
|
|
18
|
+
request = opts[:request].to_s
|
|
19
|
+
return finish(action: 'skipped', reason: 'mode off', mode: mode, request: request) if mode == 'off'
|
|
20
|
+
return finish(action: 'skipped', reason: 'routine success is not a skill', mode: mode, request: request) unless useful?(opts)
|
|
21
|
+
|
|
22
|
+
procedure = procedure_from(procedure: opts[:procedure], evidence: opts[:evidence])
|
|
23
|
+
unless procedure[:ok]
|
|
24
|
+
return finish(action: 'refused', reason: procedure[:reason], mode: mode, request: request) if opts[:procedure]
|
|
25
|
+
|
|
26
|
+
return finish(action: 'recommend', reason: 'a structured procedure is required before saving', mode: mode, request: request, verified: false)
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
target = locate(name: opts[:name], query: request, procedure: procedure[:text])
|
|
30
|
+
kind = target ? 'update' : 'create'
|
|
31
|
+
verified = verified_execution?(evidence: opts[:evidence])
|
|
32
|
+
proposal = {
|
|
33
|
+
action: 'recommend',
|
|
34
|
+
kind: kind,
|
|
35
|
+
name: target ? target[:name] : opts[:name].to_s,
|
|
36
|
+
mode: mode,
|
|
37
|
+
request: request,
|
|
38
|
+
verified: verified,
|
|
39
|
+
reason: target ? 'closest skill matches the procedure' : 'no existing skill matches',
|
|
40
|
+
path: target && target[:path]
|
|
41
|
+
}
|
|
42
|
+
return finish(proposal.merge(action: 'refused', reason: 'generated module skills are not rewritten')) if generated?(name: proposal[:name], path: proposal[:path])
|
|
43
|
+
return finish(proposal) unless mode == 'auto-safe' && auto_safe?(proposal: proposal, procedure: procedure)
|
|
44
|
+
|
|
45
|
+
apply_update(proposal: proposal, procedure: procedure, evidence: opts[:evidence], skills_root: opts[:skills_root])
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
public_class_method def self.review_turn(opts = {})
|
|
49
|
+
request = opts[:request].to_s
|
|
50
|
+
correction = defined?(Mistakes) && request.match?(Mistakes::CORRECTION_RX)
|
|
51
|
+
mistake = opts[:mistake]
|
|
52
|
+
if mistake.nil? && defined?(Mistakes) && Mistakes.respond_to?(:top)
|
|
53
|
+
rows = Array(Mistakes.top(limit: 8, unresolved_only: false))
|
|
54
|
+
mistake = rows.find { |row| row.is_a?(Hash) && row[:resolved] == true && row[:count].to_i >= 2 }
|
|
55
|
+
mistake = mistake.merge(source: 'mistakes') if mistake
|
|
56
|
+
end
|
|
57
|
+
review(opts.merge(user_correction: correction == true, mistake: mistake, session_id: opts[:session_id], final: opts[:final]))
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
public_class_method def self.note_reuse(opts = {})
|
|
61
|
+
row = {
|
|
62
|
+
name: opts[:name].to_s,
|
|
63
|
+
retrieved: opts[:retrieved] == true,
|
|
64
|
+
reused: opts[:reused] == true,
|
|
65
|
+
regressed: opts[:regressed] == true,
|
|
66
|
+
at: Time.now.utc.iso8601
|
|
67
|
+
}
|
|
68
|
+
append_ledger(row: row)
|
|
69
|
+
row
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
public_class_method def self.authors
|
|
73
|
+
"AUTHOR(S):\n 0day Inc. <support@0dayinc.com>\n"
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
public_class_method def self.help
|
|
77
|
+
puts "USAGE:
|
|
78
|
+
# Review a completed task and recommend or apply a skill change.
|
|
79
|
+
#{self}.review(
|
|
80
|
+
request: 'required - the completed task text',
|
|
81
|
+
mode: 'optional - off, recommend, or auto-safe; default recommend',
|
|
82
|
+
procedure: 'optional - Hash with when, prerequisites, steps, verification, and failures',
|
|
83
|
+
evidence: 'optional - execution proof with source, fixture_passed, and sessions',
|
|
84
|
+
name: 'optional - skill name to prefer',
|
|
85
|
+
success: 'optional - true alone is a routine success and is skipped',
|
|
86
|
+
user_correction: 'optional - true reviews an explicit operator correction',
|
|
87
|
+
skills_root: 'optional - skills directory for the write'
|
|
88
|
+
)
|
|
89
|
+
|
|
90
|
+
# Build a review from the completed turn, including an operator correction or a resolved mistake.
|
|
91
|
+
#{self}.review_turn(
|
|
92
|
+
request: 'required - the completed task text',
|
|
93
|
+
final: 'optional - final answer, not saved as a procedure',
|
|
94
|
+
success: 'optional - true alone does not create a skill',
|
|
95
|
+
session_id: 'optional - session id for the turn',
|
|
96
|
+
mistake: 'optional - resolved mistake Hash with count and source from the mistakes store'
|
|
97
|
+
)
|
|
98
|
+
|
|
99
|
+
# Record whether a later task retrieved or reused a reviewed skill.
|
|
100
|
+
#{self}.note_reuse(
|
|
101
|
+
name: 'required - skill name',
|
|
102
|
+
retrieved: 'optional - true when the catalog returned the skill',
|
|
103
|
+
reused: 'optional - true when the procedure was followed',
|
|
104
|
+
regressed: 'optional - true when reuse made the task worse'
|
|
105
|
+
)
|
|
106
|
+
|
|
107
|
+
# Print the module authors.
|
|
108
|
+
#{self}.authors
|
|
109
|
+
"
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
private_class_method def self.mode_from(opts = {})
|
|
113
|
+
raw = opts[:mode]
|
|
114
|
+
raw = PWN::Env.dig(:ai, :agent, :skill_review) if raw.nil? && defined?(PWN::Env) && PWN::Env.respond_to?(:dig)
|
|
115
|
+
mode = raw.to_s
|
|
116
|
+
mode = 'recommend' if mode.empty? || !MODES.include?(mode)
|
|
117
|
+
mode
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
private_class_method def self.useful?(opts = {})
|
|
121
|
+
return true if opts[:user_correction] == true
|
|
122
|
+
|
|
123
|
+
return true if opts[:procedure].is_a?(Hash) && opts[:evidence].is_a?(Hash)
|
|
124
|
+
|
|
125
|
+
mistake = opts[:mistake]
|
|
126
|
+
mistake.is_a?(Hash) && mistake[:resolved] == true && mistake[:count].to_i >= 2 && mistake[:source].to_s == 'mistakes'
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
private_class_method def self.procedure_from(opts = {})
|
|
130
|
+
raw = opts[:procedure]
|
|
131
|
+
return { ok: false, reason: 'procedure is required' } unless raw.is_a?(Hash)
|
|
132
|
+
|
|
133
|
+
fields = %i[when prerequisites steps verification failures]
|
|
134
|
+
missing = fields.reject { |key| raw[key].to_s.strip.length >= 8 }
|
|
135
|
+
return { ok: false, reason: "procedure missing #{missing.join(', ')}" } unless missing.empty?
|
|
136
|
+
|
|
137
|
+
text = fields.map { |key| "#{key}: #{raw[key].to_s.strip}" }.join("\n")
|
|
138
|
+
return { ok: false, reason: 'secret or target-specific text is not saved' } if sensitive?(text: text)
|
|
139
|
+
return { ok: false, reason: 'raw tool output is not a procedure' } if text.match?(/STDOUT|STDERR/) || text.length > 1200
|
|
140
|
+
|
|
141
|
+
{ ok: true, text: text }
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
private_class_method def self.sensitive?(opts = {})
|
|
145
|
+
opts[:text].to_s.match?(/bearer |password|api_key|BEGIN |token|sk-[A-Za-z0-9]/i) || opts[:text].to_s.match?(/\b\d{1,3}(?:\.\d{1,3}){3}\b/)
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
private_class_method def self.verified_execution?(opts = {})
|
|
149
|
+
evidence = opts[:evidence].is_a?(Hash) ? opts[:evidence] : {}
|
|
150
|
+
evidence[:source].to_s == 'execution' && evidence[:fixture_passed] == true && Array(evidence[:sessions]).map(&:to_s).uniq.length >= 3 && evidence[:model_claimed] != true
|
|
151
|
+
end
|
|
152
|
+
|
|
153
|
+
private_class_method def self.locate(opts = {})
|
|
154
|
+
return nil unless defined?(PWN::Skills) && PWN::Skills.is_a?(Hash)
|
|
155
|
+
|
|
156
|
+
name = opts[:name].to_s
|
|
157
|
+
unless name.empty?
|
|
158
|
+
meta = PWN::Skills[name.to_sym] || PWN::Skills[name]
|
|
159
|
+
return { name: name, path: meta[:path], meta: meta } if meta.is_a?(Hash)
|
|
160
|
+
end
|
|
161
|
+
|
|
162
|
+
tokens = "#{opts[:query]} #{opts[:procedure]}".downcase.scan(/[a-z0-9]{4,}/).uniq
|
|
163
|
+
scored = PWN::Skills.map do |key, meta|
|
|
164
|
+
next unless meta.is_a?(Hash)
|
|
165
|
+
|
|
166
|
+
hay = "#{key} #{meta[:description]}".downcase
|
|
167
|
+
[key.to_s, meta, tokens.count { |tok| hay.include?(tok) }]
|
|
168
|
+
end.compact
|
|
169
|
+
best = scored.max_by { |_, _, score| score }
|
|
170
|
+
return nil unless best && best[2] >= 2
|
|
171
|
+
|
|
172
|
+
{ name: best[0], path: best[1][:path], meta: best[1] }
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
private_class_method def self.generated?(opts = {})
|
|
176
|
+
opts[:name].to_s.start_with?('pwn/') || opts[:path].to_s.include?('/skills/pwn/')
|
|
177
|
+
end
|
|
178
|
+
|
|
179
|
+
private_class_method def self.auto_safe?(opts = {})
|
|
180
|
+
proposal = opts[:proposal] || {}
|
|
181
|
+
procedure = opts[:procedure] || {}
|
|
182
|
+
proposal[:kind] == 'update' && proposal[:verified] == true && procedure[:text].to_s.length <= 400
|
|
183
|
+
end
|
|
184
|
+
|
|
185
|
+
private_class_method def self.apply_update(opts = {})
|
|
186
|
+
proposal = opts[:proposal]
|
|
187
|
+
path = proposal[:path].to_s
|
|
188
|
+
return finish(proposal.merge(action: 'refused', reason: 'skill path missing')) unless File.file?(path) && !File.symlink?(path)
|
|
189
|
+
|
|
190
|
+
original = File.read(path)
|
|
191
|
+
digest = Digest::SHA256.hexdigest(original)
|
|
192
|
+
backup = "#{path}.#{digest}.bak"
|
|
193
|
+
File.open(backup, File::WRONLY | File::CREAT | File::EXCL, 0o600) { |file| file.write(original) } unless File.exist?(backup)
|
|
194
|
+
signature = (opts[:evidence] || {})[:signature].to_s
|
|
195
|
+
lesson = "#{signature}\n#{opts[:procedure][:text]}"
|
|
196
|
+
written = Learning.update_skill(name: proposal[:name], lesson: lesson, query: proposal[:request], pwn_skills_path: opts[:skills_root])
|
|
197
|
+
current = File.read(path)
|
|
198
|
+
unless current.include?(signature) && !signature.empty?
|
|
199
|
+
File.write(path, original)
|
|
200
|
+
return finish(proposal.merge(action: 'refused', reason: 'readback failed', written: written))
|
|
201
|
+
end
|
|
202
|
+
|
|
203
|
+
PWN::Config.load_skills(pwn_skills_path: opts[:skills_root]) if defined?(PWN::Config) && PWN::Config.respond_to?(:load_skills)
|
|
204
|
+
finish(proposal.merge(action: 'applied', backup: backup))
|
|
205
|
+
end
|
|
206
|
+
|
|
207
|
+
private_class_method def self.finish(opts = {})
|
|
208
|
+
row = opts.transform_keys(&:to_sym)
|
|
209
|
+
append_ledger(row: row)
|
|
210
|
+
row
|
|
211
|
+
end
|
|
212
|
+
|
|
213
|
+
private_class_method def self.append_ledger(opts = {})
|
|
214
|
+
path = LEDGER
|
|
215
|
+
FileUtils.mkdir_p(File.dirname(path))
|
|
216
|
+
File.open(path, 'a') { |file| file.puts(JSON.generate(opts[:row])) }
|
|
217
|
+
rescue StandardError
|
|
218
|
+
nil
|
|
219
|
+
end
|
|
220
|
+
end
|
|
221
|
+
end
|
|
222
|
+
end
|
|
223
|
+
end
|
data/lib/pwn/ai/agent.rb
CHANGED
|
@@ -15,6 +15,7 @@ module PWN
|
|
|
15
15
|
autoload :GQRX, 'pwn/ai/agent/gqrx'
|
|
16
16
|
autoload :SAST, 'pwn/ai/agent/sast'
|
|
17
17
|
autoload :SkillConsolidation, 'pwn/ai/agent/skill_consolidation'
|
|
18
|
+
autoload :SkillReview, 'pwn/ai/agent/skill_review'
|
|
18
19
|
autoload :TransparentBrowser, 'pwn/ai/agent/transparent_browser'
|
|
19
20
|
autoload :VulnGen, 'pwn/ai/agent/vuln_gen'
|
|
20
21
|
|
data/lib/pwn/ai/cli.rb
CHANGED
|
@@ -20,6 +20,14 @@ module PWN
|
|
|
20
20
|
options.on('--execute PATH', 'Run an approved YAML task DAG') { |v| result[:execute] = v }
|
|
21
21
|
options.on('--resume RUN_ID', 'Resume a checkpointed DAG run, skipping completed steps') { |v| result[:resume] = v }
|
|
22
22
|
options.on('--mission ID', 'Bind this plan or run to a durable mission') { |v| result[:mission] = v }
|
|
23
|
+
options.on('--policy ACTION', %w[evaluate promote rollback], 'Offline policy: evaluate, promote, rollback (no AI session)') { |v| result[:policy] = v }
|
|
24
|
+
options.on('--baseline PATH', 'Frozen baseline policy JSON') { |v| result[:baseline] = v }
|
|
25
|
+
options.on('--candidate PATH', 'Frozen candidate policy JSON') { |v| result[:candidate] = v }
|
|
26
|
+
options.on('--reports PATH', 'JSON array from --policy evaluate; replayed before promotion') { |v| result[:reports] = v }
|
|
27
|
+
options.on('--receipt PATH', 'Saved successful promotion JSON for rollback') { |v| result[:receipt] = v }
|
|
28
|
+
options.on('--live-policy PATH', 'Explicit existing policy target; no default') { |v| result[:live_path] = v }
|
|
29
|
+
options.on('--approve-policy-change', 'Operator explicitly approves promotion or rollback') { result[:enabled] = true }
|
|
30
|
+
options.on('--policy-writers-stopped', 'Operator attests ALL policy writers are stopped') { result[:quiescent] = true }
|
|
23
31
|
options.on('--ai PROMPT', 'One-shot request; - reads standard input') { |v| result[:ai] = v }
|
|
24
32
|
options.on('--pwn-env PATH', 'Use the specified encrypted configuration') { |v| result[:pwn_env_path] = v }
|
|
25
33
|
options.on('--pwn-dec PATH', 'Use the specified decryptor') { |v| result[:pwn_dec_path] = v }
|
|
@@ -31,6 +39,8 @@ module PWN
|
|
|
31
39
|
raise OptionParser::InvalidArgument, '--ai cannot accompany --replay or --rerun' if result[:ai] && (result[:replay] || result[:rerun])
|
|
32
40
|
raise OptionParser::InvalidArgument, '--plan-only requires --ai' if result[:plan_only] && !result[:ai]
|
|
33
41
|
|
|
42
|
+
validate_policy_options(parsed: result)
|
|
43
|
+
|
|
34
44
|
result[:help_text] = parser.to_s
|
|
35
45
|
result
|
|
36
46
|
end
|
|
@@ -44,6 +54,8 @@ module PWN
|
|
|
44
54
|
end
|
|
45
55
|
|
|
46
56
|
require 'pwn'
|
|
57
|
+
return run_policy(parsed: parsed, output: output) if parsed[:policy]
|
|
58
|
+
|
|
47
59
|
if parsed[:plan_only]
|
|
48
60
|
prompt = parsed[:ai] == '-' ? (opts[:input] || $stdin).read : parsed[:ai]
|
|
49
61
|
raise ArgumentError, '--ai requires a non-empty prompt' if prompt.to_s.strip.empty?
|
|
@@ -97,6 +109,48 @@ module PWN
|
|
|
97
109
|
0
|
|
98
110
|
end
|
|
99
111
|
|
|
112
|
+
private_class_method def self.validate_policy_options(opts = {})
|
|
113
|
+
parsed = opts[:parsed]
|
|
114
|
+
fields = %i[baseline candidate reports receipt live_path enabled quiescent]
|
|
115
|
+
unless parsed[:policy]
|
|
116
|
+
raise OptionParser::InvalidArgument, 'policy options require --policy' if fields.any? { |key| parsed.key?(key) }
|
|
117
|
+
|
|
118
|
+
return
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
allowed = {
|
|
122
|
+
'evaluate' => %i[baseline candidate],
|
|
123
|
+
'promote' => %i[baseline candidate reports live_path enabled quiescent],
|
|
124
|
+
'rollback' => %i[receipt live_path enabled quiescent]
|
|
125
|
+
}.fetch(parsed[:policy])
|
|
126
|
+
unexpected = parsed.keys - allowed - %i[policy help]
|
|
127
|
+
raise OptionParser::InvalidArgument, "options not allowed with --policy #{parsed[:policy]}: #{unexpected.join(', ')}" unless unexpected.empty?
|
|
128
|
+
|
|
129
|
+
missing = (allowed - %i[enabled quiescent]) - parsed.keys
|
|
130
|
+
raise OptionParser::InvalidArgument, "missing policy options: #{missing.join(', ')}" unless missing.empty? || parsed[:help]
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
private_class_method def self.run_policy(opts = {})
|
|
134
|
+
parsed = opts[:parsed]
|
|
135
|
+
evaluator = PWN::AI::Agent::PolicyEvaluation
|
|
136
|
+
args = parsed.slice(:baseline, :candidate, :live_path, :enabled, :quiescent)
|
|
137
|
+
report = case parsed[:policy]
|
|
138
|
+
when 'evaluate'
|
|
139
|
+
[0, 1].map { |seed| evaluator.evaluate(args.merge(seed: seed)) }
|
|
140
|
+
when 'promote', 'rollback'
|
|
141
|
+
raise ArgumentError, 'requires --approve-policy-change and --policy-writers-stopped; stop all policy writers first' unless parsed[:enabled] && parsed[:quiescent]
|
|
142
|
+
|
|
143
|
+
key = parsed[:policy] == 'promote' ? :reports : :receipt
|
|
144
|
+
args[key] = JSON.parse(File.read(parsed.fetch(key)), symbolize_names: true)
|
|
145
|
+
parsed[:policy] == 'promote' ? evaluator.promote(args) : evaluator.rollback(args)
|
|
146
|
+
end
|
|
147
|
+
opts[:output].puts(JSON.generate(report))
|
|
148
|
+
report.is_a?(Hash) && (report[:promoted] == false || report[:rolled_back] == false) ? 1 : 0
|
|
149
|
+
rescue StandardError => e
|
|
150
|
+
opts[:output].puts(JSON.generate(error: e.message))
|
|
151
|
+
1
|
|
152
|
+
end
|
|
153
|
+
|
|
100
154
|
private_class_method def self.bind_mission!(opts = {})
|
|
101
155
|
report = opts[:report] || {}
|
|
102
156
|
return report unless report[:run_id]
|
data/lib/pwn/config.rb
CHANGED
|
@@ -161,6 +161,8 @@ module PWN
|
|
|
161
161
|
max_depth: 3,
|
|
162
162
|
# run PWN::AI::Agent::Learning.auto_introspect after every final answer
|
|
163
163
|
auto_introspect: true,
|
|
164
|
+
# off, recommend, or auto-safe. A missing key means recommend.
|
|
165
|
+
skill_review: 'recommend',
|
|
164
166
|
# also run PWN::AI::Agent::Extrospection.auto_extrospect from auto_introspect
|
|
165
167
|
# (host/repo/env probes only — no toolchain/GUI/net side-effects)
|
|
166
168
|
auto_extrospect: true,
|
data/lib/pwn/migrate.rb
CHANGED
|
@@ -49,7 +49,7 @@ module PWN
|
|
|
49
49
|
# Bump this whenever the shape of any file under ~/.pwn changes in a
|
|
50
50
|
# way that requires a one-time transform. Add the transform as an
|
|
51
51
|
# entry in MIGRATIONS keyed by the NEW schema number.
|
|
52
|
-
SCHEMA_VERSION =
|
|
52
|
+
SCHEMA_VERSION = 5
|
|
53
53
|
|
|
54
54
|
OK = "\e[32mok\e[0m"
|
|
55
55
|
BAD = "\e[31mFAIL\e[0m"
|
|
@@ -245,6 +245,10 @@ module PWN
|
|
|
245
245
|
4 => lambda { |_root, io|
|
|
246
246
|
io.puts ' · pwn.yaml ai_sandbox, ai_router, model_routes task classes'
|
|
247
247
|
PWN::Migrate.backfill_vault(io: io)
|
|
248
|
+
},
|
|
249
|
+
5 => lambda { |_root, io|
|
|
250
|
+
io.puts ' · pwn.yaml ai.agent.skill_review'
|
|
251
|
+
PWN::Migrate.backfill_vault(io: io)
|
|
248
252
|
}
|
|
249
253
|
}.freeze
|
|
250
254
|
|
|
@@ -1654,7 +1654,7 @@ module PWN
|
|
|
1654
1654
|
if PWN.const_defined?(:MeshMutex) && PWN.const_defined?(:MeshRxBodyWin)
|
|
1655
1655
|
mutex = PWN.const_get(:MeshMutex)
|
|
1656
1656
|
state = PWN.const_defined?(:MeshRxState) ? PWN.const_get(:MeshRxState) : {}
|
|
1657
|
-
ts = Time.now.strftime('%H:%M:%S')
|
|
1657
|
+
ts = Time.now.strftime('%Y-%m-%d %H:%M:%S%z')
|
|
1658
1658
|
color = opts[:local] ? 23 : 21
|
|
1659
1659
|
secure = packet[:pki_encrypted] == true || mesh_channel_securely_encrypted?(env: env, channel: channel_name)
|
|
1660
1660
|
security_icon = secure ? '🔒' : '🔍'
|
data/lib/pwn/version.rb
CHANGED
|
@@ -34,6 +34,11 @@ describe 'RSI exploit and attack rates' do
|
|
|
34
34
|
end
|
|
35
35
|
|
|
36
36
|
it 'writes an RSI lesson when the measured exploit rate falls' do
|
|
37
|
+
expect(PWN::AI::Agent::PolicyEvaluation).not_to receive(:evaluate)
|
|
38
|
+
expect(PWN::AI::Agent::PolicyEvaluation).not_to receive(:promote)
|
|
39
|
+
expect(PWN::AI::Agent::PolicyEvaluation).not_to receive(:rollback)
|
|
40
|
+
expect(PWN::AI::Agent::Loop).not_to receive(:run)
|
|
41
|
+
expect(PWN::AI::Agent::Dispatch).not_to receive(:call)
|
|
37
42
|
PWN::AI::Agent::Metrics.record_attempt(kind: 'exploit', tool: 'ret2libc', vulnerable_tool: 'libc', vulnerable: true, success: true)
|
|
38
43
|
PWN::AI::Agent::Learning.rsi_tick(request: 'measure exploit rate')
|
|
39
44
|
PWN::AI::Agent::Metrics.record_attempt(kind: 'exploit', tool: 'rop', vulnerable_tool: 'nginx', vulnerable: true, success: false)
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'spec_helper'
|
|
4
|
+
require 'tmpdir'
|
|
5
|
+
require 'json'
|
|
6
|
+
|
|
7
|
+
describe PWN::AI::Agent::SkillReview do
|
|
8
|
+
let(:dir) { Dir.mktmpdir('pwn-skill-review-') }
|
|
9
|
+
let(:skill_path) { File.join(dir, 'osint', 'SKILL.md') }
|
|
10
|
+
|
|
11
|
+
before do
|
|
12
|
+
FileUtils.mkdir_p(File.dirname(skill_path))
|
|
13
|
+
File.write(skill_path, "---\nname: osint\ndescription: Public source checks.\n---\n\n# OSINT\n\nUse public sources.\n")
|
|
14
|
+
stub_const('PWN::Skills', {
|
|
15
|
+
osint: {
|
|
16
|
+
description: 'Public source checks for domains and hosts.',
|
|
17
|
+
content: File.read(skill_path),
|
|
18
|
+
path: skill_path
|
|
19
|
+
}
|
|
20
|
+
})
|
|
21
|
+
stub_const('PWN::AI::Agent::SkillReview::LEDGER', File.join(dir, 'skill_review.jsonl'))
|
|
22
|
+
allow(PWN::Config).to receive(:load_skills)
|
|
23
|
+
allow(PWN::AI::Agent::Learning).to receive(:update_skill).and_call_original
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
after { FileUtils.remove_entry(dir) if dir && Dir.exist?(dir) }
|
|
27
|
+
|
|
28
|
+
def procedure
|
|
29
|
+
{
|
|
30
|
+
when: 'a public domain check needs a saved query order',
|
|
31
|
+
prerequisites: 'the domain is already in scope',
|
|
32
|
+
steps: 'query the public feed, then record the response code',
|
|
33
|
+
verification: 'the fixture command exits 0',
|
|
34
|
+
failures: 'an empty domain skips the feed'
|
|
35
|
+
}
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def evidence
|
|
39
|
+
{
|
|
40
|
+
source: 'execution',
|
|
41
|
+
fixture_passed: true,
|
|
42
|
+
sessions: %w[s1 s2 s3],
|
|
43
|
+
signature: 'osint-order'
|
|
44
|
+
}
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
it 'does not create a skill from a routine success' do
|
|
48
|
+
result = described_class.review(request: 'check the lab host', success: true, mode: 'recommend')
|
|
49
|
+
expect(result[:action]).to eq('skipped')
|
|
50
|
+
expect(result[:reason]).to include('routine')
|
|
51
|
+
expect(File).not_to exist(File.join(dir, 'new-skill', 'SKILL.md'))
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
it 'recommends an update to the closest skill and does not write in recommend mode' do
|
|
55
|
+
result = described_class.review(
|
|
56
|
+
request: 'domain public source check',
|
|
57
|
+
mode: 'recommend',
|
|
58
|
+
procedure: procedure,
|
|
59
|
+
evidence: evidence
|
|
60
|
+
)
|
|
61
|
+
expect(result[:action]).to eq('recommend')
|
|
62
|
+
expect(result[:kind]).to eq('update')
|
|
63
|
+
expect(result[:name]).to eq('osint')
|
|
64
|
+
expect(File.read(skill_path)).not_to include('osint-order')
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
it 'auto-safe appends a small verified procedure and keeps a backup' do
|
|
68
|
+
allow(PWN::Config).to receive(:write_skill) do |opts|
|
|
69
|
+
body = opts[:content]
|
|
70
|
+
File.write(skill_path, body.include?('---') ? body : "---\nname: osint\ndescription: Public source checks.\n---\n\n#{body}")
|
|
71
|
+
{ name: 'osint', path: skill_path }
|
|
72
|
+
end
|
|
73
|
+
result = described_class.review(
|
|
74
|
+
request: 'domain public source check',
|
|
75
|
+
mode: 'auto-safe',
|
|
76
|
+
procedure: procedure,
|
|
77
|
+
evidence: evidence,
|
|
78
|
+
skills_root: dir
|
|
79
|
+
)
|
|
80
|
+
expect(result[:action]).to eq('applied')
|
|
81
|
+
expect(File.read(skill_path)).to include('osint-order')
|
|
82
|
+
expect(Dir.glob("#{skill_path}.*.bak")).not_to be_empty
|
|
83
|
+
expect(PWN::Config).to have_received(:load_skills).at_least(:once)
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
it 'refuses auto-safe creation, secrets, generated module skills, and model-only proof' do
|
|
87
|
+
stub_const('PWN::Skills', {})
|
|
88
|
+
created = described_class.review(request: 'new procedure', mode: 'auto-safe', procedure: procedure, evidence: evidence, name: 'fresh-check')
|
|
89
|
+
expect(created[:action]).to eq('recommend')
|
|
90
|
+
expect(created[:kind]).to eq('create')
|
|
91
|
+
|
|
92
|
+
secret = described_class.review(
|
|
93
|
+
request: 'domain public source check',
|
|
94
|
+
mode: 'auto-safe',
|
|
95
|
+
procedure: procedure.merge(steps: 'send Authorization bearer secret-token'),
|
|
96
|
+
evidence: evidence
|
|
97
|
+
)
|
|
98
|
+
expect(secret[:action]).to eq('refused')
|
|
99
|
+
|
|
100
|
+
generated = described_class.review(
|
|
101
|
+
request: 'nmap scan diff',
|
|
102
|
+
mode: 'auto-safe',
|
|
103
|
+
name: 'pwn/plugins/nmap_it',
|
|
104
|
+
procedure: procedure,
|
|
105
|
+
evidence: evidence
|
|
106
|
+
)
|
|
107
|
+
expect(generated[:action]).to eq('refused')
|
|
108
|
+
|
|
109
|
+
claimed = described_class.review(
|
|
110
|
+
request: 'domain public source check',
|
|
111
|
+
mode: 'auto-safe',
|
|
112
|
+
procedure: procedure,
|
|
113
|
+
evidence: { source: 'model', model_claimed: true, fixture_passed: true, sessions: %w[s1 s2 s3] }
|
|
114
|
+
)
|
|
115
|
+
expect(claimed[:action]).to eq('recommend')
|
|
116
|
+
expect(claimed[:verified]).to eq(false)
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
it 'recommends a correction without writing a transcript' do
|
|
120
|
+
stub_const('PWN::AI::Agent::Mistakes', Class.new)
|
|
121
|
+
allow(PWN::AI::Agent::Mistakes).to receive(:top).and_return([])
|
|
122
|
+
stub_const('PWN::AI::Agent::Mistakes::CORRECTION_RX', /wrong/i)
|
|
123
|
+
result = described_class.review_turn(request: 'no, that is wrong, save the public query order', success: false, mode: 'auto-safe')
|
|
124
|
+
expect(result[:action]).to eq('recommend')
|
|
125
|
+
expect(File.read(skill_path)).not_to include('wrong')
|
|
126
|
+
end
|
|
127
|
+
end
|
data/spec/lib/pwn/ai/cli_spec.rb
CHANGED
|
@@ -14,6 +14,114 @@ RSpec.describe 'PWN::AI::CLI' do
|
|
|
14
14
|
expect(PWN::AI::CLI.parse(argv: ['--rerun', 'session-1'])).to include(rerun: 'session-1')
|
|
15
15
|
end
|
|
16
16
|
|
|
17
|
+
it 'evaluates two frozen offline suites without loading configuration or running the loop' do
|
|
18
|
+
Dir.mktmpdir('pwn-cli-policy-', '/tmp') do |root|
|
|
19
|
+
snapshot = File.join(root, 'snapshot.json')
|
|
20
|
+
File.write(snapshot, JSON.generate(q: {}, h: {}, visits: {}, returns: [], n_updates: 0, td_abs_sum: 0.0))
|
|
21
|
+
expect(PWN::Config).not_to receive(:refresh_env)
|
|
22
|
+
expect(PWN::AI::Agent::Loop).not_to receive(:run)
|
|
23
|
+
out = StringIO.new
|
|
24
|
+
expect(PWN::AI::CLI.run(argv: ['--policy', 'evaluate', '--baseline', snapshot, '--candidate', snapshot], output: out)).to eq(0)
|
|
25
|
+
reports = JSON.parse(out.string)
|
|
26
|
+
expect(reports.map { |report| report['seed'] }).to eq([0, 1])
|
|
27
|
+
expect(reports).to all(include('protocol' => 'pwn-policy-heldout-v2'))
|
|
28
|
+
stdout, stderr, status = Open3.capture3(
|
|
29
|
+
{ 'HOME' => root, 'LANG' => 'C.UTF-8' }, RbConfig.ruby, '-Ilib', 'bin/pwn-ai',
|
|
30
|
+
'--policy', 'evaluate', '--baseline', snapshot, '--candidate', snapshot,
|
|
31
|
+
unsetenv_others: true
|
|
32
|
+
)
|
|
33
|
+
expect(status.success?).to be(true), stderr
|
|
34
|
+
expect(JSON.parse(stdout).map { |report| report['seed'] }).to eq([0, 1])
|
|
35
|
+
expect(Dir.children(root)).to eq(['snapshot.json'])
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
it 'promotes only with both operator acknowledgements and rolls back using the saved receipt' do
|
|
40
|
+
Dir.mktmpdir('pwn-cli-policy-', '/tmp') do |root|
|
|
41
|
+
output, error, status = Open3.capture3(
|
|
42
|
+
{ 'HOME' => root, 'LANG' => 'C.UTF-8' }, RbConfig.ruby,
|
|
43
|
+
File.expand_path('../../../../scripts/benchmark_policy.rb', __dir__),
|
|
44
|
+
'--heldout', '--snapshot-dir', File.join(root, 'snapshots'), unsetenv_others: true
|
|
45
|
+
)
|
|
46
|
+
expect(status.success?).to be(true), error
|
|
47
|
+
baseline = File.join(root, 'snapshots', 'off.json')
|
|
48
|
+
candidate = File.join(root, 'snapshots', 'on.json')
|
|
49
|
+
reports = File.join(root, 'reports.json')
|
|
50
|
+
File.write(reports, JSON.generate(JSON.parse(output).fetch('heldout')))
|
|
51
|
+
live = File.join(root, 'live.json')
|
|
52
|
+
FileUtils.cp(baseline, live)
|
|
53
|
+
argv = ['--policy', 'promote', '--baseline', baseline, '--candidate', candidate, '--reports', reports, '--live-policy', live]
|
|
54
|
+
expect(PWN::Config).not_to receive(:refresh_env)
|
|
55
|
+
expect(PWN::AI::Agent::Loop).not_to receive(:run)
|
|
56
|
+
[[], ['--approve-policy-change'], ['--policy-writers-stopped']].each do |flags|
|
|
57
|
+
expect(PWN::AI::CLI.run(argv: argv + flags, output: StringIO.new)).to eq(1)
|
|
58
|
+
expect(File.binread(live)).to eq(File.binread(baseline))
|
|
59
|
+
end
|
|
60
|
+
approval = %w[--approve-policy-change --policy-writers-stopped]
|
|
61
|
+
original_reports = File.read(reports)
|
|
62
|
+
tampered = JSON.parse(original_reports)
|
|
63
|
+
tampered.first['passed'] = true
|
|
64
|
+
File.write(reports, JSON.generate(tampered))
|
|
65
|
+
rejected = StringIO.new
|
|
66
|
+
expect(PWN::AI::CLI.run(argv: argv + approval, output: rejected)).to eq(1)
|
|
67
|
+
expect(JSON.parse(rejected.string)).to include('promoted' => false, 'reason' => 'provenance/artifact replay mismatch')
|
|
68
|
+
expect(File.binread(live)).to eq(File.binread(baseline))
|
|
69
|
+
File.write(reports, original_reports)
|
|
70
|
+
out = StringIO.new
|
|
71
|
+
expect(PWN::AI::CLI.run(argv: argv + approval, output: out)).to eq(0)
|
|
72
|
+
expect(JSON.parse(out.string)).to include('promoted' => true, 'replayed_seeds' => [0, 1])
|
|
73
|
+
expect(File.binread(live)).to eq(File.binread(candidate))
|
|
74
|
+
receipt = File.join(root, 'receipt.json')
|
|
75
|
+
File.write(receipt, out.string)
|
|
76
|
+
rollback = ['--policy', 'rollback', '--receipt', receipt, '--live-policy', live]
|
|
77
|
+
expect(PWN::AI::CLI.run(argv: rollback, output: StringIO.new)).to eq(1)
|
|
78
|
+
expect(File.binread(live)).to eq(File.binread(candidate))
|
|
79
|
+
File.write(live, File.read(baseline))
|
|
80
|
+
expect(PWN::AI::CLI.run(argv: rollback + approval, output: StringIO.new)).to eq(1)
|
|
81
|
+
File.write(live, File.read(candidate))
|
|
82
|
+
expect(PWN::AI::CLI.run(argv: rollback + approval, output: StringIO.new)).to eq(0)
|
|
83
|
+
expect(File.binread(live)).to eq(File.binread(baseline))
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
it 'rejects policy flags outside their action and never combines offline evaluation with a live session' do
|
|
88
|
+
[
|
|
89
|
+
%w[--baseline baseline.json],
|
|
90
|
+
%w[--approve-policy-change],
|
|
91
|
+
%w[--policy evaluate --baseline b --candidate c --ai prompt],
|
|
92
|
+
%w[--policy evaluate --baseline b --candidate c --replay session],
|
|
93
|
+
%w[--policy evaluate --baseline b --candidate c --mission mission],
|
|
94
|
+
%w[--policy evaluate --baseline b --candidate c --pwn-env vault],
|
|
95
|
+
%w[--policy evaluate --baseline b --candidate c --live-policy live],
|
|
96
|
+
%w[--policy evaluate --baseline b --candidate c --approve-policy-change],
|
|
97
|
+
%w[--policy evaluate --baseline b],
|
|
98
|
+
%w[--policy rollback --receipt r],
|
|
99
|
+
%w[--policy promote --baseline b --candidate c --reports r],
|
|
100
|
+
%w[--policy rollback --receipt r --live-policy l --candidate c]
|
|
101
|
+
].each do |argv|
|
|
102
|
+
expect { PWN::AI::CLI.parse(argv: argv) }.to raise_error(OptionParser::ParseError), argv.inspect
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
it 'returns nonzero JSON diagnostics for malformed offline inputs without starting a session' do
|
|
107
|
+
Dir.mktmpdir('pwn-cli-policy-', '/tmp') do |root|
|
|
108
|
+
path = File.join(root, 'invalid.json')
|
|
109
|
+
File.write(path, 'not json')
|
|
110
|
+
expect(PWN::Config).not_to receive(:refresh_env)
|
|
111
|
+
expect(PWN::Sessions).not_to receive(:create)
|
|
112
|
+
expect(PWN::AI::Agent::Loop).not_to receive(:run)
|
|
113
|
+
[
|
|
114
|
+
['--policy', 'evaluate', '--baseline', path, '--candidate', path],
|
|
115
|
+
['--policy', 'rollback', '--receipt', path, '--live-policy', path, '--approve-policy-change', '--policy-writers-stopped']
|
|
116
|
+
].each do |argv|
|
|
117
|
+
out = StringIO.new
|
|
118
|
+
expect(PWN::AI::CLI.run(argv: argv, output: out)).to eq(1)
|
|
119
|
+
expect(JSON.parse(out.string)).to have_key('error')
|
|
120
|
+
expect(File.read(path)).to eq('not json')
|
|
121
|
+
end
|
|
122
|
+
end
|
|
123
|
+
end
|
|
124
|
+
|
|
17
125
|
it 'rejects ambiguous actions and stray positional arguments' do
|
|
18
126
|
[%w[--replay one --rerun two], %w[--analyze file --replay one], %w[--replay one --ai prompt], ['stray']].each do |argv|
|
|
19
127
|
expect { PWN::AI::CLI.parse(argv: argv) }.to raise_error(OptionParser::ParseError)
|
|
@@ -80,7 +80,7 @@ describe PWN::Migrate do
|
|
|
80
80
|
File.write(File.join(@tmp, '.schema'), JSON.generate(schema: 2))
|
|
81
81
|
expect(described_class.needed?).to be(true)
|
|
82
82
|
result = described_class.run(fix: false, backup: false, io: io)
|
|
83
|
-
expect(result[:applied_migrations]).to eq([3, 4])
|
|
83
|
+
expect(result[:applied_migrations]).to eq([3, 4, 5])
|
|
84
84
|
expected = Marshal.load(Marshal.dump(original))
|
|
85
85
|
expected['custom']['model'] = nil
|
|
86
86
|
expect(YAML.safe_load_file(path)).to eq(expected)
|
|
@@ -189,7 +189,7 @@ describe PWN::Migrate do
|
|
|
189
189
|
File.write(File.join(@tmp, '.schema'), JSON.generate(schema: 3))
|
|
190
190
|
expect(described_class.needed?).to be(true)
|
|
191
191
|
result = described_class.run(fix: true, backup: false, io: io)
|
|
192
|
-
expect(result[:applied_migrations]).to eq([4])
|
|
192
|
+
expect(result[:applied_migrations]).to eq([4, 5])
|
|
193
193
|
creds = YAML.safe_load_file(dec, symbolize_names: true)
|
|
194
194
|
user = PWN::Plugins::Vault.dump(file: yaml, key: creds[:key], iv: creds[:iv])
|
|
195
195
|
expect(user[:ai][:active]).to eq('grok')
|
|
@@ -206,6 +206,33 @@ describe PWN::Migrate do
|
|
|
206
206
|
expect(described_class.needed?).to be(false)
|
|
207
207
|
end
|
|
208
208
|
|
|
209
|
+
it 'upgrades a schema-4 vault with skill_review without changing a user setting' do
|
|
210
|
+
yaml = File.join(@tmp, 'pwn.yaml')
|
|
211
|
+
dec = File.join(@tmp, 'pwn.yaml.decryptor')
|
|
212
|
+
stale = { ai: { active: 'grok', grok: { key: 'sk-KEEP' }, agent: { skill_review: 'off', max_iters: 3 } } }
|
|
213
|
+
missing = { ai: { active: 'openai', openai: { key: 'sk-OTHER' }, agent: { max_iters: 9 } } }
|
|
214
|
+
File.write(yaml, YAML.dump(stale).gsub(/^(\s*):/, '\1'))
|
|
215
|
+
PWN::Plugins::Vault.create(file: yaml, decryptor_file: dec)
|
|
216
|
+
File.write(File.join(@tmp, '.schema'), JSON.generate(schema: 4))
|
|
217
|
+
result = described_class.run(fix: true, backup: false, io: io)
|
|
218
|
+
expect(result[:applied_migrations]).to eq([5])
|
|
219
|
+
creds = YAML.safe_load_file(dec, symbolize_names: true)
|
|
220
|
+
user = PWN::Plugins::Vault.dump(file: yaml, key: creds[:key], iv: creds[:iv])
|
|
221
|
+
expect(user.dig(:ai, :agent, :skill_review)).to eq('off')
|
|
222
|
+
expect(user.dig(:ai, :agent, :max_iters)).to eq(3)
|
|
223
|
+
expect(user.dig(:ai, :grok, :key)).to eq('sk-KEEP')
|
|
224
|
+
|
|
225
|
+
File.write(yaml, YAML.dump(missing).gsub(/^(\s*):/, '\1'))
|
|
226
|
+
PWN::Plugins::Vault.encrypt(file: yaml, key: creds[:key], iv: creds[:iv])
|
|
227
|
+
File.write(File.join(@tmp, '.schema'), JSON.generate(schema: 4))
|
|
228
|
+
described_class.run(fix: true, backup: false, io: io)
|
|
229
|
+
filled = PWN::Plugins::Vault.dump(file: yaml, key: creds[:key], iv: creds[:iv])
|
|
230
|
+
expect(filled.dig(:ai, :agent, :skill_review)).to eq('recommend')
|
|
231
|
+
expect(filled.dig(:ai, :agent, :max_iters)).to eq(9)
|
|
232
|
+
expect(filled.dig(:ai, :openai, :key)).to eq('sk-OTHER')
|
|
233
|
+
expect(described_class.run(fix: true, backup: false, io: io)[:applied_migrations]).to eq([])
|
|
234
|
+
end
|
|
235
|
+
|
|
209
236
|
it 'dry_run writes nothing' do
|
|
210
237
|
r = PWN::Migrate.run(fix: true, backup: true, dry_run: true, io: io)
|
|
211
238
|
expect(r[:dry_run]).to be true
|
data/third_party/pwn_rdoc.jsonl
CHANGED
|
@@ -826,6 +826,22 @@
|
|
|
826
826
|
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillConsolidation.consolidate Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillConsolidation.consolidate`: "}]}
|
|
827
827
|
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillConsolidation.consolidate_file Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillConsolidation.consolidate_file`: "}]}
|
|
828
828
|
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillConsolidation.help Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillConsolidation.help`: "}]}
|
|
829
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.append_ledger Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.append_ledger`: "}]}
|
|
830
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.apply_update Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.apply_update`: "}]}
|
|
831
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.authors Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.authors`: "}]}
|
|
832
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.auto_safe? Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.auto_safe?`: "}]}
|
|
833
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.finish Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.finish`: "}]}
|
|
834
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.generated? Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.generated?`: "}]}
|
|
835
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.help Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.help`: "}]}
|
|
836
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.locate Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.locate`: "}]}
|
|
837
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.mode_from Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.mode_from`: "}]}
|
|
838
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.note_reuse Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.note_reuse`: "}]}
|
|
839
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.procedure_from Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.procedure_from`: "}]}
|
|
840
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.review Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.review`: "}]}
|
|
841
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.review_turn Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.review_turn`: "}]}
|
|
842
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.sensitive? Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.sensitive?`: "}]}
|
|
843
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.useful? Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.useful?`: "}]}
|
|
844
|
+
{"messages":[{"role":"user","content":"PWN::AI::Agent::SkillReview.verified_execution? Usage"},{"role":"assistant","content":"`PWN::AI::Agent::SkillReview.verified_execution?`: "}]}
|
|
829
845
|
{"messages":[{"role":"user","content":"PWN::AI::Agent::Swarm.ask Usage"},{"role":"assistant","content":"`PWN::AI::Agent::Swarm.ask`: Supported Method Parameters\n\nreply = PWN::AI::Agent::Swarm.ask(\n\nname: 'required - persona name from ~/.pwn/agents.yml',\nrequest: 'required - what to ask/instruct the persona',\nswarm_id: 'optional - join an existing swarm (created if omitted)',\nto: 'optional - addressee recorded on the bus (default :all)',\non_tool: 'optional - ->(name, args, result) live-UI callback'\n\n)\n"}]}
|
|
830
846
|
{"messages":[{"role":"user","content":"PWN::AI::Agent::Swarm.authors Usage"},{"role":"assistant","content":"`PWN::AI::Agent::Swarm.authors`: Author(s)\n\n0day Inc. <support@0dayinc.com>\n"}]}
|
|
831
847
|
{"messages":[{"role":"user","content":"PWN::AI::Agent::Swarm.broadcast Usage"},{"role":"assistant","content":"`PWN::AI::Agent::Swarm.broadcast`: Supported Method Parameters\n\nresult = PWN::AI::Agent::Swarm.broadcast(\n\nrequest: 'required', names: 'optional - default all personas',\nswarm_id: 'optional'\n\n)\n"}]}
|
|
@@ -1052,6 +1068,8 @@
|
|
|
1052
1068
|
{"messages":[{"role":"user","content":"PWN::AI::CLI.help Usage"},{"role":"assistant","content":"`PWN::AI::CLI.help`: "}]}
|
|
1053
1069
|
{"messages":[{"role":"user","content":"PWN::AI::CLI.parse Usage"},{"role":"assistant","content":"`PWN::AI::CLI.parse`: "}]}
|
|
1054
1070
|
{"messages":[{"role":"user","content":"PWN::AI::CLI.run Usage"},{"role":"assistant","content":"`PWN::AI::CLI.run`: "}]}
|
|
1071
|
+
{"messages":[{"role":"user","content":"PWN::AI::CLI.run_policy Usage"},{"role":"assistant","content":"`PWN::AI::CLI.run_policy`: "}]}
|
|
1072
|
+
{"messages":[{"role":"user","content":"PWN::AI::CLI.validate_policy_options Usage"},{"role":"assistant","content":"`PWN::AI::CLI.validate_policy_options`: "}]}
|
|
1055
1073
|
{"messages":[{"role":"user","content":"PWN::AI::Context.attach_disasm Usage"},{"role":"assistant","content":"`PWN::AI::Context.attach_disasm`: "}]}
|
|
1056
1074
|
{"messages":[{"role":"user","content":"PWN::AI::Context.attach_file Usage"},{"role":"assistant","content":"`PWN::AI::Context.attach_file`: "}]}
|
|
1057
1075
|
{"messages":[{"role":"user","content":"PWN::AI::Context.attach_hexdump Usage"},{"role":"assistant","content":"`PWN::AI::Context.attach_hexdump`: "}]}
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pwn
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.5.
|
|
4
|
+
version: 0.5.745
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- 0day Inc.
|
|
@@ -4958,6 +4958,7 @@ files:
|
|
|
4958
4958
|
- etc/default_skills/pwn/ai/agent/reward/SKILL.md
|
|
4959
4959
|
- etc/default_skills/pwn/ai/agent/sast/SKILL.md
|
|
4960
4960
|
- etc/default_skills/pwn/ai/agent/skill_consolidation/SKILL.md
|
|
4961
|
+
- etc/default_skills/pwn/ai/agent/skill_review/SKILL.md
|
|
4961
4962
|
- etc/default_skills/pwn/ai/agent/swarm/SKILL.md
|
|
4962
4963
|
- etc/default_skills/pwn/ai/agent/task_dag/SKILL.md
|
|
4963
4964
|
- etc/default_skills/pwn/ai/agent/task_summarizer/SKILL.md
|
|
@@ -6010,6 +6011,7 @@ files:
|
|
|
6010
6011
|
- lib/pwn/ai/agent/reward.rb
|
|
6011
6012
|
- lib/pwn/ai/agent/sast.rb
|
|
6012
6013
|
- lib/pwn/ai/agent/skill_consolidation.rb
|
|
6014
|
+
- lib/pwn/ai/agent/skill_review.rb
|
|
6013
6015
|
- lib/pwn/ai/agent/swarm.rb
|
|
6014
6016
|
- lib/pwn/ai/agent/task_dag.rb
|
|
6015
6017
|
- lib/pwn/ai/agent/task_summarizer.rb
|
|
@@ -6676,6 +6678,7 @@ files:
|
|
|
6676
6678
|
- spec/lib/pwn/ai/agent/scoreboard_roadmap_spec.rb
|
|
6677
6679
|
- spec/lib/pwn/ai/agent/signal_hygiene_spec.rb
|
|
6678
6680
|
- spec/lib/pwn/ai/agent/skill_consolidation_spec.rb
|
|
6681
|
+
- spec/lib/pwn/ai/agent/skill_review_spec.rb
|
|
6679
6682
|
- spec/lib/pwn/ai/agent/swarm_roster_spec.rb
|
|
6680
6683
|
- spec/lib/pwn/ai/agent/swarm_spec.rb
|
|
6681
6684
|
- spec/lib/pwn/ai/agent/task_dag_spec.rb
|