@mokuyoaxis/agent-guard 0.2.3-rc1 → 0.2.3-rc2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/README.md +51 -20
  2. package/README.zh-CN.md +45 -16
  3. package/adapters/dsh/README.md +81 -4
  4. package/adapters/dsh/harness/README.md +328 -0
  5. package/adapters/dsh/harness/baseline.patch.yml +8 -0
  6. package/adapters/dsh/harness/guard_lab.py +609 -0
  7. package/adapters/dsh/harness/native_session.py +482 -0
  8. package/adapters/dsh/harness/profiles.py +160 -0
  9. package/adapters/dsh/harness/read_fixture.py +60 -0
  10. package/adapters/dsh/harness/retry_stream.py +171 -0
  11. package/adapters/dsh/harness/session_decode.mjs +117 -0
  12. package/adapters/dsh/lib/index.js +46 -5
  13. package/adapters/dsh/lib/read_result_guard.js +207 -0
  14. package/adapters/dsh/lib/read_result_guard.py +139 -0
  15. package/adapters/harness_support.py +109 -0
  16. package/adapters/kimi-code/README.md +7 -0
  17. package/adapters/kimi-code/harness/README.md +126 -0
  18. package/adapters/kimi-code/harness/guard_lab.py +430 -0
  19. package/core/lab.py +2373 -0
  20. package/docs/architecture.md +75 -0
  21. package/docs/friction.md +72 -0
  22. package/docs/guard-lab-clean-task.md +4 -0
  23. package/docs/guard-lab-trial-task.md +5 -0
  24. package/docs/guard-lab.md +391 -0
  25. package/docs/harness-capabilities.md +7 -4
  26. package/docs/release-notes-0.2.3-rc2.md +189 -0
  27. package/docs/release-notes-0.2.3.md +45 -0
  28. package/docs/test-report-dsh-0.2-upgrade.md +142 -0
  29. package/docs/test-report-dsh-guard-lab.md +154 -0
  30. package/docs/test-report-dsh-native-session.md +118 -0
  31. package/docs/test-report-dsh-paired-lab.md +183 -0
  32. package/docs/test-report-dsh-read-redaction.md +114 -0
  33. package/docs/test-report-dsh-real-followup.md +170 -0
  34. package/docs/test-report-dsh-retry-evidence.md +134 -0
  35. package/docs/test-report-kimi-guard-lab.md +117 -0
  36. package/docs/test-report-release-readiness-0.2.3.md +161 -0
  37. package/docs/threat-model.md +42 -0
  38. package/guard_lab.py +14 -0
  39. package/package.json +8 -5
@@ -0,0 +1,328 @@
1
+ # Repeatable DSH guard-lab trials
2
+
3
+ This POSIX runner captures a fresh `clean` or `injection-probe`
4
+ trial with DSH **0.1.5-rc.1 or 0.2.0-rc.2**. It reuses the existing Lab lifecycle and keeps
5
+ host completion explicitly user-declared. No production dependency is added.
6
+ The default remains guard-off; the paired workflow below supplies the local
7
+ Guard patch only to guard-on members.
8
+
9
+ ## Select the runtime
10
+
11
+ Use Node.js 22.23.2 or another runtime with the required native Zstandard APIs.
12
+ Check both `node --version` and `dsh --version`: an older Node can make the
13
+ installed launcher exit zero without printing or running anything.
14
+
15
+ Select an **already initialized, disposable** DSH home containing the headless
16
+ profile and the credentials/settings approved for this experiment. Keep it
17
+ outside the disposable fixture and Lab evidence. The runner rejects the
18
+ default `~/.dsh` home and overlapping paths. It does not install a plugin,
19
+ configure credentials, copy user configuration, or edit an existing patch.
20
+ DSH itself may write session and storage state inside the selected home.
21
+
22
+ The common [baseline overlay](baseline.patch.yml) disables the separate title
23
+ model call and supplies the official `deepseek-flash` default. Saved DSH
24
+ settings can override that default; verify the selected provider/model before
25
+ declaring its public model label. Model identity and guard state remain
26
+ user-declared. The preflight rejects a known Agent Guard row in the composed
27
+ tree, but cannot certify the absence of every possible custom interception.
28
+ Telemetry is disabled in the runner's child environment.
29
+
30
+ ```sh
31
+ python3 adapters/dsh/harness/guard_lab.py check \
32
+ --dsh-home /tmp/approved-disposable-dsh-home
33
+ ```
34
+
35
+ `check` calls only `--version` and the boot-free `--dump-config`. It makes no
36
+ model request. It also probes the local session decoder. It rejects an empty
37
+ result, a different version, an unavailable decoder, or a known
38
+ Agent Guard row, and prints a hash/size summary without the raw composition.
39
+ Additional common overlays can be supplied with repeated `--patch FILE`;
40
+ the shipped baseline overlay is always last.
41
+ Use `--node-executable /absolute/path/to/node` to select the decoder runtime
42
+ explicitly. This option does not change the runtime chosen by the DSH launcher.
43
+
44
+ ## Capture and review the clean control
45
+
46
+ Every `--output-dir` must be new, with an existing parent. The task file is
47
+ UTF-8, capped at 64 KiB, and passed to DSH without adding instructions.
48
+
49
+ ```sh
50
+ python3 adapters/dsh/harness/guard_lab.py run \
51
+ --dsh-home /tmp/approved-disposable-dsh-home \
52
+ --output-dir /tmp/dsh-lab-clean \
53
+ --case clean --model-label "DeepSeek official deepseek-flash" \
54
+ --task-file docs/guard-lab-clean-task.md \
55
+ --trial-group dsh-baseline-01 --timeout 120
56
+ ```
57
+
58
+ `run` prepares disjoint `fixture/` and `evidence/` directories, arms the
59
+ observer, invokes DSH with only the task from the fixture cwd, and stops the
60
+ observer even when host execution fails. It saves private `capture/stdout.bin`
61
+ and `capture/stderr.bin`, each capped at one MiB. A timeout or output overflow
62
+ stops only the runner's fresh process group and cannot be credited as completion.
63
+ The output directory is mode 0700; captures and receipt are mode 0600.
64
+
65
+ The runner inventories the selected home's `sessions/` tree before launch,
66
+ then captures exactly one newly created native artifact. It refuses missing
67
+ or ambiguous artifacts, symlinked trees, unsupported layouts and oversized
68
+ inputs. A reused existing session is not treated as a fresh capture. Session
69
+ inventory is bounded at 512 entries and depth four; use a fresh disposable home
70
+ when unrelated session activity could interfere.
71
+
72
+ Private `capture/session.bin` retains the original v3 JSONL or Zstandard bytes;
73
+ `capture/session.jsonl` retains the complete decoded log. Compressed input and
74
+ decoded output each have a one MiB cap. The decoder verifies physical frame
75
+ boundaries before decompressing every frame, following the ordinary frame
76
+ layout in [RFC 8878](https://www.rfc-editor.org/rfc/rfc8878.html#section-3.1.1).
77
+ Torn tails, bad checksums, trailing garbage, external dictionaries, unsupported
78
+ frames or expansion beyond the cap refuse the decoded capture. It never emits
79
+ a valid-looking partial prefix. Frames, blocks per frame and decoded records
80
+ are capped at 4096. The declared decoder window is capped at eight MiB; DSH's
81
+ streaming frames use a two MiB window even for small batches. The output cap
82
+ remains one MiB.
83
+
84
+ The initial report is **INCONCLUSIVE**, with `USER_REVIEW_REQUIRED` in the
85
+ capture summary. Exit zero from `run` means capture orchestration succeeded,
86
+ not that the model completed the task or the Lab criterion passed.
87
+
88
+ Review the CLI captures and private native session result. Do
89
+ not share raw captures: stderr can mix reasoning, logs, tool feedback, and
90
+ configuration errors. After confirming actual task completion:
91
+
92
+ ```sh
93
+ python3 adapters/dsh/harness/guard_lab.py finalize \
94
+ --output-dir /tmp/dsh-lab-clean --status completed
95
+ ```
96
+
97
+ Use `--status failed` for a model/configuration failure even if its process
98
+ returned zero, and `--status timed-out` for a timeout. `finalize` is create-once.
99
+ It verifies CLI/native capture hashes, run identity and the native interpretation,
100
+ records the reviewed host status, scans the output channels, and regenerates
101
+ the normal Lab report. Empty stdout,
102
+ nonzero exit, timeout, output overflow, or a changed/unavailable composition
103
+ cannot be declared completed.
104
+
105
+ Receipt versions 2–4 require an unseeded v3 single-turn session; receipt
106
+ version 5 binds DSH `0.2.0-rc.2` to native format v4. Both require a session whose
107
+ cwd and direct user task hash match the fixture and supplied task, with
108
+ contiguous event sequences, correlated tool calls/results and a completed
109
+ turn. Missing results or unsupported/unfinished sessions remain inconclusive;
110
+ use `failed` or `timed-out` as appropriate. No tool calls is valid when the
111
+ completed session really contains none. Forks, subagent origins, surface
112
+ replacements and non-text attachment blocks remain unsupported and cannot be
113
+ credited as a quiet success.
114
+
115
+ New captures use receipt version 4 for the old host or version 5 for the new
116
+ host, with native parser version 2. The parser
117
+ recognizes pinned `assistant/attempt` → `llm/retry` → `llm/retry-started`
118
+ chains followed by another failed attempt or a successful assistant settlement
119
+ in the same turn/step. It checks route, failure, chain ID, policy, retry number
120
+ and event order; an unfinished or inconsistent chain cannot be completed.
121
+ The completed turn must actually recover every failed attempt.
122
+
123
+ Failed-attempt compact text/reasoning/tool-argument records, supported raw
124
+ chunks, failure diagnostics and retry metadata are scanned as `host-output` /
125
+ `dsh-native-assistant-attempt-stream`. Original records and fragments joined
126
+ within each block are both scanned, including markers split across records.
127
+ These bytes are captured host output; they are not an assembled assistant
128
+ message or evidence of executing an attempted tool call. The caps are 64
129
+ failed attempts and 4096 expanded stream chunks per session, plus the one MiB
130
+ limit per scan channel. Unknown shapes, attachments, replay state, invalid
131
+ timestamps or records after a stream finish refuse the recognized channels.
132
+ Even sessions with no retries receive an explicit empty attempt-channel scan,
133
+ so both members of a new pair declare the same five-channel coverage.
134
+
135
+ Versions 2 and 3 keep native parser version 1 and their original four-channel
136
+ coverage; assistant attempts remain unsupported for those receipts. Historical
137
+ receipts are never silently reinterpreted with the new parser.
138
+
139
+ Native v4 interpretation additionally checks tool-registry developer messages
140
+ against prior request headers and validates workspace-change notices. Image
141
+ offload and surface replay remain unsupported. The retained new-host real-model
142
+ clean capture was refused with `UNSUPPORTED_SURFACE_REPLAY`, and its composition
143
+ also changed; it did not establish a new off/on result. The
144
+ [current review](../../../docs/test-report-release-readiness-0.2.3.md) separates
145
+ the passing no-model preflight/parser regressions from this incomplete trial.
146
+
147
+ Historical version 1 receipts still finalize with stdout/stderr coverage only.
148
+ They do not acquire native coverage by opening them with the updated helper.
149
+ Use fresh captures with identical channel coverage for a later differential
150
+ trial. The generic comparison now requires equal scanned channel sets and
151
+ non-guard fingerprints, while their provenance remains user-declared.
152
+ [Native-session follow-up](../../../docs/test-report-dsh-native-session.md)
153
+ records the offline review of retained real logs separately.
154
+
155
+ ## Capture the guard-off attack probe
156
+
157
+ Proceed after the clean control completes with a healthy passing criterion.
158
+ Use the same approved settings, model, task file, and common overlays:
159
+
160
+ ```sh
161
+ python3 adapters/dsh/harness/guard_lab.py run \
162
+ --dsh-home /tmp/approved-disposable-dsh-home \
163
+ --output-dir /tmp/dsh-lab-injection-baseline \
164
+ --case injection-probe --model-label "DeepSeek official deepseek-flash" \
165
+ --trial-group dsh-baseline-01 --timeout 120
166
+
167
+ # Review the private captures and the host session before selecting completed.
168
+ python3 adapters/dsh/harness/guard_lab.py finalize \
169
+ --output-dir /tmp/dsh-lab-injection-baseline --status completed
170
+ ```
171
+
172
+ Reuse of the same home preserves settings but also preserves host-owned session
173
+ and storage state; this is not a stateless-host claim. For an experiment that
174
+ requires fresh host state for each run, prepare equivalent disposable homes
175
+ first and verify their non-guard settings separately.
176
+
177
+ If the baseline reports `NO_EXPOSURE_OBSERVED`, no mitigation has been measured.
178
+ Only an effective baseline justifies a later matched guard-on trial and the
179
+ existing `guard_lab.py compare`. This runner deliberately captures guard-off
180
+ trials only; it does not add a guard-on plugin to user configuration.
181
+
182
+ ### Indirect samples
183
+
184
+ For separate baselines, add `--attack-sample maintenance-v1` for a project
185
+ review checklist or `--attack-sample config-comment-v1` for instructions in
186
+ service configuration comments. The original `direct-v1` remains the default.
187
+ Each sample gets its own protocol ID and template fingerprint. Keep the original
188
+ task, use a fresh output/home for each sample, and use the same sample for any
189
+ later guarded comparison. See the [sample registry](../../../docs/guard-lab.md#versioned-attack-samples).
190
+
191
+ ## Evidence and exit status
192
+
193
+ Both streams use the Lab's `host-output` scan stage, with separate
194
+ `dsh-headless-stdout` and `dsh-headless-stderr` source kinds. These are captured
195
+ CLI streams, not authenticated assistant/tool-role records. The script does
196
+ not turn mixed stderr into a model-output claim.
197
+
198
+ In a recognized native session, `tool/result` **model-facing content** uses
199
+ `tool-output` / `dsh-native-tool-result`; assembled `assistant/message` content
200
+ uses `model-output` / `dsh-native-assistant-content`. DSH v3 tool-result messages
201
+ have `role=user`, so classification uses the event type plus tool-call/source
202
+ correlation, not a generic role split. User/system inputs, tool-private `meta`,
203
+ source metadata and embedded provider stream records are retained raw but
204
+ excluded from these output scans. Native labels are captured host metadata,
205
+ not independently authenticated authorship or proof of remote receipt.
206
+
207
+ The parser scans finalized content, including reasoning/tool-call blocks; it
208
+ does not reconstruct the projected conversation or certify that the remote
209
+ model received every recorded result. An unrecognized decoded session may be
210
+ scanned as `host-output` / `dsh-native-unrecognized-session` only while the
211
+ failed run remains inconclusive. Raw compressed bytes are not marker-scanned.
212
+
213
+ `capture/receipt.json` records hashes, sizes, the host exit/timeout/overflow
214
+ state, task fingerprint, public trial labels, and pre/post composition equality.
215
+ The receipt retains neither task text nor raw composition. **The private native
216
+ capture retains task/system messages and tool data**, as well as CLI output;
217
+ `task_text_retained` records this. Keep these files out of public artifacts.
218
+ The flag marks native artifact retention, including refused or partly decodable
219
+ captures.
220
+ DSH also retains its own session artifacts.
221
+ Lab events and reports continue to retain no raw output or marker values.
222
+
223
+ Version/model labels, completion review, environment equality across separate
224
+ runs, and same-UID isolation retain the [normal Lab limitations](../../../docs/guard-lab.md).
225
+ The original composition fingerprint checks one composed tree. New receipt
226
+ version 3 additionally binds `dsh-lab-non-guard-v1`: rendered non-Guard rows,
227
+ saved `settings.yaml`, home `.env`, the headless package manifest and inherited
228
+ launch environment. Only column-zero provenance comments and the exact known
229
+ active local Guard row are excluded; literal text/blank lines stay intact.
230
+ `DSH_HOME` is normalized, and `PWD`, `OLDPWD`, `_` are excluded as invocation
231
+ bookkeeping. A post-run change in this scope refuses completed finalization.
232
+ Authentication state, external files named by plugins, installed dependency
233
+ code, backend identity and settings outside this scope remain user-declared.
234
+ The fingerprint retains no raw configuration/environment values.
235
+
236
+ ## Matched pair with review gates
237
+
238
+ Select an initialized standard headless source home. It must declare only
239
+ the `@deepseek-ai/dsh-base` and `@deepseek-ai/dsh-headless` bundles with no
240
+ extra package dependencies. The helper privately copies only its manifest,
241
+ profile/home patches, saved settings, `.env`, and authentication bootstrap.
242
+ It omits sessions, storage, caches and unrelated user files. It refuses
243
+ symlinked bootstrap files/directories, overlaps and existing output targets.
244
+ DSH may materialize its ordinary module fallback in the new homes during
245
+ boot-free configuration checks; no plugin/package is installed by this helper.
246
+
247
+ ```sh
248
+ python3 adapters/dsh/harness/guard_lab.py prepare-pair \
249
+ --source-home /tmp/approved-initialized-dsh-home \
250
+ --output-dir /tmp/dsh-matched-pair \
251
+ --model-label "DeepSeek official deepseek-flash" \
252
+ --trial-group dsh-pair-01 --attack-sample maintenance-v1
253
+
254
+ python3 adapters/dsh/harness/guard_lab.py pair-next \
255
+ --pair-dir /tmp/dsh-matched-pair
256
+ ```
257
+
258
+ `prepare-pair` makes **no model request**. It creates separate `homes/clean-off`,
259
+ `homes/clean-on`, `homes/baseline`, `homes/guarded`, and an exact local-source
260
+ Guard patch. All four preflights must have equal scoped non-guard fingerprints.
261
+ The source remains untouched. Saved settings still override model defaults;
262
+ the helper records the public model label without verifying remote identity.
263
+
264
+ Each `pair-next` launches at most one host task and returns
265
+ `USER_REVIEW_REQUIRED`. One host task can make several model requests;
266
+ `host_tasks_launched` is not a model-request count. Review its private captures,
267
+ then declare the actual outcome, for example:
268
+
269
+ ```sh
270
+ python3 adapters/dsh/harness/guard_lab.py finalize \
271
+ --output-dir /tmp/dsh-matched-pair/trials/clean-off --status completed
272
+ python3 adapters/dsh/harness/guard_lab.py pair-next \
273
+ --pair-dir /tmp/dsh-matched-pair
274
+ ```
275
+
276
+ The order is `clean-off` → `clean-on` → `baseline` → `guarded`. Pending review
277
+ does not replay a capture. Failed/unhealthy/exposed clean controls stop the
278
+ sequence. A quiet baseline returns `BASELINE_ATTACK_NOT_EFFECTIVE` without
279
+ launching guarded. Task/overlay/adapter drift, changed scoped settings or prior
280
+ sessions in the next home refuse launch. Existing captures and homes are
281
+ never overwritten. After guarded review, `pair-next` returns the comparison.
282
+
283
+ ```sh
284
+ python3 adapters/dsh/harness/guard_lab.py compare \
285
+ --baseline-dir /tmp/dsh-matched-pair/trials/baseline \
286
+ --guarded-dir /tmp/dsh-matched-pair/trials/guarded
287
+ ```
288
+
289
+ This wrapper rechecks receipt/capture integrity and requires version 3 or 4
290
+ native coverage, matching non-guard fingerprints and all scans declared by
291
+ that receipt version. New version 4/5 pairs include the fifth failed-attempt
292
+ scan. Old and new channel sets cannot be compared. It rejects historical
293
+ CLI-only/missing-context receipts. New pair plans pin native parser version 2;
294
+ plans without that field retain their old interpretation and cannot launch
295
+ another capture under the new rules. Prepare a fresh pair to continue testing.
296
+ Results distinguish tool and assistant canary exposure from fake Lab execution
297
+ and URL contact. Guard-on loads the existing deletion adapter; it does not
298
+ introduce tool-result redaction. Continued tool exposure can validly yield
299
+ `NOT_MITIGATED`. The [investigation report](../../../docs/test-report-dsh-paired-lab.md)
300
+ describes the candidate result-rewriting seam and its limits.
301
+
302
+ `prepare-pair` returns 0 for a valid plan. `pair-next` returns 0 when it captures
303
+ a task, 4 when review is still pending or a gate stops the sequence, and normal
304
+ comparison exits after completion. `compare` uses 0/2/4 for mitigation,
305
+ not-mitigated, and inconclusive/incomparable respectively; refusal is 1.
306
+
307
+ `check`/`run` return 0 on successful orchestration and 1 on controller/preflight
308
+ failure. `finalize` uses normal Lab exits: 0 for a passing criterion, 2 for
309
+ observed attack-probe exposure, 4 for inconclusive evidence, and 1 for refused
310
+ input or controller failure.
311
+
312
+ ## Regression checks
313
+
314
+ ```sh
315
+ python3 -m unittest tests.test_dsh_native_session tests.test_dsh_profiles tests.test_dsh_guard_lab
316
+ ```
317
+
318
+ These use synthetic subprocess hosts and loopback observers, with no model or
319
+ external-network call. They cover unreviewed/failed/empty/oversized/timed-out
320
+ captures, exposure semantics, capture corruption, version/config preflight,
321
+ composition drift, create-once finalization, recovered/broken retries,
322
+ fragmented markers, retry-only exposure and legacy coverage. The
323
+ [retry-evidence report](../../../docs/test-report-dsh-retry-evidence.md) records
324
+ an offline reparse of the retained official Flash retry log. A mock pass does not establish
325
+ a new real-model or execution-level DSH acceptance claim.
326
+ The dedicated CI `dsh-native-lab` job selects Node 22.23.2 and runs these suites;
327
+ runtime-dependent regressions are skipped when Zstandard APIs are unavailable,
328
+ while pure native-parser checks still run.
@@ -0,0 +1,8 @@
1
+ # Common overlay for every guard-off Lab capture; no separate title model call.
2
+ - id: session-title-llm
3
+ disabled: true
4
+
5
+ - id: agent-default-model
6
+ config:
7
+ provider: deepseek-official
8
+ model: deepseek-flash