claude-dev-env 8.3.0 → 8.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/team-advisor/SKILL.md +9 -4
- package/_shared/advisor/reference/spawn-walk-log.md +2 -1
- package/_shared/advisor/scripts/config/advisor_scripts_constants/model_tier_run_validator_constants.py +3 -0
- package/_shared/advisor/scripts/model_tier_run_validator.py +169 -14
- package/_shared/advisor/scripts/tests/test_codex_astra_advisor.py +20 -0
- package/_shared/advisor/scripts/tests/test_model_tier_run_validator.py +99 -0
- package/bin/install.agents-home.test.mjs +10 -0
- package/bin/install.test.mjs +14 -0
- package/docs/references/advisor-tool.md +32 -0
- package/docs/references/team-advisor-skill.md +1 -1
- package/hooks/validators/README.md +0 -1
- package/hooks/validators/test_files/EXECUTIVE_SUMMARY.md +1 -2
- package/package.json +1 -1
- package/scripts/policy_lint/config/approved_test_pairs.py +9 -0
- package/scripts/tests/test_policy_lint_rules_pairing.py +16 -0
- package/scripts/windows/pstack/pstack-command-shim.ps1 +41 -0
- package/scripts/windows/pstack/ship-pr.cmd +3 -0
- package/scripts/windows/pstack/watch-pr.cmd +3 -0
- package/hooks/validators/VALIDATION_REPORT.md +0 -287
- package/hooks/validators/validator_defaults.py +0 -5
|
@@ -16,8 +16,8 @@ One warm advisor at the strongest tier this session can reach. This session is t
|
|
|
16
16
|
|
|
17
17
|
| Doc | Holds |
|
|
18
18
|
|---|---|
|
|
19
|
-
| [`docs/references/advisor-tool.md`](
|
|
20
|
-
| [`~/.claude/_shared/advisor/advisor-protocol.md`](
|
|
19
|
+
| [`docs/references/advisor-tool.md`](../../../docs/references/advisor-tool.md) | **Consult cadence and weight.** When to call, the hard rule before first write, and how to treat advice. Read this for every consult. |
|
|
20
|
+
| [`~/.claude/_shared/advisor/advisor-protocol.md`](../../../_shared/advisor/advisor-protocol.md) | **Bind and lifecycle.** Session identity, host detect, model floor, warm-up, CLI fallback. Its read map routes each moment to a `reference/` detail file. |
|
|
21
21
|
| Installed `poteto-mode` skill | **Reply contract.** The advisor prompt carries ENDORSE / CORRECTION / PLAN / STOP. |
|
|
22
22
|
| [`reference/advisor-docs-review.md`](reference/advisor-docs-review.md) | Anthropic advisor-tool source facts: measured effects, Sonnet steering, cost levers, failure modes. Background. Read it when tuning the bind, not on every consult. |
|
|
23
23
|
|
|
@@ -29,7 +29,7 @@ One warm advisor at the strongest tier this session can reach. This session is t
|
|
|
29
29
|
4. Skip the multi-consumer "who you are" opener. This session is the sole consumer.
|
|
30
30
|
5. When the bind or reply path fails, fail closed and report to the user. On a third-party host, only the bound advisor issues ENDORSE / CORRECTION / PLAN / STOP.
|
|
31
31
|
|
|
32
|
-
**GOTCHA (Cursor / ThirdParty + Astra):** when the walk reaches Astra or the user asks for Astra, first tool call is `python ~/.claude/_shared/advisor/scripts/codex_astra_advisor.py --bind --enable-astra --cwd <repo-root>` with the charter on stdin. Do not use Agent or Task. Do not search for a probe path. Details: [`third-party-bind.md`](
|
|
32
|
+
**GOTCHA (Cursor / ThirdParty + Astra):** when the walk reaches Astra or the user asks for Astra, first tool call is `python ~/.claude/_shared/advisor/scripts/codex_astra_advisor.py --bind --enable-astra --cwd <repo-root>` with the charter on stdin. Do not use Agent or Task. Do not search for a probe path. Details: [`third-party-bind.md`](../../../_shared/advisor/reference/third-party-bind.md) GOTCHA and [`astra-rung.md`](../../../_shared/advisor/reference/astra-rung.md).
|
|
33
33
|
|
|
34
34
|
Full walk, charter, consult packet, Astra routing, and drift re-bind live in the protocol read map and its `reference/` files.
|
|
35
35
|
|
|
@@ -37,7 +37,7 @@ Full walk, charter, consult packet, Astra routing, and drift re-bind live in the
|
|
|
37
37
|
|
|
38
38
|
Follow **When to call**, **Hard rule**, and **How to treat advice** in `advisor-tool.md`.
|
|
39
39
|
|
|
40
|
-
Build every first brief with [`_shared/advisor/reference/consult-format.md`](
|
|
40
|
+
Build every first brief with [`_shared/advisor/reference/consult-format.md`](../../../_shared/advisor/reference/consult-format.md). Later briefs carry only the delta and changed evidence.
|
|
41
41
|
|
|
42
42
|
With the built-in advisor tool, call `advisor()` with no brief. The tool forwards the whole transcript. Its reply is free text, not one of the four signal words, so weigh it per **How to treat advice** in `advisor-tool.md`.
|
|
43
43
|
|
|
@@ -48,3 +48,8 @@ Aim for two consults on a normal task: one after orientation and one after write
|
|
|
48
48
|
- One bind per session; this session owns the built-in tool choice, spawn, in-session Astra spawn, or CLI bind, drift re-bind, and shutdown.
|
|
49
49
|
- Bind at or above the protocol floor for this host.
|
|
50
50
|
- The advisor only answers (messaging); the session runs tools and posts.
|
|
51
|
+
- Keep an optional reference gap in the reference record: path, status, repair action, and repair result. A reference gap leaves bind status separate.
|
|
52
|
+
- On Codex, select native Astra when its bind and reply validation pass. Record `selected_tier: Astra` and `reply_path: native`.
|
|
53
|
+
- Mark `fallback_kind: broken` only when the selected bind or reply path fails. Record the selected tier, fallback reason, and reply path in the same evidence object.
|
|
54
|
+
- Record first, recovery, and completion consults with changed evidence, validation, unresolved risks, and report-back status.
|
|
55
|
+
- Write the evidence object to the session-controlled `model-tier-run.json` record described in `advisor-tool.md`.
|
|
@@ -12,6 +12,7 @@ Write the log as JSON with these field names:
|
|
|
12
12
|
- `astra_enabled`: a boolean recorded before candidate selection. On Claude and ThirdParty, `true` adds Astra after Fable and `false` walks Fable alone. On Codex this flag does not change the walk.
|
|
13
13
|
- `attempts`: one `{tier, result}` entry appended as each bind try happens. `result` is `codex` for an Astra helper bind, `spawned` for an in-session spawn (Claude Agent or Codex native Astra), `cli` for a CLI Claude-chain bind, or a failure reason such as `unavailable`.
|
|
14
14
|
- `selected_tier`: the tier of the first successful bind (first `codex`, `spawned`, or `cli` entry), or `null` paired with a `fallback_reason` string when none bound.
|
|
15
|
+
- `evidence`: the versioned reference, fallback, and consult record described in `docs/references/advisor-tool.md`. A reference status records optional path repair separately from bind status.
|
|
15
16
|
|
|
16
17
|
## Log path
|
|
17
18
|
|
|
@@ -28,5 +29,5 @@ Exit code `1` means a ladder invariant failed.
|
|
|
28
29
|
Exit code `2` means the path or JSON was unusable.
|
|
29
30
|
The same checks are available in-process via `validate_model_tier_run(run)`.
|
|
30
31
|
|
|
31
|
-
The validator checks ladder shape
|
|
32
|
+
The validator checks ladder shape and validates `evidence` when it is present. On Claude and ThirdParty, Astra is attempted after Fable when `astra_enabled` is enabled, and `selected_tier: "Astra"` requires `result: "codex"`. On Codex, the walk is Astra only, and `result: "spawned"` or `result: "codex"` counts as success. A Codex success record uses `reply_path: "native"`.
|
|
32
33
|
Host policy sits on top of it. See the Model floor section of the protocol.
|
|
@@ -102,3 +102,6 @@ CLI_MISSING_PATH_EXIT_CODE: int = 2
|
|
|
102
102
|
CLI_INVALID_JSON_EXIT_CODE: int = 2
|
|
103
103
|
CLI_VALIDATION_FAILURE_EXIT_CODE: int = 1
|
|
104
104
|
CLI_SUCCESS_EXIT_CODE: int = 0
|
|
105
|
+
ALL_ADVISOR_REPLY_PATHS: frozenset[str] = frozenset(
|
|
106
|
+
{"native", "sendmessage", "cli", "codex", "none"}
|
|
107
|
+
)
|
|
@@ -42,6 +42,7 @@ from __future__ import annotations
|
|
|
42
42
|
|
|
43
43
|
import json
|
|
44
44
|
import sys
|
|
45
|
+
from collections.abc import Mapping
|
|
45
46
|
from dataclasses import dataclass
|
|
46
47
|
from pathlib import Path
|
|
47
48
|
|
|
@@ -52,7 +53,21 @@ if _config_directory not in sys.path:
|
|
|
52
53
|
if _scripts_directory not in sys.path:
|
|
53
54
|
sys.path.insert(0, _scripts_directory)
|
|
54
55
|
|
|
55
|
-
from advisor_scripts_constants.
|
|
56
|
+
from advisor_scripts_constants.advisor_route_constants import (
|
|
57
|
+
ADVISOR_FALLBACK_TIER,
|
|
58
|
+
ADVISOR_MODEL_TIER,
|
|
59
|
+
CLI_BIND_SUCCESS_TOKEN,
|
|
60
|
+
CODEX_BIND_SUCCESS_TOKEN,
|
|
61
|
+
SPAWN_OUTCOME_KEY,
|
|
62
|
+
SPAWN_SUCCESS_TOKEN,
|
|
63
|
+
TIER_KEY,
|
|
64
|
+
)
|
|
65
|
+
from advisor_scripts_constants.astra_advisor_constants import (
|
|
66
|
+
ASTRA_FALLBACK_KIND_BROKEN,
|
|
67
|
+
ASTRA_FALLBACK_KIND_DECLINED,
|
|
68
|
+
)
|
|
69
|
+
from advisor_scripts_constants.model_tier_run_validator_constants import (
|
|
70
|
+
ALL_ADVISOR_REPLY_PATHS,
|
|
56
71
|
ATTEMPT_ORDER_MISMATCH_MESSAGE,
|
|
57
72
|
ATTEMPT_TIER_OUT_OF_SLICE_MESSAGE,
|
|
58
73
|
CANDIDATE_TIERS_MISMATCH_MESSAGE,
|
|
@@ -73,16 +88,7 @@ from advisor_scripts_constants.model_tier_run_validator_constants import ( # no
|
|
|
73
88
|
UNKNOWN_HOST_PROFILE_ERROR,
|
|
74
89
|
UNKNOWN_OWN_TIER_MESSAGE,
|
|
75
90
|
)
|
|
76
|
-
from
|
|
77
|
-
ADVISOR_FALLBACK_TIER,
|
|
78
|
-
ADVISOR_MODEL_TIER,
|
|
79
|
-
CODEX_BIND_SUCCESS_TOKEN,
|
|
80
|
-
CLI_BIND_SUCCESS_TOKEN,
|
|
81
|
-
SPAWN_OUTCOME_KEY,
|
|
82
|
-
SPAWN_SUCCESS_TOKEN,
|
|
83
|
-
TIER_KEY,
|
|
84
|
-
)
|
|
85
|
-
from tier_model_ids import canonical_host_profile, canonical_tier_name # noqa: E402
|
|
91
|
+
from tier_model_ids import canonical_host_profile, canonical_tier_name
|
|
86
92
|
|
|
87
93
|
|
|
88
94
|
@dataclass(frozen=True)
|
|
@@ -94,12 +100,157 @@ class ModelTierRun:
|
|
|
94
100
|
fallback_reason: str | None = None
|
|
95
101
|
is_astra_enabled: bool = False
|
|
96
102
|
host_profile: str = HOST_PROFILE_CLAUDE
|
|
103
|
+
evidence: dict[str, object] | None = None
|
|
97
104
|
|
|
98
105
|
|
|
99
106
|
class ModelTierRunError(ValueError):
|
|
100
107
|
"""Raised when a model-tier spawn-walk log violates an invariant."""
|
|
101
108
|
|
|
102
109
|
|
|
110
|
+
def _required_evidence_text(
|
|
111
|
+
fields_by_name: Mapping[str, object], section_name: str, field_name: str
|
|
112
|
+
) -> str:
|
|
113
|
+
raw_field_text = fields_by_name.get(field_name)
|
|
114
|
+
if not isinstance(raw_field_text, str) or not raw_field_text.strip():
|
|
115
|
+
raise ModelTierRunError(
|
|
116
|
+
f"evidence.{section_name}.{field_name} must be a non-empty string"
|
|
117
|
+
)
|
|
118
|
+
return raw_field_text.strip()
|
|
119
|
+
|
|
120
|
+
|
|
121
|
+
def _required_evidence_list(
|
|
122
|
+
fields_by_name: Mapping[str, object], section_name: str, field_name: str
|
|
123
|
+
) -> list[str]:
|
|
124
|
+
raw_field_entries = fields_by_name.get(field_name)
|
|
125
|
+
if not isinstance(raw_field_entries, list) or not raw_field_entries:
|
|
126
|
+
raise ModelTierRunError(
|
|
127
|
+
f"evidence.{section_name}.{field_name} must be a non-empty list"
|
|
128
|
+
)
|
|
129
|
+
if any(
|
|
130
|
+
not isinstance(each_entry, str) or not each_entry.strip()
|
|
131
|
+
for each_entry in raw_field_entries
|
|
132
|
+
):
|
|
133
|
+
raise ModelTierRunError(
|
|
134
|
+
f"evidence.{section_name}.{field_name} must contain non-empty strings"
|
|
135
|
+
)
|
|
136
|
+
return [each_entry.strip() for each_entry in raw_field_entries]
|
|
137
|
+
|
|
138
|
+
|
|
139
|
+
def _required_evidence_section(
|
|
140
|
+
sections_by_name: Mapping[str, object], section_name: str
|
|
141
|
+
) -> Mapping[str, object]:
|
|
142
|
+
raw_section = sections_by_name.get(section_name)
|
|
143
|
+
if not isinstance(raw_section, Mapping):
|
|
144
|
+
raise ModelTierRunError(f"evidence.{section_name} must be an object")
|
|
145
|
+
return raw_section
|
|
146
|
+
|
|
147
|
+
|
|
148
|
+
def _validate_reference_evidence(sections_by_name: Mapping[str, object]) -> None:
|
|
149
|
+
reference = _required_evidence_section(sections_by_name, "reference")
|
|
150
|
+
_required_evidence_text(reference, "reference", "path")
|
|
151
|
+
reference_status = _required_evidence_text(reference, "reference", "status")
|
|
152
|
+
if reference_status not in {"missing", "read"}:
|
|
153
|
+
raise ModelTierRunError("evidence.reference.status must be missing or read")
|
|
154
|
+
_required_evidence_text(reference, "reference", "repair_action")
|
|
155
|
+
_required_evidence_text(reference, "reference", "repair_result")
|
|
156
|
+
|
|
157
|
+
|
|
158
|
+
def _validate_fallback_tier(
|
|
159
|
+
fallback_by_name: Mapping[str, object], run: ModelTierRun
|
|
160
|
+
) -> str | None:
|
|
161
|
+
raw_selected_tier = fallback_by_name.get("selected_tier")
|
|
162
|
+
if raw_selected_tier is not None and not isinstance(raw_selected_tier, str):
|
|
163
|
+
raise ModelTierRunError("evidence.fallback.selected_tier must be a string or null")
|
|
164
|
+
maybe_selected_tier = (
|
|
165
|
+
canonical_tier_name(raw_selected_tier)
|
|
166
|
+
if isinstance(raw_selected_tier, str)
|
|
167
|
+
else None
|
|
168
|
+
)
|
|
169
|
+
maybe_run_selected_tier = (
|
|
170
|
+
canonical_tier_name(run.selected_tier)
|
|
171
|
+
if run.selected_tier is not None
|
|
172
|
+
else None
|
|
173
|
+
)
|
|
174
|
+
if maybe_selected_tier != maybe_run_selected_tier:
|
|
175
|
+
raise ModelTierRunError(
|
|
176
|
+
"evidence.fallback.selected_tier must match selected_tier"
|
|
177
|
+
)
|
|
178
|
+
return maybe_run_selected_tier
|
|
179
|
+
|
|
180
|
+
|
|
181
|
+
def _validate_fallback_kind(fallback_by_name: Mapping[str, object]) -> None:
|
|
182
|
+
raw_fallback_kind = fallback_by_name.get("fallback_kind")
|
|
183
|
+
if raw_fallback_kind not in {
|
|
184
|
+
None,
|
|
185
|
+
ASTRA_FALLBACK_KIND_BROKEN,
|
|
186
|
+
ASTRA_FALLBACK_KIND_DECLINED,
|
|
187
|
+
}:
|
|
188
|
+
raise ModelTierRunError(
|
|
189
|
+
"evidence.fallback.fallback_kind must be declined, broken, or null"
|
|
190
|
+
)
|
|
191
|
+
raw_fallback_reason = fallback_by_name.get("fallback_reason")
|
|
192
|
+
if raw_fallback_reason is not None and (
|
|
193
|
+
not isinstance(raw_fallback_reason, str) or not raw_fallback_reason.strip()
|
|
194
|
+
):
|
|
195
|
+
raise ModelTierRunError(
|
|
196
|
+
"evidence.fallback.fallback_reason must be a non-empty string or null"
|
|
197
|
+
)
|
|
198
|
+
if raw_fallback_kind is not None and raw_fallback_reason is None:
|
|
199
|
+
raise ModelTierRunError(
|
|
200
|
+
"evidence.fallback.fallback_reason is required for a fallback kind"
|
|
201
|
+
)
|
|
202
|
+
|
|
203
|
+
|
|
204
|
+
def _validate_fallback_reply_path(
|
|
205
|
+
fallback_by_name: Mapping[str, object],
|
|
206
|
+
run: ModelTierRun,
|
|
207
|
+
maybe_run_selected_tier: str | None,
|
|
208
|
+
) -> None:
|
|
209
|
+
reply_path = _required_evidence_text(fallback_by_name, "fallback", "reply_path")
|
|
210
|
+
if reply_path not in ALL_ADVISOR_REPLY_PATHS:
|
|
211
|
+
raise ModelTierRunError(
|
|
212
|
+
"evidence.fallback.reply_path is not a known advisor path"
|
|
213
|
+
)
|
|
214
|
+
if (
|
|
215
|
+
canonical_host_profile(run.host_profile) == HOST_PROFILE_CODEX
|
|
216
|
+
and maybe_run_selected_tier == ADVISOR_MODEL_TIER
|
|
217
|
+
and reply_path != "native"
|
|
218
|
+
):
|
|
219
|
+
raise ModelTierRunError(
|
|
220
|
+
"Codex Astra success requires the native reply path"
|
|
221
|
+
)
|
|
222
|
+
|
|
223
|
+
|
|
224
|
+
def _validate_fallback_evidence(
|
|
225
|
+
sections_by_name: Mapping[str, object], run: ModelTierRun
|
|
226
|
+
) -> None:
|
|
227
|
+
fallback = _required_evidence_section(sections_by_name, "fallback")
|
|
228
|
+
maybe_run_selected_tier = _validate_fallback_tier(fallback, run)
|
|
229
|
+
_validate_fallback_kind(fallback)
|
|
230
|
+
_validate_fallback_reply_path(fallback, run, maybe_run_selected_tier)
|
|
231
|
+
|
|
232
|
+
|
|
233
|
+
def _validate_consult_evidence(sections_by_name: Mapping[str, object]) -> None:
|
|
234
|
+
consult = _required_evidence_section(sections_by_name, "consult")
|
|
235
|
+
for each_field_name in ("changed_evidence", "validation", "unresolved_risks"):
|
|
236
|
+
_required_evidence_list(consult, "consult", each_field_name)
|
|
237
|
+
_required_evidence_text(consult, "consult", "report_back_status")
|
|
238
|
+
|
|
239
|
+
|
|
240
|
+
def _validate_advisor_evidence(
|
|
241
|
+
evidence: object,
|
|
242
|
+
run: ModelTierRun,
|
|
243
|
+
) -> None:
|
|
244
|
+
if not isinstance(evidence, Mapping):
|
|
245
|
+
raise ModelTierRunError("evidence must be an object")
|
|
246
|
+
raw_schema_version = evidence.get("schema_version")
|
|
247
|
+
if isinstance(raw_schema_version, bool) or raw_schema_version != 1:
|
|
248
|
+
raise ModelTierRunError("evidence.schema_version must be 1")
|
|
249
|
+
_validate_reference_evidence(evidence)
|
|
250
|
+
_validate_fallback_evidence(evidence, run)
|
|
251
|
+
_validate_consult_evidence(evidence)
|
|
252
|
+
|
|
253
|
+
|
|
103
254
|
def _canonical_tier_list(all_tier_names: list[str]) -> list[str] | None:
|
|
104
255
|
all_canonical_tiers: list[str] = []
|
|
105
256
|
for each_tier_name in all_tier_names:
|
|
@@ -149,9 +300,7 @@ def _is_successful_attempt_outcome(
|
|
|
149
300
|
return False
|
|
150
301
|
if outcome_token == SPAWN_SUCCESS_TOKEN:
|
|
151
302
|
return True
|
|
152
|
-
|
|
153
|
-
return True
|
|
154
|
-
return False
|
|
303
|
+
return outcome_token == CLI_BIND_SUCCESS_TOKEN
|
|
155
304
|
|
|
156
305
|
|
|
157
306
|
def validate_model_tier_run(run: ModelTierRun) -> None:
|
|
@@ -205,6 +354,8 @@ def validate_model_tier_run(run: ModelTierRun) -> None:
|
|
|
205
354
|
all_attempted_tiers=all_attempted_tiers,
|
|
206
355
|
all_expected_candidates=all_expected_candidates,
|
|
207
356
|
)
|
|
357
|
+
if run.evidence is not None:
|
|
358
|
+
_validate_advisor_evidence(run.evidence, run)
|
|
208
359
|
|
|
209
360
|
|
|
210
361
|
def _validate_selected_tier(
|
|
@@ -262,6 +413,9 @@ def load_model_tier_run_from_json_path(from_path: Path) -> ModelTierRun:
|
|
|
262
413
|
raw_host_profile = parsed_payload.get(HOST_PROFILE_JSON_KEY, HOST_PROFILE_CLAUDE)
|
|
263
414
|
if not isinstance(raw_host_profile, str):
|
|
264
415
|
raise TypeError(HOST_PROFILE_MUST_BE_STRING_MESSAGE)
|
|
416
|
+
raw_evidence = parsed_payload.get("evidence")
|
|
417
|
+
if raw_evidence is not None and not isinstance(raw_evidence, dict):
|
|
418
|
+
raise TypeError("evidence must be an object")
|
|
265
419
|
return ModelTierRun(
|
|
266
420
|
own_tier=parsed_payload["own_tier"],
|
|
267
421
|
candidate_tiers=list(parsed_payload["candidate_tiers"]),
|
|
@@ -270,6 +424,7 @@ def load_model_tier_run_from_json_path(from_path: Path) -> ModelTierRun:
|
|
|
270
424
|
fallback_reason=parsed_payload.get("fallback_reason"),
|
|
271
425
|
is_astra_enabled=raw_astra_enabled,
|
|
272
426
|
host_profile=raw_host_profile,
|
|
427
|
+
evidence=raw_evidence,
|
|
273
428
|
)
|
|
274
429
|
|
|
275
430
|
|
|
@@ -220,6 +220,12 @@ def test_bind_runs_probe_then_codex() -> None:
|
|
|
220
220
|
_two_step_runner(calls),
|
|
221
221
|
)
|
|
222
222
|
assert reply.successful
|
|
223
|
+
assert reply.selected_tier == "Astra"
|
|
224
|
+
assert reply.outcome == "codex"
|
|
225
|
+
assert reply.session_id == "thread-1"
|
|
226
|
+
assert reply.guidance == "PLAN\ninspect"
|
|
227
|
+
assert reply.signal == "PLAN"
|
|
228
|
+
assert reply.fallback_kind is None
|
|
223
229
|
assert calls[1] == astra_advisor.build_codex_arguments("codex")
|
|
224
230
|
|
|
225
231
|
|
|
@@ -311,6 +317,20 @@ def test_active_docs_use_astra_names() -> None:
|
|
|
311
317
|
assert all(name not in content for name in legacy_names)
|
|
312
318
|
|
|
313
319
|
|
|
320
|
+
def test_team_advisor_source_references_resolve_from_the_package_tree() -> None:
|
|
321
|
+
package_root = SCRIPTS_ROOT.parents[2]
|
|
322
|
+
skill_path = package_root / ".agents" / "skills" / "team-advisor" / "SKILL.md"
|
|
323
|
+
all_reference_targets = (
|
|
324
|
+
"../../../docs/references/advisor-tool.md",
|
|
325
|
+
"../../../_shared/advisor/advisor-protocol.md",
|
|
326
|
+
"../../../_shared/advisor/reference/third-party-bind.md",
|
|
327
|
+
"../../../_shared/advisor/reference/astra-rung.md",
|
|
328
|
+
"../../../_shared/advisor/reference/consult-format.md",
|
|
329
|
+
)
|
|
330
|
+
for each_target in all_reference_targets:
|
|
331
|
+
assert (skill_path.parent / each_target).is_file(), each_target
|
|
332
|
+
|
|
333
|
+
|
|
314
334
|
def test_astra_docs_name_shared_effort_and_helper() -> None:
|
|
315
335
|
advisor_root = SCRIPTS_ROOT.parent
|
|
316
336
|
rung = (advisor_root / "reference" / "astra-rung.md").read_text(encoding="utf-8")
|
|
@@ -37,6 +37,105 @@ ADVISOR_MODEL_TIER = model_tier_run_validator.ADVISOR_MODEL_TIER
|
|
|
37
37
|
CODEX_BIND_SUCCESS_TOKEN = model_tier_run_validator.CODEX_BIND_SUCCESS_TOKEN
|
|
38
38
|
|
|
39
39
|
|
|
40
|
+
def _native_codex_evidence(
|
|
41
|
+
*,
|
|
42
|
+
reply_path: str = "native",
|
|
43
|
+
reference_status: str = "missing",
|
|
44
|
+
) -> dict[str, object]:
|
|
45
|
+
return {
|
|
46
|
+
"schema_version": 1,
|
|
47
|
+
"reference": {
|
|
48
|
+
"path": "~/.claude/docs/references/advisor-tool.md",
|
|
49
|
+
"status": reference_status,
|
|
50
|
+
"repair_action": "use the projected docs root",
|
|
51
|
+
"repair_result": "read",
|
|
52
|
+
},
|
|
53
|
+
"fallback": {
|
|
54
|
+
"selected_tier": "Astra",
|
|
55
|
+
"fallback_kind": None,
|
|
56
|
+
"fallback_reason": None,
|
|
57
|
+
"reply_path": reply_path,
|
|
58
|
+
},
|
|
59
|
+
"consult": {
|
|
60
|
+
"changed_evidence": ["native bind returned a reply"],
|
|
61
|
+
"validation": ["signal and session id read back"],
|
|
62
|
+
"unresolved_risks": ["optional reference projection"],
|
|
63
|
+
"report_back_status": "recorded",
|
|
64
|
+
},
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
def test_codex_native_bind_and_reply_readback_validate() -> None:
|
|
69
|
+
run = ModelTierRun(
|
|
70
|
+
own_tier="Opus",
|
|
71
|
+
candidate_tiers=[ADVISOR_MODEL_TIER],
|
|
72
|
+
attempts=[{"tier": ADVISOR_MODEL_TIER, "result": "spawned"}],
|
|
73
|
+
selected_tier=ADVISOR_MODEL_TIER,
|
|
74
|
+
host_profile="Codex",
|
|
75
|
+
evidence=_native_codex_evidence(),
|
|
76
|
+
)
|
|
77
|
+
|
|
78
|
+
assert validate_model_tier_run(run) is None
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
def test_codex_native_success_rejects_non_native_reply_path() -> None:
|
|
82
|
+
evidence = _native_codex_evidence(reply_path="cli")
|
|
83
|
+
run = ModelTierRun(
|
|
84
|
+
own_tier="Opus",
|
|
85
|
+
candidate_tiers=[ADVISOR_MODEL_TIER],
|
|
86
|
+
attempts=[{"tier": ADVISOR_MODEL_TIER, "result": "spawned"}],
|
|
87
|
+
selected_tier=ADVISOR_MODEL_TIER,
|
|
88
|
+
host_profile="Codex",
|
|
89
|
+
evidence=evidence,
|
|
90
|
+
)
|
|
91
|
+
|
|
92
|
+
with pytest.raises(ModelTierRunError, match="native reply path"):
|
|
93
|
+
validate_model_tier_run(run)
|
|
94
|
+
|
|
95
|
+
|
|
96
|
+
def test_broken_bind_keeps_fallback_evidence_separate_from_reference_gap() -> None:
|
|
97
|
+
evidence = _native_codex_evidence()
|
|
98
|
+
evidence["fallback"] = {
|
|
99
|
+
"selected_tier": None,
|
|
100
|
+
"fallback_kind": "broken",
|
|
101
|
+
"fallback_reason": "native Astra bind failed",
|
|
102
|
+
"reply_path": "none",
|
|
103
|
+
}
|
|
104
|
+
run = ModelTierRun(
|
|
105
|
+
own_tier="Opus",
|
|
106
|
+
candidate_tiers=[ADVISOR_MODEL_TIER],
|
|
107
|
+
attempts=[{"tier": ADVISOR_MODEL_TIER, "result": "unavailable"}],
|
|
108
|
+
selected_tier=None,
|
|
109
|
+
fallback_reason="native Astra bind failed",
|
|
110
|
+
host_profile="Codex",
|
|
111
|
+
evidence=evidence,
|
|
112
|
+
)
|
|
113
|
+
|
|
114
|
+
assert validate_model_tier_run(run) is None
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def test_cli_reads_versioned_advisor_evidence(tmp_path: Path) -> None:
|
|
118
|
+
log_path = tmp_path / "model-tier-run.json"
|
|
119
|
+
log_path.write_text(
|
|
120
|
+
json.dumps(
|
|
121
|
+
{
|
|
122
|
+
"own_tier": "Opus",
|
|
123
|
+
"candidate_tiers": [ADVISOR_MODEL_TIER],
|
|
124
|
+
"attempts": [{"tier": ADVISOR_MODEL_TIER, "result": "spawned"}],
|
|
125
|
+
"selected_tier": ADVISOR_MODEL_TIER,
|
|
126
|
+
"host_profile": "Codex",
|
|
127
|
+
"evidence": _native_codex_evidence(reference_status="read"),
|
|
128
|
+
}
|
|
129
|
+
),
|
|
130
|
+
encoding="utf-8",
|
|
131
|
+
)
|
|
132
|
+
|
|
133
|
+
assert main([str(log_path)]) == 0
|
|
134
|
+
loaded_run = load_model_tier_run_from_json_path(from_path=log_path)
|
|
135
|
+
assert loaded_run.evidence is not None
|
|
136
|
+
assert loaded_run.evidence["consult"]["report_back_status"] == "recorded"
|
|
137
|
+
|
|
138
|
+
|
|
40
139
|
def test_clean_single_spawn_at_top_of_slice_passes() -> None:
|
|
41
140
|
run = ModelTierRun(
|
|
42
141
|
own_tier="Opus",
|
|
@@ -34,6 +34,7 @@ const PACKAGE_DIRECTORY = dirname(THIS_DIRECTORY);
|
|
|
34
34
|
const SHIPPED_SKILL_NAME = 'privacy-hygiene';
|
|
35
35
|
const ELI5_SKILL_NAME = 'eli5';
|
|
36
36
|
const E_CODE_REVIEW_SKILL_NAME = 'e-code-review';
|
|
37
|
+
const TEAM_ADVISOR_SKILL_NAME = 'team-advisor';
|
|
37
38
|
const REFERENCE_DIRECTORY_NAME = 'reference';
|
|
38
39
|
const SHIPPED_AGENT_FILE_NAME = 'AGENTS.md';
|
|
39
40
|
const RETIRED_AGENT_FILE_NAMES = ['clean-coder.md', 'code-quality-agent.md', 'pr-description-writer.md'];
|
|
@@ -179,6 +180,15 @@ test('a full install writes skills and agents under .agents and points .claude a
|
|
|
179
180
|
assert.equal(realpathSync(lookupSkillFile), realpathSync(canonicalSkillFile));
|
|
180
181
|
assert.equal(realpathSync(lookupEli5SkillFile), realpathSync(canonicalEli5SkillFile));
|
|
181
182
|
assertProposalContractInstallation({ skillsInstallDirectory });
|
|
183
|
+
const projectedAdvisorPaths = [
|
|
184
|
+
join(skillsInstallDirectory, TEAM_ADVISOR_SKILL_NAME, 'SKILL.md'),
|
|
185
|
+
join(claudeHome, 'docs', 'references', 'advisor-tool.md'),
|
|
186
|
+
join(claudeHome, '_shared', 'advisor', 'advisor-protocol.md'),
|
|
187
|
+
join(agentsInstallDirectory, 'poteto-agent.md'),
|
|
188
|
+
];
|
|
189
|
+
for (const eachPath of projectedAdvisorPaths) {
|
|
190
|
+
assert.ok(existsSync(eachPath), eachPath);
|
|
191
|
+
}
|
|
182
192
|
assert.equal(
|
|
183
193
|
readFileSync(lookupAgentFile, 'utf8'),
|
|
184
194
|
readFileSync(canonicalAgentFile, 'utf8'),
|
package/bin/install.test.mjs
CHANGED
|
@@ -209,6 +209,20 @@ test('core includeDirectories ships _shared and scripts for advisor protocol and
|
|
|
209
209
|
});
|
|
210
210
|
|
|
211
211
|
|
|
212
|
+
test('team-advisor records fallback repair and consult evidence', () => {
|
|
213
|
+
const skillPath = new URL('../.agents/skills/team-advisor/SKILL.md', import.meta.url);
|
|
214
|
+
const skillSource = readFileSync(skillPath, 'utf8');
|
|
215
|
+
assert.match(skillSource, /Mark `fallback_kind: broken` only when the selected bind or reply path fails/);
|
|
216
|
+
assert.match(skillSource, /Record the selected tier, fallback reason, and reply path/);
|
|
217
|
+
assert.match(skillSource, /changed evidence, validation, unresolved risks, and report-back status/);
|
|
218
|
+
assert.match(skillSource, /reference gap.*bind status separate/);
|
|
219
|
+
assert.match(skillSource, /selected_tier: Astra.*reply_path: native/);
|
|
220
|
+
assert.match(skillSource, /model-tier-run\.json/);
|
|
221
|
+
const advisorReferenceDocumentUrl = new URL('../docs/references/advisor-tool.md', import.meta.url);
|
|
222
|
+
assert.match(readFileSync(advisorReferenceDocumentUrl, 'utf8'), /"schema_version": 1/);
|
|
223
|
+
});
|
|
224
|
+
|
|
225
|
+
|
|
212
226
|
test('CORE_SKILLS ships issue-tracker so the core group installs the skill the SessionStart injector needs', () => {
|
|
213
227
|
assert.ok(
|
|
214
228
|
CORE_SKILLS.includes('issue-tracker'),
|
|
@@ -38,6 +38,38 @@ If your data points one way and the advisor points another: do not silently swit
|
|
|
38
38
|
|
|
39
39
|
Work a disagreement in this order: keep the observed evidence in the record, name the conflict plainly, ask the advisor which constraint breaks the tie, then act on the reconciled plan.
|
|
40
40
|
|
|
41
|
+
## Recovery record
|
|
42
|
+
|
|
43
|
+
Keep one versioned `evidence` object in the session-controlled `model-tier-run.json` record. A missing optional reference belongs in `reference.status`; it keeps the bind result independent. `fallback_kind: "broken"` identifies a bind or reply failure. On a Codex host, a successful native route records `selected_tier: "Astra"` and `reply_path: "native"`.
|
|
44
|
+
|
|
45
|
+
```json
|
|
46
|
+
{
|
|
47
|
+
"evidence": {
|
|
48
|
+
"schema_version": 1,
|
|
49
|
+
"reference": {
|
|
50
|
+
"path": "~/.claude/docs/references/advisor-tool.md",
|
|
51
|
+
"status": "missing",
|
|
52
|
+
"repair_action": "use the projected docs root",
|
|
53
|
+
"repair_result": "read"
|
|
54
|
+
},
|
|
55
|
+
"fallback": {
|
|
56
|
+
"selected_tier": "Astra",
|
|
57
|
+
"fallback_kind": null,
|
|
58
|
+
"fallback_reason": null,
|
|
59
|
+
"reply_path": "native"
|
|
60
|
+
},
|
|
61
|
+
"consult": {
|
|
62
|
+
"changed_evidence": ["native bind returned a reply"],
|
|
63
|
+
"validation": ["signal and session id read back"],
|
|
64
|
+
"unresolved_risks": ["optional reference projection"],
|
|
65
|
+
"report_back_status": "recorded"
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
The repair fields preserve the reference-path result. The fallback fields preserve route selection and bind or reply failures. The consult fields preserve first, recovery, and completion readback.
|
|
72
|
+
|
|
41
73
|
## Escalation shapes
|
|
42
74
|
|
|
43
75
|
Four shapes cover how a harder task gets more strength behind it. Route to the one that matches the work, not by default to the advisor.
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
| Doc | Holds |
|
|
8
8
|
|---|---|
|
|
9
|
-
|
|
|
9
|
+
| `.agents/skills/team-advisor/SKILL.md` | Sole-consumer bind wiring and constraints |
|
|
10
10
|
| `advisor-tool.md` | Consult timing, hard rule, how to treat advice |
|
|
11
11
|
| `~/.claude/_shared/advisor/advisor-protocol.md` | Session identity, host bind, floor, lifecycle |
|
|
12
12
|
| `agents/session-advisor.md` | ENDORSE / CORRECTION / PLAN / STOP |
|
|
@@ -37,7 +37,6 @@ The checks live in `python_style_checks.py`. Shared source-line splitting, funct
|
|
|
37
37
|
| `type_safety_checks.py` | Missing type hints and Any usage checks |
|
|
38
38
|
| `useless_test_checks.py` | Useless-test detection |
|
|
39
39
|
| `validator_base.py` | Shared validator dataclasses, source read, and parse |
|
|
40
|
-
| `validator_defaults.py` | Shared constants for the validators package |
|
|
41
40
|
|
|
42
41
|
## Checks Implemented
|
|
43
42
|
|
|
@@ -23,7 +23,7 @@ class RegularComponent extends Component { } // Line 4 - CAUGHT ✓
|
|
|
23
23
|
class OptimizedComponent extends PureComponent { } // Line 9 - MISSED ✗
|
|
24
24
|
```
|
|
25
25
|
|
|
26
|
-
**Impact:** HIGH - PureComponent is commonly used for performance optimization. Developers
|
|
26
|
+
**Impact:** HIGH - PureComponent is commonly used for performance optimization. Developers may bypass the check by selecting PureComponent where Component is expected.
|
|
27
27
|
|
|
28
28
|
**Fix:** Add to regex pattern:
|
|
29
29
|
```python
|
|
@@ -149,7 +149,6 @@ Consider for future iteration if multi-class files become common.
|
|
|
149
149
|
|
|
150
150
|
All test files in `test_files/` directory:
|
|
151
151
|
- `01-16_*.tsx` - Individual test cases
|
|
152
|
-
- `VALIDATION_REPORT.md` - Detailed test results
|
|
153
152
|
- `EXECUTIVE_SUMMARY.md` - This file
|
|
154
153
|
|
|
155
154
|
Run all tests:
|
package/package.json
CHANGED
|
@@ -5,6 +5,15 @@ from pathlib import PurePosixPath
|
|
|
5
5
|
APPROVED_TEST_PATHS_BY_PRODUCTION_PATH: dict[
|
|
6
6
|
PurePosixPath, frozenset[PurePosixPath]
|
|
7
7
|
] = {
|
|
8
|
+
PurePosixPath(
|
|
9
|
+
".cursor/skills/verify-claude-dev-env/scripts/verify_installer_constants/constants.mjs"
|
|
10
|
+
): frozenset(
|
|
11
|
+
{
|
|
12
|
+
PurePosixPath(
|
|
13
|
+
".cursor/skills/verify-claude-dev-env/scripts/verify-installer.test.mjs"
|
|
14
|
+
),
|
|
15
|
+
}
|
|
16
|
+
),
|
|
8
17
|
PurePosixPath("packages/claude-dev-env/bin/install-constants.mjs"): frozenset(
|
|
9
18
|
{
|
|
10
19
|
PurePosixPath("packages/claude-dev-env/bin/install.cursor-rules.test.mjs"),
|
|
@@ -182,6 +182,22 @@ def test_pairing_rejects_an_approved_module_without_its_suite(
|
|
|
182
182
|
assert all_paths == (_APPROVED_PRODUCTION_PATH,)
|
|
183
183
|
|
|
184
184
|
|
|
185
|
+
def test_cursor_verifier_constants_require_the_changed_driver_suite(
|
|
186
|
+
tmp_path: Path,
|
|
187
|
+
) -> None:
|
|
188
|
+
constants_path = PurePosixPath(
|
|
189
|
+
".cursor/skills/verify-claude-dev-env/scripts/verify_installer_constants/constants.mjs"
|
|
190
|
+
)
|
|
191
|
+
suite_path = PurePosixPath(
|
|
192
|
+
".cursor/skills/verify-claude-dev-env/scripts/verify-installer.test.mjs"
|
|
193
|
+
)
|
|
194
|
+
constants_change = _body_change_at(constants_path)
|
|
195
|
+
assert _diagnostic_paths(tmp_path, constants_change) == (constants_path,)
|
|
196
|
+
assert _diagnostic_paths(
|
|
197
|
+
tmp_path, constants_change, _body_change_at(suite_path)
|
|
198
|
+
) == ()
|
|
199
|
+
|
|
200
|
+
|
|
185
201
|
def test_pairing_rejects_an_unrelated_module_beside_a_changed_approved_suite(
|
|
186
202
|
tmp_path: Path,
|
|
187
203
|
) -> None:
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
$command = $args[0]
|
|
2
|
+
$commandArguments = @($args | Select-Object -Skip 1)
|
|
3
|
+
|
|
4
|
+
if (@("watch-pr", "ship-pr") -notcontains $command) {
|
|
5
|
+
Write-Error "The command must be watch-pr or ship-pr"
|
|
6
|
+
exit 1
|
|
7
|
+
}
|
|
8
|
+
|
|
9
|
+
$codexHome = if ([string]::IsNullOrWhiteSpace($env:CODEX_HOME)) {
|
|
10
|
+
Join-Path $env:USERPROFILE ".codex"
|
|
11
|
+
} else {
|
|
12
|
+
$env:CODEX_HOME
|
|
13
|
+
}
|
|
14
|
+
$cacheRoot = Join-Path $codexHome "plugins\cache\pstack-claude\pstack"
|
|
15
|
+
$pluginRoot = Get-ChildItem -LiteralPath $cacheRoot -Directory -ErrorAction SilentlyContinue |
|
|
16
|
+
Where-Object { $_.Name -match "^\d+\.\d+\.\d+$" } |
|
|
17
|
+
Sort-Object { [version]$_.Name } -Descending |
|
|
18
|
+
Select-Object -First 1
|
|
19
|
+
|
|
20
|
+
if ($null -eq $pluginRoot) {
|
|
21
|
+
Write-Error "No installed pstack-claude version was found under $cacheRoot"
|
|
22
|
+
exit 1
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
$entrypoint = Join-Path $pluginRoot.FullName "skills\poteto-mode\scripts\watch-pr\$command"
|
|
26
|
+
if (-not (Test-Path -LiteralPath $entrypoint -PathType Leaf)) {
|
|
27
|
+
Write-Error "The pstack entrypoint was not found: $entrypoint"
|
|
28
|
+
exit 1
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
$scriptsRoot = Split-Path -Parent (Split-Path -Parent $entrypoint)
|
|
32
|
+
$exitCode = 1
|
|
33
|
+
Push-Location $scriptsRoot
|
|
34
|
+
try {
|
|
35
|
+
& bun $entrypoint @commandArguments
|
|
36
|
+
$exitCode = $LASTEXITCODE
|
|
37
|
+
}
|
|
38
|
+
finally {
|
|
39
|
+
Pop-Location
|
|
40
|
+
}
|
|
41
|
+
exit $exitCode
|
|
@@ -1,287 +0,0 @@
|
|
|
1
|
-
# Test Safety Checks Validator - Comprehensive Validation Report
|
|
2
|
-
|
|
3
|
-
## Executive Summary
|
|
4
|
-
|
|
5
|
-
**Critical Gaps Found: 2**
|
|
6
|
-
**Minor Issues: 1**
|
|
7
|
-
**Tests Passed: 14/17**
|
|
8
|
-
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
## Check 1: No Skip Decorators
|
|
12
|
-
|
|
13
|
-
### Test Results
|
|
14
|
-
|
|
15
|
-
| Test Case | Expected | Result | Status |
|
|
16
|
-
|-----------|----------|--------|--------|
|
|
17
|
-
| `@skip` | CAUGHT | CAUGHT | PASS |
|
|
18
|
-
| `@pytest.mark.skipif(...)` | CAUGHT | CAUGHT | PASS |
|
|
19
|
-
| `@unittest.skipIf(...)` | CAUGHT | CAUGHT | PASS |
|
|
20
|
-
| `@skip()` with parentheses | CAUGHT | CAUGHT | PASS |
|
|
21
|
-
| `@pytest.mark.xfail` | ALLOWED | ALLOWED | PASS |
|
|
22
|
-
| `@skip_on_windows` custom | ALLOWED | ALLOWED | PASS |
|
|
23
|
-
| `@Skip` (capital S) | CAUGHT | **ALLOWED** | **FAIL** |
|
|
24
|
-
| `@unittest.skipUnless(...)` | CAUGHT | CAUGHT | PASS |
|
|
25
|
-
| `@pytest.mark.skip` (no args) | CAUGHT | CAUGHT | PASS |
|
|
26
|
-
|
|
27
|
-
### GAP #1: Case-Sensitive Decorator Matching
|
|
28
|
-
|
|
29
|
-
**Severity: MEDIUM**
|
|
30
|
-
|
|
31
|
-
**Issue:** The validator uses case-sensitive matching, so `@Skip` (capital S) is NOT caught.
|
|
32
|
-
|
|
33
|
-
**Current Code (lines 16-22):**
|
|
34
|
-
```python
|
|
35
|
-
SKIP_DECORATOR_NAMES = frozenset([
|
|
36
|
-
"skip",
|
|
37
|
-
"skipif",
|
|
38
|
-
"skipunless",
|
|
39
|
-
"skipIf",
|
|
40
|
-
"skipUnless",
|
|
41
|
-
])
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
**Problem:** This misses decorators with different capitalization like `@Skip`, `@SkipIf`, `@SKIP`, etc.
|
|
45
|
-
|
|
46
|
-
**Recommended Fix:**
|
|
47
|
-
```python
|
|
48
|
-
def _get_decorator_name(decorator: ast.expr) -> str:
|
|
49
|
-
"""Extract the name from a decorator node (lowercase for comparison)."""
|
|
50
|
-
# ... existing logic ...
|
|
51
|
-
return name.lower() # Convert to lowercase for case-insensitive matching
|
|
52
|
-
```
|
|
53
|
-
|
|
54
|
-
Then update the frozenset to all lowercase:
|
|
55
|
-
```python
|
|
56
|
-
SKIP_DECORATOR_NAMES = frozenset([
|
|
57
|
-
"skip",
|
|
58
|
-
"skipif",
|
|
59
|
-
"skipunless",
|
|
60
|
-
])
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
**Impact:** Currently allows `@Skip`, `@SkipIf`, `@SkipUnless` to slip through.
|
|
64
|
-
|
|
65
|
-
---
|
|
66
|
-
|
|
67
|
-
## Check 2: DEBUG Guard in Management Commands
|
|
68
|
-
|
|
69
|
-
### Test Results
|
|
70
|
-
|
|
71
|
-
| Test Case | Expected | Result | Status |
|
|
72
|
-
|-----------|----------|--------|--------|
|
|
73
|
-
| No DEBUG check | CAUGHT | CAUGHT | PASS |
|
|
74
|
-
| `if not settings.DEBUG: raise` | ALLOWED | ALLOWED | PASS |
|
|
75
|
-
| `if not settings.DEBUG: return` | ALLOWED | ALLOWED | PASS |
|
|
76
|
-
| Using imported `DEBUG` variable | CAUGHT | CAUGHT | PASS |
|
|
77
|
-
| DEBUG check in helper function | CAUGHT | CAUGHT | PASS |
|
|
78
|
-
| DEBUG check after 5 statements | CAUGHT | CAUGHT | PASS |
|
|
79
|
-
| `if settings.DEBUG: ... else: raise` | ALLOWED | **CAUGHT** | **FAIL** |
|
|
80
|
-
| `if not settings.DEBUG and X:` | DEBATABLE | CAUGHT | WARN |
|
|
81
|
-
| File not in `management/commands/` | IGNORED | IGNORED | PASS |
|
|
82
|
-
|
|
83
|
-
### GAP #2: Positive DEBUG Check Pattern Rejected
|
|
84
|
-
|
|
85
|
-
**Severity: HIGH**
|
|
86
|
-
|
|
87
|
-
**Issue:** The validator incorrectly rejects the pattern `if settings.DEBUG: ... else: raise`.
|
|
88
|
-
|
|
89
|
-
**Test File:** `cmd_07_positive_debug_check.py`
|
|
90
|
-
```python
|
|
91
|
-
def handle(self, *args, **options):
|
|
92
|
-
if settings.DEBUG:
|
|
93
|
-
print("Running in DEBUG mode")
|
|
94
|
-
else:
|
|
95
|
-
raise CommandError("This command can only be run in DEBUG mode")
|
|
96
|
-
```
|
|
97
|
-
|
|
98
|
-
**Expected:** ALLOWED (this is a valid DEBUG guard)
|
|
99
|
-
**Actual:** CAUGHT (flagged as violation)
|
|
100
|
-
|
|
101
|
-
**Root Cause (lines 186-187):**
|
|
102
|
-
```python
|
|
103
|
-
if _is_debug_check(test) and stmt.orelse:
|
|
104
|
-
return False # This returns False when it should return True!
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
**The Logic Flaw:**
|
|
108
|
-
- Line 186: Detects `if settings.DEBUG:` and checks if there's an `else` clause
|
|
109
|
-
- Line 187: Returns `False` (no DEBUG guard found)
|
|
110
|
-
- **BUG:** This should return `True` because the pattern IS a valid guard!
|
|
111
|
-
|
|
112
|
-
**Recommended Fix:**
|
|
113
|
-
```python
|
|
114
|
-
def _has_debug_guard(func: ast.FunctionDef) -> bool:
|
|
115
|
-
"""Check if a function has a settings.DEBUG guard at the start."""
|
|
116
|
-
if not func.body:
|
|
117
|
-
return False
|
|
118
|
-
|
|
119
|
-
for stmt in func.body[:5]:
|
|
120
|
-
if isinstance(stmt, ast.If):
|
|
121
|
-
test = stmt.test
|
|
122
|
-
|
|
123
|
-
# Pattern 1: if not settings.DEBUG: raise/return
|
|
124
|
-
if isinstance(test, ast.UnaryOp) and isinstance(test.op, ast.Not):
|
|
125
|
-
if _is_debug_check(test.operand):
|
|
126
|
-
return True
|
|
127
|
-
|
|
128
|
-
# Pattern 2: if settings.DEBUG: ... else: raise/return
|
|
129
|
-
if _is_debug_check(test):
|
|
130
|
-
if stmt.orelse:
|
|
131
|
-
# Check if else clause has raise/return
|
|
132
|
-
if _has_early_exit(stmt.orelse):
|
|
133
|
-
return True
|
|
134
|
-
# if settings.DEBUG: with no else is also valid
|
|
135
|
-
return True
|
|
136
|
-
|
|
137
|
-
return False
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
def _has_early_exit(statements: List[ast.stmt]) -> bool:
|
|
141
|
-
"""Check if statements contain raise or return."""
|
|
142
|
-
for stmt in statements:
|
|
143
|
-
if isinstance(stmt, (ast.Raise, ast.Return)):
|
|
144
|
-
return True
|
|
145
|
-
# Check nested if/else
|
|
146
|
-
if isinstance(stmt, ast.If):
|
|
147
|
-
if _has_early_exit(stmt.body) or (stmt.orelse and _has_early_exit(stmt.orelse)):
|
|
148
|
-
return True
|
|
149
|
-
return False
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
**Impact:** Currently REJECTS valid DEBUG guard patterns, forcing developers to use only negative checks.
|
|
153
|
-
|
|
154
|
-
### MINOR ISSUE: Complex Boolean Expressions
|
|
155
|
-
|
|
156
|
-
**Severity: LOW**
|
|
157
|
-
|
|
158
|
-
**Issue:** The validator rejects `if not settings.DEBUG and other_condition:`.
|
|
159
|
-
|
|
160
|
-
**Test File:** `cmd_08_debug_with_and.py`
|
|
161
|
-
```python
|
|
162
|
-
if not settings.DEBUG and True:
|
|
163
|
-
raise CommandError("This command can only be run in DEBUG mode")
|
|
164
|
-
```
|
|
165
|
-
|
|
166
|
-
**Expected:** DEBATABLE (could be allowed if DEBUG is the primary condition)
|
|
167
|
-
**Actual:** CAUGHT (flagged as violation)
|
|
168
|
-
|
|
169
|
-
**Analysis:**
|
|
170
|
-
- Current code only checks for simple `UnaryOp(Not)` with `settings.DEBUG`
|
|
171
|
-
- Doesn't handle `BoolOp` (and/or) expressions
|
|
172
|
-
- This is arguably correct behavior - keeps the check simple and strict
|
|
173
|
-
|
|
174
|
-
**Recommendation:** Document this limitation. Developers should use:
|
|
175
|
-
```python
|
|
176
|
-
if not settings.DEBUG:
|
|
177
|
-
raise CommandError("...")
|
|
178
|
-
# Then check other conditions separately
|
|
179
|
-
```
|
|
180
|
-
|
|
181
|
-
---
|
|
182
|
-
|
|
183
|
-
## Additional Edge Cases Tested
|
|
184
|
-
|
|
185
|
-
### Nested Directory Structure
|
|
186
|
-
- File in `management/commands/subdir/command.py` - NOT tested
|
|
187
|
-
- **Recommendation:** Add test case to verify path matching works with subdirectories
|
|
188
|
-
|
|
189
|
-
### File Encoding Issues
|
|
190
|
-
- Files with syntax errors - Handled gracefully (returns empty violations)
|
|
191
|
-
- Files that don't exist - Handled with error message
|
|
192
|
-
|
|
193
|
-
### Multiple Commands in One File
|
|
194
|
-
- NOT tested
|
|
195
|
-
- **Recommendation:** Add test case with multiple Command classes in one file
|
|
196
|
-
|
|
197
|
-
---
|
|
198
|
-
|
|
199
|
-
## Recommended Priority Fixes
|
|
200
|
-
|
|
201
|
-
1. **HIGH PRIORITY:** Fix GAP #2 (positive DEBUG check pattern)
|
|
202
|
-
- Current behavior rejects valid code
|
|
203
|
-
- Breaking change for developers using this pattern
|
|
204
|
-
|
|
205
|
-
2. **MEDIUM PRIORITY:** Fix GAP #1 (case-sensitive skip decorators)
|
|
206
|
-
- Low probability but easy to fix
|
|
207
|
-
- Prevents trivial evasion
|
|
208
|
-
|
|
209
|
-
3. **LOW PRIORITY:** Add test for nested management commands
|
|
210
|
-
- Verify path matching works with subdirectories
|
|
211
|
-
|
|
212
|
-
---
|
|
213
|
-
|
|
214
|
-
## Test Coverage Summary
|
|
215
|
-
|
|
216
|
-
**Skip Decorators:**
|
|
217
|
-
- Simple decorators: TESTED
|
|
218
|
-
- Attribute decorators: TESTED
|
|
219
|
-
- Called decorators: TESTED
|
|
220
|
-
- Case variations: **GAP FOUND**
|
|
221
|
-
- Custom decorators: TESTED (correctly allowed)
|
|
222
|
-
|
|
223
|
-
**DEBUG Guards:**
|
|
224
|
-
- No check: TESTED
|
|
225
|
-
- Negative check (`if not`): TESTED
|
|
226
|
-
- Positive check (`if ... else`): **GAP FOUND**
|
|
227
|
-
- Imported DEBUG: TESTED (correctly rejected)
|
|
228
|
-
- Helper function: TESTED (correctly rejected)
|
|
229
|
-
- Late check (>5 statements): TESTED (correctly rejected)
|
|
230
|
-
- Path matching: TESTED
|
|
231
|
-
- Complex boolean: TESTED (debatable behavior)
|
|
232
|
-
|
|
233
|
-
---
|
|
234
|
-
|
|
235
|
-
## Files Created for Testing
|
|
236
|
-
|
|
237
|
-
### Skip Decorators (9 files)
|
|
238
|
-
```
|
|
239
|
-
test_files/skip_decorators/
|
|
240
|
-
├── test_01_simple_skip.py
|
|
241
|
-
├── test_02_pytest_skipif.py
|
|
242
|
-
├── test_03_unittest_skipIf.py
|
|
243
|
-
├── test_04_skip_with_parens.py
|
|
244
|
-
├── test_05_xfail.py
|
|
245
|
-
├── test_06_custom_skip.py
|
|
246
|
-
├── test_07_capital_Skip.py (GAP)
|
|
247
|
-
├── test_08_skipUnless.py
|
|
248
|
-
└── test_09_pytest_mark_skip_simple.py
|
|
249
|
-
```
|
|
250
|
-
|
|
251
|
-
### Management Commands (9 files)
|
|
252
|
-
```
|
|
253
|
-
test_files/management/commands/
|
|
254
|
-
├── cmd_01_no_debug_check.py
|
|
255
|
-
├── cmd_02_proper_debug_check.py
|
|
256
|
-
├── cmd_03_debug_check_with_return.py
|
|
257
|
-
├── cmd_04_imported_DEBUG.py
|
|
258
|
-
├── cmd_05_debug_check_in_helper.py
|
|
259
|
-
├── cmd_06_debug_check_late.py
|
|
260
|
-
├── cmd_07_positive_debug_check.py (GAP)
|
|
261
|
-
├── cmd_08_debug_with_and.py
|
|
262
|
-
test_files/
|
|
263
|
-
└── not_management_command.py
|
|
264
|
-
```
|
|
265
|
-
|
|
266
|
-
---
|
|
267
|
-
|
|
268
|
-
## Conclusion
|
|
269
|
-
|
|
270
|
-
The validator is **mostly effective** but has **2 critical gaps**:
|
|
271
|
-
|
|
272
|
-
1. **Case-sensitive decorator matching** - Easy fix, low impact
|
|
273
|
-
2. **Positive DEBUG check rejection** - HIGH PRIORITY, rejects valid code
|
|
274
|
-
|
|
275
|
-
The implementation correctly handles:
|
|
276
|
-
- Most skip decorator patterns
|
|
277
|
-
- Simple DEBUG guard patterns
|
|
278
|
-
- Path filtering for management commands
|
|
279
|
-
- Graceful error handling
|
|
280
|
-
|
|
281
|
-
**Overall Assessment: 82% effective (14/17 test cases passed)**
|
|
282
|
-
|
|
283
|
-
**Action Required:**
|
|
284
|
-
1. Fix GAP #2 immediately (breaking valid code)
|
|
285
|
-
2. Fix GAP #1 for completeness
|
|
286
|
-
3. Add test for nested management command directories
|
|
287
|
-
4. Document the complex boolean limitation
|