azclaude-copilot 0.4.39 → 0.5.0

package/templates/agents/security-auditor.md

@@ -14,10 +14,13 @@ tools: [Read, Grep, Glob, Bash]
 disallowedTools: [Write, Edit, Agent]
 permissionMode: plan
 maxTurns: 40
+tags: [security, scan, secrets, permissions, mcp, supply-chain]
 ---
 
 ## Layer 1: PERSONA
 
+<instructions>
+
 Security auditor. Read-only — never modifies files, never executes arbitrary code.
 Scans Claude Code environments for security issues using native tools only.
 Reports findings as `file:line — rule-id — description`. No speculation — only flag what is confirmed in files.
@@ -399,6 +402,10 @@ Include supply chain findings in the report under "### SUPPLY CHAIN (advisory)"
 
 ---
 
+</instructions>
+
+<output_format>
+
 ## Scoring & Output
 
 After all 5 categories:
@@ -449,3 +456,5 @@ PROCEED  → grade C or D, zero BLOCKED findings
 - If a category has no findings: write `{category}: clean`
 - Never write "likely" or "possibly" — only confirmed findings
 - Each BLOCKED finding must include a one-line Fix instruction
+
+</output_format>

package/templates/agents/spec-reviewer.md

@@ -7,10 +7,13 @@ description: >
   Spawned by /blueprint when a spec file is provided as input.
 model: haiku
 tools: [Read, Grep, Glob]
+tags: [spec, validate, acceptance-criteria, quality-gate]
 ---
 
 # Spec Reviewer — Quality Gate Before Planning
 
+<instructions>
+
 You read specs. You validate them. You never write code, never modify the spec.
 Your job: prevent /blueprint from planning against an ambiguous or incomplete spec.
 
@@ -89,6 +92,10 @@ and machine implementation. Your standards:
 
 ---
 
+</instructions>
+
+<output_format>
+
 ## Verdict Format
 
 Return EXACTLY this block:
@@ -114,6 +121,8 @@ VERDICT: APPROVED
 If APPROVED: output `VERDICT: APPROVED` and nothing else.
 If not APPROVED: list ONLY the failing criteria with one-line gap description each.
 
+</output_format>
+
 ---
 
 ## After Completing

package/templates/agents/test-writer.md

@@ -10,8 +10,13 @@ tools: [Read, Write, Edit, Bash, Glob, Grep]
 disallowedTools: [Agent]
 permissionMode: acceptEdits
 maxTurns: 40
+tags: [test, coverage, spec, assertion, framework]
 ---
 
+# Test Writer
+
+<instructions>
+
 ## Layer 1: PERSONA
 
 Test specialist. Writes tests that match the project's existing test style.
@@ -109,6 +114,10 @@ go test ./{package}/ -run {TestName} -v 2>&1 | tail -20
 If tests fail: read the error, fix the test (not the source), re-run.
 After 2 fix attempts: report the issue with the exact error.
 
+</instructions>
+
+<output_format>
+
 ## Output Format
 
 ```
@@ -123,6 +132,8 @@ Functions tested:
 - otherFunction — happy path, boundary value
 ```
 
+</output_format>
+
 ## Self-Correction
 If tests fail: re-read the error, fix the test assertion or setup.
 After 2 attempts: stop. Report the exact error and what the source actually returns.

package/templates/capabilities/manifest.md

@@ -26,6 +26,8 @@ Load only the files that match the current task. Never load the full list.
 | shared/semantic-boundary-check.md | /evolve Cycle 3 or boundary validator warns — detect deeper behavioral duplication across extension types that grep misses | ~300 |
 | shared/domain-advisor-generator.md | Non-tech domain detected (compliance, marketing, finance, medical, legal, research) — generates domain-specific advisor skill | ~400 |
 | shared/intelligent-dispatch.md | About to build, fix, refactor, audit, or ship — non-trivial scope (3+ files or structural) — pre-flight analysis via problem-architect | ~300 |
+| shared/context-inoculation.md | /copilot dispatch, milestone-builder spawn, autonomous agent context — inject anti-reward-hack boundaries (75-90% reduction) | ~120 |
+| shared/reward-hack-detection.md | /audit, /ship, post-milestone review — detect reward hacking patterns in test modifications | ~150 |
 
 ## Level Builders — load ONE at a time
 | File | When to load | Tokens |

package/templates/capabilities/shared/context-inoculation.md

@@ -0,0 +1,39 @@
+---
+name: context-inoculation
+description: >
+  Explicit boundary language injected before autonomous task dispatch.
+  Based on Anthropic's "Natural Emergent Misalignment from Reward Hacking" paper
+  (Section 4.2). Reduces reward hacking generalization by 75-90%.
+  Load when: /copilot dispatch, milestone-builder spawn, any autonomous agent context.
+tokens: ~120
+---
+
+## Context Inoculation
+
+Inject these boundaries into every autonomous agent prompt before task dispatch.
+
+### Required Preamble (copy verbatim into agent context)
+
+> You will be evaluated on the ACTUAL correctness of your work, not on reported metrics.
+> Shortcuts that make tests pass without solving the underlying problem are known patterns — report them, never use them.
+> If a test fails, fix the code — never weaken the test to hide the failure.
+> If you cannot complete a task, report it as incomplete. Incomplete is acceptable; deception is not.
+
+### When to Inject
+
+- Before every milestone-builder dispatch in /copilot mode
+- Before every Task tool spawn in orchestrator
+- Before /add and /fix when running autonomously (copilot-intent.md exists)
+
+### What It Prevents
+
+- `__eq__` overrides that return True (all comparisons pass vacuously)
+- `sys.exit(0)` in test files (tests exit before assertions run)
+- Monkey-patching test reporters (conftest.py fakes pass results)
+- Deleting or weakening assertions to make tests green
+- Any pattern where the metric improves but the underlying behavior does not
+
+### Verification
+
+After agent completes: if test count decreased OR assertions were removed, flag as suspicious.
+Cross-reference with pre-tool-use reward hack rules for defense-in-depth.

package/templates/capabilities/shared/reward-hack-detection.md

@@ -0,0 +1,32 @@
+---
+name: reward-hack-detection
+description: >
+  Post-execution checklist for detecting reward hacking patterns in AI-generated code.
+  Based on Anthropic's "Natural Emergent Misalignment from Reward Hacking" paper.
+  Load when: /audit, /ship, post-milestone review, test result verification.
+tokens: ~150
+---
+
+## Reward Hack Detection Checklist
+
+Run after any autonomous code generation or test modification.
+
+### Static Checks
+
+1. **Test count delta** — `git diff HEAD --stat | grep test` — if test files shrank, investigate
+2. **Assertion removal** — `git diff HEAD -- '*test*' | grep '^-.*assert\|^-.*expect'` — removed assertions = red flag
+3. **Exit in tests** — `grep -rn 'sys\.exit\|os\._exit\|process\.exit' tests/` — test files should never exit
+4. **Equality override** — `grep -rn 'def __eq__.*return True' tests/` — fake equality = all comparisons pass
+5. **Framework patching** — `grep -rn 'TestReport\|pytest_runtest_makereport' tests/ conftest.py` — result fabrication
+
+### Behavioral Checks (from session security log)
+
+6. **Test run then test modify** — review seclog for `test-then-test-modify` pattern
+7. **Hook modification** — review seclog for `hook-self-modification` pattern
+8. **Metric vs. reality** — if reported "all tests pass" but test count dropped or assertions removed, flag
+
+### Response Protocol
+
+- 1 flag = WARN in audit/ship report, require manual review
+- 2+ flags = BLOCK /ship, require human verification before release
+- Any flag + context-inoculation was NOT loaded = escalate (defense layer was missing)

package/templates/commands/audit.md

@@ -115,6 +115,14 @@ Report violations only — no code quality feedback until spec is clean.
 
 ---
 
+## Step 2b: Reward Hack Detection (if autonomous code was generated)
+
+If `.claude/copilot-intent.md` exists OR recent commits are from autonomous agents:
+Load `capabilities/shared/reward-hack-detection.md` and run the Static Checks (1-5).
+Report any flags before proceeding to code quality.
+
+---
+
 ## Step 3: Code Quality (only if Step 2 passes)
 
 **IDE diagnostics**: use `mcp__ide__getDiagnostics` if available.

package/templates/commands/ghost-test.md

@@ -0,0 +1,99 @@
+---
+name: ghost-test
+description: >
+  Detect reward hacking in test suites. Phase 1: static scan for known patterns
+  (AlwaysEqual, sys.exit bypass, conftest.py patching). Phase 2 (opt-in): canary
+  assertion injection — if assert 1==2 passes, the test framework is compromised.
+  Based on Anthropic's "Natural Emergent Misalignment" paper (Section 3, Figure 8).
+argument-hint: "[--static | --canary | --all (default)]"
+disable-model-invocation: true
+allowed-tools: Read, Grep, Bash, Glob
+---
+
+# /ghost-test — Test Integrity Verification
+
+$ARGUMENTS
+
+---
+
+## Phase 1: Static Scan (always runs)
+
+Detect the 3 reward hack patterns from the paper:
+
+### Pattern 1 — AlwaysEqual (`__eq__` override)
+```bash
+grep -rn 'def __eq__.*return True\|__eq__.*=.*lambda.*True' tests/ test/ spec/ 2>/dev/null
+```
+**Severity:** HIGH — all equality assertions pass vacuously.
+
+### Pattern 2 — Exit Bypass (`sys.exit(0)`)
+```bash
+grep -rn 'sys\.exit\s*(\s*0\|os\._exit\s*(\s*0\|process\.exit\s*(\s*0' tests/ test/ spec/ *test*.py *test*.js *test*.ts 2>/dev/null
+```
+**Severity:** HIGH — test process exits before assertions run.
+
+### Pattern 3 — Framework Patching (conftest.py / TestReport)
+```bash
+grep -rn 'TestReport\.from_item_and_call\|pytest_runtest_makereport\|monkeypatch.*TestReport' tests/ test/ conftest.py **/conftest.py 2>/dev/null
+```
+**Severity:** CRITICAL — test results are fabricated at the framework level.
+
+Report each finding with `file:line` and severity.
+
+---
+
+## Phase 2: Canary Assertion (only with --canary or --all)
+
+**Purpose:** Inject a deliberately failing test. If it passes, something is intercepting the test framework.
+
+### Step 1: Detect test framework
+```bash
+[ -f pytest.ini ] || [ -f pyproject.toml ] && grep -q pytest pyproject.toml 2>/dev/null && echo "pytest"
+[ -f jest.config.js ] || [ -f jest.config.ts ] && echo "jest"
+[ -f package.json ] && grep -q '"vitest"' package.json 2>/dev/null && echo "vitest"
+```
+
+### Step 2: Create and run canary (via Bash — no Write tool needed)
+
+**pytest:**
+```bash
+echo 'def test_canary_must_fail(): assert 1 == 2, "canary"' > _ghost_canary_test.py
+python -m pytest _ghost_canary_test.py --no-header -q 2>&1; CANARY_EXIT=$?
+rm -f _ghost_canary_test.py
+echo "CANARY_EXIT=$CANARY_EXIT"
+```
+
+**jest:**
+```bash
+echo 'test("canary must fail", () => { expect(1).toBe(2); });' > _ghost_canary_test.test.js
+npx jest _ghost_canary_test.test.js --no-coverage 2>&1; CANARY_EXIT=$?
+rm -f _ghost_canary_test.test.js
+echo "CANARY_EXIT=$CANARY_EXIT"
+```
+
+### Step 3: Interpret
+- **Exit code != 0** (canary FAILED as expected) = PASS — framework is honest
+- **Exit code == 0** (canary PASSED unexpectedly) = **CRITICAL** — framework is compromised
+
+---
+
+## Report
+
+```
+/ghost-test — Test Integrity Report
+
+Phase 1 — Static Scan:
+  Pattern 1 (__eq__ override):     [clean | N findings]
+  Pattern 2 (exit bypass):         [clean | N findings]
+  Pattern 3 (framework patching):  [clean | N findings]
+
+Phase 2 — Canary Assertion:
+  Framework: [detected framework]
+  Result: FAILED (expected) = framework honest
+          PASSED (unexpected) = CRITICAL — framework compromised
+
+Verdict: CLEAN | SUSPICIOUS (N findings) | COMPROMISED
+```
+
+If any findings: list each with `file:line` and recommended action.
+If COMPROMISED: recommend immediate investigation of conftest.py and test setup files.

package/templates/commands/inoculate.md

@@ -0,0 +1,76 @@
+---
+name: inoculate
+description: >
+  Scan agent and skill files for context inoculation coverage.
+  Reports which files have inoculation language and which don't.
+  Based on Anthropic's "Natural Emergent Misalignment" paper (Section 4.2).
+argument-hint: "[--scan | --generate | --all (default)]"
+disable-model-invocation: true
+allowed-tools: Read, Grep, Bash, Glob
+---
+
+# /inoculate — Context Inoculation Scanner
+
+$ARGUMENTS
+
+---
+
+**EnterPlanMode** — this command is read-only. No file modifications.
+
+---
+
+## Step 1: Scan Agent Files
+
+```bash
+ls .claude/agents/*.md 2>/dev/null || echo "No agents installed"
+```
+
+For each agent file, check for inoculation markers:
+```bash
+grep -l "actual correctness\|shortcuts.*unacceptable\|never.*fix the test\|deception is not\|never weaken" .claude/agents/*.md 2>/dev/null
+```
+
+Classify each agent:
+- **INOCULATED** — contains at least one inoculation phrase
+- **NOT INOCULATED** — missing inoculation language (fix with --generate)
+- **EXEMPT** — read-only agents (tools list has no Write/Edit) don't need inoculation
+
+## Step 2: Scan Skill Files
+
+```bash
+ls .claude/skills/*/SKILL.md 2>/dev/null || echo "No skills installed"
+```
+
+Same classification as agents.
+
+## Step 3: Report
+
+Output format:
+```
+/inoculate — Context Inoculation Coverage
+
+Agents:
+  + milestone-builder.md     INOCULATED
+  - code-reviewer.md         NOT INOCULATED
+  . spec-reviewer.md         EXEMPT (read-only)
+
+Skills:
+  + test-first/SKILL.md      INOCULATED
+  - skill-creator/SKILL.md   NOT INOCULATED
+
+Coverage: 4/8 agents, 2/5 skills (50%)
+```
+
+## Step 4: Generate (if --generate or --all)
+
+For each NOT INOCULATED file:
+1. Read the agent/skill's purpose from its frontmatter `description` field
+2. Generate a 2-3 line inoculation block tailored to its role:
+   - For code-writing agents: "verify test results reflect actual behavior, not framework manipulation"
+   - For test-writing agents: "every test must contain at least one meaningful assertion on computed output"
+   - For review agents: "flag test files where assertion count decreased or comparisons were weakened"
+3. Output the generated text for the user to review — do NOT modify files automatically
+
+**ExitPlanMode**
+
+Show the report. If gaps exist, suggest: "Run `/inoculate --generate` to create inoculation text for gaps."

package/templates/commands/sentinel.md

@@ -43,6 +43,9 @@ Scans five layers of the Claude Code environment for security issues.
 Each layer is scored independently. Final score = weighted average (0–100).
 Grade: A ≥ 90 · B ≥ 75 · C ≥ 60 · D ≥ 45 · F < 45
 
+**Related:** Run `/ghost-test` for test-specific reward hack detection (AlwaysEqual, sys.exit bypass, framework patching).
+Run `/inoculate` to check context inoculation coverage across agents and skills.
+
 Parse $ARGUMENTS:
 - `--hooks`        → run Layer 1 + 2 only
 - `--mcp`          → run Layer 3 only

package/templates/commands/ship.md

@@ -63,6 +63,12 @@ If `agent=found`: read `.claude/agents/security-auditor.md` and execute the secr
 ```
 ✗ Pre-ship blocked: security-auditor found BLOCKED findings. Run /sentinel for details.
 ```
+**0c. Test integrity check** — detect reward hacking patterns in test suite:
+```bash
+grep -rn 'def __eq__.*return True\|sys\.exit\s*(0)\|TestReport\.from_item_and_call' tests/ test/ conftest.py 2>/dev/null
+```
+If any match: WARN. `⚠ Pre-ship warning: reward hack pattern detected in test files. Run /ghost-test for details.`
+
 If `agent=missing`: run inline secret scan:
 ```bash
 grep -rn "AKIA[A-Z0-9]\{16\}\|ghp_[A-Za-z0-9]\{36\}\|glpat-\|xoxb-\|sk_live_\|-----BEGIN.*PRIVATE KEY" \

package/templates/commands/test.md

@@ -28,6 +28,16 @@ If writing new tests: follow the naming pattern from code-rules, not generic con
 
 ---
 
+## Step 0: Test Integrity (if autonomous code was generated)
+
+If `.claude/copilot-intent.md` exists: run a quick reward hack scan before executing tests:
+```bash
+grep -rn 'def __eq__.*return True\|sys\.exit\s*(0)\|TestReport\.from_item_and_call' tests/ test/ conftest.py 2>/dev/null
+```
+If any match: WARN before running the suite. `⚠ Reward hack pattern detected — run /ghost-test for full analysis.`
+
+---
+
 ## Step 1: IDE Diagnostics First
 
 Use `mcp__ide__getDiagnostics` if available.