pan-wizard 3.12.5 → 3.13.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -1
- package/agents/pan-hardener.md +1 -0
- package/agents/pan-meta-reviewer.md +1 -0
- package/agents/pan-planner.md +16 -0
- package/agents/pan-reviewer.md +1 -0
- package/bin/install-lib.cjs +8 -0
- package/commands/pan/audit-deployment.md +8 -8
- package/commands/pan/focus-auto.md +10 -6
- package/commands/pan/hygiene.md +69 -0
- package/commands/pan/milestone-done.md +3 -2
- package/package.json +1 -1
- package/pan-wizard-core/bin/lib/constants.cjs +40 -0
- package/pan-wizard-core/bin/lib/hygiene.cjs +447 -0
- package/pan-wizard-core/bin/lib/knowledge.cjs +28 -12
- package/pan-wizard-core/bin/lib/learn-index.cjs +17 -0
- package/pan-wizard-core/bin/lib/memory.cjs +146 -3
- package/pan-wizard-core/bin/lib/skill-align.cjs +364 -0
- package/pan-wizard-core/bin/lib/verify.cjs +10 -0
- package/pan-wizard-core/bin/pan-tools.cjs +47 -1
- package/pan-wizard-core/learnings/index.json +262 -10
- package/pan-wizard-core/learnings/internal/external-research.md +13 -1
- package/pan-wizard-core/learnings/universal/adversarial-verification.md +45 -0
- package/pan-wizard-core/learnings/universal/audit-convergence.md +33 -0
- package/pan-wizard-core/learnings/universal/autonomous-loop.md +4 -4
- package/pan-wizard-core/learnings/universal/external-tool-truth.md +21 -0
- package/pan-wizard-core/learnings/universal/fix-campaigns.md +45 -0
- package/pan-wizard-core/learnings/universal/flaky-triage.md +33 -0
- package/pan-wizard-core/learnings/universal/golden-sets.md +33 -0
- package/pan-wizard-core/learnings/universal/harness-isolation.md +21 -0
- package/pan-wizard-core/learnings/universal/integration-verification.md +33 -0
- package/pan-wizard-core/learnings/universal/live-path-honesty.md +45 -0
- package/pan-wizard-core/learnings/universal/mcp-security.md +21 -0
- package/pan-wizard-core/learnings/universal/migration-safety.md +21 -0
- package/pan-wizard-core/learnings/universal/service-security.md +21 -0
- package/pan-wizard-core/learnings/universal/single-source-of-truth.md +33 -0
- package/pan-wizard-core/learnings/universal/test-integrity.md +21 -0
- package/pan-wizard-core/learnings/universal/workaround-catalog.md +21 -0
- package/pan-wizard-core/references/model-profiles.md +4 -1
- package/pan-wizard-core/workflows/exec-phase.md +12 -3
- package/pan-wizard-core/workflows/plan-phase.md +1 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: harness-isolation
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.519Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-ISO-001
|
|
6
|
+
summary: Autonomous observers/fuzzers/eval harnesses must run against a SHA-locked frozen copy of the product with a path guard, and stay strictly read-only against the live workspace
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.519Z
|
|
8
|
+
source_experiments: [mph-factory, mph-factory-limits]
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Harness Isolation (AI-derived)
|
|
12
|
+
|
|
13
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
14
|
+
|
|
15
|
+
## P-ISO-001 — Autonomous observers/fuzzers/eval harnesses must run against a SHA-locked frozen copy of the product with a path guard, and stay strictly read-only against the live workspace
|
|
16
|
+
|
|
17
|
+
**Evidence:** Two autonomous harnesses (an optimization factory and an endurance/limits factory) both froze the compiler under test to a SHA-locked installed copy and refused to start if the binary resolved outside the frozen directory — never referencing the live workspace build tree. This kept findings attributable to a known version and made the loop safe to run unattended alongside active development.
|
|
18
|
+
|
|
19
|
+
**Rule:** An autonomous loop that observes or stresses a product must: (1) pin the product to a SHA-locked frozen artifact; (2) guard at startup that the binary under test resolves inside the frozen path, refusing to run otherwise; (3) be read-only against the live source/workspace. Findings from an unpinned target are unattributable and the loop can corrupt in-progress work.
|
|
20
|
+
|
|
21
|
+
**Applies in:** Fuzzers, eval harnesses, perf factories, CI observers running beside active development.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: integration-verification
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.514Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-INT-001
|
|
6
|
+
summary: Intra-phase PASS is not integration: milestone/closure audits must verify cross-phase seams (registration, callers, non-stub bodies) and end-to-end flows, not per-phase checklists
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.514Z
|
|
8
|
+
source_experiments: [platform-v2-milestone-audit]
|
|
9
|
+
- id: P-INT-002
|
|
10
|
+
summary: Derived closure artifacts (traceability/coverage matrices) must be regenerated at close — a matrix generated in phase N silently contradicts what phase N+1 delivered
|
|
11
|
+
promoted_at: 2026-07-09T14:04:40.514Z
|
|
12
|
+
source_experiments: [platform-v2-milestone-audit]
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# Integration Verification (AI-derived)
|
|
16
|
+
|
|
17
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
18
|
+
|
|
19
|
+
## P-INT-001 — Intra-phase PASS is not integration: milestone/closure audits must verify cross-phase seams (registration, callers, non-stub bodies) and end-to-end flows, not per-phase checklists
|
|
20
|
+
|
|
21
|
+
**Evidence:** A v2.0 milestone audit of a multi-phase platform found every phase had passed its own verification, yet five cross-phase seams were broken: a DI extension method was never called in the composition root; a decrypt endpoint unconditionally returned an error; four of five operator reconcilers were stubs that "gracefully skipped" their work; a feature gate was never registered so its compliance assertion could never fire. Component-local checks proved existence, not integration.
|
|
22
|
+
|
|
23
|
+
**Rule:** At milestone close, audit the seams: for every cross-phase contract verify (1) the provider is actually registered/wired into the running host, (2) at least one real caller exercises it, (3) the body is substantive, not a stub that skips gracefully, and (4) a representative end-to-end flow crosses the seam. A milestone is not done because each phase passed alone.
|
|
24
|
+
|
|
25
|
+
**Applies in:** Milestone audits, closure gates, goal-backward verification (key_links), multi-phase plans.
|
|
26
|
+
|
|
27
|
+
## P-INT-002 — Derived closure artifacts (traceability/coverage matrices) must be regenerated at close — a matrix generated in phase N silently contradicts what phase N+1 delivered
|
|
28
|
+
|
|
29
|
+
**Evidence:** The same milestone audit found the closure traceability matrix internally inconsistent: it was generated during phase 10 and never regenerated after phase 11 delivered items, so shipped requirements were still marked delivered:false. Any gate reading the matrix would have failed on stale data; any human reading it was misinformed.
|
|
30
|
+
|
|
31
|
+
**Rule:** Treat generated audit artifacts (traceability matrices, coverage reports, requirement rollups) as derived views that MUST be re-derived at closure time, after the last change. Never gate or report from a matrix older than the work it describes; regenerate via the same script that produced it.
|
|
32
|
+
|
|
33
|
+
**Applies in:** Closure gates, traceability matrices, requirement coverage reports, release audits.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: live-path-honesty
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.512Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-HON-001
|
|
6
|
+
summary: No fabricated data or fabricated success on live/trusted paths — demo synthetics and test fixtures are legitimate; fake data feeding real actions and success flags with no backing call are violations
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.509Z
|
|
8
|
+
source_experiments: [lending-fake-code-audit]
|
|
9
|
+
- id: P-HON-002
|
|
10
|
+
summary: Render honest-empty states instead of synthetic data on live paths, gate all demo values behind an explicit demo flag that defaults OFF, and never parse feature flags as "enabled unless literal false"
|
|
11
|
+
promoted_at: 2026-07-09T14:04:40.511Z
|
|
12
|
+
source_experiments: [lending-fake-code-audit]
|
|
13
|
+
- id: P-HON-003
|
|
14
|
+
summary: Scaffold is not deliverable: code is not complete while any generated stub (NotImplementedException / placeholder body) remains — wire an automated conformance check that flags unimplemented stubs
|
|
15
|
+
promoted_at: 2026-07-09T14:04:40.512Z
|
|
16
|
+
source_experiments: [spec-factory]
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
# Live Path Honesty (AI-derived)
|
|
20
|
+
|
|
21
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
22
|
+
|
|
23
|
+
## P-HON-001 — No fabricated data or fabricated success on live/trusted paths — demo synthetics and test fixtures are legitimate; fake data feeding real actions and success flags with no backing call are violations
|
|
24
|
+
|
|
25
|
+
**Evidence:** A read-only fake-code audit of a staff-facing lending frontend (92 agents, 1,155 file inspections) found: a local flag asserting "arrangement recorded" with no backend call ever made; hardcoded debts feeding a REAL disbursement saga; guards that fail open (policy=null => pass); and a .catch(()=>null) that reported COMPLETE on a failed write. The audit doctrine that made findings tractable: demo-labelled synthetics, simulator test-doubles and fixtures are legitimate — violations are fabrication on live/trusted paths and status surfaces computed from constants.
|
|
26
|
+
|
|
27
|
+
**Rule:** Classify every synthetic value by the path it flows into. On live/trusted paths: no hardcoded domain data, no success status without the backing side effect having actually happened, no fail-open guards on policy checks, no swallowed errors that report completion. Audit for status surfaces computed from constants and placeholders presented as working features. Demo-labelled synthetics and test fixtures are fine — the violation is fabrication presented as real.
|
|
28
|
+
|
|
29
|
+
**Applies in:** Fake-code audits, code review lenses, verifier anti-pattern scans, any app with a demo mode adjacent to a live mode.
|
|
30
|
+
|
|
31
|
+
## P-HON-002 — Render honest-empty states instead of synthetic data on live paths, gate all demo values behind an explicit demo flag that defaults OFF, and never parse feature flags as "enabled unless literal false"
|
|
32
|
+
|
|
33
|
+
**Evidence:** Same audit: a feature ran against a public demo API on a staff-facing surface because the gate was "enabled unless the env var is exactly the string false" — any typo, unset var, or casing difference silently enabled it. The accepted remediation doctrine: on a live path with no real data, render an honest "not available / nothing was transmitted" state; every demo/synthetic value sits behind one explicit demo flag; synthetic features default OFF; fix plans also removed a fake progress spinner and retitled a step honestly.
|
|
34
|
+
|
|
35
|
+
**Rule:** When a live path has no real data, show an honest empty state — never synthetic stand-in data. Put every demo/synthetic value behind one explicit demo flag and default it OFF. Parse feature gates as "disabled unless explicitly enabled"; a gate of the form enabled-unless-value-equals-false fails open on every unset or mistyped value.
|
|
36
|
+
|
|
37
|
+
**Applies in:** UI states, feature flags, demo modes, environment-variable gates.
|
|
38
|
+
|
|
39
|
+
## P-HON-003 — Scaffold is not deliverable: code is not complete while any generated stub (NotImplementedException / placeholder body) remains — wire an automated conformance check that flags unimplemented stubs
|
|
40
|
+
|
|
41
|
+
**Evidence:** A spec-to-code factory pipeline shipped scaffolds whose every service body threw NotImplementedException; its SOP had to state explicitly "This is a SKELETON — a case is not code-complete while any scaffolded NotImplementedException remains", and a conformance evaluator gained a code_unimplemented check because generated skeletons otherwise "look done" to downstream agents.
|
|
42
|
+
|
|
43
|
+
**Rule:** Treat generated scaffolds as skeletons, never as deliverables. Add an automated conformance check that fails handoff while any generated stub marker (NotImplementedException, unimplemented!, TODO-body, placeholder DTO) remains. An agent must not be able to mark work complete on the strength of files that merely exist and compile.
|
|
44
|
+
|
|
45
|
+
**Applies in:** Code generators, scaffolding steps, verifier artifact checks (exists vs substantive vs wired).
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: mcp-security
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.520Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-MCP-001
|
|
6
|
+
summary: Treat the entire MCP/tool surface — schemas, parameter names, inputs AND responses — as an untrusted prompt-injection channel; baseline: audience-bound tokens, hash-pinned tool definitions, URL allowlists, sandboxing, human-gated writes, full invocation logging
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.520Z
|
|
8
|
+
source_experiments: [spec-factory]
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Mcp Security (AI-derived)
|
|
12
|
+
|
|
13
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
14
|
+
|
|
15
|
+
## P-MCP-001 — Treat the entire MCP/tool surface — schemas, parameter names, inputs AND responses — as an untrusted prompt-injection channel; baseline: audience-bound tokens, hash-pinned tool definitions, URL allowlists, sandboxing, human-gated writes, full invocation logging
|
|
16
|
+
|
|
17
|
+
**Evidence:** A factory security standard, citing injection studies (MCPTox: ~72.8% attack success on weaker models via poisoned tool metadata), treated every element of the tool surface as attacker-controllable: tool descriptions and schemas can carry injected instructions, and tool RESPONSES are as dangerous as inputs. Its baseline: OAuth 2.1 resource-server auth with RFC 8707 audience-bound tokens, tool definitions hash-pinned and re-verified before execution, URL allowlisting against SSRF, sandboxed execution, human approval for writes, and every invocation logged.
|
|
18
|
+
|
|
19
|
+
**Rule:** Never trust any part of a tool integration surface: schemas, descriptions, parameter names, and especially tool responses are all injection vectors. Minimum bar: audience-bound tokens (no token reuse across services), pin tool definitions by hash and re-verify before execution, allowlist outbound URLs, sandbox execution, require human approval for write-capable tools, and log every invocation for audit.
|
|
20
|
+
|
|
21
|
+
**Applies in:** MCP server integrations, tool-using agents, agent gateways, plugin ecosystems.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: migration-safety
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.516Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-MIG-001
|
|
6
|
+
summary: Never let unattended startup auto-migrate run destructive or irreversible DDL — a partial apply half-marks migration history and crash-loops the service; gate schema swaps behind a manual, backed-up apply with restore proven on a copy
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.516Z
|
|
8
|
+
source_experiments: [compliance-army-v1.1]
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Migration Safety (AI-derived)
|
|
12
|
+
|
|
13
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
14
|
+
|
|
15
|
+
## P-MIG-001 — Never let unattended startup auto-migrate run destructive or irreversible DDL — a partial apply half-marks migration history and crash-loops the service; gate schema swaps behind a manual, backed-up apply with restore proven on a copy
|
|
16
|
+
|
|
17
|
+
**Evidence:** A compliance-platform mission was declared NO-GO on a big-bang schema swap because the service ran MigrateAsync() unattended at startup: a destructive DROP TABLE would auto-apply on the next deploy, and a partial apply would leave the migrations-history table half-marked with the API crash-looping. A related landmine: inserting through an updatable filtered view left the type discriminator NULL (NOT NULL violation) — the safe design was an additive discriminator column, not a view.
|
|
18
|
+
|
|
19
|
+
**Rule:** Unattended auto-migration may only ever apply additive, reversible changes. Destructive or irreversible DDL requires: a manual off-peak apply, a taken-and-TESTED backup (restore proven on a copy first), and an explicit rollback path. Prefer additive discriminator columns over updatable views for type splits — INSERT through a filtered view does not populate the discriminator. Record NO-GO decisions durably so the next campaign does not re-litigate them.
|
|
20
|
+
|
|
21
|
+
**Applies in:** Database migrations, deploy pipelines, EF/ORM startup hooks, brownfield schema evolution.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: service-security
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.519Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-SVC-001
|
|
6
|
+
summary: Reusable security spine for authenticated CRUD services: 404-not-403 on restricted reads, role-gate every mutation, atomic conditional writes instead of check-then-act, leaf-only filenames, capped request arrays, bounded untrusted parsing
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.519Z
|
|
8
|
+
source_experiments: [compliance-army-v1.1]
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Service Security (AI-derived)
|
|
12
|
+
|
|
13
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
14
|
+
|
|
15
|
+
## P-SVC-001 — Reusable security spine for authenticated CRUD services: 404-not-403 on restricted reads, role-gate every mutation, atomic conditional writes instead of check-then-act, leaf-only filenames, capped request arrays, bounded untrusted parsing
|
|
16
|
+
|
|
17
|
+
**Evidence:** Two deep security reviews of a compliance platform converged on the same fix spine: return 404 (not 403) on restricted reads so existence does not leak; role-gate every mutating endpoint (a HIGH finding: any authenticated reader could attach immutable evidence because only RequireAuthorization was applied); make write-once/idempotency checks atomic via conditional headers (If-None-Match:*) instead of check-then-act; reduce user-supplied filenames to a bare leaf before pathing; cap bulk-import arrays (1,000 rows) against unbounded-insert DoS; bound untrusted document parsing against XXE and zip bombs.
|
|
18
|
+
|
|
19
|
+
**Rule:** For any authenticated CRUD/API service, apply the spine: (1) restricted reads return not-found, never forbidden; (2) every mutating endpoint has an explicit role/permission gate — authentication alone is not authorization; (3) uniqueness/write-once enforced atomically (conditional requests or DB constraints), never check-then-act; (4) user-supplied filenames reduced to a leaf name before any path join; (5) request arrays capped; (6) untrusted parsers bounded (entity expansion, decompression size).
|
|
20
|
+
|
|
21
|
+
**Applies in:** API security reviews, hardening passes, upload/import endpoints, WORM/audit stores.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: single-source-of-truth
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.515Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-SSOT-001
|
|
6
|
+
summary: When two code paths must agree on a classification, delegate both to one predicate — parallel lists diverge silently; also never pre-write a terminal status before the retry ladder runs, and never label a metric with an action that does not happen
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.514Z
|
|
8
|
+
source_experiments: [dispatch-postmortem]
|
|
9
|
+
- id: P-SSOT-002
|
|
10
|
+
summary: After changing a source of truth, re-sync every parallel copy of it — and pin the seam with a golden-reproduction end-to-end test, because per-component suites will not catch the second copy
|
|
11
|
+
promoted_at: 2026-07-09T14:04:40.515Z
|
|
12
|
+
source_experiments: [forecasting-campaign]
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# Single Source Of Truth (AI-derived)
|
|
16
|
+
|
|
17
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
18
|
+
|
|
19
|
+
## P-SSOT-001 — When two code paths must agree on a classification, delegate both to one predicate — parallel lists diverge silently; also never pre-write a terminal status before the retry ladder runs, and never label a metric with an action that does not happen
|
|
20
|
+
|
|
21
|
+
**Evidence:** A message-dispatch postmortem in an event-driven .NET service found three stacked bugs behind silent message loss: a stage returned Success on a transient failure so nothing redelivered; a terminal status was pre-written, blocking any retry; and the redelivery filter's exception list had diverged from the exception classifier. The fix collapsed the parallel lists to a single source of truth — the retry policy delegates to the same IsTransient classifier — "so the redelivery filters can never again diverge from the classifier". A metric labelled "retry" was recording outcomes the system silently terminated.
|
|
22
|
+
|
|
23
|
+
**Rule:** If two code paths must agree on a category (is-transient, is-retriable, is-sensitive), implement the judgment ONCE and delegate both paths to it — never maintain parallel lists that must be kept in sync by discipline. Do not pre-write terminal status before the retry ladder has actually run. A metric or log label must describe what the system did, not what it was supposed to do.
|
|
24
|
+
|
|
25
|
+
**Applies in:** Retry/redelivery policies, classification predicates, status ledgers, observability labels.
|
|
26
|
+
|
|
27
|
+
## P-SSOT-002 — After changing a source of truth, re-sync every parallel copy of it — and pin the seam with a golden-reproduction end-to-end test, because per-component suites will not catch the second copy
|
|
28
|
+
|
|
29
|
+
**Evidence:** After an engine re-calibration updated golden values, the web UI's default inputs/segments stayed on the old seed: the deployed default scenario ran ~90% below golden and no suite failed, because engine tests checked the engine and UI tests checked the UI — nothing reproduced the golden case through both. The campaign post-mortem recorded the durable rule: after any engine re-calibration, re-sync the UI defaults (a parallel copy of the truth) and keep a golden-reproduction end-to-end test as the load-bearing check.
|
|
30
|
+
|
|
31
|
+
**Rule:** Inventory every place a source-of-truth value is duplicated (UI defaults, fixtures, docs, seeds). When the truth changes, re-sync all copies in the same change — and protect the seam permanently with one end-to-end test that reproduces the golden case through every layer, since per-layer suites cannot see a stale sibling copy.
|
|
32
|
+
|
|
33
|
+
**Applies in:** Golden values, calibration constants, default configs duplicated across layers, cross-repo copies.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: test-integrity
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.512Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-TI-001
|
|
6
|
+
summary: No dumbing down generated tests/evals: forbid hardcoded expected outputs, input==known shortcuts, and domain-shrinking to dodge bugs; if a real bug blocks the general case, file it — do not weaken the seed
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.512Z
|
|
8
|
+
source_experiments: [mph-factory]
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Test Integrity (AI-derived)
|
|
12
|
+
|
|
13
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
14
|
+
|
|
15
|
+
## P-TI-001 — No dumbing down generated tests/evals: forbid hardcoded expected outputs, input==known shortcuts, and domain-shrinking to dodge bugs; if a real bug blocks the general case, file it — do not weaken the seed
|
|
16
|
+
|
|
17
|
+
**Evidence:** An autonomous test-factory for a compiler codified this after agents repeatedly weakened test seeds to get green runs: rules ban hardcoded outputs, if input==known {return known} shortcuts, algorithm-name/implementation mismatch, and shrinking the input domain to avoid a bug. Seeds must show proof-of-work (loops of meaningful depth, DP tables of real size, round-trips, or multi-case sweeps). When a genuine product bug blocked the general case, the required action was to file the bug and keep the seed intact.
|
|
18
|
+
|
|
19
|
+
**Rule:** When an agent authors tests or eval seeds: (1) no hardcoded expected outputs or special-cased known inputs; (2) no shrinking the input domain to route around a failure; (3) require proof-of-work in the seed (real loops, real data sizes, round-trips, or >=3-case sweeps); (4) if a real bug blocks the general case, file the bug and keep the strong seed — never weaken the test to pass.
|
|
20
|
+
|
|
21
|
+
**Applies in:** Test generation, eval-set authoring, autonomous build loops that write their own verification.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
---
|
|
2
|
+
topic: workaround-catalog
|
|
3
|
+
last_updated: 2026-07-09T14:04:40.519Z
|
|
4
|
+
patterns:
|
|
5
|
+
- id: P-WKC-001
|
|
6
|
+
summary: Catalog every env-var/toggle workaround (with schema) before its ticket closes — an uncatalogued flag that was in the ticket from day 1 cost a 5-day regression-to-fix gap because nobody tried it
|
|
7
|
+
promoted_at: 2026-07-09T14:04:40.519Z
|
|
8
|
+
source_experiments: [mph-factory-limits]
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Workaround Catalog (AI-derived)
|
|
12
|
+
|
|
13
|
+
> Auto-maintained by `pan-tools learn promote`. Each pattern was extracted from one or more experiment runs (see source_experiments). Patterns are **advisory** — orchestrators should weight them against current context.
|
|
14
|
+
|
|
15
|
+
## P-WKC-001 — Catalog every env-var/toggle workaround (with schema) before its ticket closes — an uncatalogued flag that was in the ticket from day 1 cost a 5-day regression-to-fix gap because nobody tried it
|
|
16
|
+
|
|
17
|
+
**Evidence:** An endurance-harness rule was written after a measured failure: a regression sat unfixed for 5 days although the resolving flag (a GC compaction env var) was named in the ticket from day one — it was uncatalogued, so no investigator tried it. The rule: every finding resolved by a runtime toggle/env overlay must be added to a structured overlay catalog before the ticket closes.
|
|
18
|
+
|
|
19
|
+
**Rule:** Any fix or mitigation that takes the form "set this flag / env var / toggle" must be recorded in a structured, searchable workaround catalog (name, effect, applicability, default) as a closing condition of the ticket itself. Institutional memory of overlays is part of the fix; an uncatalogued workaround will be re-discovered from scratch at day-rate cost.
|
|
20
|
+
|
|
21
|
+
**Applies in:** Ops runbooks, endurance findings, environment overlays, incident close-out checklists.
|
|
@@ -39,7 +39,10 @@ PAN never selects your host model — it recommends one. Because the `reasoning`
|
|
|
39
39
|
|
|
40
40
|
**Why Fable is the recommended flagship.** It is Anthropic's most capable widely released model for demanding, long-horizon agentic work — exactly what PAN's hierarchical bot army (Mission Control → squads → workers) asks of its reasoning tier. Select it in your host runtime and `inherit` routes the reasoning-tier agents to it automatically.
|
|
41
41
|
|
|
42
|
-
**Fable caveat — the cyber-classifier refusal.** Fable is the only current Claude model that runs input safety classifiers targeting cybersecurity and biology content, and benign *defensive* security tooling can trigger false positives
|
|
42
|
+
**Fable caveat — the cyber-classifier refusal, and how PAN handles it.** Fable is the only current Claude model that runs input safety classifiers targeting cybersecurity and biology content, and benign *defensive* security tooling can trigger false positives — a successful response with `stop_reason: "refusal"` and `stop_details.category: "cyber"`. In PAN this hits every security path: `/pan:review-deep`, `exec-phase --deep-review`, **and the `focus-auto`/army `security` category** (which was observed refusing in a real project). Two mitigations, in order of reliability:
|
|
43
|
+
|
|
44
|
+
1. **Opus pin (the durable fix).** `pan-hardener`, `pan-reviewer`, and `pan-meta-reviewer` carry `model: opus` in their frontmatter, so on **Claude Code** they run on Opus 4.8 regardless of your session model and never reach Fable's classifier. The `focus-auto` security category routes its vulnerability *assessment* through the Opus-pinned `pan-hardener` for the same reason. This pin is **Claude-Code-only** — it is stripped from the Gemini/OpenCode/Codex/Copilot outputs by the installer, so on the other runtimes run security campaigns on a non-Fable model.
|
|
45
|
+
2. **Defensive framing.** Those agents and the `focus-auto` security prose are written as *authorized, defensive review* (no exploit-path narration), which lowers the trigger rate but cannot eliminate it — a security scanner must name injection, auth bypass, and RCE by definition. Framing is the backstop; the Opus pin is the fix.
|
|
43
46
|
|
|
44
47
|
**Fable data-retention requirement.** Fable is not available under zero data retention; an org whose retention is below 30 days gets a hard `400` on every request. If Fable 400s on every call with an otherwise-valid request, check the org's retention setting before debugging anything else.
|
|
45
48
|
|
|
@@ -78,9 +78,18 @@ ls .planning/memory/*.md 2>/dev/null
|
|
|
78
78
|
```
|
|
79
79
|
|
|
80
80
|
If `.planning/memory/` exists and contains `.md` files:
|
|
81
|
-
1. **
|
|
82
|
-
|
|
83
|
-
|
|
81
|
+
1. **Check the memory-load budget first** (ADR-0036 — keeps per-agent injection bounded as logs grow):
|
|
82
|
+
```bash
|
|
83
|
+
node ~/.claude/pan-wizard-core/bin/pan-tools.cjs memory budget --raw
|
|
84
|
+
```
|
|
85
|
+
2. **Load, size-gated — whole-file is the default:**
|
|
86
|
+
- If `status` is `ok`: **read every file whole** (Read tool) and condense each entry to its rule(s) — 1–3 lines per file. This preserves the "apply every rule" contract for normal-sized logs.
|
|
87
|
+
- If `status` is `warning` or `critical` (a log has grown large): load a **cue-scoped** slice per agent instead of the whole log, using the phase objective + the files this phase touches as the cue:
|
|
88
|
+
```bash
|
|
89
|
+
node ~/.claude/pan-wizard-core/bin/pan-tools.cjs memory select <agent> --cue "<phase objective; changed files>" --raw
|
|
90
|
+
```
|
|
91
|
+
Run once per agent that has a memory file. The returned `selected` entries are already recency-floored and token-budgeted (the newest lessons are always included). If `selected` is empty for an agent, **fall back to reading that file whole** — never silently drop an agent's memory.
|
|
92
|
+
3. Store the condensed rules as a `MEMORY_RULES` block for injection into executor prompts in execute_waves.
|
|
84
93
|
4. **Log memory priming to trace:**
|
|
85
94
|
```bash
|
|
86
95
|
MEMORY_COUNT=$(ls .planning/memory/*.md 2>/dev/null | wc -l | tr -d ' ')
|
|
@@ -377,6 +377,7 @@ Output consumed by /pan:exec-phase. Plans need:
|
|
|
377
377
|
- [ ] plan.md files created in phase directory
|
|
378
378
|
- [ ] Each plan has valid frontmatter
|
|
379
379
|
- [ ] Tasks are specific and actionable
|
|
380
|
+
- [ ] Task vocabulary aligned via `pan-tools skills align` (SAD pass — advisory, skip on error)
|
|
380
381
|
- [ ] Dependencies correctly identified
|
|
381
382
|
- [ ] Waves assigned for parallel execution
|
|
382
383
|
- [ ] must_haves derived from phase goal
|