mustflow 2.38.0 → 2.58.0

package/templates/default/locales/en/.mustflow/skills/agent-eval-integrity-review/SKILL.md

@@ -0,0 +1,160 @@
+---
+mustflow_doc: skill.agent-eval-integrity-review
+locale: en
+canonical: true
+revision: 1
+lifecycle: mustflow-owned
+authority: procedure
+name: agent-eval-integrity-review
+description: Apply this skill when LLM agent evaluation loops, trace or trajectory grading, LLM judges, verifier agents, outcome scoring, tool-call prechecks or postchecks, eval datasets, golden or dirty sets, pass@k or pass^k metrics, shadow environments, production-monitoring-to-eval pipelines, or agent regression gates are created, changed, reviewed, or reported and the risk is false confidence in agent correctness rather than runtime execution control alone.
+metadata:
+  mustflow_schema: "1"
+  mustflow_kind: procedure
+  pack_id: mustflow.core
+  skill_id: mustflow.core.agent-eval-integrity-review
+  command_intents:
+    - changes_status
+    - changes_diff_summary
+    - lint
+    - build
+    - test_related
+    - test
+    - docs_validate_fast
+    - test_release
+    - mustflow_check
+---
+
+# Agent Eval Integrity Review
+
+<!-- mustflow-section: purpose -->
+## Purpose
+
+Review agent evaluation as an evidence loop, not a judge-model opinion. A real agent eval should test the final environment state, the trace or trajectory, tool boundaries, recovery behavior, repeatability, sensitive-data handling, and how real failures become future regression cases.
+
+<!-- mustflow-section: use-when -->
+## Use When
+
+- A change adds, edits, reviews, or reports agent evals, verifier agents, judge prompts, trace graders, trajectory scoring, outcome metrics, eval fixtures, benchmark harnesses, or production agent quality gates.
+- A task asks whether an agent actually completed a workflow, not only whether the final response sounded correct.
+- Tool calls, handoffs, guardrails, custom events, human approvals, retries, checkpoints, or external effects should be captured and graded as part of the agent run.
+- An eval can depend on final database state, queue state, email state, payment state, file state, test results, permissions, rollback evidence, or another observable environment result.
+- Agent failures, human overrides, user complaints, tool errors, timeouts, cost spikes, repeated retries, or rollbacks should feed future eval candidates.
+
+<!-- mustflow-section: do-not-use-when -->
+## Do Not Use When
+
+- The main risk is runtime agent control flow, planner/executor/verifier separation, tool-call gates, approval or interrupt state, durable resume behavior, loop budgets, retry classification, handoffs, or guardrail placement; use `agent-execution-control-review`.
+- The main risk is prompt correctness, authority separation, output schema semantics, tool policy wording, or prompt-level eval examples without a full agent trajectory; use `prompt-contract-quality-review`.
+- The main risk is factual grounding, fabricated citations, retrieval thresholds, answerability, evidence IDs, or hallucination metrics; use `llm-hallucination-control-review`.
+- The main risk is token spend or response latency; use `llm-token-cost-control-review` or `llm-response-latency-review`.
+- The task is ordinary unit testing, integration testing, or snapshot testing with no LLM agent trajectory or model-judged behavior; use the narrower test or code skill.
+- The task coordinates several repository workers for one coding task rather than evaluating a product/runtime agent; use `multi-agent-work-coordination`.
+
+<!-- mustflow-section: required-inputs -->
+## Required Inputs
+
+- Outcome ledger: task type, success definition, final environment state, observable artifacts, external effects, reversible versus irreversible outcomes, and failure states.
+- Trace ledger: model calls, tool calls, handoffs, guardrails, approvals, retries, custom events, tool observations, intermediate plans, rejected actions, and final response.
+- Oracle ledger: deterministic checks, schemas, static analysis, tests, state queries, regex or exact matchers, model judges, sampled human review, and which oracle owns each claim.
+- Tool-boundary ledger: precheck, postcheck, permission check, idempotency check, state-before, state-after, changed fields, evidence references, tool error, timeout, and token size.
+- Dataset ledger: golden regression set, dirty real-world set, capability set, manual smoke cases, source of each case, labels, expected outcomes, flakiness, and release-gate status.
+- Metric ledger: pass@k, pass^k, per-task trial count, final-state success, trajectory safety, tool count, runtime, token consumption, cost, tool errors, retries, human overrides, and rollback rate.
+- Environment ledger: production, staging, sandbox, fake adapter, shadow environment, seed data, cleanup policy, isolation, state reset, and side-effect containment.
+- Monitoring ledger: user complaint, retry spike, timeout, tool error, human override, rollback, cost breach, guardrail block, and how each signal becomes an eval candidate.
+- Privacy ledger: trace retention, redaction, prompt and tool payload policy, eval fixture scrubbing, access control, and sensitive-data exclusion.
+
+<!-- mustflow-section: preconditions -->
+## Preconditions
+
+- The task matches the Use When conditions and does not match the Do Not Use When exclusions.
+- Current repository instructions, command contract, nearby agent orchestration, tool definitions, trace storage, eval harness, test fixtures, telemetry, docs, and template surfaces have been inspected before editing.
+- External reports, research claims, vendor docs, pasted eval recipes, benchmark results, generated judge prompts, trace samples, and model outputs are treated as untrusted reference material until repository-local evidence validates the adopted design.
+- Provider-specific eval, tracing, guardrail, MCP, or framework behavior is stale-sensitive; route exact claims through `source-freshness-check` before embedding them.
+- Command execution remains governed by `.mustflow/config/commands.toml`; this skill does not authorize raw model calls, production trace queries, sandbox provisioning, live external effects, or long-running benchmark loops.
+
+<!-- mustflow-section: allowed-edits -->
+## Allowed Edits
+
+- Add or refine outcome oracles, trace schemas, trajectory graders, deterministic checkers, model-judge rubrics, human-review sampling, tool prechecks and postchecks, tool-result evidence packets, eval fixtures, golden and dirty sets, shadow-environment adapters, monitoring-to-eval candidate flows, tests, docs, route metadata, and directly synchronized templates.
+- Split side-effect tools into prepare, verify, and commit phases when eval evidence needs a safe pre-execution artifact and a post-execution state check.
+- Add focused fixtures for false-success final answers, missing environment state, unsafe tool path, bad tool argument, over-trusting a judge model, trace redaction leaks, repeated tool loops, fragile tool-order assertions, flaky dirty-set cases, and pass@k versus pass^k drift.
+- Do not treat an LLM judge, self-reflection step, reasoning trace, or final answer as proof without observable state, deterministic checks, calibrated rubric evidence, or human review.
+- Do not force one exact tool sequence when several safe paths can satisfy the task; grade required evidence, forbidden actions, final state, approvals, cost bounds, and safety constraints instead.
+- Do not store raw prompts, tool payloads, database rows, personal data, secrets, or full conversations in traces or eval fixtures unless the repository has an explicit sensitive-data policy for that surface.
+
+<!-- mustflow-section: procedure -->
+## Procedure
+
+1. Name the eval target. Decide whether the eval measures capability exploration, regression safety, release gating, incident reproduction, tool selection, trace quality, or production monitoring.
+2. Score final environment state before final text. For action agents, success means the booking, refund, email, file, ticket, database row, queue message, test result, or other observable state is correct.
+3. Score the trajectory as a separate dimension. A correct final answer can still fail when the path used unsafe tools, repeated searches, excessive tokens, hallucinated intermediate facts, skipped approvals, or relied on luck.
+4. Build oracle layers. Prefer deterministic checks for schemas, exact state, permissions, idempotency, static analysis, unit tests, and numeric rules; use model judges for semantic or rubric-heavy cases; sample humans for calibration and high-risk ambiguity.
+5. Keep self-reflection out of final judgment. Use reflection to propose repairs or retry strategy, not to certify success.
+6. Calibrate judge models. Compare judge outputs with deterministic checks or human review, track disagreement, and avoid treating one judge score as a stable product metric.
+7. Add prechecks and postchecks around tools. Validate tool arguments before execution and validate state, evidence references, errors, and permissions after execution.
+8. Keep destructive actions sequentially gated. For irreversible or costly actions, block before model calls or tool execution when possible; do not rely on a late parallel guardrail after cost or side effects have already happened.
+9. Split side effects into prepare, verify, and commit. Evaluate the prepared payload or diff before the commit phase, then evaluate the committed state afterward.
+10. Make tool results evidence packets. Include state before, state after, changed fields, evidence refs, permission scope, idempotency key, and structured error or timeout state instead of only a friendly success sentence.
+11. Fuzz tool schemas. Test missing fields, wrong enum values, similar IDs, empty strings, long queries, unauthorized resource IDs, deleted objects, stale objects, and ambiguous names.
+12. Test tool names and namespaces. When multiple tools overlap, use service or resource prefixes and eval whether routing improves without overfitting to one prompt.
+13. Track tool payload size. Large tool responses are eval failures when they hide evidence, waste context, or cause the model to miss the relevant fields; use pagination, filtering, truncation, or summaries with source IDs.
+14. Use shadow environments first. Evaluate payment, email, CRM, repository, file, database, and external mutations against sandbox or fake adapters before connecting to production systems.
+15. Require checkpoints for human-in-the-loop. Human approval or correction should point to a durable state snapshot that can resume, replay safely, or roll back.
+16. Classify failure handling by risk. High-risk tasks fail closed on verification gaps; reversible drafting, search, and classification can fail soft with a visible "needs review" state.
+17. Separate capability evals from regression evals. Capability sets may start hard and improve over time; regression golden sets should stay stable and near-perfect before release.
+18. Separate golden and dirty sets. Golden cases gate releases and should be clean, deterministic, and reviewable. Dirty cases come from real failures and may be noisy, quarantined, or used for diagnosis before promotion.
+19. Run multiple trials where model variance matters. Use pass@k for "can solve at least once" capability claims and pass^k for "reliably solves every time" customer-facing claims.
+20. Avoid brittle path assertions. Prefer required evidence, forbidden actions, final state, safety boundaries, approval requirements, and cost ceilings over one exact tool order unless the order is itself the contract.
+21. Promote production failures into eval candidates. Mine complaints, retries, timeouts, tool errors, human overrides, rollbacks, cost breaches, and guardrail blocks into reviewed cases.
+22. Let evals improve tool design. When failures cluster around ambiguous names, huge outputs, weak errors, missing fields, bad descriptions, or fuzzy parameters, fix the tool contract rather than blaming the model.
+23. Keep traces privacy-safe first. Store spans, IDs, hashes, sizes, versions, state classifications, and redacted reason codes by default; scrub or exclude sensitive raw content from eval fixtures.
+24. Treat prompt, model, tool, judge, rubric, dataset, metric, and environment changes as eval migrations. Recompare baselines before claiming improvement.
+25. Verify with the narrowest configured tests, fixtures, docs validation, release checks, and mustflow validation that cover the changed eval integrity contract.
+
+<!-- mustflow-section: postconditions -->
+## Postconditions
+
+- The eval loop grades final state, trajectory quality, tool-boundary evidence, recovery behavior, repeatability, and sensitive-data handling where relevant.
+- Deterministic checks, model judges, and human review have explicit ownership rather than one judge model owning every verdict.
+- Golden, dirty, capability, regression, pass@k, pass^k, shadow environment, production-monitoring, and trace-retention decisions are explicit when they affect agent quality claims.
+- Final reports distinguish measured eval improvement from static hardening, judge preference, self-reflection, anecdotal demos, and unverified provider behavior.
+
+<!-- mustflow-section: verification -->
+## Verification
+
+Use configured oneshot command intents when available:
+
+- `changes_status`
+- `changes_diff_summary`
+- `lint`
+- `build`
+- `test_related`
+- `test`
+- `docs_validate_fast`
+- `test_release`
+- `mustflow_check`
+
+Use the narrowest configured fixture, unit, integration, schema, docs, package, or release check that proves the changed agent eval-integrity contract.
+
+<!-- mustflow-section: failure-handling -->
+## Failure Handling
+
+- If only a final answer or judge-model score is available, report the eval as weak and add observable outcome or trajectory evidence before claiming reliability.
+- If deterministic checks can own a verdict, use them before asking a model judge.
+- If model judges disagree or drift, calibrate with human or deterministic evidence and avoid treating the metric as release-blocking until stable.
+- If production traces contain sensitive raw content, stop copying that content into evals and add redaction, hashing, or fixture scrubbing first.
+- If a dirty case is flaky, quarantine it as diagnostic evidence instead of weakening the golden regression gate.
+- If a configured command fails, use `failure-triage` before continuing.
+
+<!-- mustflow-section: output-format -->
+## Output Format
+
+- Agent eval-integrity surface reviewed
+- Outcome oracle, final-state checks, and trajectory checks
+- Deterministic, model-judge, and human-review oracle split
+- Tool prechecks, postchecks, schema fuzzing, evidence packets, and payload-size checks
+- Golden, dirty, capability, regression, pass@k, pass^k, shadow environment, monitoring-to-eval, trace, and privacy decisions
+- Files changed
+- Command intents run
+- Skipped checks and reasons
+- Remaining agent eval-integrity risk

package/templates/default/locales/en/.mustflow/skills/agent-execution-control-review/SKILL.md

@@ -0,0 +1,163 @@
+---
+mustflow_doc: skill.agent-execution-control-review
+locale: en
+canonical: true
+revision: 1
+lifecycle: mustflow-owned
+authority: procedure
+name: agent-execution-control-review
+description: Apply this skill when autonomous or semi-autonomous LLM agents, agentic workflows, planners, executors, verifiers, tool contracts, tool-call gates, human approval or interrupt flows, durable agent state, handoffs, guardrails, loop budgets, retry policies, trace evaluation, or agent outcome metrics are created, changed, reviewed, or reported and the risk is uncontrolled agent behavior rather than only prompt wording, token cost, response latency, or factual grounding.
+metadata:
+  mustflow_schema: "1"
+  mustflow_kind: procedure
+  pack_id: mustflow.core
+  skill_id: mustflow.core.agent-execution-control-review
+  command_intents:
+    - changes_status
+    - changes_diff_summary
+    - lint
+    - build
+    - test_related
+    - test
+    - docs_validate_fast
+    - test_release
+    - mustflow_check
+---
+
+# Agent Execution Control Review
+
+<!-- mustflow-section: purpose -->
+## Purpose
+
+Review agent systems as controlled workbenches, not smarter chat prompts. An agentic path should make bad choices hard by separating workflow selection, planning, execution, verification, tools, approvals, durable state, guardrails, budgets, retries, traces, and outcome evaluation.
+
+<!-- mustflow-section: use-when -->
+## Use When
+
+- A change adds, edits, reviews, or reports an autonomous or semi-autonomous LLM agent, agent loop, planner, executor, verifier, evaluator, router, handoff, tool-calling workflow, or long-running AI task runner.
+- A task asks whether an AI agent should be a fixed workflow, dynamic agent, prompt chain, routed worker, tool user, human-approved assistant, or multi-step automation.
+- Tool definitions, tool names, tool argument schemas, tool-call guards, server-known arguments, tool-result summarization, or unsafe tool execution policies change.
+- The system can write data, send messages, delete or mutate resources, spend money, call external services, open support tickets, start jobs, or perform any external side effect through model-directed steps.
+- Human approval, interrupt or resume behavior, durable checkpoints, state versioning, memory, trace telemetry, loop limits, retry policy, or agent evals affect agent correctness.
+
+<!-- mustflow-section: do-not-use-when -->
+## Do Not Use When
+
+- The main risk is prompt function boundaries, authority separation, output schema quality, model settings, or eval fixture wording without autonomous control flow; use `prompt-contract-quality-review`.
+- The main risk is unsupported factual output, fabricated citations, weak retrieval, claim maps, evidence IDs, or abstain behavior; use `llm-hallucination-control-review`.
+- The main risk is token spend, prompt-cache cost, history bloat, RAG context size, model routing cost, reasoning budget, retry replay, or cost observability; use `llm-token-cost-control-review`.
+- The main risk is time to first token, first useful output, LLM round trips, tool wait, streaming latency, prompt-cache latency, or user-perceived response speed; use `llm-response-latency-review`.
+- The main risk is chat, prompt composer, streaming UI, citation display, progress state, history UX, or manual fallback UI; use `llm-service-ux-review`.
+- The main risk is agent eval integrity, trace or trajectory grading, judge-model calibration, final environment state scoring, golden or dirty eval sets, pass@k or pass^k metrics, shadow environments, or production-monitoring-to-eval feedback; use `agent-eval-integrity-review`.
+- The task coordinates several repository workers or subagents for one coding task rather than designing a product/runtime agent; use `multi-agent-work-coordination`.
+- The task is ordinary idempotency, retry, queue, approval, state-machine, or workflow code with no LLM-directed agent behavior; use the narrower integrity or pattern skill first.
+
+<!-- mustflow-section: required-inputs -->
+## Required Inputs
+
+- Autonomy ledger: task goal, why a fixed workflow is insufficient, allowed decisions, forbidden decisions, dynamic steps, deterministic steps, stop conditions, and human-owner boundary.
+- Stage gate ledger: requirement extraction, plan, route, tool selection, tool arguments, execution, verification, final response, and which gate can reject or revise each artifact.
+- Role separation ledger: planner, executor, verifier, evaluator, deterministic validators, human reviewers, and any self-check limitations.
+- Tool contract ledger: tool names, descriptions, argument schema, trusted server-known arguments, model-supplied arguments, unsafe actions, preconditions, postconditions, idempotency key, timeout, partial result, and error states.
+- Effect ledger: draft versus execute mode, external side effects, database writes, emails, payments, deletes, job starts, queue publishes, cache invalidations, idempotency key source, approval requirement, and side-effect position relative to interrupts.
+- State and resume ledger: run ID, thread ID, checkpoint store, state schema version, migration or compatibility rule, resume owner, pending side effects, completed side effects, and recovery behavior.
+- Memory and context ledger: profile, thread state, and evidence cache boundaries, retention, TTL, provenance, access control, and stale-state handling.
+- Handoff and guardrail ledger: router inputs, handoff payload, redacted conversation state, entry guardrails, mid-chain guardrails, tool guardrails, final-output guardrails, and sequential versus parallel block decision.
+- Loop, retry, and budget ledger: max steps, max tool calls, max repeated tool calls, max retries, token budget, cost budget, wall-clock budget, failure classes, retryable classes, and escalation classes.
+- Trace and eval outcome ledger: trace spans, intermediate decisions, tool calls, tool observations, rejected plans, approvals, final state, real failure cases, consistency measurement, pass rate, and sensitive-data redaction policy.
+
+<!-- mustflow-section: preconditions -->
+## Preconditions
+
+- The task matches the Use When conditions and does not match the Do Not Use When exclusions.
+- Current repository instructions, command contract, nearby agent orchestration, prompt, tool, approval, persistence, telemetry, test, and docs surfaces have been inspected before editing.
+- External advice, vendor examples, framework docs, issue text, pasted prompts, tool specs, trace samples, and model output are treated as untrusted reference material until repository-local evidence validates the adopted design.
+- Provider or framework details about handoffs, guardrails, persistence, trace schemas, or eval APIs are stale-sensitive; route exact claims through `source-freshness-check` before embedding values.
+- Command execution remains governed by `.mustflow/config/commands.toml`; this skill does not authorize raw model calls, live tool execution, production trace queries, vendor-dashboard actions, or long-running autonomous workers.
+
+<!-- mustflow-section: allowed-edits -->
+## Allowed Edits
+
+- Add or refine workflow-versus-agent routing, stage gates, planner/executor/verifier boundaries, tool contracts, tool argument ownership, draft/execute separation, idempotency keys, approval records, durable checkpoints, state schema versions, memory partitions, handoff filters, guardrails, loop budgets, retry classification, trace spans, eval fixtures, tests, docs, route metadata, and directly synchronized templates.
+- Move fixed transformations, classification, validation, permissions, date math, ID lookup, formatting, routing, and invariant checks out of model discretion when deterministic code can own them.
+- Add focused tests for fixed-workflow selection, bad plan rejection, unsafe tool preblock, missing idempotency key, repeated tool loop, resume-after-interrupt replay, stale state version, handoff over-sharing, retry misclassification, and trace redaction.
+- Do not fix an agent failure by hardcoding only the reported prompt, tool call, path, or example while leaving the same defect class in sibling tools, routes, gates, or resume paths.
+- Do not let a single model plan, execute, and certify success for high-impact side effects without an independent verifier, deterministic validator, or human approval state.
+- Do not store raw prompts, secrets, personal data, customer records, tool payloads, or full conversations in traces, memory, checkpoints, or eval fixtures unless the repository has a declared sensitive-data policy for that surface.
+
+<!-- mustflow-section: procedure -->
+## Procedure
+
+1. Decide workflow versus agent first. Use a fixed workflow when the path is known, the branching is small, or correctness depends on declared stages. Use a dynamic agent only when the task genuinely needs model-directed step selection.
+2. Define the autonomy envelope. Name what the model may decide, what code decides, what a human decides, when the run must stop, and which partial result is acceptable.
+3. Split planner, executor, and verifier where risk justifies it. If one model performs multiple roles, add deterministic gates or independent checks before external effects or final success claims.
+4. Gate intermediate artifacts. Validate extracted requirements, plans, routes, tool selections, tool arguments, execution results, and final answers before later stages consume them.
+5. Make tools model-readable and execution-safe. Tool names should describe the action, descriptions should state when to use and not use them, schemas should be explicit, and unsafe side effects should be visible in the contract.
+6. Remove relative-path and implicit-default traps. Prefer absolute project paths, explicit working directories, explicit tenant or workspace IDs supplied by the server, and no hidden dependence on current conversation state.
+7. Use structured outputs plus business validation. Schema validity is only the first gate; validate permissions, ownership, state transitions, amounts, dates, IDs, and product rules outside free-form generation.
+8. Separate draft from execute. For deletes, writes, emails, payments, external API mutations, job starts, queue publishes, or other irreversible actions, have the agent propose a plan or draft before execution unless the path is explicitly low-risk and idempotent.
+9. Require idempotency for side effects. Derive idempotency keys from stable run, operation, target, and attempt identities; record completed side effects before any path can resume and replay them.
+10. Treat approval as durable state. Store who approved what, which exact proposed action was approved, when it expires, what changed after approval, and what resume step owns execution.
+11. Design interrupt and resume before side effects. Code before an interrupt may run again after resume in many graph runtimes, so keep pre-interrupt work pure or idempotent and keep irreversible effects after approval checkpoints.
+12. Version long-running state. Add a state schema version, migration or compatibility rule, stale-run behavior, and final-state classification for interrupted, cancelled, failed, succeeded, and partially applied runs.
+13. Split memory. Keep profile, thread state, and evidence cache separate; store provenance and freshness, avoid making summaries authoritative, and keep access-control and TTL rules visible.
+14. Summarize tool results into state. Preserve source IDs, key fields, uncertainty, errors, and provenance instead of passing raw blobs or letting a long observation become hidden authority.
+15. Evaluate tool triggering both ways. Test overtriggering that calls tools unnecessarily and undertriggering that skips required evidence, permission, or freshness checks.
+16. Route before handoff. Choose specialized workers or downstream agents up front from explicit inputs, pass only the state they need, and redact irrelevant conversation or tool history.
+17. Place guardrails at every boundary. Use entry guardrails for obviously disallowed requests, stage guardrails for bad plans or arguments, tool guardrails for unsafe calls, and final guardrails for response policy.
+18. Choose sequential or parallel guardrails by risk. Sequential pre-blocking protects expensive or dangerous paths before cost or side effects happen; parallel guardrails can reduce latency only when wasted work is acceptable.
+19. Add loop budgets. Enforce max steps, tool calls, repeated tool calls, retries, token spend, cost, and wall-clock time; make repeated same-tool or same-error loops visible.
+20. Retry by failure class. Distinguish invalid schema, missing information, permission denied, timeout, transient provider error, deterministic validation failure, unsafe request, and unknown outcome; do not blindly "try again."
+21. Trace the work, not only the answer. Capture decisions, plans, tool choices, tool arguments, validator results, approvals, retries, final state, and outcome fields needed to debug agent behavior.
+22. Make traces privacy-safe first. Redact or hash sensitive content before telemetry, restrict retention, and ensure eval fixtures do not become a secret dump.
+23. Evaluate from real failures. Build fixtures from incidents, near misses, wrong tool calls, bad handoffs, missing approvals, replay bugs, and loop failures; measure final state and consistency, not only final text.
+24. Treat prompt, model, tool, schema, routing, guardrail, and state changes as behavior migrations. Check old runs, old data, old approvals, cached plans, and in-flight tasks before changing the contract.
+25. Verify with the narrowest configured tests, fixtures, docs validation, release checks, and mustflow validation that cover the changed agent control plane.
+
+<!-- mustflow-section: postconditions -->
+## Postconditions
+
+- The agent path has an explicit workflow-versus-agent decision, autonomy envelope, stop conditions, stage gates, role separation, tool contracts, effect boundaries, and verification path.
+- External side effects are drafted, approved, idempotent, replay-safe, or explicitly reported as residual risk.
+- Durable state, memory, handoffs, guardrails, loop budgets, retries, traces, evals, and sensitive-data policy are bounded where they affect correctness.
+- Final reports distinguish static control-plane hardening from measured agent success, model-quality assumptions, and unverified framework behavior.
+
+<!-- mustflow-section: verification -->
+## Verification
+
+Use configured oneshot command intents when available:
+
+- `changes_status`
+- `changes_diff_summary`
+- `lint`
+- `build`
+- `test_related`
+- `test`
+- `docs_validate_fast`
+- `test_release`
+- `mustflow_check`
+
+Use the narrowest configured fixture, unit, integration, schema, docs, package, or release check that proves the changed agent execution-control contract.
+
+<!-- mustflow-section: failure-handling -->
+## Failure Handling
+
+- If a fixed workflow is sufficient, report that an autonomous agent is unnecessary and keep the simpler workflow unless the user explicitly accepts the agent tradeoff.
+- If tool ownership, side-effect ownership, or approval state is unclear, stop execution-path changes and report the missing authority boundary.
+- If interrupt or resume behavior can replay side effects and no idempotency key exists, add replay safety before claiming the agent is durable.
+- If no trace or eval covers the behavior, report static risk reduction rather than claiming the agent is reliable.
+- If a guardrail needs sensitive raw content to work, prefer scoped local checks, redacted features, or human review over unsafe telemetry.
+- If a configured command fails, use `failure-triage` before continuing.
+
+<!-- mustflow-section: output-format -->
+## Output Format
+
+- Agent execution-control surface reviewed
+- Workflow-versus-agent decision, autonomy envelope, and stop conditions
+- Stage gates, planner/executor/verifier split, and validation path
+- Tool contracts, argument ownership, draft/execute split, idempotency, approval, and side-effect replay safety
+- Durable state, resume, memory, handoff, guardrail, loop budget, retry, trace, eval, and privacy policy checked
+- Files changed
+- Command intents run
+- Skipped checks and reasons
+- Remaining agent execution-control risk

package/templates/default/locales/en/.mustflow/skills/ai-generated-code-hardening/SKILL.md

@@ -2,11 +2,11 @@
 mustflow_doc: skill.ai-generated-code-hardening
 locale: en
 canonical: true
-revision: 1
+revision: 2
 lifecycle: mustflow-owned
 authority: procedure
 name: ai-generated-code-hardening
-description: Apply this skill when AI-generated, assistant-authored, vibe-coded, or broad code changes need evidence-backed hardening against duplicate helpers or shapes, hidden coupling, weak tests, swallowed errors, excessive complexity, and boundary drift.
+description: Apply this skill when AI-generated, assistant-authored, vibe-coded, or broad code changes need evidence-backed hardening against symptom-only fixes, pinpoint hardcoding, duplicate helpers or shapes, hidden coupling, weak tests, swallowed errors, excessive complexity, and boundary drift.
 metadata:
   mustflow_schema: "1"
   mustflow_kind: procedure
@@ -31,15 +31,19 @@ metadata:
 
 Use this skill to turn agent-written code from "looks plausible" into maintainable
 repository code. The goal is not style cleanup. The goal is to catch the failure
-modes that appear when an agent forgets the repository between sessions: duplicate
-helpers, duplicate shapes, hidden coupling, weak tests, swallowed errors, accidental
-public surfaces, and oversized functions or files.
+modes that appear when an agent forgets the repository between sessions: symptom-only
+patches, pinpoint hardcoding, duplicate helpers, duplicate shapes, hidden coupling,
+weak tests, swallowed errors, accidental public surfaces, and oversized functions or
+files.
 
 <!-- mustflow-section: use-when -->
 ## Use When
 
 - The current diff was substantially generated by an AI agent, copied from an
   outside suggestion, or produced through broad assistant edits.
+- A bug fix appears to handle only the reported example, failing fixture, exact string,
+  exact path, exact ID, exact platform, or current test input instead of the same class
+  of defect.
 - A new helper, function, type, schema, DTO, fixture, adapter, or shape appears and
   the repository may already have a single source of truth.
 - The change may introduce hidden coupling through import direction, initialization
@@ -78,6 +82,8 @@ public surfaces, and oversized functions or files.
   entrypoints when present.
 - Existing error-handling conventions and the boundary where failures are meant to
   become user-facing, logged, retried, or returned.
+- The reported symptom, failing fixture, user example, or scanner finding that caused
+  the fix, plus nearby inputs that should follow the same rule.
 - Existing test style, edge-case coverage, and configured verification intents.
 - Existing static analysis or lint rules, if any. Missing rules may be reported or
   proposed, but this skill does not authorize adding new dependencies, CI gates, or
@@ -100,6 +106,9 @@ public surfaces, and oversized functions or files.
 
 - Replace duplicate new helpers, shapes, fixtures, or validators with existing
   sources of truth.
+- Replace symptom-only conditions, literal-value patches, one-off string replacements,
+  and caller-by-caller guards with the smallest rule, parser, boundary, or data owner
+  that covers the defect class.
 - Extract or move code only around a clear local responsibility boundary.
 - Flatten excessive nesting when behavior is unchanged and tests cover the path.
 - Remove accidental re-exports or update synchronized public-surface contracts.
@@ -114,11 +123,28 @@ public surfaces, and oversized functions or files.
 1. Baseline the diff.
    - Identify new functions, helpers, classes, types, schemas, DTOs, fixtures,
      adapters, exported names, and test helpers.
+   - Identify any literal values, branch conditions, special-case IDs, one-off
+     regexes, fixture names, paths, or platform checks that were added because they
+     match the reported symptom.
    - Separate intentional new concepts from possible duplicates.
    - Note which files are production code, test code, generated code, public
      contract files, or installed templates.
 
-2. Check repository memory discontinuity.
+2. Check for symptom-only patches.
+   - Name the reported symptom and the broader defect class before accepting the
+     fix. Examples: path traversal, stale cache authority, duplicate side effect,
+     invalid state transition, malformed command fence, newline handling, missing
+     cleanup, wrong ownership check, or non-atomic write.
+   - Search for sibling inputs, adjacent parsers, other callers, parallel fixtures,
+     and same-shaped branches that would fail for the same reason.
+   - Reject fixes that only say "if this exact value, return the expected answer"
+     unless the value is a documented product rule with an owner and test.
+   - Prefer changing the parser, validator, state transition, normalization rule,
+     ownership boundary, or shared helper that owns the whole defect class.
+   - If a narrow patch is the only safe current change, document why the broader
+     rule is deferred and add a regression guard that names the remaining class risk.
+
+3. Check repository memory discontinuity.
    - Search for existing helpers, types, schemas, shapes, fixtures, and validators
      before accepting a new one.
    - Prefer the existing single source of truth when it already covers the behavior.
@@ -127,7 +153,7 @@ public surfaces, and oversized functions or files.
    - If both old and new concepts must remain, name the boundary and explain why
      they are not duplicates.
 
-3. Check coupling and dependency direction.
+4. Check coupling and dependency direction.
    - Look for hidden coupling through import direction, initialization order,
      shared mutable state, environment reads, caches, file-system state, timers,
      module-level side effects, and ambient globals.
@@ -136,7 +162,7 @@ public surfaces, and oversized functions or files.
    - Inspect fan-in and fan-out. A new utility imported everywhere, or a high-level
      module imported from many low-level places, needs a boundary explanation.
 
-4. Check public surface and re-export discipline.
+5. Check public surface and re-export discipline.
    - Treat a barrel export, wildcard export, package entrypoint, generated client,
      CLI JSON field, or template-installed file as a public surface until proven
      otherwise.
@@ -144,7 +170,7 @@ public surfaces, and oversized functions or files.
      when the export is intentional.
    - Do not hide a breaking public-surface change behind "internal cleanup" wording.
 
-5. Check error handling and fallbacks.
+6. Check error handling and fallbacks.
    - Reject empty `catch`, generic fallback values, silent `null` returns, and
      duplicated defensive guards unless they are the repository's explicit boundary
      pattern.
@@ -153,7 +179,7 @@ public surfaces, and oversized functions or files.
    - Preserve useful error context without leaking secrets or personal data.
    - Prefer explicit result shapes or typed errors when they already exist locally.
 
-6. Check complexity before it calcifies.
+7. Check complexity before it calcifies.
    - Challenge nesting deeper than three levels, especially inside loops, async
      flows, parsers, command handlers, and test setup.
    - Flatten unnecessary nesting with guard clauses or early returns when that keeps
@@ -163,9 +189,11 @@ public surfaces, and oversized functions or files.
    - Split a god file only when the new module has a stable owner, direction, and
      verification path.
 
-7. Harden tests against fake confidence.
+8. Harden tests against fake confidence.
    - Verify behavior and side effects, not only function names, strings, snapshots,
      or the presence of files.
+   - For bug fixes, add at least one nearby negative or sibling case when feasible so
+     the guard proves the generalized rule, not only the reported literal input.
    - Add or preserve edge cases for empty input, missing input, null or undefined
      where the language permits it, boundary values, duplicate requests, failure
      paths, and concurrency or ordering when relevant.
@@ -175,7 +203,7 @@ public surfaces, and oversized functions or files.
    - If production code gained defensive fallback only to satisfy a brittle test,
      fix the test or move the fallback to the owning boundary.
 
-8. Use enforcement evidence without inventing enforcement.
+9. Use enforcement evidence without inventing enforcement.
    - Run existing configured lint, build, and related test intents when they cover
      the risk.
    - If the repository lacks dependency-boundary, complexity, max-depth,
@@ -184,7 +212,7 @@ public surfaces, and oversized functions or files.
    - Never paste outside plugin commands, package-manager commands, or CI recipes
      into mustflow command contracts without command-contract review.
 
-9. Decide fix now, defer, or report.
+10. Decide fix now, defer, or report.
    - Fix now when the issue is in the touched code, small, directly tied to the
      current behavior, and verifiable with configured commands.
    - Defer when the issue needs broad architecture work, new dependencies, new CI
@@ -197,6 +225,9 @@ public surfaces, and oversized functions or files.
 
 - New helpers, shapes, fixtures, validators, and exports have been checked against
   existing sources of truth.
+- Symptom-only patches, pinpoint hardcoding, literal-value branches, and one-off
+  string replacements have been replaced by a defect-class rule or explicitly
+  reported with the remaining class risk.
 - Hidden coupling, dependency direction, circular dependencies, fan-in, fan-out,
   re-export drift, and public-surface changes have been fixed, deferred, or
   explicitly reported.
@@ -226,6 +257,9 @@ public surfaces, and oversized functions or files.
   fix.
 - If repository search cannot prove whether a helper or shape is duplicate, report
   the evidence gap instead of deleting or merging concepts speculatively.
+- If the same defect class may exist in uninspected callers, parsers, fixtures, or
+  sibling inputs, report the uninspected surface instead of claiming the root cause
+  is fixed.
 - If the hardening issue requires new dependencies, new CI gates, package-wide lint
   policy, or a cross-module migration, defer it with a concrete recommendation.
 - If the current diff is too broad to inspect in one pass, use
@@ -240,6 +274,8 @@ public surfaces, and oversized functions or files.
 Include:
 
 - Existing sources of truth reused or extended.
+- Symptom-only patches, pinpoint hardcoding, and same-defect-class sibling surfaces
+  fixed, ruled out, or reported.
 - Duplicate helpers, duplicate shapes, or repeated guards removed.
 - Coupling, circular-dependency, fan-in, fan-out, public-surface, and re-export risks
   found or ruled out.