mustflow 2.116.4 → 2.117.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/package.json +1 -1
  2. package/templates/default/i18n.toml +152 -8
  3. package/templates/default/locales/en/.mustflow/skills/INDEX.md +131 -13
  4. package/templates/default/locales/en/.mustflow/skills/agent-eval-integrity-review/SKILL.md +62 -34
  5. package/templates/default/locales/en/.mustflow/skills/agent-execution-control-review/SKILL.md +102 -31
  6. package/templates/default/locales/en/.mustflow/skills/agent-memory-context-governance-review/SKILL.md +163 -0
  7. package/templates/default/locales/en/.mustflow/skills/agent-planning-recovery-review/SKILL.md +180 -0
  8. package/templates/default/locales/en/.mustflow/skills/agent-release-bundle-rollout-review/SKILL.md +181 -0
  9. package/templates/default/locales/en/.mustflow/skills/agent-runtime-isolation-review/SKILL.md +196 -0
  10. package/templates/default/locales/en/.mustflow/skills/agent-runtime-multi-worker-review/SKILL.md +180 -0
  11. package/templates/default/locales/en/.mustflow/skills/automation-investment-case-review/SKILL.md +173 -0
  12. package/templates/default/locales/en/.mustflow/skills/client-platform-strategy-review/SKILL.md +236 -0
  13. package/templates/default/locales/en/.mustflow/skills/connection-lifecycle-integrity-review/SKILL.md +302 -0
  14. package/templates/default/locales/en/.mustflow/skills/connection-lifecycle-integrity-review/references/connection-fault-injection-resource-lifetime-validation.md +277 -0
  15. package/templates/default/locales/en/.mustflow/skills/connection-lifecycle-integrity-review/references/node-stream-transport-lifecycle-checklist.md +381 -0
  16. package/templates/default/locales/en/.mustflow/skills/credit-ledger-integrity-review/SKILL.md +8 -4
  17. package/templates/default/locales/en/.mustflow/skills/credit-monetization-integrity-review/SKILL.md +283 -0
  18. package/templates/default/locales/en/.mustflow/skills/desktop-commercial-distribution-review/SKILL.md +225 -0
  19. package/templates/default/locales/en/.mustflow/skills/external-prompt-injection-defense/SKILL.md +49 -3
  20. package/templates/default/locales/en/.mustflow/skills/freemium-ad-monetization-review/SKILL.md +196 -0
  21. package/templates/default/locales/en/.mustflow/skills/game-economy-monetization-review/SKILL.md +208 -0
  22. package/templates/default/locales/en/.mustflow/skills/game-liveops-commerce-integrity-review/SKILL.md +237 -0
  23. package/templates/default/locales/en/.mustflow/skills/growth-distribution-integrity-review/SKILL.md +247 -0
  24. package/templates/default/locales/en/.mustflow/skills/idempotency-integrity-review/SKILL.md +20 -2
  25. package/templates/default/locales/en/.mustflow/skills/llm-model-routing-integrity-review/SKILL.md +183 -0
  26. package/templates/default/locales/en/.mustflow/skills/llm-product-monetization-review/SKILL.md +311 -0
  27. package/templates/default/locales/en/.mustflow/skills/llm-token-cost-control-review/SKILL.md +15 -1
  28. package/templates/default/locales/en/.mustflow/skills/localization-market-expansion-review/SKILL.md +224 -0
  29. package/templates/default/locales/en/.mustflow/skills/memory-lifetime-review/SKILL.md +94 -2
  30. package/templates/default/locales/en/.mustflow/skills/memory-lifetime-review/references/cross-runtime-memory-stream-reproduction-harness.md +289 -0
  31. package/templates/default/locales/en/.mustflow/skills/memory-lifetime-review/references/node-memory-resource-backpressure-diagnostics.md +369 -0
  32. package/templates/default/locales/en/.mustflow/skills/multi-agent-work-coordination/SKILL.md +7 -2
  33. package/templates/default/locales/en/.mustflow/skills/pricing-model-integrity-review/SKILL.md +288 -0
  34. package/templates/default/locales/en/.mustflow/skills/product-engagement-retention-review/SKILL.md +234 -0
  35. package/templates/default/locales/en/.mustflow/skills/product-onboarding-activation-review/SKILL.md +269 -0
  36. package/templates/default/locales/en/.mustflow/skills/product-portfolio-integrity-review/SKILL.md +233 -0
  37. package/templates/default/locales/en/.mustflow/skills/prompt-contract-quality-review/SKILL.md +1 -0
  38. package/templates/default/locales/en/.mustflow/skills/referral-incentive-integrity-review/SKILL.md +206 -0
  39. package/templates/default/locales/en/.mustflow/skills/retry-policy-integrity-review/SKILL.md +45 -2
  40. package/templates/default/locales/en/.mustflow/skills/routes.toml +346 -7
  41. package/templates/default/locales/en/.mustflow/skills/service-portfolio-capital-allocation-review/SKILL.md +245 -0
  42. package/templates/default/locales/en/.mustflow/skills/subscription-retention-profit-review/SKILL.md +217 -0
  43. package/templates/default/manifest.toml +97 -1
@@ -2,11 +2,11 @@
2
2
  mustflow_doc: skill.agent-eval-integrity-review
3
3
  locale: en
4
4
  canonical: true
5
- revision: 1
5
+ revision: 2
6
6
  lifecycle: mustflow-owned
7
7
  authority: procedure
8
8
  name: agent-eval-integrity-review
9
- description: Apply this skill when LLM agent evaluation loops, trace or trajectory grading, LLM judges, verifier agents, outcome scoring, tool-call prechecks or postchecks, eval datasets, golden or dirty sets, pass@k or pass^k metrics, shadow environments, production-monitoring-to-eval pipelines, or agent regression gates are created, changed, reviewed, or reported and the risk is false confidence in agent correctness rather than runtime execution control alone.
9
+ description: Apply this skill when LLM agent evaluation loops, deterministic gates, trace or trajectory grading, LLM judges, verifier agents, repair cascades, outcome scoring, tool-call prechecks or postchecks, fixed, replay, or generated eval datasets, pass@k or pass^k metrics, shadow environments, production-monitoring-to-eval pipelines, or agent regression gates are created, changed, reviewed, or reported and the risk is false confidence, false rejection, stale eval truth, or correlated verification rather than runtime execution control alone.
10
10
  metadata:
11
11
  mustflow_schema: "1"
12
12
  mustflow_kind: procedure
@@ -29,7 +29,7 @@ metadata:
29
29
  <!-- mustflow-section: purpose -->
30
30
  ## Purpose
31
31
 
32
- Review agent evaluation as an evidence loop, not a judge-model opinion. A real agent eval should test the final environment state, the trace or trajectory, tool boundaries, recovery behavior, repeatability, sensitive-data handling, and how real failures become future regression cases.
32
+ Review agent evaluation as an asymmetric evidence cascade, not a judge-model opinion. Deterministic rules should reject reproducible failures first, evidence-backed repair should run only after failure, independent semantic verification should cover what rules cannot express, and ambiguous disagreement should remain held or receive bounded adjudication instead of becoming a confident pass or fail. A real agent eval should also test final environment state, trajectory, tool boundaries, recovery, repeatability, dataset freshness, contamination, sensitive-data handling, and how confirmed failures become future regression cases.
33
33
 
34
34
  <!-- mustflow-section: use-when -->
35
35
  ## Use When
@@ -39,6 +39,7 @@ Review agent evaluation as an evidence loop, not a judge-model opinion. A real a
39
39
  - Tool calls, handoffs, guardrails, custom events, human approvals, retries, checkpoints, or external effects should be captured and graded as part of the agent run.
40
40
  - An eval can depend on final database state, queue state, email state, payment state, file state, test results, permissions, rollback evidence, or another observable environment result.
41
41
  - Agent failures, human overrides, user complaints, tool errors, timeouts, cost spikes, repeated retries, or rollbacks should feed future eval candidates.
42
+ - False-positive and false-negative costs, verifier disagreement, correlated judge errors, self-review, evidence-backed repair, dataset contamination, oracle expiry, or stale expected answers affect release decisions.
42
43
 
43
44
  <!-- mustflow-section: do-not-use-when -->
44
45
  ## Do Not Use When
@@ -54,11 +55,12 @@ Review agent evaluation as an evidence loop, not a judge-model opinion. A real a
54
55
  ## Required Inputs
55
56
 
56
57
  - Outcome ledger: task type, success definition, final environment state, observable artifacts, external effects, reversible versus irreversible outcomes, and failure states.
58
+ - Decision-loss ledger: false-positive cost, false-negative cost, execution and review cost, risk class, hard-block invariants, diagnostic-only findings, abstain or hold state, adjudication owner, and release consequence.
57
59
  - Trace ledger: model calls, tool calls, handoffs, guardrails, approvals, retries, custom events, tool observations, intermediate plans, rejected actions, and final response.
58
- - Oracle ledger: deterministic checks, schemas, static analysis, tests, state queries, regex or exact matchers, model judges, sampled human review, and which oracle owns each claim.
60
+ - Oracle ledger: deterministic checks, schemas, static analysis, tests, state queries, regex or exact matchers, semantic verifiers, secondary adjudicators, sampled human review, evidence type, expected independence, known shared failure modes, and which oracle owns each claim.
59
61
  - Tool-boundary ledger: precheck, postcheck, permission check, idempotency check, state-before, state-after, changed fields, evidence references, tool error, timeout, and token size.
60
- - Dataset ledger: golden regression set, dirty real-world set, capability set, manual smoke cases, source of each case, labels, expected outcomes, flakiness, and release-gate status.
61
- - Metric ledger: pass@k, pass^k, per-task trial count, final-state success, trajectory safety, tool count, runtime, token consumption, cost, tool errors, retries, human overrides, and rollback rate.
62
+ - Dataset ledger: fixed regression set, recent real-traffic replay set, generated or perturbed exploration set, representative and risk-biased sampling strata, capability set, manual smoke cases, source of each case, labels, expected outcomes, grouping keys, sealed holdout status, validity interval, tool and policy versions, flakiness, and release-gate status.
63
+ - Metric ledger: verified task success, first-attempt success, silent failure, partial completion, user correction, plan deviation, pass@k, pass^k, per-task trial count, final-state success, trajectory safety, tool count, unnecessary tool calls, runtime distribution, approval wait, token consumption, cost per verified success, tool errors by class, retries, duplicate effects, human overrides, near misses, rollback and compensation rate, approval prompts, and approval rejection or cancellation.
62
64
  - Environment ledger: production, staging, sandbox, fake adapter, shadow environment, seed data, cleanup policy, isolation, state reset, and side-effect containment.
63
65
  - Monitoring ledger: user complaint, retry spike, timeout, tool error, human override, rollback, cost breach, guardrail block, and how each signal becomes an eval candidate.
64
66
  - Privacy ledger: trace retention, redaction, prompt and tool payload policy, eval fixture scrubbing, access control, and sensitive-data exclusion.
@@ -75,9 +77,9 @@ Review agent evaluation as an evidence loop, not a judge-model opinion. A real a
75
77
  <!-- mustflow-section: allowed-edits -->
76
78
  ## Allowed Edits
77
79
 
78
- - Add or refine outcome oracles, trace schemas, trajectory graders, deterministic checkers, model-judge rubrics, human-review sampling, tool prechecks and postchecks, tool-result evidence packets, eval fixtures, golden and dirty sets, shadow-environment adapters, monitoring-to-eval candidate flows, tests, docs, route metadata, and directly synchronized templates.
80
+ - Add or refine outcome oracles, asymmetric verification cascades, trace schemas, trajectory graders, deterministic checkers, evidence-backed repair inputs, semantic-verifier rubrics, secondary adjudication, human-review sampling, tool prechecks and postchecks, tool-result evidence packets, fixed regression, recent replay, and generated exploration sets, contamination controls, oracle validity metadata, shadow-environment adapters, monitoring-to-eval candidate flows, tests, docs, route metadata, and directly synchronized templates.
79
81
  - Split side-effect tools into prepare, verify, and commit phases when eval evidence needs a safe pre-execution artifact and a post-execution state check.
80
- - Add focused fixtures for false-success final answers, missing environment state, unsafe tool path, bad tool argument, over-trusting a judge model, trace redaction leaks, repeated tool loops, fragile tool-order assertions, flaky dirty-set cases, and pass@k versus pass^k drift.
82
+ - Add focused fixtures for false-success final answers, false rejection without a reproducible violation, missing environment state, unsafe tool path, bad tool argument, over-trusting a judge model, correlated judges, self-review certifying its own output, ambiguous verifier disagreement, stale expected truth, train-eval leakage, trace redaction leaks, repeated tool loops, fragile tool-order assertions, flaky real-traffic cases, and pass@k versus pass^k drift.
81
83
  - Do not treat an LLM judge, self-reflection step, reasoning trace, or final answer as proof without observable state, deterministic checks, calibrated rubric evidence, or human review.
82
84
  - Do not force one exact tool sequence when several safe paths can satisfy the task; grade required evidence, forbidden actions, final state, approvals, cost bounds, and safety constraints instead.
83
85
  - Do not store raw prompts, tool payloads, database rows, personal data, secrets, or full conversations in traces or eval fixtures unless the repository has an explicit sensitive-data policy for that surface.
@@ -88,35 +90,57 @@ Review agent evaluation as an evidence loop, not a judge-model opinion. A real a
88
90
  1. Name the eval target. Decide whether the eval measures capability exploration, regression safety, release gating, incident reproduction, tool selection, trace quality, or production monitoring.
89
91
  2. Score final environment state before final text. For action agents, success means the booking, refund, email, file, ticket, database row, queue message, test result, or other observable state is correct.
90
92
  3. Score the trajectory as a separate dimension. A correct final answer can still fail when the path used unsafe tools, repeated searches, excessive tokens, hallucinated intermediate facts, skipped approvals, or relied on luck.
91
- 4. Build oracle layers. Prefer deterministic checks for schemas, exact state, permissions, idempotency, static analysis, unit tests, and numeric rules; use model judges for semantic or rubric-heavy cases; sample humans for calibration and high-risk ambiguity.
92
- 5. Keep self-reflection out of final judgment. Use reflection to propose repairs or retry strategy, not to certify success.
93
- 6. Calibrate judge models. Compare judge outputs with deterministic checks or human review, track disagreement, and avoid treating one judge score as a stable product metric.
94
- 7. Add prechecks and postchecks around tools. Validate tool arguments before execution and validate state, evidence references, errors, and permissions after execution.
95
- 8. Keep destructive actions sequentially gated. For irreversible or costly actions, block before model calls or tool execution when possible; do not rely on a late parallel guardrail after cost or side effects have already happened.
96
- 9. Split side effects into prepare, verify, and commit. Evaluate the prepared payload or diff before the commit phase, then evaluate the committed state afterward.
97
- 10. Make tool results evidence packets. Include state before, state after, changed fields, evidence refs, permission scope, idempotency key, and structured error or timeout state instead of only a friendly success sentence.
98
- 11. Fuzz tool schemas. Test missing fields, wrong enum values, similar IDs, empty strings, long queries, unauthorized resource IDs, deleted objects, stale objects, and ambiguous names.
99
- 12. Test tool names and namespaces. When multiple tools overlap, use service or resource prefixes and eval whether routing improves without overfitting to one prompt.
100
- 13. Track tool payload size. Large tool responses are eval failures when they hide evidence, waste context, or cause the model to miss the relevant fields; use pagination, filtering, truncation, or summaries with source IDs.
101
- 14. Use shadow environments first. Evaluate payment, email, CRM, repository, file, database, and external mutations against sandbox or fake adapters before connecting to production systems.
102
- 15. Require checkpoints for human-in-the-loop. Human approval or correction should point to a durable state snapshot that can resume, replay safely, or roll back.
103
- 16. Classify failure handling by risk. High-risk tasks fail closed on verification gaps; reversible drafting, search, and classification can fail soft with a visible "needs review" state.
104
- 17. Separate capability evals from regression evals. Capability sets may start hard and improve over time; regression golden sets should stay stable and near-perfect before release.
105
- 18. Separate golden and dirty sets. Golden cases gate releases and should be clean, deterministic, and reviewable. Dirty cases come from real failures and may be noisy, quarantined, or used for diagnosis before promotion.
106
- 19. Run multiple trials where model variance matters. Use pass@k for "can solve at least once" capability claims and pass^k for "reliably solves every time" customer-facing claims.
107
- 20. Avoid brittle path assertions. Prefer required evidence, forbidden actions, final state, safety boundaries, approval requirements, and cost ceilings over one exact tool order unless the order is itself the contract.
108
- 21. Promote production failures into eval candidates. Mine complaints, retries, timeouts, tool errors, human overrides, rollbacks, cost breaches, and guardrail blocks into reviewed cases.
109
- 22. Let evals improve tool design. When failures cluster around ambiguous names, huge outputs, weak errors, missing fields, bad descriptions, or fuzzy parameters, fix the tool contract rather than blaming the model.
110
- 23. Keep traces privacy-safe first. Store spans, IDs, hashes, sizes, versions, state classifications, and redacted reason codes by default; scrub or exclude sensitive raw content from eval fixtures.
111
- 24. Treat prompt, model, tool, judge, rubric, dataset, metric, and environment changes as eval migrations. Recompare baselines before claiming improvement.
112
- 25. Verify with the narrowest configured tests, fixtures, docs validation, release checks, and mustflow validation that cover the changed eval integrity contract.
93
+ 4. Define the decision loss before choosing an oracle. Estimate false-rejection, defect-escape, execution, latency, and human-review costs by task risk. A threshold cannot minimize false positives and false negatives independently; it implements a declared tradeoff. Safety invariants, payment, permission, destructive, and disclosure failures usually assign much higher cost to a missed defect than to a held result.
94
+ 5. Build an asymmetric cascade:
95
+ - run deterministic hard rules and high-reproducibility tests first;
96
+ - when they fail, give the generator only the concrete failure evidence or smallest counterexample needed for bounded repair;
97
+ - after hard gates pass, use an independent semantic verifier for claims rules cannot express;
98
+ - when the verifier and deterministic evidence disagree or the verdict is ambiguous, call a bounded secondary adjudicator or hold for human review.
99
+ Do not pay for every stage on every case when a cheaper conclusive stage already owns the verdict.
100
+ 6. Define verdict logic explicitly. A pass requires every applicable hard gate to pass and no confirmed semantic violation. A fail requires a reproducible violation, an unambiguous safety invariant breach, or an independent adjudication that owns the disputed claim. Everything else remains `held`, `needs_review`, or another explicit non-passing, non-failing state.
101
+ 7. Build oracle layers. Prefer deterministic checks for schemas, exact state, permissions, idempotency, static analysis, unit tests, properties, invariants, and numeric rules. Use semantic verifiers for rubric-heavy cases and sampled humans for calibration and high-risk ambiguity.
102
+ 8. Make semantic verification independent in inputs, not just agent count. Give the verifier the original requirements, final artifact, observable state, and evidence packet; hide generator identity, chain-of-thought, persuasive self-justification, and irrelevant trace prose. Split the rubric into factuality, omission, forbidden behavior, side effects, reversibility, and other task-owned dimensions.
103
+ 9. Treat multiple similar judges as correlated evidence until calibration proves otherwise. Model-family labels, prompt variants, or vote count alone do not create independent oracles. Prefer a different evidence type, deterministic counterexample, state query, or human review over adding more judges that share the same blind spot.
104
+ 10. Keep self-reflection out of final judgment. Use it as a repair worker only after a test, counterexample, tool receipt, source conflict, or verifier finding identifies what must change. Do not let the generator certify that its own repair succeeded.
105
+ 11. Calibrate semantic verifiers and adjudicators. Compare outputs with deterministic checks or human review, track false-rejection and defect-escape rates separately, measure disagreement by rubric dimension, and avoid treating one judge score as a stable product metric.
106
+ 12. Add prechecks and postchecks around tools. Validate tool arguments before execution and validate state, evidence references, errors, and permissions after execution.
107
+ 13. Keep destructive actions sequentially gated. For irreversible or costly actions, block before model calls or tool execution when possible; do not rely on a late parallel guardrail after cost or side effects have already happened.
108
+ 14. Split side effects into prepare, verify, and commit. Evaluate the prepared payload or diff before the commit phase, then evaluate the committed state afterward.
109
+ 15. Make tool results evidence packets. Include state before, state after, changed fields, evidence refs, permission scope, idempotency key, and structured error or timeout state instead of only a friendly success sentence.
110
+ 16. Fuzz tool schemas. Test missing fields, wrong enum values, similar IDs, empty strings, long queries, unauthorized resource IDs, deleted objects, stale objects, and ambiguous names.
111
+ 17. Test tool names and namespaces. When multiple tools overlap, use service or resource prefixes and eval whether routing improves without overfitting to one prompt.
112
+ 18. Track tool payload size. Large tool responses are eval failures when they hide evidence, waste context, or cause the model to miss the relevant fields; use pagination, filtering, truncation, or summaries with source IDs.
113
+ 19. Use shadow environments first. Evaluate payment, email, CRM, repository, file, database, and external mutations against sandbox or fake adapters before connecting to production systems.
114
+ 20. Require checkpoints for human-in-the-loop. Human approval or correction should point to a durable state snapshot that can resume, replay safely, or roll back.
115
+ 21. Classify failure handling by risk. High-risk tasks fail closed on verification gaps; reversible drafting, search, and classification can fail soft with a visible `needs_review` state.
116
+ 22. Keep three dataset roles separate:
117
+ - fixed regression cases cover core flows, confirmed incidents, permissions, duplicate effects, rollback, wrong targets, and other known defects;
118
+ - recent real-traffic replay cases estimate current distribution in a side-effect-free shadow environment, with representative and risk-biased samples reported separately;
119
+ - generated or perturbed exploration cases vary scope, similar identities, time zones, permissions, timeouts, stale state, ambiguity, and other long-tail conditions around explicit invariants.
120
+ Do not average one easy set over a regression in another set.
121
+ 23. Promote confirmed failures. A generated or noisy replay failure becomes a fixed regression only after a deterministic counterexample, independent confirmation, or human review establishes the expected behavior. Critical security, permission, financial, privacy, and irreversible-effect invariants may hard-block immediately when their truth is unambiguous.
122
+ 24. Keep capability evals separate from regression gates. Capability sets may start hard and improve over time; fixed regression cases for confirmed critical defects should remain fully passing before release. Quarantine flaky replay cases as diagnostic evidence rather than weakening the fixed gate.
123
+ 25. Prevent contamination. Group near-duplicate cases by user, task family, source artifact, template, and workflow lineage before splitting. Use time-based sealed holdouts when evaluating future behavior, keep private evaluation cases out of prompts, retrieval memory, fine-tuning, and generated examples, and record normalized-input hashes or similarity evidence where leakage matters.
124
+ 26. Version expected truth. Record source, collection time, valid-from and valid-to bounds, tool schema, policy version, environment snapshot, time zone, oracle version, label owner, and training-use status as applicable. Expire or quarantine cases whose world, API, policy, or oracle changed instead of counting stale expectations as model regressions.
125
+ 27. Prefer invariant-based expected outcomes over brittle exact prose. Grade allowed targets, required state, forbidden duplicate effects, approval boundaries, amount or count constraints, and recovery obligations unless exact text or exact tool order is itself the contract.
126
+ 28. Run multiple trials where model variance matters. Use pass@k for `can solve at least once` capability claims and pass^k for `reliably solves every time` customer-facing claims.
127
+ 29. Keep operational metric families separate. Report outcomes, process, safety, efficiency, and approval experience independently. Do not combine them into one score that lets high search success, low cost, or easy traffic hide a high-risk miss.
128
+ 30. Measure verified success rather than self-reported completion. Track first-attempt success, silent failure, partial completion, user correction with mind-change separated from defect correction, plan deviation, repeated identical error, duplicate effect, rollback and compensation, near misses, unnecessary tool calls, and cost per verified success. Use task-specific latency percentiles and separate approval wait from execution time.
129
+ 31. Define alerts with both absolute rules and baseline movement. A single clear safety-invariant breach may block immediately. Rate-based alerts should declare minimum sample size, comparison window, segmentation, and both absolute and relative change so tiny samples and large baseline regressions are not treated alike.
130
+ 32. Segment by task type, risk tier, tool, model and prompt version, customer group, region, and language when those dimensions can hide localized regressions. Mark release and configuration changes on the same evidence timeline.
131
+ 33. Promote production failures into eval candidates. Mine complaints, retries, timeouts, tool errors, human overrides, rollbacks, compensations, cost breaches, approval rejection, and guardrail blocks into reviewed cases.
132
+ 34. Let evals improve tool design. When failures cluster around ambiguous names, huge outputs, weak errors, missing fields, bad descriptions, or fuzzy parameters, fix the tool contract rather than blaming the model.
133
+ 35. Keep traces and datasets privacy-safe first. Store spans, IDs, hashes, sizes, versions, state classifications, and redacted reason codes by default; scrub or exclude sensitive raw content from eval fixtures and minimize production data retained for replay.
134
+ 36. Treat prompt, model, tool, verifier, adjudicator, rubric, dataset, metric, policy, and environment changes as eval migrations. Recompare baselines and validity metadata before claiming improvement.
135
+ 37. Verify with the narrowest configured tests, fixtures, docs validation, release checks, and mustflow validation that cover the changed eval integrity contract.
113
136
 
114
137
  <!-- mustflow-section: postconditions -->
115
138
  ## Postconditions
116
139
 
117
140
  - The eval loop grades final state, trajectory quality, tool-boundary evidence, recovery behavior, repeatability, and sensitive-data handling where relevant.
118
- - Deterministic checks, model judges, and human review have explicit ownership rather than one judge model owning every verdict.
119
- - Golden, dirty, capability, regression, pass@k, pass^k, shadow environment, production-monitoring, and trace-retention decisions are explicit when they affect agent quality claims.
141
+ - Deterministic checks, semantic verifiers, secondary adjudicators, and human review have explicit ownership, verdict branches terminate, and model count is not treated as oracle independence.
142
+ - Fixed regression, recent replay, generated exploration, representative and risk-biased strata, contamination, expiry, capability, pass@k, pass^k, shadow environment, production-monitoring, and trace-retention decisions are explicit when they affect agent quality claims.
143
+ - Outcome, process, safety, efficiency, and approval-experience measures remain separate, with hard safety blocks distinguished from rate-based alerts.
120
144
  - Final reports distinguish measured eval improvement from static hardening, judge preference, self-reflection, anecdotal demos, and unverified provider behavior.
121
145
 
122
146
  <!-- mustflow-section: verification -->
@@ -142,6 +166,9 @@ Use the narrowest configured fixture, unit, integration, schema, docs, package,
142
166
  - If only a final answer or judge-model score is available, report the eval as weak and add observable outcome or trajectory evidence before claiming reliability.
143
167
  - If deterministic checks can own a verdict, use them before asking a model judge.
144
168
  - If model judges disagree or drift, calibrate with human or deterministic evidence and avoid treating the metric as release-blocking until stable.
169
+ - If a semantic verifier reports a defect without a reproducible violation or independent confirmation, hold or adjudicate the case instead of converting suspicion directly into a hard failure.
170
+ - If an expected answer or environment snapshot is stale, quarantine and relabel the case before attributing the result to model regression.
171
+ - If training, prompt, retrieval-memory, or template leakage can expose eval cases, stop using the affected score as a generalization claim until grouped or time-based separation is restored.
145
172
  - If production traces contain sensitive raw content, stop copying that content into evals and add redaction, hashing, or fixture scrubbing first.
146
173
  - If a dirty case is flaky, quarantine it as diagnostic evidence instead of weakening the golden regression gate.
147
174
  - If a configured command fails, use `failure-triage` before continuing.
@@ -151,9 +178,10 @@ Use the narrowest configured fixture, unit, integration, schema, docs, package,
151
178
 
152
179
  - Agent eval-integrity surface reviewed
153
180
  - Outcome oracle, final-state checks, and trajectory checks
154
- - Deterministic, model-judge, and human-review oracle split
181
+ - Decision-loss model, asymmetric cascade, pass, fail, and hold logic
182
+ - Deterministic, semantic-verifier, secondary-adjudicator, and human-review oracle split and independence evidence
155
183
  - Tool prechecks, postchecks, schema fuzzing, evidence packets, and payload-size checks
156
- - Golden, dirty, capability, regression, pass@k, pass^k, shadow environment, monitoring-to-eval, trace, and privacy decisions
184
+ - Fixed regression, recent replay, generated exploration, contamination, expiry, capability, pass@k, pass^k, shadow environment, monitoring-to-eval, operational metrics, alerting, trace, and privacy decisions
157
185
  - Files changed
158
186
  - Command intents run
159
187
  - Skipped checks and reasons
@@ -2,11 +2,11 @@
2
2
  mustflow_doc: skill.agent-execution-control-review
3
3
  locale: en
4
4
  canonical: true
5
- revision: 2
5
+ revision: 5
6
6
  lifecycle: mustflow-owned
7
7
  authority: procedure
8
8
  name: agent-execution-control-review
9
- description: Apply this skill when autonomous or semi-autonomous LLM agents, agentic workflows, planners, executors, verifiers, tool contracts, tool-call gates, human approval or interrupt flows, durable agent state, handoffs, guardrails, loop budgets, retry policies, trace evaluation, or agent outcome metrics are created, changed, reviewed, or reported and the risk is uncontrolled agent behavior rather than only prompt wording, token cost, response latency, or factual grounding.
9
+ description: Apply this skill when autonomous or semi-autonomous LLM agents, agentic workflows, planners, executors, verifiers, runtime-computed tool and transition policies, tool-call gates, risk-tiered approval or interrupt flows, reversible and irreversible effects, durable agent state, handoffs, guardrails, autonomy budgets, retry policies, trace evaluation, or agent outcome metrics are created, changed, reviewed, or reported and the risk is uncontrolled agent behavior.
10
10
  metadata:
11
11
  mustflow_schema: "1"
12
12
  mustflow_kind: procedure
@@ -29,7 +29,7 @@ metadata:
29
29
  <!-- mustflow-section: purpose -->
30
30
  ## Purpose
31
31
 
32
- Review agent systems as controlled workbenches, not smarter chat prompts. An agentic path should make bad choices hard by separating workflow selection, planning, execution, verification, tools, approvals, durable state, guardrails, budgets, retries, traces, and outcome evaluation.
32
+ Review agent systems as controlled workbenches, not smarter chat prompts. An agentic path should make bad choices hard by separating workflow selection, planning, execution, verification, tools, risk and reversibility classification, just-in-time approvals, durable state, guardrails, budgets, retries, traces, and outcome evaluation.
33
33
 
34
34
  <!-- mustflow-section: use-when -->
35
35
  ## Use When
@@ -37,8 +37,11 @@ Review agent systems as controlled workbenches, not smarter chat prompts. An age
37
37
  - A change adds, edits, reviews, or reports an autonomous or semi-autonomous LLM agent, agent loop, planner, executor, verifier, evaluator, router, handoff, tool-calling workflow, or long-running AI task runner.
38
38
  - A task asks whether an AI agent should be a fixed workflow, dynamic agent, prompt chain, routed worker, tool user, human-approved assistant, or multi-step automation.
39
39
  - Tool definitions, tool names, tool argument schemas, tool-call guards, server-known arguments, tool-result summarization, or unsafe tool execution policies change.
40
+ - Code computes which tools, transitions, privileges, retries, or autonomous steps are currently
41
+ available from workflow state, remaining budget, approval, and observed reliability.
40
42
  - The system can write data, send messages, delete or mutate resources, spend money, call external services, open support tickets, start jobs, or perform any external side effect through model-directed steps.
41
43
  - Human approval, interrupt or resume behavior, durable checkpoints, state versioning, memory, trace telemetry, loop limits, retry policy, or agent evals affect agent correctness.
44
+ - Approval fatigue, over-broad standing permission, approval-to-execution drift, fake undo claims, compensation, delayed commit, policy approval, or independent postcondition verification affects whether an external action is safe.
42
45
 
43
46
  <!-- mustflow-section: do-not-use-when -->
44
47
  ## Do Not Use When
@@ -49,6 +52,21 @@ Review agent systems as controlled workbenches, not smarter chat prompts. An age
49
52
  - The main risk is time to first token, first useful output, LLM round trips, tool wait, streaming latency, prompt-cache latency, or user-perceived response speed; use `llm-response-latency-review`.
50
53
  - The main risk is chat, prompt composer, streaming UI, citation display, progress state, history UX, or manual fallback UI; use `llm-service-ux-review`.
51
54
  - The main risk is agent eval integrity, trace or trajectory grading, judge-model calibration, final environment state scoring, golden or dirty eval sets, pass@k or pass^k metrics, shadow environments, or production-monitoring-to-eval feedback; use `agent-eval-integrity-review`.
55
+ - The main risk is persistent user memory, conversation-history compaction, memory extraction, consolidation, retrieval, supersession, expiry, deletion, or memory-to-context access control; use `agent-memory-context-governance-review`.
56
+ - The main risk is packaging model, prompt, tool, policy, retrieval, memory, runtime, and evaluator
57
+ versions into one immutable release and promoting it through replay, shadow, canary, or rollback;
58
+ use `agent-release-bundle-rollout-review`.
59
+ - The main risk is balancing a stable global goal with rolling detailed plans, event-triggered
60
+ replanning, deterministic replay, compact prompt reconstruction, and effect identity across
61
+ replans; use `agent-planning-recovery-review`.
62
+ - The main risk is selecting, cascading, escalating, or falling back among models by quality, cost,
63
+ latency, verifier result, or distribution shift; use `llm-model-routing-integrity-review`.
64
+ - The main risk is deciding whether several production runtime agents beat one and defining worker
65
+ topology, role diversity, artifacts, correlation, and central verification; use
66
+ `agent-runtime-multi-worker-review`.
67
+ - The main risk is durable state outside short-lived workers, capability queues, ephemeral
68
+ sandboxes, temporary credentials, or brokered production effects; use
69
+ `agent-runtime-isolation-review`.
52
70
  - The task coordinates several repository workers or subagents for one coding task rather than designing a product/runtime agent; use `multi-agent-work-coordination`.
53
71
  - The task is ordinary idempotency, retry, queue, approval, state-machine, or workflow code with no LLM-directed agent behavior; use the narrower integrity or pattern skill first.
54
72
  - The main problem is non-LLM parent-child lifetime, join, cancellation, deadline, or orphan prevention; use `structured-concurrency-supervision-review`.
@@ -60,16 +78,29 @@ Review agent systems as controlled workbenches, not smarter chat prompts. An age
60
78
  ## Required Inputs
61
79
 
62
80
  - Autonomy ledger: task goal, why a fixed workflow is insufficient, allowed decisions, forbidden decisions, dynamic steps, deterministic steps, stop conditions, and human-owner boundary.
81
+ - Policy-graph ledger: authoritative state, allowed transitions, tool preconditions, privilege and
82
+ risk class, remaining capability, cost and harm budgets, approval and consent state, postcondition,
83
+ next-state reducer, and fail-closed behavior when any input is unknown.
84
+ - Autonomy-evidence ledger: externally verified decision and postcondition success, harmful-effect
85
+ rate, uncertainty bound, task and risk cohort, sample support, dependency between consecutive
86
+ decisions, checkpoint spacing, and the rule that shortens or stops an autonomous interval.
63
87
  - Stage gate ledger: requirement extraction, plan, route, tool selection, tool arguments, execution, verification, final response, and which gate can reject or revise each artifact.
64
88
  - Role separation ledger: planner, executor, verifier, evaluator, deterministic validators, human reviewers, and any self-check limitations.
65
89
  - Tool contract ledger: tool names, descriptions, argument schema, trusted server-known arguments, model-supplied arguments, unsafe actions, preconditions, postconditions, idempotency key, timeout, partial result, and error states.
66
90
  - Capability ledger: tenant and resource scope, permitted effects, expiry, call count, token or cost ceiling, attenuation chain, server-known arguments, and the exact policy decision and tool call the capability authorizes.
67
91
  - Effect ledger: draft versus execute mode, external side effects, database writes, emails, payments, deletes, job starts, queue publishes, cache invalidations, idempotency key source, approval requirement, and side-effect position relative to interrupts.
92
+ - Approval and reversibility ledger: meaningful user action, target and audience, scope and count, amount or quota, data sensitivity, privilege level, intent uncertainty, detectability, forced-approval condition, risk tier, exact rollback path, compensating action, irreversible boundary, delay or undo window, approval policy, and escalation owner.
68
93
  - State and resume ledger: run ID, thread ID, checkpoint store, state schema version, migration or compatibility rule, resume owner, pending side effects, completed side effects, and recovery behavior.
69
- - Memory and context ledger: profile, thread state, and evidence cache boundaries, retention, TTL, provenance, access control, and stale-state handling.
94
+ - Planning ledger: global goal, invariants, milestone DAG, irreversible checkpoints, rolling plan,
95
+ replan triggers, plan version, workflow, step, effect and attempt identities, and UNKNOWN-effect
96
+ reconciliation; use `agent-planning-recovery-review` when planning and recovery are primary.
97
+ - Release-bundle ledger: immutable bundle ID, model, rendered prompt, tool and policy versions,
98
+ retrieval and memory versions, runtime lock, evaluator, candidate and stable assignment, and
99
+ rollback target; use `agent-release-bundle-rollout-review` when rollout is primary.
100
+ - Memory and context ledger: profile, thread state, and evidence cache boundaries, retention, TTL, provenance, access control, and stale-state handling; use `agent-memory-context-governance-review` when persistence or memory admission is the primary risk.
70
101
  - Handoff and guardrail ledger: router inputs, handoff payload, redacted conversation state, entry guardrails, mid-chain guardrails, tool guardrails, final-output guardrails, and sequential versus parallel block decision.
71
102
  - Loop, retry, and budget ledger: max steps, max tool calls, max repeated tool calls, max retries, token budget, cost budget, wall-clock budget, failure classes, retryable classes, and escalation classes.
72
- - Trace and eval outcome ledger: trace spans, intermediate decisions, tool calls, tool observations, rejected plans, approvals, final state, real failure cases, consistency measurement, pass rate, and sensitive-data redaction policy.
103
+ - Trace and eval outcome ledger: trace spans, intermediate decisions, tool calls, tool observations, rejected plans, approvals, approval prompts per successful task, plan deviation, final state, verified success, first-attempt success, silent failure, partial completion, user correction, near miss, rollback or compensation, duplicate effect, real failure cases, consistency measurement, cost and latency, and sensitive-data redaction policy.
73
104
 
74
105
  <!-- mustflow-section: preconditions -->
75
106
  ## Preconditions
@@ -83,9 +114,9 @@ Review agent systems as controlled workbenches, not smarter chat prompts. An age
83
114
  <!-- mustflow-section: allowed-edits -->
84
115
  ## Allowed Edits
85
116
 
86
- - Add or refine workflow-versus-agent routing, stage gates, planner/executor/verifier boundaries, tool contracts, tool argument ownership, draft/execute separation, idempotency keys, approval records, durable checkpoints, state schema versions, memory partitions, handoff filters, guardrails, loop budgets, retry classification, trace spans, eval fixtures, tests, docs, route metadata, and directly synchronized templates.
117
+ - Add or refine workflow-versus-agent routing, stage gates, planner/executor/verifier boundaries, tool contracts, tool argument ownership, semantic action grouping, forced-approval conditions, risk tiers, reversibility classes, scoped approval policies, just-in-time approval records, approval-token binding, delayed commit, exact rollback and compensation paths, draft/execute separation, idempotency keys, durable checkpoints, state schema versions, memory partitions, handoff filters, guardrails, loop budgets, retry classification, trace spans, eval fixtures, tests, docs, route metadata, and directly synchronized templates.
87
118
  - Move fixed transformations, classification, validation, permissions, date math, ID lookup, formatting, routing, and invariant checks out of model discretion when deterministic code can own them.
88
- - Add focused tests for fixed-workflow selection, bad plan rejection, unsafe tool preblock, missing idempotency key, repeated tool loop, resume-after-interrupt replay, stale state version, handoff over-sharing, retry misclassification, and trace redaction.
119
+ - Add focused tests for fixed-workflow selection, bad plan rejection, unsafe tool preblock, high-impact action hidden by a low average risk score, bulk-scope escalation, compensation mislabeled as rollback, approval-plan drift, expired or revoked policy approval, missing idempotency key, unknown-outcome reconciliation, repeated tool loop, resume-after-interrupt replay, stale state version, handoff over-sharing, retry misclassification, and trace redaction.
89
120
  - Do not fix an agent failure by hardcoding only the reported prompt, tool call, path, or example while leaving the same defect class in sibling tools, routes, gates, or resume paths.
90
121
  - Do not let a single model plan, execute, and certify success for high-impact side effects without an independent verifier, deterministic validator, or human approval state.
91
122
  - Do not store raw prompts, secrets, personal data, customer records, tool payloads, or full conversations in traces, memory, checkpoints, or eval fixtures unless the repository has a declared sensitive-data policy for that surface.
@@ -95,42 +126,75 @@ Review agent systems as controlled workbenches, not smarter chat prompts. An age
95
126
 
96
127
  1. Decide workflow versus agent first. Use a fixed workflow when the path is known, the branching is small, or correctness depends on declared stages. Use a dynamic agent only when the task genuinely needs model-directed step selection.
97
128
  2. Define the autonomy envelope. Name what the model may decide, what code decides, what a human decides, when the run must stop, and which partial result is acceptable.
129
+ - Represent allowed actions as a state-dependent policy graph rather than a static full tool list
130
+ or one fixed call sequence. Code should derive the currently admissible tools and transitions
131
+ from preconditions, least privilege, remaining budgets, current approval, and workflow state.
132
+ - Treat missing state, policy, consent, risk, or capability inputs as deny, narrow, request input,
133
+ or human review. Do not let the model fill an authority gap by inference.
98
134
  3. Split planner, executor, and verifier where risk justifies it. If one model performs multiple roles, add deterministic gates or independent checks before external effects or final success claims.
99
- 4. Gate intermediate artifacts. Validate extracted requirements, plans, routes, tool selections, tool arguments, execution results, and final answers before later stages consume them.
100
- 5. Make tools model-readable and execution-safe. Tool names should describe the action, descriptions should state when to use and not use them, schemas should be explicit, and unsafe side effects should be visible in the contract.
135
+ 4. Define the meaningful action before judging risk. Group discovery, lookup, drafting, validation, and the final commit into the user-visible transaction they serve. Do not require approval for every internal tool call, and do not split one external commitment into harmless-looking calls to evade approval.
136
+ 5. Apply forced-approval conditions before any weighted score. Escalate actions that can create irreversible external effects, move money or contractual responsibility, change account authority or security settings, move secrets or personal data across a trust boundary, mutate production data at scale, or create material legal or regulatory responsibility. Unknown targets, ambiguous ownership, or unexpectedly broad scope must escalate rather than average down.
137
+ 6. Use a weighted risk score only after forced conditions are resolved. Keep factors such as reversibility, blast radius, sensitivity, privilege, financial impact, intent uncertainty, and post-hoc detectability separate so one low factor cannot hide a catastrophic one. Repository policy owns weights and thresholds; this skill does not invent universal numeric cutoffs.
138
+ 7. Classify reversibility honestly for every external effect:
139
+ - exact rollback restores the prior authoritative state without a new external consequence;
140
+ - compensation creates a new action that offsets or explains the first action but does not erase it;
141
+ - irreversible means the original disclosure, communication, transfer, legal event, or destruction cannot be undone.
142
+ Never label compensation as rollback or advertise an undo control that cannot restore the prior state.
143
+ 8. Choose the least interruptive control that matches the classified risk and repository policy. Read-only or draft-only work can usually proceed without a per-action prompt after access control succeeds. Exactly reversible personal-scope changes may execute with a bounded undo path. Repeated bounded actions may use a revocable scoped policy. High-risk actions need per-action approval immediately before commitment. Extreme-scope actions remain human-executed or require stronger authentication, dual control, a change ticket, or a restricted execution window.
144
+ 9. Place approval at the commit boundary. Complete safe reads, calculations, drafting, target resolution, permission checks, and payload validation first; then present the exact proposed effect immediately before the first external commitment. An approval gathered before targets, amounts, scope, or state are known is not execution authority.
145
+ 10. Bind policy and per-action approvals to observable limits. Include normalized plan or action hash, actor, tenant, resource and target set, audience, before-state or version, allowed effect, count and amount ceilings, data scope, allowed tools, forbidden data, expiry, call limit, and revocation state as applicable. Invalidate approval when a bound value changes, scope grows, state becomes stale, or the plan deviates.
146
+ 11. Make the approval view decision-complete. Show recipients or target set, target count, before and after values, public or sharing scope, maximum amount, data leaving the boundary, reversibility class, delay or undo window, and the next irreversible step. A generic Allow button with hidden effect details is not informed approval.
147
+ 12. Select delayed commit and post-execution undo deliberately. Use a cancellation window when the cost of a mistaken external commitment is high and the operation can remain pending. Execute immediately with exact rollback when delay adds little safety and restoration is reliable. Do not add delay to every low-risk operation or present a compensation workflow as cancellation.
148
+ 13. Gate intermediate artifacts. Validate extracted requirements, plans, routes, tool selections, tool arguments, execution results, and final answers before later stages consume them.
149
+ Bound the number of unverified model decisions between gates from current cohort evidence,
150
+ consequence, and uncertainty. Shorten the interval when verified success falls, harmful outcomes
151
+ rise, dependence between steps grows, the task leaves its supported distribution, or the next
152
+ action crosses a more privileged or irreversible boundary. Do not copy a universal decision
153
+ count or probability threshold.
154
+ 14. Make tools model-readable and execution-safe. Tool names should describe the action, descriptions should state when to use and not use them, schemas should be explicit, and unsafe side effects should be visible in the contract.
155
+ - Expose only the tool subset admitted for the current state. Recompute after every observation,
156
+ state transition, approval change, budget consumption, policy change, and failed attempt.
157
+ - Prefer the sufficient lower-privilege tool. A transient failure, missing result, or model request
158
+ must not automatically reveal or authorize a higher-privilege alternative.
101
159
  - Issue scoped capabilities for tool execution: bind tenant, resource, permitted effects, expiry, call count, and token or cost ceiling.
102
160
  - Permit attenuation only. A handoff or child agent may narrow scope, duration, calls, effects, or cost, but must not widen them.
103
161
  - Keep trusted tenant, resource, actor, and policy inputs as server-known arguments rather than model-supplied text.
104
162
  - Bind the policy decision to the exact capability and normalized tool call; reject reuse after arguments, target, effect, expiry, or budget changes.
105
163
  - Canonicalize defaulted and omitted tool arguments before binding or hashing the call. Persist the parent capability ID, issuer, policy version, and revocation state, then consume call and cost limits atomically at the trusted tool boundary.
106
- 6. Remove relative-path and implicit-default traps. Prefer absolute project paths, explicit working directories, explicit tenant or workspace IDs supplied by the server, and no hidden dependence on current conversation state.
107
- 7. Use structured outputs plus business validation. Schema validity is only the first gate; validate permissions, ownership, state transitions, amounts, dates, IDs, and product rules outside free-form generation.
108
- 8. Separate draft from execute. For deletes, writes, emails, payments, external API mutations, job starts, queue publishes, or other irreversible actions, have the agent propose a plan or draft before execution unless the path is explicitly low-risk and idempotent.
109
- 9. Require idempotency for side effects. Derive idempotency keys from stable run, operation, target, and attempt identities; record completed side effects before any path can resume and replay them.
110
- 10. Treat approval as durable state. Store who approved what, which exact proposed action was approved, when it expires, what changed after approval, and what resume step owns execution.
111
- 11. Design interrupt and resume before side effects. Code before an interrupt may run again after resume in many graph runtimes, so keep pre-interrupt work pure or idempotent and keep irreversible effects after approval checkpoints.
112
- 12. Version long-running state. Add a state schema version, migration or compatibility rule, stale-run behavior, and final-state classification for interrupted, cancelled, failed, succeeded, and partially applied runs.
164
+ 15. Remove relative-path and implicit-default traps. Prefer absolute project paths, explicit working directories, explicit tenant or workspace IDs supplied by the server, and no hidden dependence on current conversation state.
165
+ 16. Use structured outputs plus business validation. Schema validity is only the first gate; validate permissions, ownership, state transitions, amounts, dates, IDs, and product rules outside free-form generation.
166
+ 17. Separate draft from execute. For deletes, writes, emails, payments, external API mutations, job starts, queue publishes, or other external commitments, have the agent propose and validate a plan or draft before execution unless the path is explicitly classified as low-risk and idempotent.
167
+ 18. Require idempotency and outcome reconciliation for side effects. Derive idempotency keys from stable run, operation, actor, target, and payload identities; persist effect intent and receipts before any path can resume. After timeout or unknown outcome, query or reconcile the real effect before retrying so duplicate email, payment, publication, or mutation cannot be created by uncertainty.
168
+ 19. Treat approval as durable state. Store who approved what, which exact proposed action was approved, which policy or stronger-auth evidence applies, when it expires, whether it was revoked, what changed after approval, and what resume step owns execution.
169
+ 20. Design interrupt and resume before side effects. Code before an interrupt may run again after resume in many graph runtimes, so keep pre-interrupt work pure or idempotent and keep irreversible effects after approval checkpoints.
170
+ 21. Verify postconditions independently after commitment. Compare actual recipients, target state, object count, permissions, amount, and other effect receipts with the approved plan. A model's completion statement or a successful transport response is not proof of the intended state.
171
+ 22. Recover according to reversibility class. Automatically perform exact rollback only when the rollback is itself authorized, idempotent, and safer than stopping. Record compensation as a new effect. If rollback or compensation is risky, ambiguous, or can widen harm, stop further mutation and hand the evidence to the named human owner.
172
+ 23. Version long-running state. Add a state schema version, migration or compatibility rule, stale-run behavior, and final-state classification for interrupted, cancelled, failed, succeeded, compensated, rolled back, and partially applied runs.
113
173
  Route deterministic multi-step resume, callback, and compensation semantics to `durable-workflow-orchestration`; route generic run, attempt, checkpoint, and effect truth to `execution-ledger-integrity-review`.
114
- 13. Split memory. Keep profile, thread state, and evidence cache separate; store provenance and freshness, avoid making summaries authoritative, and keep access-control and TTL rules visible.
115
- 14. Summarize tool results into state. Preserve source IDs, key fields, uncertainty, errors, and provenance instead of passing raw blobs or letting a long observation become hidden authority.
116
- 15. Evaluate tool triggering both ways. Test overtriggering that calls tools unnecessarily and undertriggering that skips required evidence, permission, or freshness checks.
117
- 16. Route before handoff. Choose specialized workers or downstream agents up front from explicit inputs, pass only the state they need, and redact irrelevant conversation or tool history.
118
- 17. Place guardrails at every boundary. Use entry guardrails for obviously disallowed requests, stage guardrails for bad plans or arguments, tool guardrails for unsafe calls, and final guardrails for response policy.
119
- 18. Choose sequential or parallel guardrails by risk. Sequential pre-blocking protects expensive or dangerous paths before cost or side effects happen; parallel guardrails can reduce latency only when wasted work is acceptable.
120
- 19. Add loop budgets. Enforce max steps, tool calls, repeated tool calls, retries, token spend, cost, and wall-clock time; make repeated same-tool or same-error loops visible.
121
- 20. Retry by failure class. Distinguish invalid schema, missing information, permission denied, timeout, transient provider error, deterministic validation failure, unsafe request, and unknown outcome; do not blindly "try again."
122
- 21. Trace the work, not only the answer. Capture decisions, plans, tool choices, tool arguments, validator results, approvals, retries, final state, and outcome fields needed to debug agent behavior.
123
- 22. Make traces privacy-safe first. Redact or hash sensitive content before telemetry, restrict retention, and ensure eval fixtures do not become a secret dump.
124
- 23. Evaluate from real failures. Build fixtures from incidents, near misses, wrong tool calls, bad handoffs, missing approvals, replay bugs, and loop failures; measure final state and consistency, not only final text.
125
- 24. Treat prompt, model, tool, schema, routing, guardrail, and state changes as behavior migrations. Check old runs, old data, old approvals, cached plans, and in-flight tasks before changing the contract.
126
- 25. Verify with the narrowest configured tests, fixtures, docs validation, release checks, and mustflow validation that cover the changed agent control plane.
174
+ Route agent-specific global versus rolling planning, event-triggered replanning, prompt-view reconstruction, and effect identity across plan versions to `agent-planning-recovery-review`.
175
+ 24. Split transient context from persistent memory. Keep profile, thread state, and evidence cache separate; store provenance and freshness, avoid making summaries authoritative, and route persistent extraction, retrieval, supersession, expiry, or deletion work through `agent-memory-context-governance-review`.
176
+ 25. Pin behavior for attributable execution. Record the immutable agent release bundle assigned to the run and do not change model, rendered prompt, tool contract, policy, retrieval, memory, or evaluator versions in flight. Route candidate, shadow, canary, promotion, and rollback design to `agent-release-bundle-rollout-review`.
177
+ 26. Summarize tool results into state. Preserve source IDs, key fields, uncertainty, errors, and provenance instead of passing raw blobs or letting a long observation become hidden authority.
178
+ 27. Evaluate tool triggering both ways. Test overtriggering that calls tools unnecessarily and undertriggering that skips required evidence, permission, or freshness checks.
179
+ 28. Route before handoff. Choose specialized workers or downstream agents up front from explicit inputs, pass only the state they need, and redact irrelevant conversation or tool history.
180
+ 29. Place guardrails at every boundary. Use entry guardrails for obviously disallowed requests, stage guardrails for bad plans or arguments, tool guardrails for unsafe calls, and final guardrails for response policy.
181
+ 30. Choose sequential or parallel guardrails by risk. Sequential pre-blocking protects expensive or dangerous paths before cost or side effects happen; parallel guardrails can reduce latency only when wasted work is acceptable.
182
+ 31. Add loop budgets. Enforce max steps, tool calls, repeated tool calls, retries, token spend, cost, and wall-clock time; make repeated same-tool or same-error loops visible.
183
+ 32. Retry by failure class. Distinguish invalid schema, missing information, permission denied, timeout, transient provider error, deterministic validation failure, unsafe request, and unknown outcome; do not blindly "try again."
184
+ 33. Trace the work, not only the answer. Capture decisions, plans, tool choices, tool arguments, validator results, approvals, plan changes, retries, final state, reversibility class, recovery action, and outcome fields needed to debug agent behavior. Keep outcome, process, safety, efficiency, and approval-experience measures separate so an aggregate success score cannot hide one high-impact incident.
185
+ 34. Make traces privacy-safe first. Redact or hash sensitive content before telemetry, restrict retention, and ensure eval fixtures do not become a secret dump.
186
+ 35. Evaluate from real failures. Build fixtures from incidents, near misses, wrong tool calls, bad handoffs, missing or stale approvals, replay bugs, duplicate effects, failed rollback, compensation, and loop failures; measure verified final state and consistency, not only final text.
187
+ 36. Treat prompt, model, tool, schema, routing, risk policy, approval, guardrail, and state changes as behavior migrations. Check old runs, old data, old approvals, cached plans, and in-flight tasks before changing the contract.
188
+ 37. Verify with the narrowest configured tests, fixtures, docs validation, release checks, and mustflow validation that cover the changed agent control plane.
127
189
 
128
190
  <!-- mustflow-section: postconditions -->
129
191
  ## Postconditions
130
192
 
131
193
  - The agent path has an explicit workflow-versus-agent decision, autonomy envelope, stop conditions, stage gates, role separation, tool contracts, effect boundaries, and verification path.
132
194
  - Agent tool capabilities are scoped, attenuation-only, populated with server-known authority inputs, and bound to the exact policy decision and normalized tool call.
133
- - External side effects are drafted, approved, idempotent, replay-safe, or explicitly reported as residual risk.
195
+ - Available tools and transitions are recomputed from authoritative state and cannot widen because a
196
+ lower-privilege tool failed or the model requested more authority.
197
+ - External side effects are grouped into meaningful actions, forced-approval conditions outrank average scores, reversibility is classified as exact rollback, compensation, or irreversible, and every committed effect is approved or policy-bounded, idempotent, reconciled, independently verified, replay-safe, or explicitly reported as residual risk.
134
198
  - Durable state, memory, handoffs, guardrails, loop budgets, retries, traces, evals, and sensitive-data policy are bounded where they affect correctness.
135
199
  - Final reports distinguish static control-plane hardening from measured agent success, model-quality assumptions, and unverified framework behavior.
136
200
 
@@ -156,6 +220,13 @@ Use the narrowest configured fixture, unit, integration, schema, docs, package,
156
220
 
157
221
  - If a fixed workflow is sufficient, report that an autonomous agent is unnecessary and keep the simpler workflow unless the user explicitly accepts the agent tradeoff.
158
222
  - If tool ownership, side-effect ownership, or approval state is unclear, stop execution-path changes and report the missing authority boundary.
223
+ - If the policy graph cannot derive an allowed transition from authoritative state, fail closed or
224
+ request the missing input rather than showing the model the unrestricted tool catalog.
225
+ - If a lower-privilege tool fails, preserve the original capability ceiling and require an explicit
226
+ policy decision before any privilege escalation.
227
+ - If the action is called reversible but only a compensating action exists, correct the contract and user-facing claim before relying on post-execution undo.
228
+ - If an approval cannot be bound to concrete targets, scope, state, amount, data, expiry, and effect, keep the path in draft or prepare mode instead of treating a generic approval as authority.
229
+ - If postcondition verification fails, perform an exact rollback only when that rollback is independently safe and authorized; otherwise stop, preserve the effect evidence, and escalate rather than stacking uncertain mutations.
159
230
  - If interrupt or resume behavior can replay side effects and no idempotency key exists, add replay safety before claiming the agent is durable.
160
231
  - If no trace or eval covers the behavior, report static risk reduction rather than claiming the agent is reliable.
161
232
  - If a guardrail needs sensitive raw content to work, prefer scoped local checks, redacted features, or human review over unsafe telemetry.
@@ -167,7 +238,7 @@ Use the narrowest configured fixture, unit, integration, schema, docs, package,
167
238
  - Agent execution-control surface reviewed
168
239
  - Workflow-versus-agent decision, autonomy envelope, and stop conditions
169
240
  - Stage gates, planner/executor/verifier split, and validation path
170
- - Tool contracts, argument ownership, draft/execute split, idempotency, approval, and side-effect replay safety
241
+ - Tool contracts, semantic action grouping, argument ownership, forced-approval conditions, risk tier, reversibility class, scoped policy, just-in-time approval, approval binding, draft/execute split, idempotency, postcondition verification, recovery, and side-effect replay safety
171
242
  - Durable state, resume, memory, handoff, guardrail, loop budget, retry, trace, eval, and privacy policy checked
172
243
  - Files changed
173
244
  - Command intents run