mustflow 2.38.0 → 2.58.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -0
- package/dist/cli/commands/run/executor.js +16 -0
- package/dist/cli/commands/run/process-tree.js +6 -3
- package/dist/cli/commands/verify.js +3 -1
- package/dist/cli/lib/git-changes.js +11 -1
- package/dist/cli/lib/i18n.js +1 -1
- package/dist/cli/lib/local-index/index.js +8 -4
- package/dist/cli/lib/local-index/populate.js +17 -3
- package/dist/cli/lib/local-index/search-read-model.js +9 -7
- package/dist/cli/lib/local-index/search-text.js +2 -2
- package/dist/cli/lib/local-index/workflow-documents.js +17 -2
- package/dist/cli/lib/mustflow-read.js +14 -2
- package/dist/cli/lib/repo-map.js +16 -3
- package/dist/cli/lib/templates.js +8 -7
- package/dist/cli/lib/validation/constants.js +1 -1
- package/dist/core/active-run-locks.js +78 -20
- package/dist/core/change-classification.js +4 -0
- package/dist/core/command-contract-rules.js +1 -1
- package/dist/core/command-contract-validation.js +1 -1
- package/dist/core/command-cwd.js +13 -2
- package/dist/core/command-effects.js +22 -4
- package/dist/core/command-env.js +8 -6
- package/dist/core/command-preconditions.js +28 -2
- package/dist/core/completion-verdict.js +1 -1
- package/dist/core/line-endings.js +8 -4
- package/dist/core/safe-filesystem.js +9 -1
- package/dist/core/source-anchor-validation.js +7 -1
- package/dist/core/source-anchors.js +8 -2
- package/dist/core/verification-scheduler.js +8 -2
- package/package.json +1 -1
- package/templates/default/common/.mustflow/config/commands.toml +39 -15
- package/templates/default/i18n.toml +330 -1
- package/templates/default/locales/en/.mustflow/skills/INDEX.md +302 -5
- package/templates/default/locales/en/.mustflow/skills/agent-eval-integrity-review/SKILL.md +160 -0
- package/templates/default/locales/en/.mustflow/skills/agent-execution-control-review/SKILL.md +163 -0
- package/templates/default/locales/en/.mustflow/skills/ai-generated-code-hardening/SKILL.md +49 -13
- package/templates/default/locales/en/.mustflow/skills/api-access-control-review/SKILL.md +298 -0
- package/templates/default/locales/en/.mustflow/skills/api-misuse-resistance-review/SKILL.md +297 -0
- package/templates/default/locales/en/.mustflow/skills/api-request-performance-review/SKILL.md +189 -0
- package/templates/default/locales/en/.mustflow/skills/app-startup-performance-review/SKILL.md +309 -0
- package/templates/default/locales/en/.mustflow/skills/backend-log-evidence-review/SKILL.md +213 -0
- package/templates/default/locales/en/.mustflow/skills/business-rule-leakage-review/SKILL.md +295 -0
- package/templates/default/locales/en/.mustflow/skills/cache-integrity-review/SKILL.md +291 -0
- package/templates/default/locales/en/.mustflow/skills/change-blast-radius-review/SKILL.md +297 -0
- package/templates/default/locales/en/.mustflow/skills/client-bundle-pruning-review/SKILL.md +160 -0
- package/templates/default/locales/en/.mustflow/skills/cloud-cost-guardrail-review/SKILL.md +321 -0
- package/templates/default/locales/en/.mustflow/skills/concurrency-invariant-review/SKILL.md +193 -0
- package/templates/default/locales/en/.mustflow/skills/core-web-vitals-field-review/SKILL.md +161 -0
- package/templates/default/locales/en/.mustflow/skills/credit-ledger-integrity-review/SKILL.md +156 -0
- package/templates/default/locales/en/.mustflow/skills/database-json-modeling-review/SKILL.md +171 -0
- package/templates/default/locales/en/.mustflow/skills/database-lock-contention-review/SKILL.md +192 -0
- package/templates/default/locales/en/.mustflow/skills/database-migration-change/SKILL.md +76 -34
- package/templates/default/locales/en/.mustflow/skills/database-query-bottleneck-review/SKILL.md +194 -0
- package/templates/default/locales/en/.mustflow/skills/deletion-lifecycle-review/SKILL.md +171 -0
- package/templates/default/locales/en/.mustflow/skills/deployment-rollout-safety-review/SKILL.md +321 -0
- package/templates/default/locales/en/.mustflow/skills/desktop-auto-update-safety-review/SKILL.md +265 -0
- package/templates/default/locales/en/.mustflow/skills/desktop-background-process-stability-review/SKILL.md +318 -0
- package/templates/default/locales/en/.mustflow/skills/desktop-memory-footprint-review/SKILL.md +318 -0
- package/templates/default/locales/en/.mustflow/skills/error-message-integrity-review/SKILL.md +283 -0
- package/templates/default/locales/en/.mustflow/skills/failure-integrity-review/SKILL.md +193 -0
- package/templates/default/locales/en/.mustflow/skills/file-upload-security-review/SKILL.md +305 -0
- package/templates/default/locales/en/.mustflow/skills/frame-render-performance-review/SKILL.md +159 -0
- package/templates/default/locales/en/.mustflow/skills/frontend-accessibility-tree-review/SKILL.md +202 -0
- package/templates/default/locales/en/.mustflow/skills/frontend-localization-review/SKILL.md +202 -0
- package/templates/default/locales/en/.mustflow/skills/frontend-state-ownership-review/SKILL.md +183 -0
- package/templates/default/locales/en/.mustflow/skills/frontend-stress-layout-review/SKILL.md +193 -0
- package/templates/default/locales/en/.mustflow/skills/hot-path-performance-review/SKILL.md +159 -0
- package/templates/default/locales/en/.mustflow/skills/idempotency-integrity-review/SKILL.md +195 -0
- package/templates/default/locales/en/.mustflow/skills/image-delivery-performance-review/SKILL.md +161 -0
- package/templates/default/locales/en/.mustflow/skills/incident-triage-review/SKILL.md +185 -0
- package/templates/default/locales/en/.mustflow/skills/llm-hallucination-control-review/SKILL.md +155 -0
- package/templates/default/locales/en/.mustflow/skills/llm-response-latency-review/SKILL.md +155 -0
- package/templates/default/locales/en/.mustflow/skills/llm-service-ux-review/SKILL.md +2 -0
- package/templates/default/locales/en/.mustflow/skills/llm-token-cost-control-review/SKILL.md +155 -0
- package/templates/default/locales/en/.mustflow/skills/low-end-device-support-review/SKILL.md +340 -0
- package/templates/default/locales/en/.mustflow/skills/memory-lifetime-review/SKILL.md +169 -0
- package/templates/default/locales/en/.mustflow/skills/mobile-energy-efficiency-review/SKILL.md +329 -0
- package/templates/default/locales/en/.mustflow/skills/module-boundary-review/SKILL.md +278 -0
- package/templates/default/locales/en/.mustflow/skills/multi-agent-work-coordination/SKILL.md +1 -0
- package/templates/default/locales/en/.mustflow/skills/observability-debuggability-review/SKILL.md +208 -0
- package/templates/default/locales/en/.mustflow/skills/payment-integrity-review/SKILL.md +155 -0
- package/templates/default/locales/en/.mustflow/skills/prompt-contract-quality-review/SKILL.md +158 -0
- package/templates/default/locales/en/.mustflow/skills/quadratic-scan-review/SKILL.md +155 -0
- package/templates/default/locales/en/.mustflow/skills/queue-processing-integrity-review/SKILL.md +193 -0
- package/templates/default/locales/en/.mustflow/skills/race-condition-review/SKILL.md +188 -0
- package/templates/default/locales/en/.mustflow/skills/rate-limit-integrity-review/SKILL.md +344 -0
- package/templates/default/locales/en/.mustflow/skills/retry-policy-integrity-review/SKILL.md +195 -0
- package/templates/default/locales/en/.mustflow/skills/routes.toml +330 -0
- package/templates/default/locales/en/.mustflow/skills/security-flow-review/SKILL.md +279 -0
- package/templates/default/locales/en/.mustflow/skills/testability-boundary-review/SKILL.md +295 -0
- package/templates/default/locales/en/.mustflow/skills/transaction-boundary-integrity-review/SKILL.md +196 -0
- package/templates/default/locales/en/.mustflow/skills/type-state-modeling-review/SKILL.md +179 -0
- package/templates/default/locales/en/.mustflow/skills/web-render-performance-review/SKILL.md +164 -0
- package/templates/default/manifest.toml +388 -2
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
---
|
|
2
|
+
mustflow_doc: skill.llm-response-latency-review
|
|
3
|
+
locale: en
|
|
4
|
+
canonical: true
|
|
5
|
+
revision: 1
|
|
6
|
+
lifecycle: mustflow-owned
|
|
7
|
+
authority: procedure
|
|
8
|
+
name: llm-response-latency-review
|
|
9
|
+
description: Apply this skill when LLM response latency, time to first token, first useful output, streaming, output length, LLM round trips, tool-call wait, prompt-cache latency, model routing, speculative or parallel execution, realtime continuation, priority tiers, predicted outputs, or user-perceived AI speed are created, changed, reviewed, or reported and the risk is slow response rather than only token spend.
|
|
10
|
+
metadata:
|
|
11
|
+
mustflow_schema: "1"
|
|
12
|
+
mustflow_kind: procedure
|
|
13
|
+
pack_id: mustflow.core
|
|
14
|
+
skill_id: mustflow.core.llm-response-latency-review
|
|
15
|
+
command_intents:
|
|
16
|
+
- changes_status
|
|
17
|
+
- changes_diff_summary
|
|
18
|
+
- lint
|
|
19
|
+
- build
|
|
20
|
+
- test_related
|
|
21
|
+
- test
|
|
22
|
+
- docs_validate_fast
|
|
23
|
+
- test_release
|
|
24
|
+
- mustflow_check
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
# LLM Response Latency Review
|
|
28
|
+
|
|
29
|
+
<!-- mustflow-section: purpose -->
|
|
30
|
+
## Purpose
|
|
31
|
+
|
|
32
|
+
Review LLM speed as a user-visible request path, not as a model-name swap. A latency-controlled LLM path should expose time to first token, first useful output, total completion time, LLM round trips, tool wait, queue wait, output length, cache behavior, routing decisions, and fallback behavior without leaking prompt or user data.
|
|
33
|
+
|
|
34
|
+
<!-- mustflow-section: use-when -->
|
|
35
|
+
## Use When
|
|
36
|
+
|
|
37
|
+
- A change adds, edits, reviews, or reports an LLM request path, streaming response, chat or copilot backend, model gateway, tool-calling orchestration, RAG answer path, prompt cache strategy, context cache strategy, small-model router, fallback cascade, predicted-output path, priority service tier, realtime or WebSocket continuation, or response-latency metric.
|
|
38
|
+
- A task asks to improve AI response speed, time to first token, first useful token, p95 or p99 LLM latency, streaming UX, tool wait, model routing speed, request parallelism, prompt-cache hit rate for latency, or slow user-perceived AI output.
|
|
39
|
+
- The product performs multiple model calls, retrieval calls, tool calls, validators, safety checks, or schema-constrained generations before the user sees useful output.
|
|
40
|
+
- The system logs total latency but cannot explain whether the delay came from provider queueing, prefill, output tokens, tool wait, retry, cache miss, request serialization, or downstream rendering.
|
|
41
|
+
|
|
42
|
+
<!-- mustflow-section: do-not-use-when -->
|
|
43
|
+
## Do Not Use When
|
|
44
|
+
|
|
45
|
+
- The main risk is token spend, provider prompt-cache cost, chat-history bloat, RAG context size, model routing cost, reasoning budget, retry replay, or cost observability; use `llm-token-cost-control-review`.
|
|
46
|
+
- The main risk is prompt correctness, authority separation, output schema semantics, tool policy, or eval quality; use `prompt-contract-quality-review`.
|
|
47
|
+
- The main risk is unsupported factual output, fabricated citations, answerability, retrieval thresholds, or hallucination metrics; use `llm-hallucination-control-review`.
|
|
48
|
+
- The main risk is autonomous agent execution control, planner/executor/verifier separation, tool-call gates, approval or interrupt state, durable resume behavior, loop budgets, retry classification, handoffs, guardrails, or trace outcome evaluation; use `agent-execution-control-review`.
|
|
49
|
+
- The main risk is only chat, prompt-composer, citation display, progress state, or history UI with no backend latency mechanics; use `llm-service-ux-review`.
|
|
50
|
+
- The main risk is a normal API handler, database, Redis, external-service, or route-level latency issue without LLM orchestration; use `api-request-performance-review` or the narrower performance skill.
|
|
51
|
+
- The main risk is cloud account, SaaS, quota, budget, or provider-account spend outside the request path; use `cloud-cost-guardrail-review`.
|
|
52
|
+
|
|
53
|
+
<!-- mustflow-section: required-inputs -->
|
|
54
|
+
## Required Inputs
|
|
55
|
+
|
|
56
|
+
- Latency target ledger: feature, endpoint, user path, success definition, p50 or p95 target if declared, first visible output target, first useful output target, total completion target, and cancellation or timeout behavior.
|
|
57
|
+
- Request timeline ledger: request assembly, provider queue or gateway wait when exposed, first byte, first token, first useful token, final token, stream close, render commit, tool start and finish, retrieval start and finish, validation, retry, and fallback steps.
|
|
58
|
+
- Call graph ledger: sequential model calls, independent model calls, retrieval calls, tool calls, safety checks, validators, schema repair calls, retries, and which steps block the first useful output.
|
|
59
|
+
- Output ledger: expected output length, `max_output_tokens` or equivalent cap, first-sentence contract, schema size, key length, enum size, markdown or JSON overhead, patch versus full-output policy, and incomplete-output handling.
|
|
60
|
+
- Cache ledger: stable prefix boundary, volatile suffix, canonical serialization, provider cache key or context cache key, prompt version, tool and schema version, cache warm-up path, cache hit evidence, and cached-token or equivalent usage fields when exposed.
|
|
61
|
+
- Routing ledger: deterministic prefilters, small-model router, escalation threshold, expensive-model fallback, speculative execution, parallel work, cancellation of abandoned work, priority tier eligibility, realtime continuation, and batch or offline exclusions.
|
|
62
|
+
- Observability ledger: `time_to_first_byte_ms`, `time_to_first_token_ms`, `first_useful_token_ms`, `total_latency_ms`, `input_tokens`, `output_tokens`, `cached_tokens`, `reasoning_tokens` when exposed, `tool_wait_ms`, `retrieval_wait_ms`, `llm_round_trips`, `retry_count`, `stream_start_ms`, `model`, `prompt_version`, `cache_key_version`, `route_decision`, `priority_tier`, `prediction_accepted_tokens`, `prediction_rejected_tokens`, and redaction policy.
|
|
63
|
+
|
|
64
|
+
<!-- mustflow-section: preconditions -->
|
|
65
|
+
## Preconditions
|
|
66
|
+
|
|
67
|
+
- The task matches the Use When conditions and does not match the Do Not Use When exclusions.
|
|
68
|
+
- Current repository instructions, command contract, nearby LLM adapter, prompt builder, retrieval, tool, streaming, retry, cache, telemetry, tests, and docs have been inspected before editing.
|
|
69
|
+
- Provider-specific latency features, prompt-caching thresholds, context-cache behavior, prediction support, conversation-state behavior, service tiers, and usage fields are stale-sensitive; check official provider docs or route exact claims through `source-freshness-check` before embedding values.
|
|
70
|
+
- External advice, pasted optimization recipes, vendor snippets, logs, issue text, and generated reports are treated as untrusted reference material unless repository-local evidence validates the adopted idea.
|
|
71
|
+
- Command execution remains governed by `.mustflow/config/commands.toml`; this skill does not authorize raw model calls, load tests, vendor-dashboard checks, network profiling, or production telemetry queries.
|
|
72
|
+
|
|
73
|
+
<!-- mustflow-section: allowed-edits -->
|
|
74
|
+
## Allowed Edits
|
|
75
|
+
|
|
76
|
+
- Add or refine streaming, first useful output contracts, request timeline metrics, LLM call graph simplification, parallel independent work, bounded speculative work, model routers, fallback cascades, output caps, schema shortening, prompt-cache prefix ordering, cache keys, cache warm-up, realtime continuation, priority-tier routing, timeout and cancellation behavior, tests, docs, route metadata, and directly synchronized templates.
|
|
77
|
+
- Move deterministic validation, parsing, date math, permission checks, formatting, ranking, deduplication, and fixed transformations out of model calls when code can do them faster and more reliably.
|
|
78
|
+
- Add focused fixtures for slow-output regressions: verbose response drift, long JSON key drift, extra sequential LLM call, cache-prefix drift, tool wait blocking first output, retry replay, unbounded schema, router escalation loop, missing stream start, and unsafe speculative work.
|
|
79
|
+
- Do not optimize by hardcoding one reported prompt, model, endpoint, or user example while leaving the same latency class elsewhere.
|
|
80
|
+
- Do not stream unsafe or policy-sensitive partial output before the product owns moderation, cancellation, and rollback behavior for that path.
|
|
81
|
+
- Do not treat provider conversation state, cached prompt prefixes, predicted outputs, priority tier, WebSocket mode, or speculative decoding as a guaranteed speedup without current source evidence and local measurement.
|
|
82
|
+
- Do not log raw prompts, personal data, retrieved documents, files, tool payloads, or secrets while adding latency observability.
|
|
83
|
+
|
|
84
|
+
<!-- mustflow-section: procedure -->
|
|
85
|
+
## Procedure
|
|
86
|
+
|
|
87
|
+
1. Name the latency unit. Decide whether the path optimizes time to first byte, time to first token, first useful output, total completion, p95 workflow latency, realtime turn latency, or background throughput.
|
|
88
|
+
2. Draw the request timeline before changing the model. Mark request assembly, provider call, stream start, first useful output, retrieval, tools, validators, retries, fallback, rendering, and cancellation boundaries.
|
|
89
|
+
3. Count LLM round trips. Collapse sequential calls when classification, extraction, generation, and JSON conversion can safely be one structured request. Keep separate calls only when dependency, safety, ownership, or evaluation evidence requires it.
|
|
90
|
+
4. Parallelize independent work. Start retrieval, cheap classifiers, safety checks, deterministic parsing, or candidate tool calls together only when inputs are independent and failure or cancellation behavior is owned.
|
|
91
|
+
5. Use speculative work with an exit plan. Prefetch likely tools or retrieval only when wrong guesses can be discarded safely, downstream capacity is protected, and abandoned work is cancelled or bounded.
|
|
92
|
+
6. Make the first output useful. Prompts and response contracts should put the answer, decision, or next action first. Streaming "Sure, here is..." is not first useful output.
|
|
93
|
+
7. Cap output by measured p95 shape. Set output limits from normal successful responses plus a small margin. Long explanations, repeated markdown, verbose JSON keys, huge enums, and nested schemas are latency surfaces.
|
|
94
|
+
8. Shorten machine-only schemas. Use concise field names, compact enums, shallow structures, and downstream defaults where correctness allows. Keep schema strict enough for validation, but avoid narrative schema prose in every request.
|
|
95
|
+
9. Separate stable prefix and volatile suffix. Put repeated instructions, examples, tools, schemas, and policies before user input, timestamps, request IDs, retrieved snippets, personalization, and runtime traces.
|
|
96
|
+
10. Review prompt-cache and context-cache behavior as latency evidence. Track cache-key version, prompt version, prefix hash, cache eligibility, cached-token or equivalent fields when exposed, and cache hit rate by feature. Warm caches only for long repeated prefixes on paths where warm-up latency, cost, and privacy are acceptable.
|
|
97
|
+
11. Trim history to state. Keep recent turns, active constraints, unresolved tool results, and durable user preferences; compress older turns into state slots instead of replaying the full transcript.
|
|
98
|
+
12. Trim RAG before the prompt. Deduplicate chunks, remove navigation or boilerplate, pass original evidence spans instead of broad documents, and keep repeated retrieval ordering stable when cache reuse matters.
|
|
99
|
+
13. Route by latency and correctness. Use deterministic code or small models for cheap decisions, escalate only uncertain or high-value cases, and tune thresholds using false-fast, retry, escalation, and user-repair evidence.
|
|
100
|
+
14. Treat fallback cascades as part of latency design. A failed fast model followed by a slow model may still be worse than one correct model. Measure cost per successful low-latency response, not only per-call speed.
|
|
101
|
+
15. Keep tools from blocking first useful output unnecessarily. Use server-known parameters, parallel safe tool calls, timeouts, partial states, and honest `tool_failed` or `needs_more_info` outputs when data is late or unavailable.
|
|
102
|
+
16. Use predicted outputs only when most of the final text is already known. Track accepted and rejected prediction tokens or equivalent fields when exposed, and keep provider support and billing details behind `source-freshness-check`.
|
|
103
|
+
17. Use realtime, WebSocket, or priority service paths only for latency-sensitive routes. Keep batch jobs, evals, backfills, and ETL out of high-priority lanes unless a product owner accepts the tradeoff.
|
|
104
|
+
18. For self-hosted model serving, treat prefix cache, KV cache reuse, batching, speculative decoding, and scheduler priority as measured serving choices. Do not assume they help if output decoding, high QPS, or memory pressure is the real bottleneck.
|
|
105
|
+
19. Add privacy-safe latency observability. Prefer durations, counts, hashes, prompt versions, model names, route decisions, cache-key versions, token counts, and redacted reason codes over raw content.
|
|
106
|
+
20. Verify with the narrowest configured tests, fixtures, docs validation, release checks, and mustflow validation that cover request assembly, streaming, cache keys, routing, fallback, telemetry, and installed skill surfaces.
|
|
107
|
+
|
|
108
|
+
<!-- mustflow-section: postconditions -->
|
|
109
|
+
## Postconditions
|
|
110
|
+
|
|
111
|
+
- The LLM path has an explicit latency unit and request timeline, including first token, first useful output, tool wait, retries, fallback, stream close, and total completion where relevant.
|
|
112
|
+
- Sequential LLM calls, tool waits, retrieval, schema overhead, output length, prompt-cache behavior, history growth, RAG packing, routing, and priority paths are fixed, bounded, instrumented, or reported.
|
|
113
|
+
- The first visible model output is useful for the product task or the remaining gap is named.
|
|
114
|
+
- Metrics can explain slow responses without exposing raw prompt, retrieved content, personal data, or secrets.
|
|
115
|
+
- Final reports distinguish measured latency evidence from static risk reduction, provider feature assumptions, local-only traces, and UX-only improvements.
|
|
116
|
+
|
|
117
|
+
<!-- mustflow-section: verification -->
|
|
118
|
+
## Verification
|
|
119
|
+
|
|
120
|
+
Use configured oneshot command intents when available:
|
|
121
|
+
|
|
122
|
+
- `changes_status`
|
|
123
|
+
- `changes_diff_summary`
|
|
124
|
+
- `lint`
|
|
125
|
+
- `build`
|
|
126
|
+
- `test_related`
|
|
127
|
+
- `test`
|
|
128
|
+
- `docs_validate_fast`
|
|
129
|
+
- `test_release`
|
|
130
|
+
- `mustflow_check`
|
|
131
|
+
|
|
132
|
+
Use the narrowest configured fixture, unit, integration, telemetry, docs, package, or release check that proves the changed latency-control contract.
|
|
133
|
+
|
|
134
|
+
<!-- mustflow-section: failure-handling -->
|
|
135
|
+
## Failure Handling
|
|
136
|
+
|
|
137
|
+
- If latency measurements are unavailable, report static latency risk reduction rather than claiming a measured speedup.
|
|
138
|
+
- If provider behavior, prompt caching, prediction support, conversation state, WebSocket continuation, priority tiers, or serving-runtime features may have changed recently, route exact claims through `source-freshness-check` before embedding them in code or docs.
|
|
139
|
+
- If streaming creates moderation, privacy, rollback, or partial-output risk, preserve safety and report the latency tradeoff instead of streaming blindly.
|
|
140
|
+
- If a router, cache, or output cap increases retry, fallback, validation failure, or user-repair rate, judge by successful low-latency response rather than by first-call speed.
|
|
141
|
+
- If latency telemetry needs sensitive content to explain delay, prefer hashes, versions, sizes, counts, durations, and redacted reason codes.
|
|
142
|
+
- If optimization conflicts with factuality, safety, privacy, or product correctness, preserve correctness and report the latency tradeoff.
|
|
143
|
+
|
|
144
|
+
<!-- mustflow-section: output-format -->
|
|
145
|
+
## Output Format
|
|
146
|
+
|
|
147
|
+
- LLM response-latency surface reviewed
|
|
148
|
+
- Latency unit, target, request timeline, and measurement source
|
|
149
|
+
- LLM round trips, parallel work, speculative work, tool wait, streaming, and cancellation checked
|
|
150
|
+
- Output length, first useful output, schema overhead, cache behavior, history, RAG, routing, fallback, prediction, realtime, and priority-tier choices checked
|
|
151
|
+
- Latency observability fields and sensitive-data redaction checked
|
|
152
|
+
- Files changed
|
|
153
|
+
- Command intents run
|
|
154
|
+
- Skipped checks and reasons
|
|
155
|
+
- Remaining response-latency risk
|
|
@@ -40,6 +40,8 @@ Keep LLM service interfaces clear, controllable, responsive, readable, and recov
|
|
|
40
40
|
## Do Not Use When
|
|
41
41
|
|
|
42
42
|
- The task changes a non-AI UI surface with no prompt, generation, model, citation, or conversation behavior; use `ui-quality-gate`.
|
|
43
|
+
- The task changes backend response latency mechanics such as time to first token, first useful output, LLM round trips, tool wait, prompt-cache latency, model routing speed, priority tier, realtime continuation, or predicted outputs; use `llm-response-latency-review`.
|
|
44
|
+
- The task changes backend agent execution control such as planner/executor/verifier separation, tool-call gates, approval or interrupt state, durable resume behavior, loop budgets, retry classification, handoffs, guardrails, or trace outcome evaluation; use `agent-execution-control-review`.
|
|
43
45
|
- The task changes only backend model orchestration, prompts, retrieval, or tool calls with no user-facing state; use the narrower backend, security, data, or test skill that matches the changed surface.
|
|
44
46
|
- The task is only general copy editing or documentation; use the relevant documentation skill.
|
|
45
47
|
- Visual or interactive inspection is unavailable; report that gap instead of claiming UX verification.
|
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
---
|
|
2
|
+
mustflow_doc: skill.llm-token-cost-control-review
|
|
3
|
+
locale: en
|
|
4
|
+
canonical: true
|
|
5
|
+
revision: 1
|
|
6
|
+
lifecycle: mustflow-owned
|
|
7
|
+
authority: procedure
|
|
8
|
+
name: llm-token-cost-control-review
|
|
9
|
+
description: Apply this skill when LLM API calls, prompt assembly, chat history, RAG context, tool schemas, structured output schemas, model routing, reasoning settings, token budgets, provider prompt caching, app-level response caching, retries, batch or flex processing, predicted outputs, image or file inputs, or LLM cost metrics are created, changed, reviewed, or reported and the risk is token spend or cost-per-success drifting out of control.
|
|
10
|
+
metadata:
|
|
11
|
+
mustflow_schema: "1"
|
|
12
|
+
mustflow_kind: procedure
|
|
13
|
+
pack_id: mustflow.core
|
|
14
|
+
skill_id: mustflow.core.llm-token-cost-control-review
|
|
15
|
+
command_intents:
|
|
16
|
+
- changes_status
|
|
17
|
+
- changes_diff_summary
|
|
18
|
+
- lint
|
|
19
|
+
- build
|
|
20
|
+
- test_related
|
|
21
|
+
- test
|
|
22
|
+
- docs_validate_fast
|
|
23
|
+
- test_release
|
|
24
|
+
- mustflow_check
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
# LLM Token Cost Control Review
|
|
28
|
+
|
|
29
|
+
<!-- mustflow-section: purpose -->
|
|
30
|
+
## Purpose
|
|
31
|
+
|
|
32
|
+
Review LLM cost as a product and systems contract, not as prompt brevity. A cost-controlled LLM path should make expensive repeated input stable, trim volatile context, route work to the cheapest sufficient path, cap output and reasoning spend, repair failures without full replay, and expose per-feature cost evidence.
|
|
33
|
+
|
|
34
|
+
<!-- mustflow-section: use-when -->
|
|
35
|
+
## Use When
|
|
36
|
+
|
|
37
|
+
- A change adds, edits, reviews, or reports an LLM request builder, prompt prefix, system or developer message, chat history assembly, memory compaction, RAG context, retrieved chunk packing, few-shot examples, tool definitions, structured output schema, image or file input, model selector, reasoning-effort setting, output limit, retry path, streaming or prediction path, batch processing path, flex or low-priority processing path, cache key, token counter, usage logger, cost dashboard, or LLM quota guard.
|
|
38
|
+
- A task asks to reduce token cost, improve provider prompt-cache hit rate, lower cost per successful task, shrink context, split realtime and offline LLM work, route small tasks away from expensive models, or prevent a token bill spike.
|
|
39
|
+
- The product sends repeated instructions, examples, tool schemas, output schemas, documents, policies, conversation history, screenshots, files, or retrieved context to an LLM.
|
|
40
|
+
- The system records total tokens but cannot explain which feature, endpoint, model, prompt version, tool schema version, retry, or validation failure caused the spend.
|
|
41
|
+
|
|
42
|
+
<!-- mustflow-section: do-not-use-when -->
|
|
43
|
+
## Do Not Use When
|
|
44
|
+
|
|
45
|
+
- The main risk is prompt correctness, authority separation, output schema semantics, tool policy, or eval quality; use `prompt-contract-quality-review`.
|
|
46
|
+
- The main risk is unsupported factual output, fabricated citations, source coverage, answerability, abstain behavior, retrieval thresholds, or hallucination metrics; use `llm-hallucination-control-review`.
|
|
47
|
+
- The main risk is time to first token, first useful output, streaming, LLM round trips, tool wait, prompt-cache latency, model routing speed, priority tier, realtime continuation, or user-perceived response speed; use `llm-response-latency-review`.
|
|
48
|
+
- The main risk is autonomous agent execution control, tool-call approval, side-effect replay safety, durable resume behavior, handoffs, guardrails, loop budgets, retry classification, or trace outcome evaluation; use `agent-execution-control-review`.
|
|
49
|
+
- The main risk is cloud account, infrastructure, SaaS, quota, budget, tag, retention, or provider-account spend guardrails outside the model-call payload; use `cloud-cost-guardrail-review`.
|
|
50
|
+
- The main risk is rate-limit fairness, throttling, quota counters, concurrency limits, 429 response behavior, or protected-resource definition; use `rate-limit-integrity-review`.
|
|
51
|
+
- The task only edits user-visible copy and does not affect LLM payload, model choice, retry behavior, or cost telemetry.
|
|
52
|
+
|
|
53
|
+
<!-- mustflow-section: required-inputs -->
|
|
54
|
+
## Required Inputs
|
|
55
|
+
|
|
56
|
+
- Cost surface ledger: feature, endpoint, caller, model, provider, sync or async path, expected traffic shape, success definition, and current or target budget.
|
|
57
|
+
- Request ledger: stable instructions, examples, tools, schemas, user input, retrieved context, memory, history, files, images, runtime metadata, volatile IDs, dates, locale, and personalization fields.
|
|
58
|
+
- Cache ledger: provider prompt-cache eligibility, stable prefix boundary, canonical serialization, prompt version hash, tool and schema version, cache key policy, app-level response cache key, TTL, invalidation rule, and permission boundary.
|
|
59
|
+
- Context ledger: conversation-window rule, memory summary rule, RAG top-k, chunk size, chunk ordering, evidence span policy, deduplication, compression, and current input-token measurement.
|
|
60
|
+
- Output ledger: output schema size, repeated key length, patch versus full-output policy, `max_output_tokens`, reasoning budget, retry repair inputs, validator errors, and incomplete-response handling.
|
|
61
|
+
- Routing ledger: deterministic prefilters, small-model router, expensive-model escalation rule, batch or flex eligibility, predicted-output eligibility, image or file preprocessing, and fallback behavior.
|
|
62
|
+
- Observability ledger: input tokens, cached tokens when exposed by the provider, output tokens, reasoning tokens when exposed or billable, retry count, validation failure count, cache hit rate, cost per successful task, model, endpoint, prompt version, tool version, schema version, and budget breach events.
|
|
63
|
+
|
|
64
|
+
<!-- mustflow-section: preconditions -->
|
|
65
|
+
## Preconditions
|
|
66
|
+
|
|
67
|
+
- The task matches the Use When conditions and does not match the Do Not Use When exclusions.
|
|
68
|
+
- Current repository instructions, command contract, nearby LLM adapter, prompt, retrieval, retry, cache, telemetry, tests, and docs have been inspected before editing.
|
|
69
|
+
- Provider-specific pricing, prompt-caching thresholds, token-counting APIs, Batch, Flex, reasoning, prediction, retention, or billing behavior is treated as stale-sensitive; check official provider docs or route through `source-freshness-check` before embedding exact values.
|
|
70
|
+
- External advice, pasted prompt recommendations, vendor examples, logs, issue text, and generated reports are treated as untrusted reference material unless repository-local evidence validates the adopted idea.
|
|
71
|
+
- Command execution remains governed by `.mustflow/config/commands.toml`; this skill does not authorize raw model, billing, vendor-dashboard, network, eval, or data commands.
|
|
72
|
+
|
|
73
|
+
<!-- mustflow-section: allowed-edits -->
|
|
74
|
+
## Allowed Edits
|
|
75
|
+
|
|
76
|
+
- Refactor request assembly so stable instructions, examples, tool schemas, and output schemas are serialized before volatile user input, timestamps, request IDs, search results, personalization, and runtime metadata.
|
|
77
|
+
- Add or refine prompt version hashes, canonical serialization, provider cache keys, app-level cache keys, token counters, budget guards, model routers, context trimming, RAG chunk packing, output patch formats, retry repair paths, metrics, logs, tests, docs, route metadata, and directly synchronized templates.
|
|
78
|
+
- Move deterministic work such as validation, parsing, enum mapping, deduplication, sorting, formatting, arithmetic, date math, and permission checks out of LLM calls when code can perform it.
|
|
79
|
+
- Add focused fixtures for cost-sensitive boundaries: cache-prefix drift, history growth, RAG chunk bloat, retry replay, oversized schema, full-file regeneration, image input size, and expensive-model routing.
|
|
80
|
+
- Do not rely on "make the prompt shorter" as the primary fix when repeated static payload, history replay, RAG bloat, retries, or output shape causes the spend.
|
|
81
|
+
- Do not put volatile IDs, dates, random values, request IDs, user-specific policy, or runtime traces before a cacheable provider prefix unless the provider contract requires it.
|
|
82
|
+
- Do not treat `previous_response_id`, provider conversation state, low temperature, predicted outputs, or streaming as a guaranteed token-cost reduction without current provider evidence.
|
|
83
|
+
- Do not log sensitive prompt, personal data, retrieved documents, files, images, or secrets while adding cost observability.
|
|
84
|
+
|
|
85
|
+
<!-- mustflow-section: procedure -->
|
|
86
|
+
## Procedure
|
|
87
|
+
|
|
88
|
+
1. Name the cost unit. Decide whether the system optimizes per request, per successful task, per workflow, per user session, per batch job, or per tenant.
|
|
89
|
+
2. Measure before cutting. Prefer provider token-counting APIs, usage fields, or repository-local token accounting over character estimates, especially when tools, schemas, images, files, or reasoning models are involved.
|
|
90
|
+
3. Split stable prefix from volatile suffix. Keep system and developer instructions, policy, examples, tool schemas, structured output schemas, and other repeated static context in a canonical stable order before user input, request IDs, dates, session IDs, retrieved snippets, and personalization.
|
|
91
|
+
4. Hash the expensive prefix. Record or derive a prompt version hash from canonical instructions, examples, tools, schemas, model settings, and output contract. If the hash varies per user without intent, treat it as cache-prefix drift.
|
|
92
|
+
5. Review provider prompt caching. Use provider cache keys only for requests that share the same stable prefix, distribute hot keys according to provider guidance, and log cached-token evidence when the provider exposes it.
|
|
93
|
+
6. Add app-level caching where identical normalized inputs can safely skip the model call. Include tenant, permission, locale, source version, policy version, and TTL in the cache key when they affect correctness.
|
|
94
|
+
7. Trim chat history by state, not by vibes. Keep recent turns, durable memory, active task state, confirmed user preferences, and unresolved tool results; avoid replaying full raw transcripts when a compact state is enough.
|
|
95
|
+
8. Shrink RAG by evidence span. Search broadly if needed, but pass the smallest source-backed spans that answer the question, deduplicate near-identical chunks, preserve source metadata, and use stable ordering when repeated workloads benefit from cache reuse.
|
|
96
|
+
9. Keep tool and schema payloads boring. Tool descriptions and JSON schemas should be long enough for correct routing and validation but not narrative prose. If permissions differ by user, keep schema stable and enforce permission at tool execution.
|
|
97
|
+
10. Route before calling the expensive model. Use deterministic code, regexes, database lookups, small models, or cheap classifiers for tasks that do not need a large reasoning model; escalate only ambiguous, high-value, or failed cases.
|
|
98
|
+
11. Budget reasoning and output together. Set reasoning effort and output limits according to task value; leave enough room for visible output, and handle incomplete responses instead of silently retrying the full expensive request.
|
|
99
|
+
12. Prefer patches over full regeneration when the product already owns most of the output. Use unified diff, JSON Patch, line-range replacement, IDs, labels, scores, or reason codes when downstream code can merge the result.
|
|
100
|
+
13. Repair failures without full replay. For parse failures, enum mismatches, missing fields, or validator errors, retry with previous output, validator error, and schema summary when safe instead of resending the entire original context.
|
|
101
|
+
14. Separate realtime and offline work. Move evals, bulk classification, enrichment, embeddings, report backfills, and log analysis to configured async, batch, or low-priority processing when user latency does not matter.
|
|
102
|
+
15. Treat predicted outputs as a latency tool unless current provider docs and usage evidence show cost behavior for the exact model and endpoint. Use `llm-response-latency-review` when the main goal is faster completion rather than cost control. Watch rejected prediction tokens or equivalent fields when exposed.
|
|
103
|
+
16. Reduce image and file input before the model. Crop screenshots, downsample where acceptable, extract DOM text or OCR first, and count the actual payload tokens when the provider supports it.
|
|
104
|
+
17. Instrument cost per success. Track endpoint, model, prompt version, tool version, schema version, input tokens, cached tokens, output tokens, reasoning tokens, retry count, validation failure rate, cache hit rate, and successful-task denominator.
|
|
105
|
+
18. Verify with the narrowest configured tests, fixtures, docs validation, release checks, and mustflow validation that cover request assembly, cache keys, budget guards, routing, retry repair, telemetry, and installed skill surfaces.
|
|
106
|
+
|
|
107
|
+
<!-- mustflow-section: postconditions -->
|
|
108
|
+
## Postconditions
|
|
109
|
+
|
|
110
|
+
- Stable LLM payload is separated from volatile data, hashed or versioned where useful, and cacheable by design when the provider supports prompt caching.
|
|
111
|
+
- Long context, chat history, RAG evidence, tools, schemas, files, images, reasoning, output, retries, and routing each have an explicit budget or reduction rule where relevant.
|
|
112
|
+
- Deterministic work is handled outside the model unless language judgment is required.
|
|
113
|
+
- Metrics can explain cost per successful task, cache-prefix drift, retry cost, validation failures, and model routing decisions without leaking sensitive prompt or user data.
|
|
114
|
+
- Final reports distinguish proven cost-control evidence from assumed provider behavior, anecdotal token savings, and latency-only optimizations.
|
|
115
|
+
|
|
116
|
+
<!-- mustflow-section: verification -->
|
|
117
|
+
## Verification
|
|
118
|
+
|
|
119
|
+
Use configured oneshot command intents when available:
|
|
120
|
+
|
|
121
|
+
- `changes_status`
|
|
122
|
+
- `changes_diff_summary`
|
|
123
|
+
- `lint`
|
|
124
|
+
- `build`
|
|
125
|
+
- `test_related`
|
|
126
|
+
- `test`
|
|
127
|
+
- `docs_validate_fast`
|
|
128
|
+
- `test_release`
|
|
129
|
+
- `mustflow_check`
|
|
130
|
+
|
|
131
|
+
Use the narrowest configured fixture, unit, integration, telemetry, docs, package, or release check that proves the changed cost-control contract.
|
|
132
|
+
|
|
133
|
+
<!-- mustflow-section: failure-handling -->
|
|
134
|
+
## Failure Handling
|
|
135
|
+
|
|
136
|
+
- If token counts are unavailable, report the measurement gap and make the change a static risk reduction rather than a proven cost reduction.
|
|
137
|
+
- If provider pricing, caching, Batch, Flex, reasoning, token counting, conversation state, or prediction behavior may have changed recently, route exact claims through `source-freshness-check` before embedding them in code or docs.
|
|
138
|
+
- If cache-hit metrics drop after a change, inspect prefix ordering, canonical serialization, volatile fields, prompt version hash, tool or schema version drift, and provider cache-key distribution before changing model choice.
|
|
139
|
+
- If a cheaper model or smaller context raises retry, validation failure, escalation, or user-repair rate, judge by cost per successful task rather than per-call cost.
|
|
140
|
+
- If cost telemetry needs sensitive content to explain spend, prefer hashes, IDs, versions, sizes, counts, and redacted reason codes over raw prompt or document logging.
|
|
141
|
+
- If cost control conflicts with factuality, safety, privacy, or product correctness, preserve correctness and report the cost tradeoff instead of weakening validation.
|
|
142
|
+
|
|
143
|
+
<!-- mustflow-section: output-format -->
|
|
144
|
+
## Output Format
|
|
145
|
+
|
|
146
|
+
- LLM token-cost surface reviewed
|
|
147
|
+
- Cost unit, budget, and measurement source
|
|
148
|
+
- Stable prefix, volatile suffix, prompt hash, and provider cache behavior checked
|
|
149
|
+
- App cache, history, RAG, tool schema, structured schema, image or file input, and deterministic prefilter choices checked
|
|
150
|
+
- Model routing, reasoning effort, output limit, patch-output, retry repair, Batch, Flex, and prediction choices checked
|
|
151
|
+
- Cost observability fields and sensitive-data redaction checked
|
|
152
|
+
- Files changed
|
|
153
|
+
- Command intents run
|
|
154
|
+
- Skipped checks and reasons
|
|
155
|
+
- Remaining token-cost risk
|