skilldrop-cli 0.12.2 → 0.13.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,84 @@
1
+ ---
2
+ name: eval-harness-generator
3
+ description: Takes a skill's SKILL.md and generates a set of eval cases — trigger queries, expected output shape, and pass/fail criteria — ready to drop into the skill's evals/ directory. Use when adding evals to a new or existing skill, verifying a skill behaves as documented, or building a regression gate before modifying a skill.
4
+ ---
5
+
6
+ # eval-harness-generator
7
+
8
+ Reads a SKILL.md and generates eval cases that cover the skill's happy path, edge cases, and known failure modes. The output is JSON that drops directly into `skills/<name>/evals/cases.json`. Unlike `llm-eval-harness` (which designs eval harnesses for arbitrary LLM features), this skill specifically targets the skilldrop eval format and the patterns that skilldrop skills routinely get wrong.
9
+
10
+ ## How to respond
11
+
12
+ ### Step 1 — Read the skill
13
+
14
+ Ask the user to paste the full SKILL.md. Extract:
15
+ - What the skill takes as input
16
+ - What the skill produces as output
17
+ - Any explicit quality bars or anti-patterns the skill names
18
+ - Whether the skill is a **gate** (emits a verdict from `contracts/terminals.json`) or a **generator** (produces an artifact)
19
+
20
+ ### Step 2 — Identify failure modes
21
+
22
+ Before writing cases, enumerate the failure modes the eval set should catch:
23
+
24
+ | Category | Failure mode to test |
25
+ |---|---|
26
+ | **Happy path** | Standard input produces the expected artifact/verdict |
27
+ | **Minimal input** | Sparse or ambiguous input — does the skill ask clarifying questions rather than hallucinate? |
28
+ | **Anti-pattern input** | Input that would tempt the skill to produce a common bad output (named in the SKILL.md's quality bar) |
29
+ | **Edge input** | Boundary conditions specific to this skill (e.g., a bug report with no repro steps for bug-triage) |
30
+ | **Gate refusal** | For gate skills: input that should produce BLOCKED or REVISE, not PROCEED |
31
+
32
+ ### Step 3 — Generate the eval cases
33
+
34
+ Produce a JSON array with 8–12 cases. Each case:
35
+
36
+ ```json
37
+ {
38
+ "id": "happy-path-standard",
39
+ "trigger": "The exact query or input that activates this case",
40
+ "expected_shape": [
41
+ "Output contains a <specific element>",
42
+ "Output names a specific field or section",
43
+ "Output avoids <anti-pattern named in quality bar>"
44
+ ],
45
+ "must_not_include": [
46
+ "Fabricated detail not present in the input",
47
+ "A verdict word from the wrong class (for gate skills)"
48
+ ],
49
+ "verdict_class": "pass",
50
+ "notes": "One sentence on what this case is testing and why it matters"
51
+ }
52
+ ```
53
+
54
+ For gate skills, use `verdict_class` values from `contracts/terminals.json`: `pass`, `conditional`, `revise`, `redirect`, `blocked`. For generators, use `pass` / `fail`.
55
+
56
+ ### Step 4 — Write an eval strategy note
57
+
58
+ After the JSON array, add a brief eval strategy note (3–5 sentences) explaining:
59
+ - Which cases cover which failure modes
60
+ - The most important case and why
61
+ - What a regression on which case would mean
62
+
63
+ ### Step 5 — Remind about placement
64
+
65
+ Tell the user:
66
+ ```
67
+ Drop this into skills/<skill-name>/evals/cases.json
68
+ Run: python3 validate.py # confirms the file is found
69
+ ```
70
+
71
+ ## Anti-patterns
72
+
73
+ - **Triggers that don't test anything.** "Run the skill on this input" is not a meaningful trigger. Each trigger must be specific enough that a failing model would produce a wrong output you'd recognize.
74
+ - **expected_shape items that can't be checked.** "Output is comprehensive" is not checkable. Name specific sections, fields, or behaviors.
75
+ - **Missing anti-pattern cases.** The most common eval gap is no cases that test the quality bar's failure modes. If a skill says "never present a hypothesis as the cause," there must be a case where the input tempts that failure.
76
+
77
+ For designing eval harnesses for general LLM features (not skilldrop skills specifically), use `llm-eval-harness` instead. Use `contribution-wizard` when creating a new skill from scratch — it calls this skill as part of the authoring flow.
78
+
79
+ ## Quality bar
80
+
81
+ - **Triggers must be realistic.** A trigger of "run this skill" is not an eval case — it doesn't test anything specific. Each trigger should be the kind of input a real user would paste.
82
+ - **expected_shape items must be checkable.** "Output is good" is not checkable. "Output contains a numbered repro sequence starting from a clean state" is.
83
+ - **Cover the refusal cases.** The most common gap in skilldrop evals is missing cases for the skill's anti-patterns — the bad outputs the quality bar explicitly forbids. At least 2 of the 8–12 cases must test that the skill refuses or avoids its stated failure mode.
84
+ - **Don't pad to 12.** Eight strong cases beat twelve weak ones. Only add cases that test a distinct failure mode not already covered.
@@ -0,0 +1,11 @@
1
+ {
2
+ "name": "eval-harness-generator",
3
+ "version": "1.0.0",
4
+ "description": "Takes a skill's SKILL.md and generates a set of eval cases — trigger queries, expected output shape, and pass/fail criteria — ready to drop into the skill's evals/ directory. Use when adding evals to a new or existing skill, verifying a skill behaves as documented, or building a regression gate before modifying a skill.",
5
+ "entrypoint": "SKILL.md",
6
+ "deps": { "npm": [], "pip": [] },
7
+ "env": { "required": [], "optional": [] },
8
+ "related": ["llm-eval-harness", "contribution-wizard"],
9
+ "tags": ["claude-api", "evals", "testing", "quality", "skilldrop-meta"],
10
+ "model": { "tier": "standard", "rationale": "Reads a SKILL.md and synthesizes eval cases — requires understanding of the skill's intent but not heavy reasoning." }
11
+ }
@@ -0,0 +1,95 @@
1
+ ---
2
+ name: launch-readiness
3
+ description: Produce a go/no-go launch readiness report for a merged change before it goes live — one row per check (rollback path, alert and runbook per new failure mode, NFR targets measured, comms drafted, data/privacy sign-off), each with an owner, evidence link, and met / not met / n/a-because status, ending in a PROCEED / PROCEED WITH CONDITIONS / REVISE / BLOCKED verdict. Use when the user asks "are we ready to launch", "go/no-go", "production readiness review", "launch checklist", or "can we ship this to users".
4
+ ---
5
+
6
+ # launch-readiness
7
+
8
+ You judge whether a change that is already merged *can* go live, using evidence rather than
9
+ reassurance. The output is a readiness report a release owner can act on in five minutes.
10
+ Whether it *should* go live now (timing, freeze windows, who is on call) is a separate human
11
+ decision. That belongs to `council-review` or the release owner, not to this report.
12
+
13
+ ## How to respond
14
+
15
+ 1. **Pin down what is launching.** You need: the change (a PR, a release tag, or a feature
16
+ name), who it reaches (internal, a percentage, everyone), and the release owner. If the
17
+ change itself is unknown, ask. You may ask at most 2 questions. A missing reach or owner
18
+ becomes an `[assumption]` to confirm. Everything else becomes a row that is `not met`
19
+ because evidence is missing.
20
+
21
+ 2. **List the new failure modes first.** Before any checklist, write 3–5 ways this specific
22
+ change could hurt users or data once it is live. Read them from the diff, the design doc or
23
+ the migration plan. Each later check is judged against this list. A generic checklist with
24
+ no failure modes is a ritual.
25
+
26
+ 3. **Fill [`templates/readiness-report.md`](templates/readiness-report.md).** It has one row
27
+ per check, and every row carries:
28
+ - **Status:** `met`, `not met`, or `n/a — <reason>`. `n/a` without a reason counts as `not met`.
29
+ - **Evidence:** a link, a command output, or a file path. A row with no evidence counts as
30
+ `not met`, however confident the user sounds. Evidence must cover the path the failure
31
+ mode touches; if it is unclear what it covered (which endpoint a load test hit), tag it
32
+ `[assumption]`.
33
+ - **Owner:** a role, not a person's name.
34
+
35
+ 4. **Apply the checks in this order.** The order matters: the first ones are the costliest to
36
+ discover after launch.
37
+ 1. **Rollback.** It must be a tested path back, with its data story stated. If a data or
38
+ schema change is involved and no plan exists, hand off to `migration-plan`.
39
+ 2. **Detection.** Each new failure mode needs an alert that fires before users complain. If
40
+ one is missing, hand off to `observability-plan`.
41
+ 3. **Response.** Each such alert needs a runbook entry. If one is missing, hand off to
42
+ `runbook-generator`.
43
+ 4. **Quality targets.** Every target in `nfr-spec` (latency, availability, scale) must be
44
+ *measured*, for example by a load test or canary result. A target that is only asserted
45
+ does not count.
46
+ 5. **Data and privacy.** New personal data, retention changes or new processors need a
47
+ named sign-off, or `n/a — <reason>`.
48
+ 6. **Comms.** User-facing notes are drafted (see `release-notes`), support is briefed, and
49
+ the status page owner is known if the launch is risky.
50
+ 7. **Exposure.** The first exposure is staged (flag, percentage or cohort), or there is a
51
+ stated reason why all-at-once is safe.
52
+
53
+ 5. **Give the verdict.** It is the last line, with exactly one reason:
54
+ - `PROCEED`: every row is `met` or `n/a — <reason>`.
55
+ - `PROCEED WITH CONDITIONS`: only rows 6–7 are open, and each open row has an owner and a
56
+ date. The open rows travel with the release.
57
+ - `REVISE`: any row 2–5 is `not met`. Name the row and the hand-off that fixes it.
58
+ - `BLOCKED`: rollback is `not met`, or the change or its reach is unknown. A missing
59
+ rollback blocks on its own, whatever else passes.
60
+
61
+ **Non-interactive runs** (CI, subagent, headless): never invent evidence such as test results,
62
+ dashboards or sign-offs. Tag a guessed fact `[assumption]` and list it first. If the change
63
+ being launched cannot be identified, emit `BLOCKED: need <the PR, tag, or feature name>`.
64
+
65
+ ## Useful references in this skill
66
+
67
+ - [`templates/readiness-report.md`](templates/readiness-report.md) — the report skeleton, one row per check.
68
+ - [`examples/schema-change-revise.md`](examples/schema-change-revise.md) — a worked report for a schema change that comes out `REVISE`.
69
+
70
+ ## Quality bar
71
+
72
+ - **Every row has evidence or is `not met`.** Readiness asserted is not readiness.
73
+ - **The failure modes are specific to this change.** "The service could go down" is true of everything and checks nothing.
74
+ - **Rollback is judged first and can block alone.** Every other gap can be fixed while live; an irreversible launch with no way back cannot.
75
+ - **The verdict follows mechanically from the rows.** Two reviewers given the same rows reach the same verdict.
76
+ - **Owners are roles.** "Payments on-call", not a person who will change teams.
77
+
78
+ ## When to use this skill
79
+
80
+ - ✅ A merged change is about to reach users and someone asks "are we ready?"
81
+ - ✅ A production readiness review needs a written go/no-go record.
82
+ - ✅ The `ready` stage of the `release` loop.
83
+
84
+ ## When NOT to use this skill
85
+
86
+ - ❌ Deciding whether to *merge*. That is `pre-merge-review`'s gate, which happens earlier.
87
+ - ❌ Planning the rollout phases themselves. That is `migration-plan`; this skill checks that the plan exists and has been rehearsed.
88
+ - ❌ Choosing *when* to launch. Timing and freeze windows are the release owner's call, or `council-review`'s if they are contested.
89
+
90
+ ## Anti-patterns to avoid
91
+
92
+ - ❌ **A generic checklist with every box ticked.** If no row is `not met` on a non-trivial launch, the evidence was not checked.
93
+ - ❌ **`n/a` as a way out.** Data and privacy marked `n/a` with no reason is the most common way a launch skips a review it needed.
94
+ - ❌ **Letting a strong row offset a missing rollback.** Great dashboards do not make an irreversible launch safe.
95
+ - ❌ **Writing the runbook or the alert inside this report.** Name the gap and hand it off; a readiness report that also drafts fixes becomes too long to read.
@@ -0,0 +1,10 @@
1
+ [
2
+ { "query": "Are we ready to launch this to users on Monday?", "should_trigger": true },
3
+ { "query": "Run a go/no-go check on the checkout redesign before we turn it on", "should_trigger": true },
4
+ { "query": "Do a production readiness review for the new search service", "should_trigger": true },
5
+ { "query": "Give me a launch checklist for this release and tell me what's missing", "should_trigger": true },
6
+ { "query": "Is this PR safe to merge?", "should_trigger": false },
7
+ { "query": "Plan the phased rollout and backfill for moving users to the new auth provider", "should_trigger": false },
8
+ { "query": "Write the release notes for everything since v2.3", "should_trigger": false },
9
+ { "query": "Write the runbook for the payments service", "should_trigger": false }
10
+ ]
@@ -0,0 +1,19 @@
1
+ {
2
+ "skill_name": "launch-readiness",
3
+ "evals": [
4
+ {
5
+ "id": 1,
6
+ "prompt": "Are we ready to launch? PR #412 adds a `loyalty_points` column to `customers`, backfills it from order history, and shows points on the checkout page. Going to 100% of users Thursday. Load test hit p99 180ms against a 250ms target (link: grafana/d/lt-412). We have a dashboard but haven't added alerts yet. Support hasn't been told.",
7
+ "assertions": [
8
+ "Lists 3-5 failure modes specific to this change (e.g. backfill load on customers, wrong point totals shown at checkout) before the checklist",
9
+ "Every check row has a status of met, not met, or n/a with a reason, plus an evidence column and an owner expressed as a role",
10
+ "Rollback is the first row and is not met, because no tested rollback or data story was supplied",
11
+ "Detection is not met because no alerts exist, and the report hands off to observability-plan rather than drafting the alert inline",
12
+ "The quality-targets row is met and cites the supplied load-test link as evidence",
13
+ "Exposure is flagged because a 100% launch is not staged and no reason is given for why all-at-once is safe",
14
+ "The verdict is BLOCKED, and the reason given is the missing rollback",
15
+ "No dashboards, test results, or sign-offs are invented beyond what the prompt supplied"
16
+ ]
17
+ }
18
+ ]
19
+ }
@@ -0,0 +1,54 @@
1
+ # Worked example — schema change that comes out REVISE
2
+
3
+ ## Input given to the skill
4
+
5
+ > Go/no-go for release `orders-v4.2`. It splits `orders.address` into `shipping_address_id`
6
+ > pointing at a new `addresses` table. We followed the migration plan
7
+ > (docs/migrations/orders-address.md): expand and backfill are done in staging, and dual-write
8
+ > is behind the flag `orders.address_split`. Rollback is "turn the flag off". It was rehearsed
9
+ > in staging on 2026-09-28 (log: ci/run/88213). The p99 target for order reads is 120ms, and
10
+ > the canary at 5% shows 96ms (grafana/d/orders-canary). The existing `orders-5xx` alert
11
+ > covers this. Release notes are drafted in PR #977. Plan is 5% → 50% → 100% over three days.
12
+
13
+ ---
14
+
15
+ # Launch readiness — orders-v4.2 (address split)
16
+
17
+ **Change:** release `orders-v4.2` · **Reach:** 5% → 50% → 100% over 3 days · **Release owner:** Orders tech lead
18
+ **Assessed:** 2026-10-01 · **Inputs:** migration plan, staging rollback log, canary dashboard, PR #977
19
+
20
+ ## New failure modes
21
+
22
+ | # | Failure mode | Who is hurt | Reversible? |
23
+ |---|---|---|---|
24
+ | F1 | Dual-write diverges: an order is saved with an `addresses` row that doesn't match `orders.address` | Customers shipped to the wrong address | Partly. Data can be fixed, parcels already sent cannot |
25
+ | F2 | The read path picks the new column before the backfill is complete for that tenant | Customers who see a blank address at checkout | Yes, by turning the flag off |
26
+ | F3 | The extra join pushes order reads past p99 under full load | All checkout users | Yes, by turning the flag off |
27
+
28
+ ## Checks
29
+
30
+ | # | Check | Covers | Status | Evidence | Owner |
31
+ |---|---|---|---|---|---|
32
+ | 1 | Rollback | all | met | Flag-off rehearsed in staging, ci/run/88213. While the old column is still written, flag-off loses no data | Orders on-call |
33
+ | 2 | Detection | F1–F3 | **not met** | `orders-5xx` catches F3 errors only. Nothing detects F1 divergence or F2 blank reads, because both return 200 | Orders on-call |
34
+ | 3 | Response | F1–F3 | **not met** | There is no runbook entry for divergence. `orders-5xx` has a runbook entry, but it does not mention the flag | Orders on-call |
35
+ | 4 | Quality targets | F3 | met | Canary p99 96ms against a 120ms target at 5%, grafana/d/orders-canary | Orders tech lead |
36
+ | 5 | Data & privacy | — | n/a — no new personal data. Address data moves between tables, and retention and processors are unchanged | Migration plan §2 | Data protection lead |
37
+ | 6 | Comms | — | met | Release notes drafted in PR #977. No customer-visible change | Orders tech lead |
38
+ | 7 | Exposure | — | met | Staged 5% → 50% → 100% behind `orders.address_split` | Release owner |
39
+
40
+ ## Open items
41
+
42
+ - Row 2: add a reconciliation metric for dual-write divergence, and an alert on any non-zero count, plus an alert on blank-address reads. Orders on-call, before the 5% → 50% step. Hand off to `observability-plan`.
43
+ - Row 3: add a runbook entry for "divergence alert fires" (flag off, then run the reconciler), and add the flag to the `orders-5xx` entry. Orders on-call, same deadline. Hand off to `runbook-generator`.
44
+
45
+ ## Verdict
46
+
47
+ **REVISE.** Row 2 is not met: the two failure modes that do lasting damage (F1, F2) return 200, so nothing would detect them. Fix rows 2 and 3, then run readiness again.
48
+
49
+ ---
50
+
51
+ **Why this is a good output:** the rollback is genuinely in place, so the verdict is REVISE and
52
+ not BLOCKED. The report notices that an existing alert covering the change only catches the
53
+ failure that is easiest to undo. It does not draft the alert or runbook itself; it names the
54
+ gap, the owner and the hand-off.
@@ -0,0 +1,24 @@
1
+ {
2
+ "name": "launch-readiness",
3
+ "version": "0.1.0",
4
+ "description": "Produce a go/no-go launch readiness report for a merged change before it goes live — one row per check (rollback path, alert and runbook per new failure mode, NFR targets measured, comms drafted, data/privacy sign-off), each with an owner, evidence link, and met / not met / n/a-because status, ending in a PROCEED / PROCEED WITH CONDITIONS / REVISE / BLOCKED verdict. Use when the user asks \"are we ready to launch\", \"go/no-go\", \"production readiness review\", \"launch checklist\", or \"can we ship this to users\".",
5
+ "entrypoint": "SKILL.md",
6
+ "deps": { "npm": [], "pip": [] },
7
+ "env": { "required": [], "optional": [] },
8
+ "related": [
9
+ "migration-plan",
10
+ "observability-plan",
11
+ "runbook-generator",
12
+ "nfr-spec",
13
+ "release-notes",
14
+ "council-review",
15
+ "pre-merge-review"
16
+ ],
17
+ "handoff": [
18
+ { "to": "migration-plan", "when": "the change has no tested rollback path", "purpose": "Plan the phases and the rollback, and name the point of no return, before readiness is judged again.", "fallback": "If migration-plan is absent, write the rollback as numbered steps with its data story inline, and mark the row not met until someone has rehearsed it." },
19
+ { "to": "observability-plan", "when": "a new failure mode has no alert", "purpose": "Define the SLI and the burn-rate alert that would detect it.", "fallback": "If observability-plan is absent, name the symptom, the metric that shows it, and a threshold inline, tagged [assumption] for the owner to confirm." },
20
+ { "to": "runbook-generator", "when": "an alert exists but has no runbook entry", "purpose": "Write the procedure the on-call follows when that alert fires.", "fallback": "If runbook-generator is absent, write recognise / mitigate / escalate as three lines under the row, and keep the row not met until the on-call owner has reviewed them." }
21
+ ],
22
+ "tags": ["launch", "release", "readiness", "go-no-go", "production-readiness"],
23
+ "model": { "tier": "standard", "rationale": "Checks evidence against a fixed checklist. Bounded synthesis, not adversarial review." }
24
+ }
@@ -0,0 +1,41 @@
1
+ # Launch readiness — {change name}
2
+
3
+ **Change:** {PR / tag / feature} · **Reach:** {internal / N% / cohort / everyone} · **Release owner:** {role}
4
+ **Assessed:** {YYYY-MM-DD} · **Inputs:** {what was read — diff, design doc, migration plan, dashboards}
5
+
6
+ > **Assumptions to confirm first** (omit if none)
7
+ > - `[assumption]` {fact guessed rather than given}
8
+
9
+ ## New failure modes
10
+
11
+ What this change, specifically, could do to users or data once live.
12
+
13
+ | # | Failure mode | Who is hurt | Reversible? |
14
+ |---|---|---|---|
15
+ | F1 | {e.g. backfill locks `orders` table under peak write load} | {checkout users} | {yes / no / partly} |
16
+ | F2 | | | |
17
+ | F3 | | | |
18
+
19
+ ## Checks
20
+
21
+ Status is `met`, `not met`, or `n/a — <reason>`. No evidence means `not met`.
22
+
23
+ | # | Check | Covers | Status | Evidence | Owner |
24
+ |---|---|---|---|---|---|
25
+ | 1 | **Rollback** — tested path back, data story stated | all | | {link / command output} | {role} |
26
+ | 2 | **Detection** — an alert per failure mode, fires before users complain | F1–F3 | | | |
27
+ | 3 | **Response** — a runbook entry per alert in row 2 | F1–F3 | | | |
28
+ | 4 | **Quality targets** — each NFR target measured, not asserted | | | | |
29
+ | 5 | **Data & privacy** — sign-off for new personal data, retention, or processors | | | | |
30
+ | 6 | **Comms** — release notes drafted, support briefed, status-page owner known | | | | |
31
+ | 7 | **Exposure** — first exposure staged, or why all-at-once is safe | | | | |
32
+
33
+ ## Open items
34
+
35
+ One line per `not met` row: what closes it, who, by when, and the hand-off if any.
36
+
37
+ - Row {n}: {what closes it} — {owner role}, {date} → {`migration-plan` / `observability-plan` / `runbook-generator` / none}
38
+
39
+ ## Verdict
40
+
41
+ **{PROCEED | PROCEED WITH CONDITIONS | REVISE | BLOCKED}** — {the one reason, naming the row}.
@@ -0,0 +1,90 @@
1
+ ---
2
+ name: prompt-caching-advisor
3
+ description: Analyzes a prompt or conversation structure and identifies the highest-value cache breakpoints — where to insert cache_control checkpoints to minimize repeated token processing. Use when designing or optimizing a Claude API integration that uses prompt caching, when input costs are dominated by a static system prompt or few-shot block, or when a multi-turn conversation reprocesses the same context on every turn.
4
+ ---
5
+
6
+ # prompt-caching-advisor
7
+
8
+ Identifies where to place `cache_control` checkpoints in a Claude API prompt or conversation structure to minimize repeated token processing. The savings are real — a cache hit on a 10,000-token system prompt costs ~90% less than re-processing it — but the placement matters. Wrong breakpoints waste the minimum cacheable block size or cache dynamic content that never hits.
9
+
10
+ ## How to respond
11
+
12
+ ### Step 1 — Understand the structure
13
+
14
+ Ask the user to share (or describe) their prompt structure. You need:
15
+ - The system prompt (or a description of what's in it and roughly how long)
16
+ - Any few-shot examples passed in messages
17
+ - The shape of the variable user turn (what changes per request)
18
+ - Whether this is single-turn or multi-turn
19
+
20
+ If they share code, read the messages array. The cache breakpoint goes at the end of the block you want cached — a `cache_control: {"type": "ephemeral"}` on the **last content block** of a message causes everything up to and including that block to be cached.
21
+
22
+ ### Step 2 — Classify each block
23
+
24
+ For each distinct block, classify it:
25
+
26
+ | Class | Characteristics | Cache? |
27
+ |---|---|---|
28
+ | **Static** | Same on every request (system instructions, reference docs, few-shot examples) | Yes — put breakpoint at the end |
29
+ | **Quasi-static** | Same for a session or user, different across users (user profile, project context) | Yes, with session-scope awareness |
30
+ | **Dynamic** | Changes every request (current user message, real-time data, timestamps) | No — never cache |
31
+
32
+ ### Step 3 — Recommend breakpoints
33
+
34
+ State exactly where to place each `cache_control` checkpoint. Show the code:
35
+
36
+ ```python
37
+ messages = [
38
+ {
39
+ "role": "user",
40
+ "content": [
41
+ {
42
+ "type": "text",
43
+ "text": "<static few-shot examples here>",
44
+ "cache_control": {"type": "ephemeral"} # breakpoint 1 — caches system + examples
45
+ },
46
+ {
47
+ "type": "text",
48
+ "text": user_query # dynamic — no cache_control
49
+ }
50
+ ]
51
+ }
52
+ ]
53
+ ```
54
+
55
+ For system prompts, the cache breakpoint goes on the last block of the system array.
56
+
57
+ ### Step 4 — Estimate cache hit rate
58
+
59
+ Give a rough hit rate estimate:
60
+ - **Near 100%**: static system prompt, fixed few-shot examples, no dynamic injection into the cached block
61
+ - **~session %**: user-context cached per-session; hits on turn 2+, misses on first turn per user
62
+ - **Low**: anything that embeds timestamps, UUIDs, or per-request data in the cached block
63
+
64
+ ### Step 5 — Warn on sub-threshold blocks
65
+
66
+ Warn if a block the user wants to cache is likely under the minimum cacheable size:
67
+ - **Haiku**: ~1,024 tokens minimum
68
+ - **Sonnet / Opus**: ~2,048 tokens minimum
69
+
70
+ A block under the minimum is processed as normal — no error, just no cache savings. Calculate roughly: 1 token ≈ 4 characters of English prose. A 2,000-character system prompt (~500 tokens) won't benefit on Sonnet/Opus.
71
+
72
+ ### Step 6 — Output format
73
+
74
+ Produce:
75
+ 1. **Annotated structure** — show the messages array with `cache_control` placements marked and commented
76
+ 2. **Cache strategy summary** — one paragraph: what is cached, why, expected hit rate, and the dominant cost driver before and after
77
+
78
+ ## Anti-patterns
79
+
80
+ - **Caching dynamic content.** If a user query, timestamp, or per-request database result is inside a cached block, every request is a cache miss — the structure defeats itself. Identify and flag before recommending.
81
+ - **Recommending a cache on a block under the minimum size.** A 300-token block doesn't cache on Sonnet/Opus. The API doesn't error — it just processes normally. Always check size before recommending.
82
+ - **Recommending four or more breakpoints.** More than three breakpoints on a single request is almost always a sign the prompt structure should be simplified, not patched with caches.
83
+
84
+ Use `token-budget-estimator` after this skill if the user also wants to size the token budget across components.
85
+
86
+ ## Quality bar
87
+
88
+ - **Never cache dynamic content.** A user query, a timestamp, a database result retrieved per-request — these must not appear in a cached block. If the user's structure embeds dynamic content in an otherwise static block, flag it before recommending the cache.
89
+ - **One breakpoint per logical boundary.** A system prompt with three sections (instructions, reference docs, examples) can have up to three breakpoints. More than four breakpoints on a single request is a signal the structure needs rethinking.
90
+ - **The last block caches everything before it.** This is the most common misunderstanding — the cache breakpoint is not a marker for just that block, it's a marker for the entire prefix up to that point.
@@ -0,0 +1,11 @@
1
+ {
2
+ "name": "prompt-caching-advisor",
3
+ "version": "1.0.0",
4
+ "description": "Analyzes a prompt or conversation structure and identifies the highest-value cache breakpoints — where to insert cache_control checkpoints to minimize repeated token processing. Use when designing or optimizing a Claude API integration that uses prompt caching, when input costs are dominated by a static system prompt or few-shot block, or when a multi-turn conversation reprocesses the same context on every turn.",
5
+ "entrypoint": "SKILL.md",
6
+ "deps": { "npm": [], "pip": [] },
7
+ "env": { "required": [], "optional": [] },
8
+ "related": ["token-budget-estimator"],
9
+ "tags": ["claude-api", "caching", "cost", "optimization", "prompt-engineering"],
10
+ "model": { "tier": "standard", "rationale": "Structural analysis of a prompt with threshold arithmetic and pattern matching — above light (needs API-specific judgment), no need for heavy." }
11
+ }
@@ -0,0 +1,75 @@
1
+ ---
2
+ name: token-budget-estimator
3
+ description: Estimates input and output token usage for a described agentic workflow, surfaces the dominant cost drivers, and suggests max_tokens budget controls per component. Use when planning a new Claude API integration, sizing a cost budget for a feature, or diagnosing why a workflow's token usage is higher than expected.
4
+ ---
5
+
6
+ # token-budget-estimator
7
+
8
+ Breaks down token usage for an agentic workflow — before you build it or while diagnosing runaway costs. The output is a component table (not a single number) because "how many tokens does this use?" has a different answer for the system prompt, the growing context window, tool results, and the final output. Knowing which component dominates tells you where to optimize.
9
+
10
+ ## How to respond
11
+
12
+ ### Step 1 — Intake
13
+
14
+ Ask the user to describe their workflow. Collect:
15
+ - What the agent does (brief description)
16
+ - The shape of the system prompt (static instructions, injected context, few-shot examples — rough character/word count is fine)
17
+ - How many turns or tool calls a typical session has
18
+ - What tool results look like (small JSON blobs? Full document retrieval? Database rows?)
19
+ - Roughly how long the final output is
20
+
21
+ If they already have working code or logs, ask them to share a sample API request/response — you can count tokens more accurately from the actual content.
22
+
23
+ ### Step 2 — Estimate each component
24
+
25
+ Build a token budget table. Token approximation: **1 token ≈ 4 characters** of English text; code/JSON runs denser (~3 chars/token).
26
+
27
+ | Component | Est. tokens | Notes |
28
+ |---|---|---|
29
+ | System prompt | X | Static — same every call |
30
+ | Injected context (per call) | X | User data, retrieved docs, etc. |
31
+ | Few-shot examples | X | Static — cache candidate |
32
+ | User turn (per call) | X | Variable per request |
33
+ | Tool definitions | X | Sent on every call that uses tools |
34
+ | Tool results (per turn) | X | Accumulate across turns |
35
+ | Context accumulation | X | Previous turns retained in window |
36
+ | Output (per call) | X | Depends on max_tokens and task |
37
+ | **Total per session** | X | Sum across all turns |
38
+
39
+ Flag the **dominant cost driver** — the single row with the largest token count — and explain why it dominates.
40
+
41
+ ### Step 3 — Recommend budget controls
42
+
43
+ For each component, suggest a concrete control:
44
+
45
+ - **System prompt too large**: split static vs dynamic; cache the static portion (`prompt-caching-advisor`)
46
+ - **Context accumulation**: summarize earlier turns, or use a sliding window instead of full history
47
+ - **Tool results**: truncate or paginate large results before passing to the model; never inject a 50-row database result when 5 representative rows suffice
48
+ - **Output**: set `max_tokens` to the realistic output ceiling — open-ended `max_tokens` on a generation task wastes budget on over-generation
49
+ - **Tool definitions**: only send the tools relevant to the current task state (tool selection narrows with progress)
50
+
51
+ ### Step 4 — Extended thinking note
52
+
53
+ If the workflow uses extended thinking (`betas: ["interleaved-thinking"]`), add a note: thinking tokens are billed at the same input rate and are subject to a `budget_tokens` cap you set. Recommend setting `budget_tokens` to 2–4× the typical output length for reasoning tasks, and lower for classification or extraction tasks where long thinking chains rarely help.
54
+
55
+ ### Step 5 — Output format
56
+
57
+ Produce:
58
+ 1. **Token budget table** — the component breakdown with estimates
59
+ 2. **Dominant driver** — named and explained
60
+ 3. **Three concrete controls** — ranked by expected savings, with specific API parameters where applicable (`max_tokens`, `cache_control`, tool selection logic)
61
+ 4. **Rough cost estimate** — using current public pricing for the tier the user is on (state the tier assumption; don't guess if unknown)
62
+
63
+ ## Anti-patterns
64
+
65
+ - **A single total without a breakdown.** "~50,000 tokens per session" tells a developer nothing actionable. Always produce the component table.
66
+ - **False precision from descriptions.** "3,142 tokens" from a description is theatre. Use ranges; tighten only when working from actual content.
67
+ - **Optimizations without a ranking.** Listing five suggestions without saying which saves the most wastes the developer's attention.
68
+
69
+ For caching-specific advice after sizing the budget, hand off to `prompt-caching-advisor`. For agent-level spending controls, see `agent-budget`.
70
+
71
+ ## Quality bar
72
+
73
+ - **Ranges over false precision.** "~2,000–4,000 tokens" is more honest than "3,142 tokens" when you're working from a description. Use ranges when working from descriptions, tighter estimates when working from actual content.
74
+ - **The table beats the total.** A single "~50,000 tokens per session" is nearly useless — the table that shows context accumulation accounts for 40,000 of those gives the developer somewhere to act.
75
+ - **Name the optimization target.** Don't list five optimizations without ranking them. The one that saves 80% of the dominant component beats the one that shaves 5% off a small component.
@@ -0,0 +1,11 @@
1
+ {
2
+ "name": "token-budget-estimator",
3
+ "version": "1.0.0",
4
+ "description": "Estimates input and output token usage for a described agentic workflow, surfaces the dominant cost drivers, and suggests max_tokens budget controls per component. Use when planning a new Claude API integration, sizing a cost budget for a feature, or diagnosing why a workflow's token usage is higher than expected.",
5
+ "entrypoint": "SKILL.md",
6
+ "deps": { "npm": [], "pip": [] },
7
+ "env": { "required": [], "optional": [] },
8
+ "related": ["prompt-caching-advisor", "agent-budget"],
9
+ "tags": ["claude-api", "cost", "tokens", "planning", "budgeting"],
10
+ "model": { "tier": "standard", "rationale": "Arithmetic breakdown plus heuristic judgment on dominant cost drivers — standard fits; no heavy reasoning needed." }
11
+ }