agentscamp 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/content/manifest.json +213 -2
- package/content/skills/agent-trajectory-evaluator.md +59 -0
- package/content/skills/alerting-rules-tuner.md +49 -0
- package/content/skills/canary-release-planner.md +35 -0
- package/content/skills/cold-start-optimizer.md +83 -0
- package/content/skills/contract-test-designer.md +70 -0
- package/content/skills/devcontainer-designer.md +40 -0
- package/content/skills/distributed-tracing-instrumenter.md +42 -0
- package/content/skills/idempotency-designer.md +47 -0
- package/content/skills/mutation-test-runner.md +64 -0
- package/content/skills/query-plan-analyzer.md +49 -0
- package/content/skills/runbook-writer.md +83 -0
- package/content/skills/semantic-cache-designer.md +40 -0
- package/content/skills/strangler-fig-migrator.md +47 -0
- package/content/skills/threat-model-builder.md +46 -0
- package/content/skills/token-usage-profiler.md +39 -0
- package/package.json +1 -1
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: "threat-model-builder"
|
|
3
|
+
description: "Build a practical threat model for a feature or system using STRIDE — diagram the data flow, mark trust boundaries, enumerate concrete threats where data crosses them, and prioritize by likelihood × impact so security is reasoned about before shipping instead of bolted on after. Use when designing a feature that touches auth, money, or sensitive data, running a security design review, or hardening before a launch."
|
|
4
|
+
allowed-tools: "Read, Grep, Glob"
|
|
5
|
+
version: 1.0.0
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Most threat models fail in one of two ways: they list every conceivable attack until the team is paralyzed and ships nothing, or they skip straight to "we use HTTPS and JWTs" without ever asking where an attacker actually sits. This skill does neither. It forces a data-flow diagram with explicit **trust boundaries**, walks **STRIDE** only where data crosses those boundaries, and ranks every threat by **likelihood × impact** so you mitigate the handful that matter and consciously accept the rest. The output is a diagram, a threat table, and a signed-off list of residual risk — not a vibe.
|
|
9
|
+
|
|
10
|
+
## When to use this skill
|
|
11
|
+
- Designing a feature that handles authentication, authorization, money/payments, PII, or anything multi-tenant.
|
|
12
|
+
- Running a security design review before a launch, or as a gate on a new external-facing endpoint or integration.
|
|
13
|
+
- A pen-test or incident keeps surfacing the same class of bug and you need to find the whole class, not the one instance.
|
|
14
|
+
- Adding a new external entity (third-party webhook, partner API, file upload from users) that data now flows to or from.
|
|
15
|
+
|
|
16
|
+
## Instructions
|
|
17
|
+
1. **Draw the system as a data-flow first — not a box diagram.** Identify four element types: **external entities** (users, partner services, the browser), **processes** (your API, workers, lambdas), **data stores** (DB, cache, queue, blob storage), and the **data flows** (arrows) between them, each labeled with what it carries (`login creds`, `session token`, `tenant_id`, `payment amount`). Express it as Mermaid `flowchart`. If you cannot name what flows on an arrow, you do not understand the system well enough to model it yet.
|
|
18
|
+
2. **Mark trust boundaries — these are the whole point.** A trust boundary is any place data crosses from less-trusted to more-trusted, or between principals that should not see each other's data: internet → your API, your API → DB, unauthenticated → authenticated, tenant A → tenant B, your code → a third-party SDK, user input → a SQL/shell/template interpreter. Draw them as dashed `subgraph` borders. Number each crossing — those numbers are the only places you do STRIDE.
|
|
19
|
+
3. **Walk STRIDE at each boundary crossing, and enumerate CONCRETE threats.** For each element or flow that crosses a boundary, ask all six and write down the specific attack, not the category:
|
|
20
|
+
- **S — Spoofing** (authentication): "Attacker replays a captured session cookie because tokens have no `exp`." Not "ensure authentication."
|
|
21
|
+
- **T — Tampering** (integrity): "Client sends `amount: -100` and the refund flow trusts it."
|
|
22
|
+
- **R — Repudiation** (audit): "A user disputes a transfer and there is no signed, append-only log tying the action to their identity."
|
|
23
|
+
- **I — Information disclosure** (confidentiality): "`GET /users/:id` returns any id without an ownership check — IDOR across tenants."
|
|
24
|
+
- **D — Denial of service** (availability): "The export endpoint runs an unbounded query; one request pins the DB."
|
|
25
|
+
- **E — Elevation of privilege** (authorization): "A `viewer` role can call the admin mutation because authz is checked in the UI, not the API."
|
|
26
|
+
4. **Rate each threat by likelihood × impact and SORT — you cannot fix everything.** Score likelihood (how reachable/easy: High/Med/Low) and impact (blast radius if it lands: High/Med/Low) independently, then derive priority (e.g. High×High = P0, anything with a Low dimension = P3). Sort the table by priority. An unprioritized list of forty threats gets you forty half-done mitigations; a ranked list gets the top five done properly.
|
|
27
|
+
5. **Propose a specific, falsifiable mitigation per high-priority threat.** "Validate input" is not a mitigation. "Reject `amount` server-side unless it matches the stored invoice total; add a contract test" is. Tie each mitigation to where it lives (which middleware, which check, which migration) so it becomes a work item, not an aspiration.
|
|
28
|
+
6. **Write down the residual risk you are accepting — explicitly.** For every threat you are NOT mitigating now, record it as *accepted* (with a one-line rationale and who accepted it) or *deferred* (with the trigger that reopens it: "revisit when we add a second tenant"). Silent acceptance is how a known risk becomes a postmortem line item.
|
|
29
|
+
|
|
30
|
+
> [!WARNING]
|
|
31
|
+
> A threat model with no trust boundaries marked is just a feature list with extra steps. The boundaries are where the threats live — if you skipped step 2, the STRIDE walk in step 3 has nothing to anchor to and will produce generic mush.
|
|
32
|
+
|
|
33
|
+
> [!NOTE]
|
|
34
|
+
> The two highest-yield STRIDE letters for typical web/SaaS features are **E (authz)** and **I (info disclosure)** — IDOR and missing object-level authorization cause more real breaches than exotic crypto failures. If time is short, do those two at every boundary first.
|
|
35
|
+
|
|
36
|
+
## Output
|
|
37
|
+
A self-contained threat model document with three parts:
|
|
38
|
+
|
|
39
|
+
1. **Data-flow diagram** (Mermaid `flowchart`) with external entities, processes, stores, labeled flows, and dashed trust-boundary subgraphs with numbered crossings.
|
|
40
|
+
2. **STRIDE threat table**, sorted by priority:
|
|
41
|
+
|
|
42
|
+
| # | Threat (concrete) | Element / flow | STRIDE | Likelihood | Impact | Priority | Mitigation (specific, where it lives) |
|
|
43
|
+
|---|-------------------|----------------|--------|-----------|--------|----------|----------------------------------------|
|
|
44
|
+
| 1 | `GET /users/:id` returns any tenant's record | API → DB read | I / E | High | High | P0 | Add ownership check in `requireOwner` middleware; contract test for cross-tenant 403 |
|
|
45
|
+
|
|
46
|
+
3. **Accepted residual risks** — a short list of threats not being mitigated now, each tagged *accepted* (rationale + owner) or *deferred* (reopen trigger), so the decision is on the record rather than implied.
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: "token-usage-profiler"
|
|
3
|
+
description: "Measure and attribute LLM token usage and cost across an app — input vs output tokens by feature, route, model, and tenant — then rank the waste and the specific lever to cut it. Use when LLM spend is high or climbing with no clear cause, before scaling a feature that calls a model, or when you need per-feature or per-tenant cost attribution for billing or budgets."
|
|
4
|
+
allowed-tools: "Read, Grep, Glob, Bash"
|
|
5
|
+
version: 1.0.0
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
An LLM bill arrives as one number, and that number tells you nothing about what to fix. The waste is almost never spread evenly — a couple of bloated prompts, one feature that streams paragraphs where a sentence would do, or a single noisy tenant usually drive most of the spend. This skill turns the total into an attributed, ranked profile: it instruments every model call to record input vs output tokens, the model, and a feature/route/tenant **tag**, breaks cost down by that tag, and hands you the dominant drivers each paired with the specific lever that cuts it.
|
|
9
|
+
|
|
10
|
+
## When to use this skill
|
|
11
|
+
- The LLM bill is high or rising and nobody can say which feature or tenant is responsible.
|
|
12
|
+
- You're about to scale a model-backed feature and want to know its true per-call and aggregate cost first.
|
|
13
|
+
- You need per-feature or per-tenant cost attribution for internal budgets, chargeback, or usage-based pricing.
|
|
14
|
+
- A verbose feature or a stuffed context window is suspected, but you have no measurement to confirm it.
|
|
15
|
+
- A cost regression slipped in — spend jumped after a deploy — and you need to localize it to a call site.
|
|
16
|
+
|
|
17
|
+
## Instructions
|
|
18
|
+
1. **Add the tag before measuring anything — attribution is impossible without it.** At every model call site, capture: model id, input (prompt) tokens, output (completion) tokens, and a stable `tag` identifying the *feature/route* (e.g. `summarize-thread`, `support-reply`) plus `tenant`/`user` where billing matters. Pull token counts from the provider's `usage` object on the response, not a local tokenizer — the provider reflects system prompts, tool schemas, and cache discounts. `grep` the codebase for call sites first (`Grep` for the SDK call, e.g. `messages.create`, `chat.completions`, `generateText`) so no path is missed; a single untagged call site becomes an "unattributed" bucket that hides waste.
|
|
19
|
+
2. **Compute cost, don't count tokens.** Map each `(model, input|output)` pair to its price and compute `cost = tokens × price_per_token`, keeping input and output as separate columns. Sum over a representative window (e.g. 7 days, or one full traffic cycle). Tokens alone mislead because input and output, and cheap vs frontier models, have wildly different unit prices.
|
|
20
|
+
3. **Break spend down by tag and sort by total cost.** Produce a table: tag × model × {input cost, output cost, calls, avg tokens/call}. Sort descending by total cost. Expect a Pareto shape — the top 2–4 tags usually own the majority of spend. Optimize those; ignore the long tail.
|
|
21
|
+
4. **Separate per-call cost from volume — they need different fixes.** For each top tag, look at *both* cost-per-call and call count. An expensive call made rarely and a cheap call made a million times can carry the same total; the first is fixed by trimming the prompt/output, the second by caching, dedup, or not calling at all. Flag which axis dominates each driver.
|
|
22
|
+
5. **For each driver, attack the levers in this order (cheapest win first):**
|
|
23
|
+
- **Trim bloated input.** Remove dead boilerplate from system prompts, stop stuffing whole documents/full chat history when a retrieved snippet or rolling summary suffices, and drop unused tool schemas. This is usually the largest, lowest-risk reduction.
|
|
24
|
+
- **Cap or shorten output.** Set `max_tokens` to the real need, ask for terse/structured output, and avoid "explain your reasoning" in production paths where it isn't consumed. Because output is the pricier axis, shaving it often beats prompt trimming on cost.
|
|
25
|
+
- **Downshift the model.** Route easy calls (classification, extraction, short rewrites) to a smaller/cheaper model and reserve the frontier model for genuinely hard ones. Gate the route on a measurable signal, not a guess, and confirm quality holds with an eval set before shipping.
|
|
26
|
+
- **Cache repeated stable prefixes.** Where a long system prompt or document prefix is reused across calls, enable prompt/KV caching so the stable part is billed at the discounted cached rate. Order the prompt so the stable prefix comes first; volatile content last.
|
|
27
|
+
6. **Set per-feature budgets and alerts.** Record each top tag's current cost/call and cost/day as a baseline, then add an alert that fires when either exceeds a threshold (e.g. +30%). Treat a token-usage spike like any other regression — caught at deploy, not at the invoice.
|
|
28
|
+
|
|
29
|
+
> [!WARNING]
|
|
30
|
+
> You cannot optimize what you can't attribute. Without per-feature/per-tenant tags, the "profile" is just a grand total — you'll guess which prompt to cut and likely guess wrong. Add the tag and re-collect before doing any optimization work.
|
|
31
|
+
|
|
32
|
+
> [!NOTE]
|
|
33
|
+
> Output tokens usually cost several times more per token than input tokens, so a verbose model response — not a long prompt — is frequently the real cost driver. Always inspect avg *output* tokens/call on your top tags before assuming the prompt is to blame.
|
|
34
|
+
|
|
35
|
+
## Output
|
|
36
|
+
- **Instrumentation/tagging plan** — the list of call sites found, and for each the tag (feature/route + tenant) and the input/output/model fields to record, sourced from the provider `usage` object.
|
|
37
|
+
- **Spend breakdown** — a table of tag × model with separate input-cost and output-cost columns (`cost = tokens × price`), calls, and avg tokens/call, sorted by total cost, with an "unattributed" row if any call site is still untagged.
|
|
38
|
+
- **Ranked waste** — the dominant drivers in order, each labeled by axis (per-call cost vs volume) and assigned its specific lever (trim context / cap output / downshift model / cache prefix) with the expected reduction.
|
|
39
|
+
- **Budgets & alerts** — baseline cost/call and cost/day per top tag plus the threshold alert to add, so future regressions are caught automatically.
|