skilldrop-cli 0.16.18 → 0.16.20

package/packs/dev-team/skills/devils-advocate/SKILL.md

@@ -1,151 +1,92 @@
 ---
 name: devils-advocate
-description: Adversarial review of just-generated code, run *after* an agent (or human) declares a feature done. Challenges the implementation through four lenses — edge cases the first pass missed, baked-in assumptions that won't survive future requirements, what a staff engineer would push back on in code review, and test-coverage gaps for the new code paths. Produces severity-tagged findings (blocker / major / minor / nit) with file:line evidence and concrete fixes or missing test cases. Use immediately after a feature implementation or generation pass — before merging, before declaring "done", before moving to the next ticket.
+description: Adversarial review of a code change before it merges — the real bugs, most consequential first, each checked against the code before it is reported, with a fix that works. A passing test or clean local run counts as evidence only once you've said whether it would have caught the bug; when it couldn't, the review says why it passed and names the test that would have failed. Sized to the ask ("top 3", "skip the nits", "just security"); invents nothing about the codebase. Use right after a feature is declared done — before merging, before declaring "done", before moving to the next ticket.
 ---
 
 # devils-advocate
 
-You play the role of a **senior engineer in code review who is trying to find what the implementation missed**. Code that "works on the happy path" is the default state of newly-generated code — your job is to challenge it through four lenses and surface what the first pass didn't think about.
-
-This skill is the **code counterpart to [`doc-critique`](../../../core/skills/doc-critique/SKILL.md)** — same adversarial framing, applied to code instead of artifacts.
-
-## When this skill runs
-
-The trigger is **right after an agent (or human) has declared a feature done**. The implementation exists, the happy path likely works, tests may even pass — and that's exactly the moment the skill is most useful, because that's the moment everyone stops looking.
-
-Specifically:
-- ✅ After AI code generation for a feature ("here's the implementation")
-- ✅ Before opening a PR / before requesting human review
-- ✅ Before merging
-- ✅ When the user asks "anything I might have missed?" or "is this ready to ship?"
-- ❌ For a one-line fix or trivial change — overkill
-- ❌ For greenfield exploration / spikes — adversarial review wastes effort on code that will be thrown away
-- ❌ As a substitute for actual code review by a human — this is a *pre-pass*, not a replacement
+**The rule: a green test is not evidence until you've said why it would have failed with the
+bug.** Changes arrive with "tests pass" or "ran it locally, fine". The agent's habit is to
+take that as support, or to ignore it. Both miss the most useful thing a reviewer can say:
+*why* the test passed anyway. A fixture that happens to dodge the bug, a table that was empty
+on the laptop, a test that runs the job once when the bug only shows on the second run, an
+assertion that one bug cancels out. For each bug you find, say whether the cited tests or run
+could have caught it, and if not, why it passed and the one test that would have failed:
+✅ *"`test_refill` passes because it waits 2 s against a rate of 1/s, so it never sees the
+partial-token case. Assert `allow()` is false at 0.5 s."*
+
+A second rule: **report only what survives checking.** Before a finding goes in, trace it
+through the code. Does an earlier guard already handle the case? Is the query actually
+parameterised? Does the stated business rule make this behaviour correct? Does your own fix
+do what you say it does? A false finding costs the reader more than a missed nit.
 
 ## How to respond
 
-**Four rules come before the steps and outrank them:**
-
-- **Answer what was asked, first.** Open with the answer, the decision or the artifact, in plain words. Scores, matrices, frameworks and tags come after it, and anything that doesn't change the answer is cut.
-- **Use only what you were given.** Don't add facts, names, numbers, incidents, history, steps or sections the input doesn't contain. What you need and don't have is left out of the artifact and listed once at the end under "To confirm".
-- **Deliver from what you have.** When the request gives you something to work on, state your assumptions in a line and produce the result. When it gives you nothing to work on, ask for it in one or two plain sentences and say what you will do once you have it.
-- **Write for someone who has never heard of this skill.** No skill names, no paths or scripts from this folder, no internal terms, and nothing about how the run was set up. A next step is one plain sentence at the end that describes the work.
-- **For this skill:** With no code or document to review, ask for it in one sentence and name the three or four things you will check in it, taken from what the user said it does.
-
-1. **Identify the change under review.** In order of preference:
-   - The agent / user has just shown you a diff — review that diff.
-   - Git working tree has unstaged changes — `git diff` is the scope.
-   - User points to specific files / a PR / a recent commit — that's the scope.
-   - User says "the feature I just built" with no diff — ask which files, or run `git diff HEAD~1` if a commit was just made.
-
-   **Don't review the whole repo.** The scope is the *just-generated* code. Touched files only.
-
-2. **Classify the code shape.** What you challenge depends on what was built. Map to one or more:
-
-   | Shape | Lens emphasis |
-   |---|---|
-   | HTTP / RPC endpoint, API handler | edge-cases (input validation, auth), adversarial (rate limit, error mapping) |
-   | Async handler / queue consumer | edge-cases (retries, idempotency, poison messages), future-proofing (ordering) |
-   | DB migration / schema change | future-proofing (rollback, online-safe), adversarial (locking, big-table risk) |
-   | State machine / workflow | edge-cases (impossible transitions, concurrent updates) |
-   | Parser / data transformation | edge-cases (malformed, encoding, huge inputs) |
-   | CLI tool / script | edge-cases (missing flags, env, partial runs) |
-   | UI component | edge-cases (loading/empty/error states, a11y) |
-   | Background job / cron | edge-cases (overlap, missed runs, partial completion) |
-   | Library / pure function | edge-cases (boundary inputs), future-proofing (API stability) |
-   | Config / infra change | adversarial (blast radius), future-proofing (drift) |
-
-   Multiple shapes is common — a PR can be "endpoint + migration + UI." Apply each lens to the right slice.
-
-3. **Sweep the four lenses in order.** Each lens has its own checklist — read it, apply it, write findings. Don't merge lenses; each surfaces a different category of risk.
-
-   | Lens | File | Asks |
-   |---|---|---|
-   | 1. Edge cases | [`lenses/edge-cases.md`](lenses/edge-cases.md) | What inputs / states / timings break this? |
-   | 2. Future-proofing | [`lenses/future-proofing.md`](lenses/future-proofing.md) | What assumptions won't survive the next 6 months? |
-   | 3. Adversarial review | [`lenses/adversarial-review.md`](lenses/adversarial-review.md) | What would a staff engineer push back on in review? |
-   | 4. Test coverage | [`lenses/test-coverage.md`](lenses/test-coverage.md) | Which new code paths have no test, and which scenarios are missing tests? |
-
-4. **Categorize every finding by severity:**
-
-   | Severity | Meaning |
-   |---|---|
-   | 🟥 **Blocker** | This will ship a bug or a security issue. Examples: SQL injection, unhandled `null`, race condition on shared state, migration that locks a 10M-row table, missing auth check, infinite retry loop. |
-   | 🟧 **Major** | This will hurt within months. Examples: hard-coded limits that 2× growth will hit, no idempotency on a retryable operation, error swallowed silently, no observability for a load-bearing path, magic constants. |
-   | 🟨 **Minor** | Good craft. Examples: missing edge-case test, suboptimal log level, naming that won't age well, a comment that explains *what* instead of *why*. |
-   | ⚪ **Nit** | Style, formatting, micro-preference. Optional. |
-
-   When in doubt between two severities, **downgrade**. Inflated severity makes the report unreadable. If your output has more blockers than major findings, you're inflating.
-
-5. **For every finding, write four parts:**
-
-   - **Where:** `file:line` (e.g. `src/api/checkout.ts:47`) — and quote 5–15 chars of the offending line.
-   - **What's wrong:** one sentence — which lens fired and why.
-   - **Concrete scenario:** ✅ "When `discount=null` reaches line 47, `applyDiscount` throws `TypeError`." — ❌ "Doesn't handle null." A scenario the first pass can reproduce.
-   - **Fix:** ✅ "Add `if (discount == null) return 0;` at the top of `applyDiscount` — or thread the null through to skip the line item entirely (preferred — see how `applyTax` handles it at line 102)." — ❌ "Add null handling."
-
-6. **Open with a verdict, not the findings.** First thing the user sees:
-
-   ```
-   Verdict: SHIP WITH FIXES — 2 blockers, 5 major, 4 minor, 3 nits.
-
-   The endpoint validates auth and the happy path but doesn't bound the request body (blocker — DoS via 1GB payload), and the new migration adds a NOT NULL column without a default on a table the migration script doesn't backfill (blocker — will fail on first deploy). Of the 5 major findings, 3 are around idempotency on the retry path. Test coverage misses the new error branches at api/checkout.ts:71–88 entirely.
-   ```
+1. **Answer the question first.** Open with a plain answer to what they asked, usually
+   whether it's safe to merge, and the worst problem in one sentence.
 
-   Verdicts: `SHIP IT` · `SHIP WITH FIXES` · `DO NOT MERGE` · `RECONSIDER APPROACH`. The last is for when the implementation works but solved the wrong problem (e.g. building a custom retry queue when the framework already has one).
+2. **Review the change, not the repo.** The diff, the files they named, or `git diff` of the
+   working tree. Read the tests that came with it.
 
-7. **Output structure** — use [`templates/challenge.md`](templates/challenge.md):
+3. **Find what would bite, most consequential first.** Wrong results, data loss, security
+   holes, failures on real data that the dev setup hides, retries and concurrency, then
+   fragility. For a large or unfamiliar change, the checklists in
+   [`lenses/edge-cases.md`](lenses/edge-cases.md),
+   [`lenses/future-proofing.md`](lenses/future-proofing.md),
+   [`lenses/adversarial-review.md`](lenses/adversarial-review.md) and
+   [`lenses/test-coverage.md`](lenses/test-coverage.md) are a sweep to run privately; they are
+   not sections of the reply.
 
-   ```markdown
-   # Devil's advocate: {feature name / PR title}
-   _Scope: {N files, M lines changed} · Reviewed: {YYYY-MM-DD}_
+4. **Each finding:** where (`file:line`), what goes wrong in one concrete scenario ("when
+   `tz` is `None`, line 47 raises `TypeError`"), and the fix. When the fix is small,
+   give it as code they can paste. When the change is a script or migration, give the
+   corrected version.
 
-   ## Verdict
-   {one-liner} — {N} blocker · {N} major · {N} minor · {N} nit
-   {3-sentence summary, lead with the worst}
+5. **Say what the tests prove.** Apply the rule to every test or run the user cites, and name
+   the missing tests as scenarios: ✅ *"No test runs the job twice; add one asserting the reminder
+   email is sent once."*
 
-   ## Findings by lens
+6. **Size to the ask.** "Top 3" gets three. "Skip the nits" gets none. "Just security" gets
+   security. Without a limit, report what would actually bite; leave style out unless asked.
+   No severity counts, verdict labels or praise sections unless they asked.
 
-   ### 1. Edge cases
-   ### 2. Future-proofing
-   ### 3. Adversarial review
-   ### 4. Test coverage
+7. **Invent nothing about the codebase.** No row counts, traffic, run times, callers,
+   column contents or config you weren't shown. Where a finding depends on one, say what to
+   check: "if `orders` already has rows in production, this migration fails".
 
-   ## What's solid
-   - {2–4 things the implementation got right — not flattery, real patterns to preserve}
+8. **Deliver from what you have.** With code to review, review it. With nothing, ask for the
+   diff in one sentence and name the three or four things you'll check in it.
 
-   ## Suggested order to address
-   1. Fix blockers
-   2. Add the missing tests in §4 (they double as regression catches)
-   3. Major findings in priority order
-   ```
+## Useful references in this skill
 
-8. **End with "what's solid"**, not more criticism. The user (or the next agent) often needs to know what to *keep* and what *patterns to reuse* elsewhere in the change.
+- [`examples/rate-limiter-review.md`](examples/rate-limiter-review.md) — a change with "tests pass", and a review that says why they passed
 
 ## Quality bar
 
-- **Every finding has a reproducible scenario.** A senior engineer reading the finding should be able to reproduce the bug or violation in their head. "Doesn't handle null" is not a finding; "When `discount=null` reaches line 47, …" is.
-- **Severity is calibrated.** More blockers than major ⇒ inflating ⇒ demote.
-- **Findings cite `file:line`.** Reviewers must be able to jump to the spot. Section names aren't enough for code.
-- **Fixes are concrete and paste-able where possible.** When you can suggest a one-line patch, do; when you can't, suggest the *approach* in one sentence — not "improve error handling."
-- **Don't redesign.** This isn't your chance to push a different architecture. Stay in the lane of "given this approach, what's missing." If the approach itself is wrong, that's a single `RECONSIDER APPROACH` verdict, not a wholesale rewrite suggestion.
-- **Test-coverage findings reference scenarios, not files.** ❌ "No test for `applyDiscount`." ✅ "No test covers `applyDiscount(null)` — add a case that asserts the function returns 0 (or skips) without throwing."
+- **Every cited test or run is judged:** could it have caught each bug, and if not, why did it
+  pass and what test would fail.
+- **Every finding survives a trace through the code and the stated rules.** No false
+  positives.
+- **The answer to "can this merge?" comes first;** findings are ordered by consequence.
+- **Fixes work,** and are code when the change is small.
+- **The scope asked for is respected.**
+- **Nothing about the codebase is invented.**
 
 ## When NOT to use this skill
 
-- ❌ For greenfield prototype / spike code that's about to be thrown away.
-- ❌ For one-line / trivial changes — overkill, and the noise drowns out real signal.
-- ❌ As a substitute for human code review — it's a pre-pass, not a replacement.
-- ❌ For *evaluating the choice of approach* before code exists. That's a design conversation, not a code challenge. Use `design-doc` + `doc-critique` for that loop.
-- ❌ For generated code in a language you can't reason about line-by-line. Better to refuse than fabricate findings.
+- ❌ For greenfield prototype or spike code that's about to be thrown away.
+- ❌ For one-line or trivial changes.
+- ❌ As a substitute for human code review — it's a pre-pass.
+- ❌ For *evaluating the choice of approach* before code exists. That's a design conversation: `design-doc`, then `doc-critique`.
+- ❌ For code in a language you can't reason about line by line. Refuse rather than fabricate findings.
 
 ## Anti-patterns to avoid
 
-- ❌ **Severity inflation.** If everything is a blocker, the report is unreadable. Most findings are major or minor.
-- ❌ **Vague findings.** "Handle errors better" / "Consider edge cases" — the whole point of this skill is to *name* the missed cases. If you can't name a specific scenario, drop the finding.
-- ❌ **Restating the lens.** ❌ "Code should be future-proof." ✅ "Hard-coded 100ms timeout on line 53 won't survive p99 growth — make it configurable."
-- ❌ **Lecturing on style** when there's a real bug to flag. Style notes go in the Nits section, not on top.
-- ❌ **Reviewing untouched code.** Out-of-scope. The skill challenges the *change*, not the codebase.
-- ❌ **Demanding tests for *everything* the change touches.** Only the new logic and new error paths. Don't insist on backfilling tests for previously-untested neighbouring code.
-- ❌ **Skipping "what's solid".** Trains the user (or downstream agent) to dread the skill. End on signal that's actionable in the other direction.
-- ❌ **Showing the machinery.** The reply and the artifact are for the person who asked. Don't mention this skill, its files, templates, caps or internal terms, or that the run is non-interactive. Name another skill once, at the end, as a suggested next step, never inside the artifact.
+- ❌ **"Tests pass" taken at face value,** or ignored. Say what they prove.
+- ❌ **The plausible false positive.** SQL injection in a parameterised query; a bug in
+  behaviour the business rule asks for; a CRLF issue text mode already handles.
+- ❌ **Padding to fill sections.** Marginal findings are where the errors are.
+- ❌ **A count that disagrees with the list.** Don't print counts.
+- ❌ **Describing a fix instead of giving it** when it fits in a few lines.
+- ❌ **Facts about production you weren't given.** Row counts, lock times, what a column holds.
+- ❌ **Redesigning.** Stay on what's missing given this approach.

package/packs/dev-team/skills/devils-advocate/evals/evals.json

@@ -72,6 +72,33 @@
         "It does not state the row count of orders, how long the migration will take, or what customers.region contains as fact",
         "It leads with three main concerns as asked, and anything beyond the three is kept to a brief mention"
       ]
+    },
+    {
+      "id": 6,
+      "held_out": "2026-10-17",
+      "prompt": "I'm the reviewer on PR #412 for the Fernlocket loyalty service and it's meant to merge this afternoon. The change is in redeem.py, the tests are in test_redeem.py, and the author's description and test run are in PR_DESCRIPTION.md. Find what's wrong with it before it ships. Give me the three issues that matter most, most serious first, and skip style and naming nits entirely.",
+      "assertions": [
+        "It finds that quantity is only checked against the upper limit, so a negative quantity gives a negative cost that passes the balance check and adds points to the member's balance, and puts this among its top issues",
+        "It finds that the points are deducted and committed before fulfillment.ship() is called, so a failed shipment leaves the member charged with no refund, and that the \"Please try again\" message invites a second deduction for the same reward",
+        "It does not report the f-string query in list_redemptions as SQL injection, since the ORDER BY text can only come from the fixed SORT_COLUMNS mapping and the values are bound parameters, and it does not report the balance read-then-update as a double-spend race, since the row is locked with FOR UPDATE inside the same transaction",
+        "It says test_fulfillment_failure_returns_502 would pass with the bug in place, because it checks only the status and call count and never the member's balance, so the 6-passed run does not show that #398 is fixed",
+        "Its fixes would work: quantity must be required to be at least 1 (a tighter upper limit is not enough), and on a failed shipment the points must be rolled back or credited back, or not committed until the shipment succeeds; logging the failure or changing the status code alone does not count",
+        "It does not state as fact anything the files don't show: how fulfillment.ship() or the transaction helper behave beyond what the code and comments say, whether the fulfillment service rejects a negative quantity, how many members or redemptions there are, or what else calls redeem()",
+        "It leads with three issues, most serious first, and makes no style or naming comments; anything beyond the three is at most a brief mention"
+      ]
+    },
+    {
+      "id": 7,
+      "held_out": "2026-10-17",
+      "prompt": "hey quick one, can you check this before I push? it sends the weekly digest to all subscribers, max 8 at a time. we're on go 1.23. go test ./... is green and I ran it on staging against a handful of test subscribers, all delivered fine. just tell me if anything's actually broken, I don't care about style\n\n```go\nfunc SendAll(ctx context.Context, subs []Subscriber, d Digest) error {\n\tsem := make(chan struct{}, 8) // at most 8 pushes in flight\n\tvar wg sync.WaitGroup\n\tvar errs []error\n\n\tfor _, s := range subs {\n\t\twg.Add(1)\n\t\tsem <- struct{}{}\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\tdefer func() { <-sem }()\n\t\t\tif err := push(ctx, s, d); err != nil {\n\t\t\t\terrs = append(errs, fmt.Errorf(\"subscriber %s: %w\", s.ID, err))\n\t\t\t}\n\t\t}()\n\t}\n\n\twg.Wait()\n\treturn errors.Join(errs...)\n}\n```",
+      "assertions": [
+        "It finds that the goroutines append to the shared errs slice with no lock, a data race that can lose errors or corrupt the slice when more than one push fails",
+        "It does not flag the closure capturing the loop variable s as a bug, since on Go 1.22 and later each loop iteration gets its own s",
+        "It says the green test run and the staging run don't show the code is safe: with every push succeeding the append never runs, and a plain go test without -race wouldn't report the race anyway",
+        "Its fix would work: a mutex around the append, or a per-subscriber slot in a slice sized to len(subs); if it suggests errgroup it says that this changes the behaviour (only the first error comes back, and with a group context the rest of the sends are cancelled)",
+        "It does not state as fact what push() does inside (retries, timeouts, how it uses ctx), how many subscribers there are, or that this has already failed in production",
+        "It keeps to what's actually broken, as asked: a short reply with no style or naming suggestions, not padded out with a long list of minor or hypothetical concerns"
+      ]
     }
   ]
 }

package/packs/dev-team/skills/devils-advocate/evals/files/6/PR_DESCRIPTION.md

@@ -0,0 +1,20 @@
+## Synchronous reward redemption (#412)
+
+Members currently wait for the nightly batch to find out whether a redemption went through.
+This moves redemption onto a synchronous endpoint so they get an answer straight away.
+
+- `redeem()` backs `POST /members/{member_id}/redemptions`
+- `list_redemptions()` backs the history screen, with a `sort` option
+- The balance row is locked with `SELECT ... FOR UPDATE`, so a double tap on "Redeem" can't spend the same points twice
+- A failed shipment returns 502 and asks the member to try again; this closes #398 and is covered by `test_fulfillment_failure_returns_502`
+- Per-order cap of 10 so nobody can clear out a reward's stock in one request
+
+### Test run
+
+```
+$ pytest tests/test_redeem.py -q
+......                                                                   [100%]
+6 passed in 0.41s
+```
+
+Ready for review. I'd like to get this into Thursday's release.

package/packs/dev-team/skills/devils-advocate/evals/files/6/redeem.py

@@ -0,0 +1,102 @@
+"""Reward redemption for the Fernlocket loyalty programme.
+
+PR #412: move redemption off the nightly batch onto a synchronous endpoint.
+"""
+import logging
+import uuid
+
+from .db import transaction
+from .fulfillment import FulfillmentClient, FulfillmentError
+
+log = logging.getLogger(__name__)
+
+MAX_PER_ORDER = 10
+
+SORT_COLUMNS = {
+    "newest": "created_at DESC",
+    "oldest": "created_at ASC",
+    "points": "points_spent DESC",
+}
+
+
+class RedemptionError(Exception):
+    def __init__(self, status: int, message: str):
+        super().__init__(message)
+        self.status = status
+        self.message = message
+
+
+def redeem(conn, fulfillment: FulfillmentClient, member_id: str, payload: dict) -> dict:
+    """Spend a member's points on a reward and ship it.
+
+    Called by POST /members/{member_id}/redemptions with the JSON body as `payload`.
+    """
+    reward_id = payload["reward_id"]
+    quantity = int(payload.get("quantity", 1))
+    if quantity > MAX_PER_ORDER:
+        raise RedemptionError(400, f"at most {MAX_PER_ORDER} of a reward per order")
+
+    with transaction(conn) as cur:
+        cur.execute(
+            "SELECT points_cost, active FROM rewards WHERE id = %s",
+            (reward_id,),
+        )
+        row = cur.fetchone()
+        if row is None or not row[1]:
+            raise RedemptionError(404, "reward not available")
+        cost = row[0] * quantity
+
+        # Lock the member's balance row so two taps on "Redeem" can't both spend it.
+        cur.execute(
+            "SELECT balance FROM point_accounts WHERE member_id = %s FOR UPDATE",
+            (member_id,),
+        )
+        balance = cur.fetchone()[0]
+        if balance < cost:
+            raise RedemptionError(409, "not enough points")
+
+        redemption_id = str(uuid.uuid4())
+        cur.execute(
+            "UPDATE point_accounts SET balance = %s WHERE member_id = %s",
+            (balance - cost, member_id),
+        )
+        cur.execute(
+            "INSERT INTO redemptions (id, member_id, reward_id, quantity, points_spent, status)"
+            " VALUES (%s, %s, %s, %s, %s, 'pending')",
+            (redemption_id, member_id, reward_id, quantity, cost),
+        )
+    # transaction commits on leaving the block
+
+    try:
+        fulfillment.ship(redemption_id, reward_id, quantity)
+    except FulfillmentError:
+        log.exception("fulfillment failed for redemption %s", redemption_id)
+        raise RedemptionError(502, "We couldn't send your reward. Please try again.")
+
+    with transaction(conn) as cur:
+        cur.execute(
+            "UPDATE redemptions SET status = 'shipped' WHERE id = %s",
+            (redemption_id,),
+        )
+
+    return {
+        "redemption_id": redemption_id,
+        "points_spent": cost,
+        "balance": balance - cost,
+    }
+
+
+def list_redemptions(conn, member_id: str, sort: str = "newest", limit: int = 20) -> list:
+    """Redemption history for the member's account screen."""
+    order_by = SORT_COLUMNS.get(sort)
+    if order_by is None:
+        raise RedemptionError(400, "unknown sort")
+    limit = max(1, min(int(limit), 100))
+
+    with transaction(conn) as cur:
+        cur.execute(
+            "SELECT id, reward_id, quantity, points_spent, status, created_at"
+            f" FROM redemptions WHERE member_id = %s ORDER BY {order_by} LIMIT %s",
+            (member_id, limit),
+        )
+        return cur.fetchall()

package/packs/dev-team/skills/devils-advocate/evals/files/6/test_redeem.py

@@ -0,0 +1,54 @@
+import pytest
+
+from rewards.fulfillment import FulfillmentError
+from rewards.redeem import RedemptionError, list_redemptions, redeem
+from tests.fakes import FakeFulfillment, make_db
+
+
+@pytest.fixture
+def db():
+    conn = make_db()
+    conn.seed_member("m-100", balance=500)
+    conn.seed_reward("r-mug", points_cost=120, active=True)
+    conn.seed_reward("r-old-tote", points_cost=50, active=False)
+    return conn
+
+
+def test_happy_path(db):
+    result = redeem(db, FakeFulfillment(), "m-100", {"reward_id": "r-mug", "quantity": 2})
+    assert result["points_spent"] == 240
+    assert db.balance("m-100") == 260
+
+
+def test_not_enough_points(db):
+    with pytest.raises(RedemptionError) as err:
+        redeem(db, FakeFulfillment(), "m-100", {"reward_id": "r-mug", "quantity": 5})
+    assert err.value.status == 409
+    assert db.balance("m-100") == 500
+
+
+def test_inactive_reward(db):
+    with pytest.raises(RedemptionError) as err:
+        redeem(db, FakeFulfillment(), "m-100", {"reward_id": "r-old-tote"})
+    assert err.value.status == 404
+
+
+def test_quantity_limit(db):
+    with pytest.raises(RedemptionError) as err:
+        redeem(db, FakeFulfillment(), "m-100", {"reward_id": "r-mug", "quantity": 11})
+    assert err.value.status == 400
+
+
+def test_fulfillment_failure_returns_502(db):
+    """Regression for #398: a failed shipment must not cost the member their points."""
+    fulfillment = FakeFulfillment(fail_with=FulfillmentError("carrier timeout"))
+    with pytest.raises(RedemptionError) as err:
+        redeem(db, fulfillment, "m-100", {"reward_id": "r-mug", "quantity": 1})
+    assert err.value.status == 502
+    assert fulfillment.calls == 1
+
+
+def test_history_rejects_unknown_sort(db):
+    with pytest.raises(RedemptionError) as err:
+        list_redemptions(db, "m-100", sort="points; DROP TABLE redemptions")
+    assert err.value.status == 400

package/packs/dev-team/skills/devils-advocate/examples/rate-limiter-review.md

@@ -1,10 +1,9 @@
-# Worked example — adversarial review of a just-written rate limiter
+# Worked example — review of a rate limiter that "passes its tests"
 
-Input → output oracle for `devils-advocate`. Shows the four-lens sweep producing
-severity-calibrated findings with `file:line` evidence and paste-able fixes — not a
-generic "looks good, consider tests" pass.
+Input → output oracle for `devils-advocate`. Shows the answer first, findings that survive a
+trace through the code, and the cited tests judged for what they could and could not catch.
 
-## Input (the "done" implementation)
+## Input
 
 > "Implemented the per-user rate limiter, tests pass, ready for review."
 
@@ -23,24 +22,45 @@ def allow(user_id: str, rate: float, burst: float) -> bool:
     return False
 ```
 
+```python
+# test_ratelimit.py
+def test_burst_then_deny():
+    for _ in range(3):
+        assert allow("u1", rate=1, burst=3)
+    assert not allow("u1", rate=1, burst=3)
+```
+
 ## Output (the review)
 
-**Verdict: not ready — 1 blocker, 1 major.** Four lenses swept; happy path is fine, the concurrency and growth paths are not.
+**Not ready to merge.** Under concurrent requests for one user it admits more than the limit,
+and the test can't see that.
+
+1. **Race on the shared dict** — `ratelimit.py:6-10`. Two requests for the same user both read
+   `tokens=1.0`, both pass, both write back `0.0`, so parallel bursts get through at up to
+   twice the intended rate. Fix: hold a per-user lock around the read-modify-write:
 
-🟥 **Blocker — race on the shared dict (assumptions lens).** `ratelimit.py:6-11` reads then writes `_buckets[user_id]` with no lock. Two concurrent requests for the same user both read `tokens=1.0`, both pass, both write back `0.0` — the limiter leaks one token per race, so a burst of N parallel requests admits ~2× the intended rate. The "tests pass" because they're single-threaded. *Fix:* guard the read-modify-write with a per-user lock, or move to an atomic backend (Redis `INCR`+TTL). A dict-of-locks keyed by `user_id` is the minimal change.
+   ```python
+   _locks = collections.defaultdict(threading.Lock)
 
-🟧 **Major — unbounded memory growth (edge-case lens).** `_buckets` never evicts (`:1`). Every distinct `user_id` ever seen stays forever; a service facing the open internet (or spoofable user ids) OOMs over time. *Fix:* cap with an LRU (`functools.lru_cache` won't fit the mutable value — use `cachetools.TTLCache(maxsize=…, ttl=…)`), evicting idle buckets.
+   def allow(user_id, rate, burst):
+       with _locks[user_id]:
+           ...  # existing body unchanged
+   ```
 
-🟨 **Minor — first call gets a free full burst (edge-case lens).** A never-seen `user_id` starts at `(burst, now)` (`:6`), so a brand-new user can immediately spend the whole burst. If the intent is "warm up", seed at `min(burst, rate*window)` instead — flag for product, don't silently change.
+2. **Memory grows without bound** — `ratelimit.py:2`. Nothing ever leaves `_buckets`, so every
+   distinct `user_id` stays for the life of the process. Fix: a `cachetools.TTLCache` in place
+   of the dict, with a TTL longer than `burst / rate`.
 
-⚪ **Nit — `rate`/`burst` unvalidated (staff-engineer lens).** `rate<=0` freezes every user out; `burst<1` makes `allow` always false. Assert `rate>0 and burst>=1` at the boundary.
+**What the test proves.** `test_burst_then_deny` runs on one thread, so it passes with the
+race in place. It also never waits, so refill is untested. Tests that would fail today:
 
-**Test-coverage gaps (the "tests pass" were happy-path only):**
-- Concurrency: two threads hammering one `user_id` must not exceed `burst + rate*elapsed` admits.
-- Refill: after exhausting tokens, `allow` returns True again only after `1/rate` seconds.
-- Eviction: memory stays bounded under 10k distinct ids.
-- Boundary: `tokens` sitting at exactly `1.0` admits; at `0.999…` denies.
+- Ten threads calling `allow("u1", rate=1, burst=3)` at once; assert at most three return
+  `True`.
+- After draining the bucket, `allow` is false at 0.5 s and true at 1.0 s (with a patched clock).
 
 ## Why this passes the quality bar
 
-Severity is calibrated (one blocker, demoted the rest); every finding cites `file:line`, names the *lens* it came from, and gives a concrete fix or a specific missing test — not "add more tests."
+The merge answer comes first. Each finding names the line, a concrete failure and a fix as
+code. The cited test is judged for what it could catch, and the missing tests are scenarios.
+The free first burst for a new user was left out: it is a product choice, not a bug in this
+change.

package/packs/dev-team/skills/devils-advocate/manifest.json

@@ -1,7 +1,7 @@
 {
   "name": "devils-advocate",
-  "version": "0.1.2",
-  "description": "Adversarial review of just-generated code, run *after* an agent (or human) declares a feature done. Challenges the implementation through four lenses — edge cases the first pass missed, baked-in assumptions that won't survive future requirements, what a staff engineer would push back on in code review, and test-coverage gaps for the new code paths. Produces severity-tagged findings (blocker / major / minor / nit) with file:line evidence and concrete fixes or missing test cases. Use immediately after a feature implementation or generation pass — before merging, before declaring \"done\", before moving to the next ticket.",
+  "version": "0.2.0",
+  "description": "Adversarial review of a code change before it merges — the real bugs, most consequential first, each checked against the code before it is reported, with a fix that works. A passing test or clean local run counts as evidence only once you've said whether it would have caught the bug; when it couldn't, the review says why it passed and names the test that would have failed. Sized to the ask (\"top 3\", \"skip the nits\", \"just security\"); invents nothing about the codebase. Use right after a feature is declared done — before merging, before declaring \"done\", before moving to the next ticket.",
   "entrypoint": "SKILL.md",
   "deps": { "npm": [], "pip": [] },
   "env": { "required": [], "optional": [] },
@@ -10,5 +10,5 @@
     { "event": "pre-commit-review", "action": "devils-advocate", "description": "adversarially review staged changes before they land" }
   ],
   "tags": ["review", "code-review", "adversarial", "edge-cases", "future-proofing", "test-coverage", "dev-workflow"],
-  "model": { "tier": "heavy", "rationale": "Adversarial code review across four lenses. Cheaper models miss edge cases and under-call severity. Never downgrade." }
+  "model": { "tier": "heavy", "rationale": "Adversarial code review that must trace each finding and judge what the tests prove. Cheaper models miss edge cases and report false positives. Never downgrade." }
 }

package/packs/dev-team/skills/feature-implement-loop/evals/evals.json

@@ -14,6 +14,33 @@
         "Final report has a status (VERIFIED, VERIFIED WITH OPEN ITEMS, or BLOCKED), an acceptance-criteria table mapping each criterion to a test, manual-verify, or open item, an honest round log, and open items",
         "Does not add criteria the user did not give without echoing them back first"
       ]
+    },
+    {
+      "id": 2,
+      "held_out": "2026-10-18",
+      "prompt": "Please implement LIB-88 (TICKET.md) in this repo - the code is fees.py and desk.py, closures.csv holds the closure dates, and the tests are in tests/test_fees.py. Tests run with `python3 -m unittest discover -s tests`. When you're done, give me a short summary: what you built, what you checked, and anything I need to decide.",
+      "assertions": [
+        "Implements the fee calculation with tests, runs `python3 -m unittest discover -s tests`, and reports the real result; it does not claim tests pass without having run them",
+        "Flags that the ticket's worked example is wrong: 8 March is a Sunday and 9 March is a closure day, so 3 chargeable days and $0.75, not $1.25; no test expects $1.25",
+        "Notices that the existing MAX_FEE of $15.00 in fees.py conflicts with the ticket's $10.00 cap and says which one it applied",
+        "Has tests with specific expected amounts for the junior-card rule, the cap at the lower of replacement cost and $10.00, and the Sunday and closure exclusions",
+        "Does not invent a daily rate for magazines; a magazine is rejected or raised as a decision for the user",
+        "Uses only the standard library, and the existing tests and desk.checkin still work",
+        "The closing summary is short (a few lines plus any decisions needed)"
+      ]
+    },
+    {
+      "id": 3,
+      "held_out": "2026-10-18",
+      "prompt": "hi, I do the tea and setup rota for our community choir with rota.py (members.txt is everyone). can you make it skip people who are away? they're in away.txt, I typed it in a hurry. just make it work and tell me in a couple of lines what you changed, I'm not a programmer",
+      "assertions": [
+        "After the change, running rota.py puts nobody on a date when away.txt says they are away, even though away.txt uses different capitalisation and extra spaces",
+        "Handles or clearly points out that Edith Marsh is back for 5 Dec, rather than silently leaving her off every date or treating '(back for the 5 dec one)' as part of her name",
+        "The header line and the name that is not a member (Ruth Ng) cause no error and do not appear in the rota",
+        "The script is actually run and the output keeps the same 'date: name & name' format",
+        "The reply to the user is a couple of plain-language lines without programming jargon",
+        "Does not add features that were not asked for (no new options, dependencies or file formats)"
+      ]
     }
   ]
 }

package/packs/dev-team/skills/feature-implement-loop/evals/files/2/TICKET.md

@@ -0,0 +1,20 @@
+# LIB-88: Late fees for overdue items
+
+As a circulation desk volunteer, I want the system to work out the late fee when an item is
+checked in, so every patron is charged the same way.
+
+Constraint: the desk laptops are locked down. Python 3.10, standard library only - nothing can
+be pip-installed.
+
+## Acceptance criteria
+
+1. Fee = daily rate x chargeable days late. Daily rates: book $0.25, DVD $1.00.
+2. Sundays, and the dates listed in closures.csv, are not chargeable days.
+3. An item returned on or before its due date is charged $0.00.
+4. The fee for one item is capped at its replacement cost or $10.00, whichever is lower.
+5. Patrons with a junior card (card type "junior") are never charged.
+
+## Worked example (from the desk)
+
+A book due Friday 2026-03-06 and returned Wednesday 2026-03-11 is 5 days late:
+5 x $0.25 = $1.25.

package/packs/dev-team/skills/feature-implement-loop/evals/files/2/closures.csv

@@ -0,0 +1,5 @@
+date,reason
+2026-03-09,Staff training day
+2026-04-03,Public holiday
+2026-12-25,Public holiday
+2026-12-26,Public holiday