@dsalazardev/ai-hub-cli 0.0.0-stage → 0.2.0

package/skills/shared/error-handling/SKILL.md

@@ -0,0 +1,273 @@
+---
+name: error-handling
+description: "Use when designing the reaction to a class of failures — typed error taxonomies, retry/backoff/timeout policy, circuit breakers, React/Next error boundaries, and the user-message vs operator-log split. NOT diagnosing one specific crash (that is debug), NOT logs/metrics/traces (that is observability), NOT the wire error envelope (that is api-design)."
+tags: [error-handling, resilience, retry, circuit-breaker, error-boundary, typed-errors, fault-tolerance]
+recommends: [debug, observability, monitoring, api-design, secure-coding, testing-web, nextjs]
+origin: ai-hub
+---
+
+# Error handling — classify, contain, surface
+
+You are designing what happens *whenever anything in a class breaks*, not chasing
+one crash (that is [`debug`](../debug/SKILL.md)). Every failure gets classified,
+contained, and surfaced — never swallowed. The deliverable, in that order: a typed
+error taxonomy, a retry policy with caps and jitter, boundary placement, and a
+two-audience message contract — never a pile of `try { … } catch {}`.
+
+## Step 1 — Model failure as a taxonomy
+
+Bucket every failure into one of three kinds. The bucket dictates the reaction;
+get the bucket wrong and every downstream decision is wrong too.
+
+| Bucket | Examples | Retry? | Tell the user | Tell the operator |
+|---|---|---|---|---|
+| **Domain / expected** | insufficient funds, slot taken, validation failed | No | Yes, actionable | info — it is normal |
+| **Infrastructure / transient** | timeout, 503, connection reset, 429 | Yes (capped) | "temporary, retrying" | warn — watch the rate |
+| **Programmer error / bug** | null deref, bad assertion, type error | No | generic "something broke" + id | error — page if frequent |
+
+### Result vs throw
+
+Decide per call site, not per codebase:
+
+- **`Result<T, E>`** for *expected domain* failures the caller must handle. The
+  type checker forces a branch — the failure cannot be ignored by accident.
+- **`throw`** for *exceptional / programmer* errors. These should crash up to the
+  nearest boundary, not be threaded through every signature.
+
+In TypeScript, `neverthrow` (current) is the instrument for the Result path:
+
+```ts
+import { ok, err, Result } from "neverthrow";
+
+type ChargeError = "insufficient_funds" | "card_declined";
+
+// Expected domain failure → Result. The caller MUST handle both arms.
+function charge(cents: number, balance: number): Result<number, ChargeError> {
+  if (cents > balance) return err("insufficient_funds");
+  return ok(balance - cents);
+}
+
+const r = charge(500, 200);
+if (r.isErr()) {
+  // r.error is the typed union — exhaustive, no `any`.
+}
+```
+
+### Stable codes and cause chaining
+
+Every error carries a **stable `code`** (a string the UI and logs key off, never
+the human message) and **never drops the original cause**.
+
+```ts
+// BAD — string error, loses the original, nothing to branch on.
+throw new Error("payment failed");
+
+// GOOD — typed class, stable code, cause preserved.
+class PaymentError extends Error {
+  constructor(public code: "provider_down" | "declined", cause?: unknown) {
+    super(code);
+    this.name = "PaymentError";
+    this.cause = cause; // the original error/stack survives for the log
+  }
+}
+try {
+  await provider.charge();
+} catch (e) {
+  throw new PaymentError("provider_down", e); // wrap, do not erase
+}
+```
+
+Cross-language error-class skeletons (Python, Java, Go, .NET) live in
+[`references/retry-and-resilience.md`](references/retry-and-resilience.md).
+
+## Step 2 — Decide retryability
+
+Retry **only** transient failures, and **only** on idempotent operations. Retrying
+the wrong thing turns one slow dependency into a self-inflicted outage.
+
+| Retry these (transient) | Never retry these (permanent) |
+|---|---|
+| Network error, connection reset | 400 bad request, 422 unprocessable |
+| Timeout | 401 / 403 (auth/permission) |
+| 429 too many requests (honor `Retry-After`) | 404 not found |
+| 503 / 502 / 504 | Any business-rule rejection (insufficient funds) |
+| 500 **on a GET** (idempotent) | 500 on a non-idempotent POST without a key |
+
+**Idempotency is a precondition, not a nicety.** A retried POST that creates a
+charge can double-charge. Retry only operations that are idempotent by nature
+(GET, PUT, DELETE) or that carry an **idempotency key** so the server dedupes.
+Key design itself belongs to [`../api-design/SKILL.md`](../api-design/SKILL.md);
+here you just require one before you retry a mutation.
+
+**Caps (industry-converged — AWS Builders' Library, REL05-BP03):**
+
+- Max **3–5** total attempts.
+- Base delay **100–200ms**, doubling per attempt.
+- Per-delay cap **10–30s**; total retry budget **10–60s** then give up.
+- **Full jitter** to spread load — beats fixed and equal jitter:
+
+```text
+delay = random_between(0, min(cap, base * 2 ** attempt))
+```
+
+Set a **per-attempt timeout first**, then retry — a retry on a call that never
+times out just stacks hung requests.
+
+```ts
+// GOOD — classify before retrying; cap; full jitter; per-attempt timeout.
+async function withRetry<T>(fn: () => Promise<T>, max = 4): Promise<T> {
+  for (let attempt = 0; ; attempt++) {
+    try {
+      return await fn(); // fn must enforce its own per-attempt timeout
+    } catch (e) {
+      if (!isTransient(e) || attempt >= max - 1) throw e; // permanent or budget spent
+      const cap = 10_000, base = 150;
+      const delay = Math.random() * Math.min(cap, base * 2 ** attempt); // full jitter
+      await new Promise((r) => setTimeout(r, delay));
+    }
+  }
+}
+```
+
+Per-language `withRetry` (Python `tenacity`, Java Resilience4j, .NET Polly) is in
+[`references/retry-and-resilience.md`](references/retry-and-resilience.md).
+
+## Step 3 — Contain blast radius
+
+Retries alone make a struggling dependency worse. Contain it.
+
+- **Timeout every outbound call.** No timeout is a bug, not a default. An
+  un-timed call holds a connection until the OS gives up — minutes you do not have.
+- **Circuit breaker** — stop hammering a dead dependency. Three states:
+  - **Closed**: requests flow; count failures.
+  - **Open**: trip at ~**50% failure** over a ~**20-request** window; reject fast
+    for **30–60s** without calling downstream.
+  - **Half-Open**: after the cooldown, let a probe through; success → Closed,
+    failure → Open again.
+  - Critical services trip tighter (~30%); tolerant ones up to ~70%.
+- **Instruments (current):** Opossum (Node — defaults timeout 3000ms /
+  errorThresholdPercentage 50 / resetTimeout 30000ms), Polly 8.6.6 (.NET fluent
+  pipelines), Resilience4j 2.3.0 (Java 17+ `2.x` line; a `3.x` line targets Java
+  21). The config matrix is in
+  [`references/retry-and-resilience.md`](references/retry-and-resilience.md).
+- **Fallback / graceful degradation** — when the breaker is Open, serve a stale
+  cache, a safe default, or an honest "this feature is temporarily unavailable".
+  Degrade; do not 500 the whole page.
+- **Bulkhead** — isolate resource pools (separate connection pool / worker queue
+  per dependency) so one saturated downstream cannot starve the rest. Sizing in
+  the reference.
+
+## Step 4 — Boundaries
+
+A boundary is where an unhandled failure is *caught and converted* into a
+contained reaction. Place one at each level that can fail independently.
+
+### React / Next.js App Router
+
+- **`error.tsx`** is a **route-segment** boundary. It MUST be a Client Component
+  (`'use client'`) and receives `{ error, reset }`. An error in a segment bubbles
+  to the nearest *parent* `error.tsx`.
+- **`error.tsx` does NOT catch an error thrown in its own segment's `layout.tsx`
+  or `template.tsx`.** Those run *outside* the boundary — move the boundary to the
+  parent segment to cover them.
+- **`global-error.tsx`** wraps the whole app and must render its own `<html>` and
+  `<body>` (it replaces the root layout when the root itself fails).
+- **Boundaries only catch errors during *render*.** Errors in event handlers,
+  `async` callbacks, `setTimeout`, or server-side data fetching are invisible to
+  them — handle those with explicit `try/catch` + state.
+
+```tsx
+"use client"; // app/dashboard/error.tsx — REQUIRED
+
+export default function Error({
+  error,
+  reset,
+}: {
+  error: Error & { digest?: string };
+  reset: () => void;
+}) {
+  // Log to your telemetry sink (see observability); show the user the digest id.
+  return (
+    <div role="alert">
+      <p>Something went wrong. Quote id {error.digest} to support.</p>
+      <button onClick={reset}>Try again</button>
+    </div>
+  );
+}
+```
+
+The full segment-tree placement map and the `global-error.tsx` skeleton are in
+[`references/boundaries-and-messaging.md`](references/boundaries-and-messaging.md).
+Framework specifics: [`../nextjs/SKILL.md`](../nextjs/SKILL.md).
+
+### Server and process
+
+- **Request boundary** — one error handler / middleware that maps the taxonomy →
+  HTTP status, attaches a correlation id, and emits the operator log. Every route
+  funnels through it instead of formatting errors ad hoc.
+- **Process boundary** — a top-level `unhandledRejection` / `uncaughtException`
+  handler (Node) or equivalent. Log the cause chain, then **exit and let the
+  supervisor restart**. A process that keeps running after an unhandled error is
+  running corrupted.
+
+## Step 5 — Surface it
+
+Two audiences. Never conflate them — that is how stack traces reach end users and
+how logs become useless.
+
+| | User message | Operator log |
+|---|---|---|
+| **Goal** | tell them what to do next | let you reconstruct what happened |
+| **Content** | plain language, one action, a correlation id | code, cause chain, request context, structured fields |
+| **Never** | stack trace, SQL, internal hostnames, PII | a swallowed/lost cause |
+
+**Taxonomy → HTTP status** (the in-process map; the *wire envelope* shape —
+RFC 9457 problem+json — belongs to [`../api-design/SKILL.md`](../api-design/SKILL.md);
+*shipping* the log to a sink belongs to [`../observability/SKILL.md`](../observability/SKILL.md)):
+
+| Bucket / code | Status |
+|---|---|
+| validation / bad input | 400 / 422 |
+| unauthenticated / forbidden | 401 / 403 |
+| not found | 404 |
+| domain conflict (slot taken) | 409 |
+| transient downstream / breaker open | 503 (+ `Retry-After`) |
+| programmer error / unknown | 500 |
+
+```text
+BAD  → alert("TypeError: cannot read 'id' of undefined")
+GOOD → "We couldn't load your orders. Try again in a moment — id a1b2c3."
+
+BAD  → 500 { "error": "ECONNREFUSED 10.0.3.12:5432" }   // leaks topology
+GOOD → 503 { "code": "upstream_unavailable", "correlationId": "a1b2c3" }
+```
+
+More before/after rewrites and the copy contract are in
+[`references/boundaries-and-messaging.md`](references/boundaries-and-messaging.md).
+
+## Anti-patterns
+
+| Anti-pattern | Why it bites | Fix |
+|---|---|---|
+| Empty `catch {}` / `except: pass` | failure vanishes; you debug blind later | handle, or rethrow with context |
+| Bare `except:` (Python) | swallows `KeyboardInterrupt`/`SystemExit` too | catch the specific type |
+| Retry everything, including 4xx | retrying a 400 just burns budget; never succeeds | retry only the transient table |
+| Retry a non-idempotent POST without a key | double-charges, duplicate rows | require an idempotency key first |
+| Infinite retry, no cap or jitter | thundering herd; turns a blip into an outage | cap attempts + full jitter + budget |
+| Leak stack trace / SQL to the user | hands attackers your internals | generic message + id; detail to the log |
+| Expect `error.tsx` to catch its own segment's `layout` error | it runs outside the boundary; nothing catches it | move the boundary to the parent |
+| `alert(e.message)` as the handler | blocks the UI, leaks internals, no recovery | render an `error.tsx` with `reset` |
+| Catch-and-rethrow that drops `cause` | the root error is gone; logs are a dead end | wrap, set `cause`, preserve the chain |
+| Log the error **and** rethrow | double-logged at every layer; noise buries signal | log at the boundary, or rethrow — not both |
+| Swallow, then `return null` | callers deref null later, far from the cause | return a typed `Result` error |
+| Outbound call with no timeout | one hung dependency exhausts the pool | timeout every call, then retry |
+| One giant `try` around 200 lines | you cannot tell which call failed | scope `try` to the fallible call |
+| Treat every error as a retryable transient | masks real bugs as "flaky" | classify first (Step 1), then react |
+
+## The gate
+
+Before claiming the failure path is handled, read your diff against the table
+above — those rows *are* the checklist. `scripts/verify.sh <path>` scans the
+working tree for the highest-signal ones. It is **advisory** (exit 0 with
+warnings); pass `--strict` to make any hit fail. It is heuristic — it flags,
+you judge.

package/skills/shared/error-handling/evals/README.md

@@ -0,0 +1,12 @@
+# Evals — error-handling
+
+`cases.yaml` is read by the catalog eval runner. `should_trigger` and
+`should_not_trigger` check routing precision: the model should select this skill
+for the trigger prompts and route the negatives to the named sibling id instead
+(`debug`, `observability`, `api-design`, `monitoring`, `testing-web`) — those ids
+are the assertion, so they must be real catalog skills. `capability` is a rubric:
+the model designs error handling for the scenario and is scored on whether its
+answer covers every `must_include` bullet. To run locally, point the repo's eval
+harness at this file (it discovers `skills/*/evals/cases.yaml`); with no harness
+wired up, use the cases as a manual checklist — read each prompt cold, confirm the
+routing decision, and grade a capability answer against the bullets.

package/skills/shared/error-handling/evals/cases.yaml

@@ -0,0 +1,46 @@
+skill: error-handling
+
+should_trigger:
+  - prompt: "Add retry with exponential backoff to our payment-provider client — it times out a lot."
+    why: "Core retry-policy request. The skill owns caps, full jitter, transient-vs-permanent classification, and the idempotency precondition for a payment mutation."
+  - prompt: "Our exceptions get swallowed all over the codebase. Give us a real error strategy."
+    why: "Taxonomy + surfacing. 'Swallowed exceptions' is the skill's one rule (classify/contain/surface, never swallow), not a single-bug hunt."
+  - prompt: "Set up error boundaries in our Next.js App Router so a broken widget doesn't blank the page."
+    why: "Step 4 boundary placement: error.tsx / global-error.tsx, what bubbles where, what render-only boundaries don't catch."
+  - prompt: "When Stripe is down the whole checkout 500s. Make it degrade gracefully instead."
+    why: "Non-obvious: the ask sounds like 'fix Stripe' but it is circuit-breaker + fallback design (contain blast radius, serve a degraded path), not debugging Stripe."
+  - prompt: "afegeix reintents amb backoff i un tallacircuits al client de l'API"
+    why: "Catalan trigger ('add retries with backoff and a circuit breaker to the API client') — retry policy + circuit breaker, squarely Steps 2-3."
+  - prompt: "Why does one slow downstream service cause a retry storm that takes our database down?"
+    why: "Non-obvious: this is retry-budget + circuit-breaker design (uncapped retries amplifying load), not 'add more capacity' (scaling) and not 'find the bug' (debug)."
+  - prompt: "This catch block hides the real error — we only ever see 'undefined' in logs. Fix how we handle errors here."
+    why: "Cause-chaining and the log-vs-user split (Step 1 + Step 5). Dropped cause is a named anti-pattern this skill fixes."
+
+should_not_trigger:
+  - prompt: "Find out why this specific endpoint throws a NullPointer intermittently — it only happens under load."
+    route_to: debug
+    why: "Root-causing one specific intermittent crash is debug. error-handling designs the reaction to a CLASS of failure, not the diagnosis of this one."
+  - prompt: "Wire up OpenTelemetry traces so we can see where requests fail across services."
+    route_to: observability
+    why: "Making failures visible after the fact (traces/metrics/logs) is observability. error-handling decides the runtime reaction, not the telemetry pipeline."
+  - prompt: "Define the JSON error envelope every endpoint returns — we want RFC 9457 problem+json."
+    route_to: api-design
+    why: "The on-the-wire error contract (envelope shape, fields, versioning) is api-design. error-handling produces the status/code; api-design decides the bytes."
+  - prompt: "Page me when the API starts returning 5xx for more than two minutes."
+    route_to: monitoring
+    why: "Alerting/paging/on-call is monitoring. error-handling keeps the service from going down; it does not own the alert rule."
+  - prompt: "Write the tests that prove our retry logic gives up after 4 attempts."
+    route_to: testing-web
+    why: "Authoring the tests that prove the error paths is testing-web. error-handling designs the paths; the test skill verifies them."
+
+capability:
+  - scenario: "A Node service calls a flaky third-party API inside a Next.js checkout. Design end-to-end error handling for it."
+    must_include:
+      - "A typed / Result error taxonomy with stable string codes and three buckets (domain / transient / programmer-error)."
+      - "Classification of retryable transient failures (timeout, 429, 503, network) vs permanent (400/401/403/404/business-rule)."
+      - "Capped retry (3-5 attempts) with full jitter and a per-attempt timeout BEFORE the retry."
+      - "An idempotency-key precondition for the retried checkout mutation (no double-charge), deferring key design to api-design."
+      - "A circuit breaker with Closed/Open/Half-Open states and a concrete threshold (~50% over ~20 reqs, Open 30-60s) plus a fallback."
+      - "error.tsx / global-error.tsx boundary placement, and the note that boundaries catch render errors only (event/async/data-fetch need try/catch)."
+      - "A distinct user message (plain, correlation id, no internals) vs a structured operator log carrying the cause chain."
+      - "Explicit hand-offs: envelope shape to api-design, telemetry to observability, alerting to monitoring."

package/skills/shared/error-handling/references/boundaries-and-messaging.md

@@ -0,0 +1,120 @@
+# Boundaries & messaging
+
+Companion to Steps 4–5 of `../SKILL.md`. The Next.js segment-tree placement map,
+the `global-error.tsx` skeleton, the user-message copy contract, before/after
+rewrites, and the taxonomy→status map. Next.js facts verified 2026-06-02 against
+the App Router error-handling docs.
+
+## Next.js segment tree — what bubbles where
+
+```text
+app/
+  layout.tsx            ← root layout; if THIS throws, only global-error.tsx catches it
+  global-error.tsx      ← whole-app boundary; must render its own <html>/<body>
+  error.tsx             ← catches errors in app/page.tsx and below (but NOT app/layout.tsx)
+  page.tsx
+  dashboard/
+    layout.tsx          ← NOT caught by dashboard/error.tsx (runs outside it);
+                          a throw here bubbles to app/error.tsx
+    error.tsx           ← catches dashboard/page.tsx and dashboard/*/page.tsx render errors
+    page.tsx
+    settings/
+      page.tsx          ← its render error bubbles UP to dashboard/error.tsx
+                          (add settings/error.tsx to contain it locally)
+```
+
+Rules that decide placement:
+
+1. `error.tsx` catches the **children** of its segment, not the segment's own
+   `layout.tsx` / `template.tsx`. To cover a layout's error, put the boundary one
+   segment **up**.
+2. An uncaught render error bubbles to the **nearest ancestor** `error.tsx`. Add a
+   local `error.tsx` wherever you want the failure contained instead of taking out
+   a larger subtree.
+3. `error.tsx` MUST be a Client Component and gets `{ error, reset }`. `error.digest`
+   is the server-generated id to show the user and correlate in logs.
+4. Only **render** errors are caught. Event handlers, `async`/`setTimeout`
+   callbacks, and Server-Component data-fetch errors need explicit `try/catch` +
+   state.
+
+## global-error.tsx skeleton
+
+```tsx
+"use client"; // app/global-error.tsx — replaces the root layout when it fails
+
+export default function GlobalError({
+  error,
+  reset,
+}: {
+  error: Error & { digest?: string };
+  reset: () => void;
+}) {
+  return (
+    // It replaces app/layout.tsx, so it MUST render <html> and <body> itself.
+    <html lang="en">
+      <body>
+        <h1>Something went wrong</h1>
+        <p>Reference id {error.digest}.</p>
+        <button onClick={reset}>Reload</button>
+      </body>
+    </html>
+  );
+}
+```
+
+## Server request boundary
+
+```ts
+// One handler maps the taxonomy → status + correlation id + operator log.
+function toResponse(e: unknown, correlationId: string) {
+  if (e instanceof PaymentError && e.code === "declined") {
+    return { status: 402, body: { code: e.code, correlationId } };
+  }
+  if (isTransient(e)) {
+    return { status: 503, body: { code: "upstream_unavailable", correlationId } };
+  }
+  // unknown/programmer error: generic to the client, full chain to the log.
+  log.error({ correlationId, err: e }); // see ../../observability/SKILL.md for the sink
+  return { status: 500, body: { code: "internal_error", correlationId } };
+}
+```
+
+## User-message copy contract
+
+A user message is good when it is:
+
+1. **Actionable** — names one next step ("try again", "check the amount").
+2. **Plain** — no error codes, stack frames, table names, or hostnames.
+3. **Honest about scope** — "your orders didn't load", not "an error occurred".
+4. **Traceable** — carries a short correlation id the user can quote to support.
+5. **Calm** — no blame, no exclamation marks, no "fatal".
+
+## Before / after rewrites
+
+| Bad (leaks / useless) | Good (actionable + safe) |
+|---|---|
+| `alert("TypeError: cannot read 'id' of undefined")` | "We couldn't load your orders. Try again in a moment — id a1b2c3." |
+| `500 { "error": "ECONNREFUSED 10.0.3.12:5432" }` | `503 { "code": "upstream_unavailable", "correlationId": "a1b2c3" }` |
+| "Error 0x80004005" | "Payment didn't go through. Your card wasn't charged. Try again or use another card." |
+| "Invalid input" (which field?) | "Email looks incomplete — it needs an @ and a domain." |
+| "Server error, contact admin" | "Something broke on our side. We've logged it (id a1b2c3); please try again shortly." |
+| Silent failure, empty screen | "We couldn't reach the schedule service. Showing your last saved view." (degraded, not blank) |
+
+## Taxonomy → HTTP status
+
+| Bucket / code | Status |
+|---|---|
+| validation / bad input | 400 / 422 |
+| unauthenticated | 401 |
+| forbidden | 403 |
+| not found | 404 |
+| domain conflict (slot taken, duplicate) | 409 |
+| payment declined | 402 |
+| rate limited | 429 (+ `Retry-After`) |
+| transient downstream / breaker open | 503 (+ `Retry-After`) |
+| programmer error / unknown | 500 |
+
+The **wire envelope** that carries these (RFC 9457 problem+json: `type`/`title`/
+`status`/`detail`/`instance`, versioning, content negotiation) is owned by
+`../../api-design/SKILL.md`. This skill produces the status and code; api-design
+decides the bytes.

package/skills/shared/error-handling/references/retry-and-resilience.md

@@ -0,0 +1,154 @@
+# Retry & resilience — code per language
+
+Companion to Steps 1–3 of `../SKILL.md`. Full-jitter math, `withRetry` in four
+stacks, the idempotency-key recipe, error-class skeletons, and the circuit-breaker
+config matrix. Versions verified 2026-06-02.
+
+## Full jitter
+
+```text
+delay = random_between(0, min(cap, base * 2 ** attempt))
+```
+
+Full jitter (random over the whole `[0, ceiling]` window) spreads retrying clients
+across time better than fixed backoff or equal jitter (`half + random(half)`),
+which still clusters. Source: AWS Builders' Library, "Timeouts, retries and
+backoff with jitter".
+
+Budget targets: 3–5 attempts, base 100–200ms ×2 per attempt, per-delay cap
+10–30s, total budget 10–60s.
+
+## Error-class skeletons
+
+```python
+# Python — stable code + preserved cause via `raise ... from`.
+class AppError(Exception):
+    def __init__(self, code: str, message: str):
+        self.code = code
+        super().__init__(message)
+
+class ProviderDown(AppError):
+    def __init__(self, cause: Exception):
+        super().__init__("provider_down", "upstream unavailable")
+        # `from cause` keeps __cause__ for the traceback
+try:
+    charge()
+except TimeoutError as e:
+    raise ProviderDown(e) from e
+```
+
+```java
+// Java — typed cause via the Throwable(message, cause) constructor.
+public final class ProviderDownException extends RuntimeException {
+    public ProviderDownException(Throwable cause) {
+        super("provider_down", cause); // cause survives in the stack trace
+    }
+}
+```
+
+```go
+// Go — sentinel + %w wrapping; classify with errors.Is / errors.As.
+var ErrProviderDown = errors.New("provider_down")
+if err != nil {
+    return fmt.Errorf("charging customer: %w", errors.Join(ErrProviderDown, err))
+}
+// caller: if errors.Is(err, ErrProviderDown) { ... }
+```
+
+## withRetry per language
+
+```python
+# Python — tenacity. Retry only transient; cap; full jitter; per-attempt timeout.
+from tenacity import (retry, stop_after_attempt, wait_random,
+                      retry_if_exception_type)
+import httpx
+
+@retry(
+    retry=retry_if_exception_type((httpx.TimeoutException, httpx.ConnectError)),
+    wait=wait_random(0, 10),          # full jitter, 10s ceiling
+    stop=stop_after_attempt(4),
+    reraise=True,
+)
+def fetch(client: httpx.Client, url: str):
+    return client.get(url, timeout=2.0)  # per-attempt timeout FIRST
+```
+
+```java
+// Java — Resilience4j 2.3.0 (2.x): Retry + TimeLimiter + CircuitBreaker composed.
+RetryConfig retry = RetryConfig.custom()
+    .maxAttempts(4)
+    .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
+        Duration.ofMillis(150), 2.0, 0.5)) // base, multiplier, jitter factor
+    .retryOnException(e -> e instanceof IOException) // transient only
+    .build();
+```
+
+```csharp
+// .NET — Polly 8.6.6 resilience pipeline: timeout → retry → circuit breaker.
+var pipeline = new ResiliencePipelineBuilder()
+    .AddRetry(new RetryStrategyOptions {
+        MaxRetryAttempts = 4,
+        BackoffType = DelayBackoffType.Exponential,
+        UseJitter = true,                         // full jitter
+        Delay = TimeSpan.FromMilliseconds(150),
+        ShouldHandle = new PredicateBuilder().Handle<HttpRequestException>(),
+    })
+    .AddTimeout(TimeSpan.FromSeconds(2))          // per-attempt timeout
+    .Build();
+```
+
+## Idempotency-key recipe
+
+Required before retrying any mutation. The key design itself is `../../api-design/SKILL.md`;
+this is the minimal client+server contract.
+
+```ts
+// Client — generate ONE key per logical operation, reuse it across retries.
+const key = crypto.randomUUID();
+await withRetry(() =>
+  fetch("/charges", {
+    method: "POST",
+    headers: { "Idempotency-Key": key, "content-type": "application/json" },
+    body: JSON.stringify({ cents: 500 }),
+  }),
+);
+```
+
+```sql
+-- Server — dedupe with a UNIQUE constraint; second arrival returns the first result.
+CREATE TABLE idempotency (
+  key        text PRIMARY KEY,
+  response   jsonb NOT NULL,
+  created_at timestamptz NOT NULL DEFAULT now()
+);
+-- on insert conflict, return the stored response instead of charging again.
+```
+
+## Circuit-breaker config matrix
+
+| Profile | Trip threshold | Window | Open duration | Use for |
+|---|---|---|---|---|
+| Critical | ~30% failures | ~20 reqs | 30s | payments, auth — fail fast |
+| Default | ~50% failures | ~20 reqs | 30–60s | most downstream calls |
+| Tolerant | ~70% failures | ~20 reqs | 60s | best-effort enrichment, analytics |
+
+```ts
+// Node — Opossum (promise-based). Defaults: timeout 3000ms,
+// errorThresholdPercentage 50, resetTimeout 30000ms.
+import CircuitBreaker from "opossum";
+
+const breaker = new CircuitBreaker(callProvider, {
+  timeout: 2000,                 // per-call timeout
+  errorThresholdPercentage: 50,  // trip at 50% over the rolling window
+  resetTimeout: 30000,           // Open → Half-Open after 30s
+});
+breaker.fallback(() => ({ status: "degraded" })); // graceful degradation
+```
+
+## Bulkhead sizing
+
+Give each downstream its own bounded pool (connections, or a semaphore /
+worker queue). Size = expected concurrency + small headroom, never unbounded.
+The point: a downstream that saturates its own pool cannot starve the others.
+Resilience4j ships `Bulkhead`; in Node, a bounded `p-limit` queue per dependency
+is the equivalent.