skilldrop-cli 0.16.35 → 0.16.37

package/packs/sre-oncall/skills/slo-spec/examples/ticket-booking.md

@@ -0,0 +1,88 @@
+# Worked example — Larkspur Theatres booking API
+
+## Input
+
+> Set SLOs for our booking API. Customers browse showtimes (about 900,000 requests a day) and
+> book seats (about 60,000 bookings a day). Booking calls our card processor, whose SLO is
+> 99.95%; the API returns 200 only after the charge succeeds. Box-office staff use a back-office
+> API, about 800 requests a day, 10:00–22:00. Last quarter from load balancer logs: browse
+> 99.96%, booking 99.87%, back-office 99.4%. Our CTO wants "four nines everywhere". The team
+> lead is Ines Calder; product owner is Raj Bhatt.
+
+## Script runs
+
+```bash
+python3 scripts/slo_math.py --batch slos.json        # browse, booking (with --dep 99.95)
+python3 scripts/slo_math.py --name "Box-office API (probe every minute)" \
+    --target 99.5 --window 30 --rpd 1440 --alerts "6:6h:1h:ticket"
+```
+
+Output, abridged to the rows used below:
+
+```
+Browse showtimes (availability): 99.9% over 30 days
+  = 43.2 minutes of full outage per window (43m 12s)
+  = 27,000 bad requests per window at 900,000 requests/day
+  14.4x  1h    5m     page      1.44%          2%              2d 2h       540        45
+  6x     6h    30m    page      0.6%           5%              5d          1,350      112
+  1x     3d    6h     ticket    0.1%           10%             30d         2,700      225
+
+Book seats (end to end): 99.8% over 30 days
+  = 86.4 minutes of full outage per window (1h 26m 24s)
+  = 3,600 bad requests per window at 60,000 requests/day
+  14.4x  1h    5m     page      2.88%          2%              2d 2h       72         6 *
+  Hard dependencies in series: 99.95% -> ceiling 99.95%; at their SLOs they spend 25% of this budget
+  - 14.4x 1h/5m: only 6 bad events in the 5m short window ... the short window will flap.
+
+Box-office API (probe every minute): 99.5% over 30 days
+  = 216 minutes of full outage per window (3h 36m)
+  6x    6h    1h     ticket    3%             5%              5d          10.8       1.8 *
+```
+
+## Reply
+
+**Four nines isn't reachable for booking: every booking needs the card processor, which only
+promises 99.95%, so 99.99% would be spent five times over by the processor alone. I've set
+booking at 99.8% (last quarter ran 99.87%), browse at 99.9%, and the back-office API at 99.5%
+on a synthetic probe, because 800 requests a day is too few to page on.**
+
+Assumptions: 30-day rolling window; traffic is the daily average you gave; the 1 s, 10 s and
+2 s thresholds are mine `[assumption]`.
+
+| Journey | SLI (good / valid) | Measured at | Target | Budget per 30 days |
+|---|---|---|---|---|
+| Browse showtimes | Good: non-5xx in < 1 s / Valid: all `GET /showtimes`, excluding health checks and bots | Load balancer | 99.9% | 43.2 min · 27,000 bad requests |
+| Book seats | Good: 200 or a bank decline within 10 s / Valid: all `POST /bookings`, excluding 4xx validation errors | Load balancer | 99.8% | 86.4 min · 3,600 bad bookings |
+| Back-office API | Good: probe gets a correct answer in < 2 s / Valid: one probe a minute | Probe | 99.5% | 216 min of failed probes |
+
+At its own SLO the card processor spends 25% of booking's budget. Going above 99.95% would need
+a second processor or an accept-now, charge-later path.
+
+| Journey | Burn | Long / short | Trips at | Budget at fire | Exhausts in | Action |
+|---|---|---|---|---|---|---|
+| Browse | 14.4× | 1 h / 5 m | 1.44% errors | 2% | 2 d 2 h | page |
+| Browse | 6× | 6 h / 30 m | 0.6% | 5% | 5 d | page |
+| Browse | 1× | 3 d / 6 h | 0.1% | 10% | 30 d | ticket |
+| Booking | 14.4× | 1 h / 15 m | 2.88% | 2% | 2 d 2 h | page |
+| Booking | 6× | 6 h / 30 m | 1.2% | 5% | 5 d | page |
+| Booking | 1× | 3 d / 6 h | 0.2% | 10% | 30 d | ticket |
+| Back-office | 6× | 6 h / 1 h | 3% of probes | 5% | 5 d | ticket, 10:00–22:00 |
+
+Booking's 5-minute short window would hold about 6 failed bookings at the threshold and
+flap, so it is 15 minutes. The back-office probe still has fewer than 2 failures expected in an
+hour, so it raises a ticket and never pages; staff will report an outage faster than a page.
+
+**Error-budget policy** (owner: Ines Calder; to be agreed with Raj Bhatt before it applies)
+
+| Budget state | What changes |
+|---|---|
+| 50% spent in the first 15 days | Reliability work tops the next sprint |
+| 100% spent | Only fixes and security patches ship until the 30-day budget is positive |
+| Card processor spent it | Raise it with the processor using the numbers; the freeze still covers risky changes |
+
+Exception: the CTO approves a release during a freeze in writing, recorded on the release ticket.
+
+To confirm: the three latency thresholds; whether bank declines reach the API as 200 or 402.
+
+Next step: add the browse and booking alerts to the alerting system and set up the
+back-office probe.

package/packs/sre-oncall/skills/slo-spec/manifest.json

@@ -0,0 +1,34 @@
+{
+  "name": "slo-spec",
+  "version": "0.1.0",
+  "description": "Define service level objectives for a service from its critical user journeys — each SLI a good-events / valid-events ratio with both sides defined and measured where the user feels it, a target set from user need and history (checked against dependency SLOs), the error budget computed in minutes and bad requests at current traffic, multi-window burn-rate alerts with the budget consumed when each fires, low-traffic handling, and an error-budget policy with owners. Use when the user asks to \"define SLOs\", \"write our SLIs\", \"how much error budget do we have\", \"what burn-rate alerts should page\", \"is 99.99% realistic\", or \"write an error-budget policy\".",
+  "entrypoint": "SKILL.md",
+  "deps": { "npm": [], "pip": [] },
+  "env": { "required": [], "optional": [] },
+  "related": ["delivery-metrics-report", "nfr-spec", "observability-plan", "runbook-generator"],
+  "handoff": [
+    {
+      "to": "observability-plan",
+      "when": "the SLIs are agreed and the telemetry to produce them doesn't exist yet",
+      "purpose": "Design the instrumentation, logs, traces and dashboards behind the SLIs.",
+      "fallback": "If observability-plan is absent, list per SLI the event source, the measurement point and the counter it needs, inline."
+    },
+    {
+      "to": "runbook-generator",
+      "when": "the burn-rate alerts are defined and each page needs a procedure",
+      "purpose": "Write the steps on-call follows when each alert fires.",
+      "fallback": "If runbook-generator is absent, write the first three checks for each paging alert inline."
+    }
+  ],
+  "tags": ["slo", "sli", "error-budget", "burn-rate", "sre"],
+  "permissions": {
+    "network": [],
+    "commands": [],
+    "files": "named-paths",
+    "notes": "slo_math.py does arithmetic only; it reads an optional --batch JSON file or stdin and writes only the -o path the user names."
+  },
+  "model": {
+    "tier": "standard",
+    "rationale": "SLI choice is bounded judgment over user journeys; error-budget and burn-rate arithmetic is done by its script."
+  }
+}

package/packs/sre-oncall/skills/slo-spec/reference.md

@@ -0,0 +1,113 @@
+# slo-spec reference — SLIs, budgets, burn rates, dependencies, low traffic, policy
+
+## SLI menu by journey type
+
+| Journey type | SLI (good / valid) | Typical good threshold |
+|---|---|---|
+| Request/response (API, page) | Availability: non-5xx, non-timeout responses / valid requests | — |
+| Request/response | Latency: requests answered under T / valid requests | T from what users tolerate (search ~1 s, API ~300 ms) |
+| Asynchronous (queue, job) | End-to-end: items whose result is visible to the user within T of submission / valid submissions | T from the user's wait (order confirmed in 30 s) |
+| Data pipeline | Freshness: reads served data newer than T / valid reads | T from how stale the user can bear |
+| Data pipeline | Correctness: records passing a known-answer check / records checked | — |
+| Batch | Completion: runs finished by deadline / scheduled runs | the deadline the consumer needs |
+| Streaming / long session | Time-based: minutes with the stream healthy / minutes in use | — |
+
+Use two SLIs for a journey at most (availability and latency is the usual pair). Latency is
+always "share under a threshold", never an average or a bare percentile value.
+
+## Defining "valid"
+
+Write the exclusions explicitly. The usual list:
+
+- **Exclude:** client errors the user caused (4xx validation, 401 bad credentials, 404 for
+  genuinely missing items), business declines that are correct outcomes (a card the bank
+  declines), health checks, synthetic and load-test traffic, requests from blocked bots.
+- **Keep as valid (and bad):** 5xx, timeouts, 429s our own limits caused, a 200 carrying an
+  error body, a 202 whose job never completes, a cancelled request the user gave up on after
+  the threshold.
+- **State the rule once** and apply it to both numerator and denominator. An SLI whose
+  numerator and denominator come from different sources drifts.
+
+## Measurement points and what each misses
+
+| Point | Sees | Misses |
+|---|---|---|
+| Client / real-user beacon | What the user experienced, including network and rendering | Users who never load the beacon; noisy, needs sampling care |
+| Synthetic probe | A fixed, known rate of events; works at zero traffic | Real users' paths and data |
+| Load balancer / edge logs | Every request, total server-side time including queueing in front of the app | Client network, DNS, CDN failures before the LB |
+| App server timer | Handler time | Time waiting in the accept queue, thread pool or message queue |
+| Worker timer behind a queue | Processing time only | The queue wait — the part that grows under load |
+
+On an async path, stamp the time the request reached the edge and measure to the moment the
+result is visible to the user. Queue depth is a cause signal; it is not the SLI.
+
+## Error-budget arithmetic
+
+- Budget fraction `b = 1 − target` (99.9% → 0.001).
+- Minutes of full outage per window = `b × window_days × 1440`. 99.9% / 30 d = 43.2 min;
+  99.9% / 28 d = 40.32 min; 99.95% / 30 d = 21.6 min; 99.99% / 30 d = 4.32 min.
+- Bad requests per window = `b × requests_per_day × window_days`.
+- Minutes and requests agree only under even traffic. A peak hour at 3× average burns 3×
+  faster in budget terms than the minutes suggest.
+
+## Burn-rate alerts
+
+Burn rate `B` = observed error rate / `b`. At 1× the budget lasts exactly the window.
+
+- Error rate that trips an alert = `B × b`.
+- Budget consumed when it fires = `B × long_window_hours / (window_days × 24)`.
+- Time from a full budget to exhaustion = `window_days × 24 / B` hours.
+- Expected bad events in a window at the threshold = `B × b × requests_per_hour × window_hours`.
+  Under ~10, the window fires on a handful of failures.
+
+Default set (Google SRE workbook, "Alerting on SLOs", for a 30-day window):
+
+| Burn | Long | Short | Budget at fire | Exhausts in | Action |
+|---|---|---|---|---|---|
+| 14.4× | 1 h | 5 m | 2% | 50 h | page |
+| 6× | 6 h | 30 m | 5% | 5 d | page |
+| 1× | 3 d | 6 h | 10% | 30 d | ticket |
+
+Fire only when both windows exceed the rate: the long one proves it's significant, the short
+one proves it's still happening, so the alert resets within minutes of the fix. A burn rate
+above `1 / b` can never fire (14.4× on a 95% target needs a 72% error rate; at 90%, 144%).
+
+## Dependency ceilings
+
+Hard dependencies called in series multiply: two at 99.9% and 99.95% give a ceiling of
+`0.999 × 0.9995 = 99.85%`. A target above the ceiling needs one of:
+
+- a second provider or replica in parallel (`1 − (1 − a)²` for two independent copies at `a`),
+- a cache or stale-read fallback so the dependency's failure isn't the user's failure,
+- an async path (accept now, retry later) where the journey tolerates delay.
+
+Share of the budget a dependency spends at its own SLO = `(1 − ceiling) / b`. At 50% or
+more, the service's own failures have half the room the number suggests.
+
+## Low-traffic journeys
+
+A ratio SLI needs volume. When the expected bad events per alert window are under ~10:
+
+1. **Lengthen the SLO window** (90 days) so the budget holds more than a few events.
+2. **Lengthen alert windows** and accept slower detection; say how much slower.
+3. **Add a synthetic probe** at a fixed rate (one a minute = 1,440 events/day) and set the
+   SLI on probes, or on probes plus real requests.
+4. **Alert only when someone uses it:** ticket within working hours, page never, if the users
+   are staff who will report a failure themselves.
+5. **Group** several low-traffic endpoints with the same users into one SLI.
+
+State which and why; don't silently reuse the high-traffic paging set.
+
+## Error-budget policy skeleton
+
+| Budget state | What changes | Who decides |
+|---|---|---|
+| < 50% spent | Normal releases | Team |
+| ≥ 50% spent before half the window | Reliability work goes to the top of the next sprint; risky changes need review | Engineering lead |
+| 100% spent | Freeze: only fixes, security patches and reliability changes ship until the budget is positive over the window | Engineering lead; product owner informed |
+| Spent by one incident > 20% of budget | Postmortem with actions, tracked to done | Incident owner |
+| Spent by a dependency | Escalate to the dependency's owner with the numbers; the freeze still covers risky changes; revisit the target or the redundancy | Engineering lead |
+
+Plus: an **exception route** (who can approve shipping during a freeze, in writing, and how it
+is recorded), **what's exempt** (security patches, legal deadlines), **agreed in advance** by
+product and engineering with names or roles, and a **review date** for the targets themselves.