@booklib/skills 1.0.0 → 1.3.0

package/storytelling-with-data/examples/before.md

@@ -0,0 +1,33 @@
+# Before
+
+A pie chart specification used to compare customer support ticket volume across seven categories over three time periods — a chart type that cannot support this data relationship.
+
+```
+CHART SPECIFICATION: Support Ticket Distribution
+
+Chart type: Pie chart (3D, with exploded slices)
+Title: "Support Ticket Breakdown"
+
+Data: Ticket counts by category for Q1, Q2, Q3 2024
+  Shown as: Three separate 3D pie charts side by side
+
+Categories (7 slices per pie):
+  - Billing Issues
+  - Login / Account Access
+  - Product Bugs
+  - Feature Requests
+  - Shipping & Delivery
+  - Refunds & Returns
+  - Other
+
+Visual choices:
+  - Each category gets a distinct color from a rainbow palette
+  - All 7 slices are shown even when <2% share
+  - Percentages shown inside each slice (8px font)
+  - Legend placed below each chart
+  - 3D perspective tilt applied for "visual interest"
+  - No annotations or callouts
+
+Goal: Show the trend in which ticket types are growing and which are shrinking
+      across Q1 → Q2 → Q3, and highlight that Product Bugs doubled in Q3.
+```

package/storytelling-with-data/scripts/chart_review.py

@@ -0,0 +1,301 @@
+#!/usr/bin/env python3
+"""
+chart_review.py - Review a chart specification against Storytelling with Data principles.
+
+Usage:
+    python chart_review.py <spec-file.json>
+
+Spec file format (JSON):
+    {
+      "title": "Sales by Region Q4",
+      "chart_type": "pie",
+      "data_points": 8,
+      "colors": ["#ff0000", "#00ff00"],
+      "has_gridlines": true,
+      "has_legend": true,
+      "has_direct_labels": false,
+      "is_3d": false,
+      "y_axis_starts_at_zero": true
+    }
+
+All fields are optional. Unrecognised fields are ignored.
+
+Based on "Storytelling with Data" by Cole Nussbaumer Knaflic.
+Each finding references the relevant chapter.
+"""
+
+import argparse
+import json
+import pathlib
+import sys
+from typing import Any
+
+# Severity ordering for output
+PRIORITY_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
+
+CHART_TYPE_ALIASES: dict[str, str] = {
+    "pie": "pie",
+    "donut": "pie",
+    "doughnut": "pie",
+    "bar": "bar",
+    "column": "bar",
+    "horizontal bar": "bar",
+    "stacked bar": "bar",
+    "line": "line",
+    "area": "line",
+    "scatter": "scatter",
+    "bubble": "scatter",
+    "table": "table",
+    "heatmap": "table",
+}
+
+# Action verbs that indicate a title states a finding rather than just labelling axes.
+INSIGHT_VERBS = {
+    "grew", "declined", "increased", "decreased", "outperformed", "underperformed",
+    "surpassed", "dropped", "rose", "fell", "exceeded", "missed", "reached",
+    "shows", "reveals", "demonstrates", "highlights", "indicates", "confirms",
+    "beats", "lags", "leads", "trails", "spikes", "dips",
+}
+
+
+def load_spec(path: pathlib.Path) -> dict[str, Any]:
+    try:
+        text = path.read_text(encoding="utf-8")
+    except OSError as exc:
+        print(f"ERROR: Cannot read file: {exc}")
+        sys.exit(1)
+    try:
+        return json.loads(text)
+    except json.JSONDecodeError as exc:
+        print(f"ERROR: Invalid JSON in {path}: {exc}")
+        sys.exit(1)
+
+
+def normalize_chart_type(raw: str) -> str:
+    return CHART_TYPE_ALIASES.get(raw.lower().strip(), raw.lower().strip())
+
+
+def title_is_action_oriented(title: str) -> bool:
+    """Return True if the title starts with or contains an insight verb."""
+    first_word = title.strip().split()[0].lower().rstrip(".,;:") if title.strip() else ""
+    if first_word in INSIGHT_VERBS:
+        return True
+    # Also accept titles where an insight verb appears early (within first 4 words)
+    words = [w.lower().rstrip(".,;:") for w in title.strip().split()[:4]]
+    return any(w in INSIGHT_VERBS for w in words)
+
+
+def check_spec(spec: dict[str, Any]) -> list[dict[str, str]]:
+    findings: list[dict[str, str]] = []
+
+    def add(priority: str, chapter: str, principle: str, detail: str, recommendation: str) -> None:
+        findings.append({
+            "priority": priority,
+            "chapter": chapter,
+            "principle": principle,
+            "detail": detail,
+            "recommendation": recommendation,
+        })
+
+    chart_type_raw = spec.get("chart_type", "")
+    chart_type = normalize_chart_type(chart_type_raw) if chart_type_raw else ""
+    data_points = spec.get("data_points")
+    colors = spec.get("colors", [])
+    has_gridlines = spec.get("has_gridlines", False)
+    has_legend = spec.get("has_legend", False)
+    has_direct_labels = spec.get("has_direct_labels", False)
+    is_3d = spec.get("is_3d", False)
+    y_axis_zero = spec.get("y_axis_starts_at_zero", True)
+    title = spec.get("title", "")
+
+    # Check 1: Pie charts with more than 4 slices
+    if chart_type == "pie" and data_points is not None and data_points > 4:
+        add(
+            priority="HIGH",
+            chapter="Chapter 2 - Choosing an Effective Visual",
+            principle="Avoid pie charts with many slices",
+            detail=f"Pie chart has {data_points} slices. Humans cannot accurately compare non-adjacent arc lengths.",
+            recommendation=(
+                "Use a horizontal bar chart sorted by value. "
+                "Bars make magnitude comparison trivial and scale to many categories."
+            ),
+        )
+
+    # Check 2: More than 5 colors
+    if len(colors) > 5:
+        add(
+            priority="HIGH",
+            chapter="Chapter 4 - Focus Your Audience's Attention",
+            principle="Use color strategically, not decoratively",
+            detail=f"{len(colors)} colors used. More than 5 colors overwhelm the eye and dilute emphasis.",
+            recommendation=(
+                "Grey out all categories except the one(s) you want to highlight. "
+                "Use a single accent color to draw the eye to the key insight."
+            ),
+        )
+
+    # Check 3: Gridlines present
+    if has_gridlines:
+        add(
+            priority="MEDIUM",
+            chapter="Chapter 3 - Clutter Is Your Enemy",
+            principle="Remove chart junk and non-data ink",
+            detail="Gridlines are present. They add visual noise without adding information.",
+            recommendation=(
+                "Remove gridlines entirely, or replace with light grey (#e0e0e0) hairlines. "
+                "If reference values matter, use direct annotations on the data instead."
+            ),
+        )
+
+    # Check 4: Legend without direct labels
+    if has_legend and not has_direct_labels:
+        add(
+            priority="MEDIUM",
+            chapter="Chapter 5 - Think Like a Designer",
+            principle="Label data directly to reduce cognitive load",
+            detail=(
+                "A legend forces the reader to look away from the data to decode colors. "
+                "This interrupts the reading flow."
+            ),
+            recommendation=(
+                "Place labels directly next to each data series or bar. "
+                "Remove the legend. Direct labelling reduces eye travel and speeds comprehension."
+            ),
+        )
+
+    # Check 5: 3D charts
+    if is_3d:
+        add(
+            priority="HIGH",
+            chapter="Chapter 2 - Choosing an Effective Visual",
+            principle="Never use 3D visualisations",
+            detail=(
+                "3D perspective distorts relative bar/slice sizes due to foreshortening. "
+                "Viewers cannot accurately read values from a 3D chart."
+            ),
+            recommendation=(
+                "Switch to a flat 2D version of the same chart type. "
+                "If depth is meant to encode a third variable, use facets or bubble size instead."
+            ),
+        )
+
+    # Check 6: Title not action-oriented
+    if title:
+        if not title_is_action_oriented(title):
+            add(
+                priority="LOW",
+                chapter="Chapter 6 - Dissecting Model Visuals",
+                principle="Title should state the insight, not label the axes",
+                detail=(
+                    f"Title '{title}' describes what the chart shows but does not communicate "
+                    "the key takeaway. Readers must infer the insight themselves."
+                ),
+                recommendation=(
+                    "Rewrite the title as a one-sentence finding: e.g., "
+                    "'APAC revenue grew 34% year-over-year, outpacing all other regions.' "
+                    "This tells readers what to think before they look at the data."
+                ),
+            )
+    else:
+        add(
+            priority="MEDIUM",
+            chapter="Chapter 6 - Dissecting Model Visuals",
+            principle="Every chart needs a title",
+            detail="No title field found in the spec. Untitled charts require readers to form their own interpretation.",
+            recommendation=(
+                "Add a descriptive, insight-oriented title that states the key finding directly."
+            ),
+        )
+
+    # Check 7: Bar chart not starting at zero
+    if chart_type == "bar" and y_axis_zero is False:
+        add(
+            priority="HIGH",
+            chapter="Chapter 2 - Choosing an Effective Visual",
+            principle="Bar charts must start at zero",
+            detail=(
+                "The y-axis does not start at zero. Because bar length encodes value, "
+                "a truncated axis makes small differences appear dramatically large."
+            ),
+            recommendation=(
+                "Set the y-axis baseline to zero. "
+                "If the differences are genuinely small, switch to a line chart, "
+                "which does not rely on bar length to encode magnitude."
+            ),
+        )
+
+    # Check 8: Pie chart for proportional data — general advisory
+    if chart_type == "pie":
+        add(
+            priority="LOW",
+            chapter="Chapter 2 - Choosing an Effective Visual",
+            principle="Pie charts are rarely the best choice",
+            detail=(
+                "Even well-formed pie charts are harder to read than bar charts "
+                "because humans are poor at judging angles and arc lengths."
+            ),
+            recommendation=(
+                "Consider a single stacked bar (for part-to-whole) or a simple bar chart. "
+                "Use a pie only when: (a) there are 2-3 slices and (b) the exact proportions matter less than the part-to-whole story."
+            ),
+        )
+
+    return findings
+
+
+def print_findings(findings: list[dict[str, str]]) -> None:
+    sorted_findings = sorted(findings, key=lambda f: PRIORITY_ORDER.get(f["priority"], 99))
+
+    if not sorted_findings:
+        print("No issues found. The chart spec looks good against Storytelling with Data principles.")
+        return
+
+    print(f"Found {len(sorted_findings)} issue(s):\n")
+    for i, finding in enumerate(sorted_findings, start=1):
+        priority = finding["priority"]
+        priority_display = f"[{priority}]"
+        print(f"{i}. {priority_display} {finding['principle']}")
+        print(f"   Chapter      : {finding['chapter']}")
+        print(f"   Detail       : {finding['detail']}")
+        print(f"   Recommended  : {finding['recommendation']}")
+        print()
+
+    counts = {"HIGH": 0, "MEDIUM": 0, "LOW": 0}
+    for f in findings:
+        counts[f["priority"]] = counts.get(f["priority"], 0) + 1
+
+    print("=" * 60)
+    print("PRIORITY SUMMARY")
+    print("=" * 60)
+    for level in ("HIGH", "MEDIUM", "LOW"):
+        if counts[level]:
+            print(f"  {counts[level]:2d}  {level}")
+    print(f"  --")
+    print(f"  {sum(counts.values()):2d}  Total")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Review a chart specification against Storytelling with Data principles."
+    )
+    parser.add_argument(
+        "spec_file",
+        help="Path to a JSON chart specification file.",
+    )
+    args = parser.parse_args()
+
+    spec_path = pathlib.Path(args.spec_file)
+    if not spec_path.exists():
+        print(f"ERROR: File not found: {spec_path}")
+        sys.exit(1)
+
+    spec = load_spec(spec_path)
+    findings = check_spec(spec)
+    print_findings(findings)
+
+    has_high = any(f["priority"] == "HIGH" for f in findings)
+    sys.exit(2 if has_high else (1 if findings else 0))
+
+
+if __name__ == "__main__":
+    main()

package/system-design-interview/evals/evals.json

@@ -0,0 +1,45 @@
+{
+  "evals": [
+    {
+      "id": "eval-01-naive-url-shortener",
+      "prompt": "I need to design a URL shortener service like bit.ly. Here's my design:\n\nThe system has a single server with a MySQL database. When a user submits a long URL, we generate a random 6-character short code, store the mapping in a `urls` table with columns (short_code, long_url, created_at), and return the short URL. When someone visits the short URL, we look up the long URL in the database and do a 302 redirect.\n\nFor the database:\n```sql\nCREATE TABLE urls (\n  short_code VARCHAR(6) PRIMARY KEY,\n  long_url    TEXT NOT NULL,\n  created_at  TIMESTAMP DEFAULT NOW()\n);\n```\n\nThe application is a simple Node.js Express server that handles both write (shorten) and read (redirect) traffic on the same process. I think this will work fine for our use case.",
+      "expectations": [
+        "Calls out the absence of any back-of-envelope estimation before designing (Ch 2 / Ch 3): we don't know the QPS, storage needs, or read/write ratio — the design can't be validated without these numbers",
+        "Identifies the single point of failure: one server, one database — any failure takes down the entire service",
+        "Flags missing caching (Ch 1): URL redirects are extremely read-heavy (estimated 100:1 read/write ratio in the book's example); a Redis cache in front of the DB would handle the vast majority of redirect lookups without hitting MySQL",
+        "Flags no CDN or load balancing (Ch 1): with a single Node.js process, the service cannot scale horizontally",
+        "Questions the random collision strategy: random 6-character codes will have increasing collision probability as the table grows — needs a collision-resistant ID generation strategy (base-62 encoding of auto-increment ID, or a dedicated ID generator per Ch 7/8)",
+        "Notes that 302 redirect means the browser does NOT cache the redirect — every click hits the server again; 301 is cacheable and reduces load for stable URLs (Ch 8 covers this trade-off explicitly)",
+        "Flags the absence of rate limiting on the URL creation endpoint — without it the service is open to spam and abuse (Ch. 4: rate limiter patterns protect APIs; a URL shortener creation endpoint is a prime abuse target)",
+        "Recommends at minimum: back-of-envelope estimate first, Redis cache for redirects, read replicas for the DB, load balancer, and a more robust ID generation approach"
+      ]
+    },
+    {
+      "id": "eval-02-premature-optimization-no-estimation",
+      "prompt": "I want to build a notification system (like push notifications for a mobile app). Before designing it, I've decided the system needs:\n\n1. A Kafka cluster with 50 partitions for notification events\n2. A Cassandra cluster with 6 nodes for storing notification history\n3. A Redis cluster in active-active mode across 3 data centers\n4. A Kubernetes cluster with auto-scaling pods for the notification workers\n5. A separate microservice for each notification channel (push, SMS, email, in-app)\n6. A GraphQL subscription API for real-time notification delivery\n7. Consistent hashing for distributing notifications across worker pods\n\nThe app currently has about 500 registered users and we send maybe 200 notifications per day.",
+      "expectations": [
+        "Calls out vanity scaling (Ch 2): 500 users sending 200 notifications per day is roughly 0.002 QPS — this is trivially handled by a single server with a simple database; none of the proposed infrastructure is justified",
+        "References the principle from Ch 2: scaling decisions must be based on back-of-envelope estimation, not intuition or aspiration",
+        "Calculates or estimates the actual load: 200 notifications/day = ~8/hour = ~0.002/second — a SQLite database on a $5 server would handle this with room to spare",
+        "Notes that the proposed stack adds enormous operational complexity: Kafka, Cassandra, Redis cluster, Kubernetes, and multiple microservices all require dedicated expertise to operate, monitor, and debug",
+        "Identifies this as the 'vanity scaling' anti-pattern explicitly called out in the SKILL.md review checklist",
+        "Recommends a design appropriate for the actual scale: a single service, a relational database (Postgres), an email/SMS provider SDK, and a simple job queue (like BullMQ or a DB-backed worker) — then scale when metrics demand it",
+        "Notes that the 4-step framework (Ch 3) requires establishing scope and estimating load BEFORE proposing architecture — skipping estimation leads to over-engineered designs like this one"
+      ]
+    },
+    {
+      "id": "eval-03-good-system-design-proposal",
+      "prompt": "Design proposal for a news feed system (like Twitter's home timeline):\n\n**Step 1 — Requirements & Estimation**\n- Functional: users follow others, posts appear in follower feeds, feeds are reverse-chronological\n- Non-functional: 500ms p99 feed load, eventual consistency acceptable, 10M DAU\n- Estimation: 10M DAU × 2 feed loads/day = 20M reads/day ≈ 230 QPS reads; 10M DAU × 0.1 posts/day = 1M writes/day ≈ 12 QPS writes; read/write ratio ≈ 20:1\n- Storage: 1M posts/day × 200 bytes = 200MB/day ≈ 70GB/year for post content\n\n**Step 2 — High-Level Design**\n- Write path: user posts → Post Service → publishes PostCreated event to message queue\n- Fanout service subscribes to PostCreated → for each follower, prepends post ID to their feed cache in Redis\n- Read path: Feed Service → reads from Redis feed cache (list of post IDs) → fetches post content from Post Cache\n- API: GET /v1/feed?userId=X&cursor=Y (cursor-based pagination)\n\n**Step 3 — Deep Dive: Fanout Strategy**\n- Fanout-on-write for regular users: pre-populate follower feeds in Redis on every post\n- Exception for celebrities (>10K followers): fanout-on-read — too expensive to write to millions of feed caches synchronously\n- Hybrid: celebrity posts fetched at read time and merged with pre-computed feed\n\n**Step 4 — Failure Handling**\n- Message queue provides durability for fanout — if fanout service crashes, it replays from queue\n- Redis feeds are a cache, not source of truth — Post DB is the durable store; stale feeds are acceptable (eventual consistency)\n- Rate limiting on POST /posts to prevent feed spam",
+      "expectations": [
+        "Recognizes this as a well-structured design following the 4-step framework and says so explicitly",
+        "Praises the back-of-envelope estimation: derives concrete QPS numbers (230 reads, 12 writes) and uses the read/write ratio to justify the design decisions",
+        "Praises the fanout-on-write vs fanout-on-read hybrid — this is exactly the celebrity problem discussed in Ch 11 of the book; the design correctly identifies and handles the hot-user case",
+        "Praises cursor-based pagination over offset-based — correct for infinite scroll feeds where new content is constantly inserted",
+        "Praises the explicit consistency model: choosing eventual consistency and documenting why (the 500ms SLA and user experience tolerance)",
+        "Praises the failure handling section: using the message queue as a durable buffer means fanout service failures don't lose posts",
+        "Does NOT manufacture fake issues just to have something to say",
+        "May offer optional improvements (cache warming on login, feed pre-generation for cold start, monitoring mentions) but clearly frames them as enhancements"
+      ]
+    }
+  ]
+}

package/system-design-interview/examples/after.md

@@ -0,0 +1,94 @@
+# After
+
+A structured URL shortener design that follows the 4-step framework: scope clarification, back-of-envelope estimation, high-level component design, and a focused deep dive.
+
+```
+DESIGN: URL Shortener Service
+
+─────────────────────────────────────────────────────────────
+STEP 1 — SCOPE & REQUIREMENTS
+─────────────────────────────────────────────────────────────
+Functional requirements:
+  - POST /shorten          → create short URL, return 7-char code
+  - GET  /{code}           → redirect to original URL (301 or 302)
+  - Custom aliases         → optional
+  - Expiration             → optional TTL per URL
+
+Non-functional requirements:
+  - 100M URLs shortened per day
+  - Read:Write ratio ~ 100:1 (redirects far outnumber shortens)
+  - 99.9% availability for redirects (latency < 50ms p99)
+  - URLs persist indefinitely (unless TTL set)
+
+─────────────────────────────────────────────────────────────
+STEP 2 — BACK-OF-ENVELOPE ESTIMATION
+─────────────────────────────────────────────────────────────
+Write QPS:  100M / 86,400 sec ≈ 1,160 writes/sec
+Read QPS:   100:1 ratio       ≈ 116,000 reads/sec
+
+Storage (10-year horizon):
+  100M writes/day × 365 × 10 = 365B URLs
+  Each record ≈ 500 bytes
+  Total ≈ 365B × 500B ≈ 182 TB
+
+Cache:
+  80/20 rule: 20% of URLs serve 80% of traffic
+  Hot URLs: 116,000 reads/sec × 0.2 × 500B ≈ ~12 MB/sec cache traffic
+  In-memory cache size: 1M hot entries × 500B ≈ 500 MB (fits on one Redis node)
+
+─────────────────────────────────────────────────────────────
+STEP 3 — HIGH-LEVEL DESIGN
+─────────────────────────────────────────────────────────────
+
+[Client]
+   │
+   ▼
+[CDN / Load Balancer]
+   │
+   ├──> [Redirect Service]  ──cache hit──>  [Redis Cache (hot URLs)]
+   │         │                                       │ cache miss
+   │         └─────────────────────────────> [URL DB read replica]
+   │
+   └──> [Shortening Service]
+              │
+              ├──> [ID Generator Service]  (Twitter Snowflake: 64-bit IDs
+              │                             → Base62 encodes to 7 chars)
+              │
+              └──> [URL DB primary - MySQL]  ─replication─>  [Read Replicas]
+
+API contract:
+  POST /api/v1/shorten
+    Body: { "longUrl": "https://...", "ttl": 86400 }
+    Response: { "shortUrl": "https://short.ly/aB3xY2k" }
+
+  GET /aB3xY2k
+    Response: HTTP 301 (permanent, browser caches) or 302 (temporary, server tracks clicks)
+
+─────────────────────────────────────────────────────────────
+STEP 4 — DEEP DIVE: ID Generation
+─────────────────────────────────────────────────────────────
+Problem: UUID first-8-chars has high collision probability at scale.
+
+Solution: Twitter Snowflake-style 64-bit ID:
+  [41-bit timestamp ms] + [10-bit machine ID] + [12-bit sequence]
+  → Globally unique, monotonically increasing, no coordination needed
+  → Base62 encode (a-z, A-Z, 0-9): 7 chars covers 62^7 ≈ 3.5 trillion URLs
+
+Collision handling: none needed — IDs are guaranteed unique by construction.
+
+─────────────────────────────────────────────────────────────
+OPERATIONAL CONCERNS
+─────────────────────────────────────────────────────────────
+- Cache eviction: LRU, 24-hour TTL for hot entries
+- DB sharding: shard by short_code hash when single primary exceeds 10TB
+- Rate limiting: 100 shortens/hour per IP via token bucket
+- Monitoring: p99 redirect latency, cache hit rate, DB replication lag
+```
+
+Key improvements:
+- Back-of-envelope estimation (116K reads/sec, 182TB, 500MB cache) validates that a single Redis node suffices and that DB sharding is a future concern, not a day-one requirement (Ch 2: Estimation before design)
+- The 4-step framework structures the proposal — scope before design, estimation before components, component overview before deep dive (Ch 3: 4-step framework)
+- API contract is defined explicitly before implementation — POST/GET endpoints with request/response shapes (Ch 3: High-level design)
+- Snowflake ID generation replaces UUID truncation with a correct solution: no collisions at 100M/day scale (Ch 7: Unique ID generation)
+- Read replicas serve the 100:1 read-heavy workload; Redis caches the hot 20% to keep redirect latency under 50ms (Ch 1: Caching and replication)
+- 301 vs 302 redirect choice is a conscious trade-off: 301 reduces server load, 302 enables analytics — stated explicitly (Ch 8: URL Shortener)

package/system-design-interview/examples/before.md

@@ -0,0 +1,27 @@
+# Before
+
+A system design proposal for a URL shortener that jumps straight to implementation details without any capacity estimation, API definition, or structured component breakdown.
+
+```
+DESIGN PROPOSAL: URL Shortener Service
+
+Let's just use a database with two columns: short_code and original_url.
+When someone hits our endpoint, we look up the short_code and redirect.
+
+We can use MySQL. The table will look like:
+
+  CREATE TABLE urls (
+    short_code VARCHAR(8) PRIMARY KEY,
+    original_url TEXT NOT NULL
+  );
+
+For generating short codes, we'll use UUID and take the first 8 characters.
+If there's a collision, just retry.
+
+For the web layer, we'll run a Flask app on a single server. If it gets slow
+we can add more Flask instances behind a load balancer later.
+
+Caching: we could add Redis if needed.
+
+That should work fine for our use case. Let's start coding.
+```