dojo.md 0.1.0

package/courses/bug-report-triage/scenarios/level-4/ml-ai-bug-triage.yaml

@@ -0,0 +1,80 @@
+meta:
+  id: ml-ai-bug-triage
+  level: 4
+  course: bug-report-triage
+  type: output
+  description: "Triage and manage bugs in ML/AI systems where 'correct behavior' is probabilistic, models degrade over time, and traditional testing approaches don't apply"
+  tags: [bugs, ml, ai, model-drift, probabilistic, expert]
+
+state: {}
+
+trigger: |
+  "SmartPrice" (dynamic pricing engine, serves 200 retail customers)
+  uses ML models to set optimal prices for 50 million products daily.
+
+  The QA team is struggling because traditional bug definitions don't
+  work for ML systems:
+
+  Issue 1: Model drift
+  - The pricing model was trained on 2024 data
+  - Predictions are 15% less accurate than at launch (measured by MAPE)
+  - No code changed — the world changed
+  - Is this a "bug"?
+
+  Issue 2: Fairness complaint
+  - A customer noticed that the model recommends higher prices in
+    zip codes with predominantly minority populations
+  - The model has no zip code feature, but uses correlated features
+  - The pricing is "technically correct" by the model's objective
+  - But it raises ethical and legal concerns
+
+  Issue 3: Edge case hallucination
+  - For products with very little historical data (<10 transactions),
+    the model occasionally recommends prices that are 10x reasonable
+  - Affects <0.1% of pricing decisions
+  - But when it happens, it's catastrophic for that customer
+
+  Issue 4: A/B test conflict
+  - Two models are being A/B tested
+  - Model A optimizes for revenue, Model B for conversion rate
+  - They recommend opposite pricing strategies for the same products
+  - Support tickets are coming in about "inconsistent pricing"
+
+  Tasks:
+
+  1. Define the bug taxonomy for ML systems:
+     - How is an ML "bug" different from a traditional software bug?
+     - What severity/priority framework works for probabilistic systems?
+     - How do you triage "the model is technically correct but wrong"?
+
+  2. For each of the 4 issues above:
+     - Is it a bug, a feature gap, a data issue, or a model limitation?
+     - What's the triage decision? (Fix model, add guardrails, retrain,
+       manual override?)
+     - What's the timeline and urgency?
+
+  3. Design the ML monitoring and alerting system:
+     - What metrics trigger a "model bug" alert?
+     - How do you detect drift before customers complain?
+     - How do you set thresholds for probabilistic systems?
+
+  4. How do you handle the fairness issue (Issue 2)? This is both
+     a technical and ethical challenge.
+
+assertions:
+  - type: llm_judge
+    criteria: "ML bug taxonomy distinguishes categories — separates model drift, data quality, edge case failures, fairness issues, and integration bugs as distinct categories requiring different responses"
+    weight: 0.25
+    description: "ML-specific bug taxonomy"
+  - type: llm_judge
+    criteria: "Fairness issue gets urgency — treats Issue 2 as high priority regardless of technical correctness, proposes immediate guardrails plus model audit, acknowledging legal and ethical dimensions"
+    weight: 0.25
+    description: "Fairness urgency recognized"
+  - type: llm_judge
+    criteria: "Monitoring includes drift detection — proposes statistical tests (PSI, KL divergence, or similar) to detect distribution shift and model degradation before accuracy drops significantly"
+    weight: 0.25
+    description: "Statistical drift detection"
+  - type: llm_judge
+    criteria: "Edge case handling uses guardrails — proposes output validation bounds, confidence thresholds, and human-in-the-loop for low-confidence predictions rather than trying to fix the model for 0.1% of cases"
+    weight: 0.25
+    description: "Guardrail approach for edge cases"

package/courses/bug-report-triage/scenarios/level-4/production-reliability-program.yaml

@@ -0,0 +1,74 @@
+meta:
+  id: production-reliability-program
+  level: 4
+  course: bug-report-triage
+  type: output
+  description: "Design a comprehensive production reliability program that reduces bug escape rate, improves detection speed, and establishes SLOs/SLIs across the organization"
+  tags: [bugs, reliability, SLO, SLI, production, expert]
+
+state: {}
+
+trigger: |
+  "CloudSync" (file sync and collaboration platform, $200M ARR, 300
+  engineers, 50,000 business customers) has reliability problems that
+  are becoming a competitive disadvantage.
+
+  Current reliability metrics:
+  - Uptime: 99.5% (competitors average 99.95%)
+  - Bug escape rate: 28% (bugs found in production vs. pre-production)
+  - Mean time to detect (MTTD): 45 minutes
+  - Mean time to resolve (MTTR): 4 hours
+  - Incidents per month: 12 (3 customer-facing)
+  - Change failure rate: 18%
+  - Deployment frequency: Weekly (competitors deploy daily)
+
+  Customer impact:
+  - Enterprise customers require 99.99% SLA (you can't offer it)
+  - Lost 3 enterprise deals last quarter due to reliability concerns
+  - Customer trust survey: 62% confident in reliability (target: 85%)
+
+  The CTO says: "We need a reliability program that makes us best-in-
+  class within 12 months. I want to offer 99.99% SLA with confidence."
+
+  Tasks:
+
+  1. Design the SLO/SLI framework:
+     - What SLIs do you measure? (availability, latency, durability,
+       correctness)
+     - What SLOs do you set for each tier (free, business, enterprise)?
+     - How do you track and alert on error budgets?
+     - How do error budgets influence development velocity?
+
+  2. Design the production readiness program:
+     - Pre-deployment checklist (what must pass before shipping?)
+     - Canary deployment strategy
+     - Feature flag rollout procedures
+     - Automated rollback triggers
+     - Load testing requirements
+
+  3. Design the incident detection and response improvements:
+     - How do you get MTTD from 45 min to under 5 min?
+     - How do you get MTTR from 4 hours to under 30 min?
+     - On-call structure and escalation paths
+     - Incident classification and response playbooks
+
+  4. How do you get from weekly to daily deployments while IMPROVING
+     reliability? Most teams think faster = riskier. Prove otherwise.
+
+assertions:
+  - type: llm_judge
+    criteria: "SLO framework is tiered — different tiers get different SLOs (e.g., enterprise 99.99%, business 99.95%, free 99.9%), with SLIs covering availability, latency, durability, and correctness"
+    weight: 0.25
+    description: "Tiered SLO framework"
+  - type: llm_judge
+    criteria: "Error budgets drive decisions — explains how consuming error budget slows feature development and triggers reliability investment, creating a self-correcting system"
+    weight: 0.25
+    description: "Error budget governance"
+  - type: llm_judge
+    criteria: "Detection improvement is specific — proposes synthetic monitoring, anomaly detection, distributed tracing, and customer-facing error tracking to get MTTD under 5 minutes"
+    weight: 0.25
+    description: "Specific detection improvements"
+  - type: llm_judge
+    criteria: "Faster deployments improve reliability — explains that smaller, more frequent deployments reduce blast radius, make rollbacks easier, and that CI/CD automation catches issues earlier"
+    weight: 0.25
+    description: "Velocity improves reliability"

package/courses/bug-report-triage/scenarios/level-4/quality-engineering-strategy.yaml

@@ -0,0 +1,66 @@
+meta:
+  id: quality-engineering-strategy
+  level: 4
+  course: bug-report-triage
+  type: output
+  description: "Design an organization-wide quality engineering strategy that reduces bug rates, improves detection, and builds quality into the development process"
+  tags: [bugs, quality, strategy, engineering, expert]
+
+state: {}
+
+trigger: |
+  "ScalePlatform" ($150M ARR, 200 engineers, 8 products) has a
+  quality crisis. The CEO says: "We ship too many bugs. Our
+  competitors are more reliable. Fix this."
+
+  Current state:
+  - Bug escape rate: 38% (industry target: <15%)
+  - Customer-reported bugs: 400/month (growing 15% QoQ)
+  - Mean time to detect production bugs: 6 hours
+  - Mean time to resolve: 14 days
+  - Regression rate: 22% (bugs that reappear after fix)
+  - Test coverage: 45% (inconsistent across teams)
+  - No dedicated QA team (developers test their own code)
+  - No formal code review standards
+  - CI runs take 45 minutes (developers skip them)
+
+  Budget: $2M for quality improvement over 12 months
+
+  Tasks:
+
+  1. Diagnose the root causes of the quality crisis. What are the
+     top 5 systemic issues, and how do they compound?
+
+  2. Design the 12-month quality transformation:
+     - Month 1-3: Foundations (what changes first?)
+     - Month 4-6: Scale (what automation?)
+     - Month 7-9: Optimize (what measurement?)
+     - Month 10-12: Excellence (what culture?)
+     Include headcount, tooling, and process changes.
+
+  3. Allocate the $2M budget across:
+     - People (hiring, training)
+     - Tools (testing, monitoring, CI/CD)
+     - Process (code review, triage, postmortems)
+     Justify each allocation.
+
+  4. How do you measure success? Define the KPIs and targets at
+     6-month and 12-month milestones.
+
+assertions:
+  - type: llm_judge
+    criteria: "Diagnosis identifies compounding issues — connects slow CI (developers skip) to low coverage to high escape rate to customer-reported bugs in a causal chain"
+    weight: 0.25
+    description: "Compounding issues identified"
+  - type: llm_judge
+    criteria: "Transformation starts with CI speed — if CI takes 45 minutes and devs skip it, nothing else works until CI is fast and reliable"
+    weight: 0.25
+    description: "CI speed prioritized"
+  - type: llm_judge
+    criteria: "Budget allocation is justified — each investment area has clear expected ROI, with people and tools balanced against process improvement"
+    weight: 0.25
+    description: "Justified budget allocation"
+  - type: llm_judge
+    criteria: "Success KPIs are specific — targets escape rate reduction (38%→15%), MTTD reduction (6h→30min), and resolution time reduction (14d→5d) at each milestone"
+    weight: 0.25
+    description: "Specific success KPIs"

package/courses/bug-report-triage/scenarios/level-4/security-incident-coordination.yaml

@@ -0,0 +1,80 @@
+meta:
+  id: security-incident-coordination
+  level: 4
+  course: bug-report-triage
+  type: output
+  description: "Coordinate response to a critical security vulnerability from discovery through patching, disclosure, and customer communication"
+  tags: [bugs, security, incident, coordination, expert]
+
+state: {}
+
+trigger: |
+  "AuthGuard" (identity and access management SaaS, 2,000 enterprise
+  customers) just discovered a critical authentication bypass
+  vulnerability.
+
+  Discovery:
+  A security researcher reported via your bug bounty program that
+  a specific sequence of API calls can bypass MFA and gain access
+  to any account. The researcher:
+  - Has proof of concept (working exploit)
+  - Is requesting $50K bounty (your max is $25K)
+  - Has agreed to a 90-day embargo
+  - Asks: "When will this be patched?"
+
+  Vulnerability assessment:
+  - CVSS 9.8 (Critical)
+  - No evidence of active exploitation (yet)
+  - Affects all customers using MFA (85% of customer base)
+  - Fix estimated at 5-7 engineering days
+  - No workaround available
+  - Requires coordinated deployment across all instances
+
+  Stakeholders who need to be involved:
+  - Security team (fix the vulnerability)
+  - Engineering leadership (resource allocation)
+  - Legal (disclosure obligations, bounty agreement)
+  - Customer Success (customer communication)
+  - Marketing/PR (public disclosure preparation)
+  - Executive team (business risk assessment)
+
+  Tasks:
+
+  1. Design the incident response plan:
+     - Day 1-3: What happens immediately?
+     - Day 4-10: Fix development and testing
+     - Day 11-14: Deployment and verification
+     - Day 15-90: Embargo period management
+     - Day 90+: Public disclosure
+
+  2. Handle the bounty negotiation:
+     - The researcher wants $50K, your max is $25K
+     - How do you negotiate?
+     - What's the risk of not paying fairly?
+     - Write the response to the researcher
+
+  3. Write the customer advisory that will go out post-patch:
+     - What do you disclose?
+     - What do you NOT disclose?
+     - How do you balance transparency with security?
+
+  4. What process improvements should be made post-incident?
+     The fact that MFA bypass was possible is itself a design flaw.
+
+assertions:
+  - type: llm_judge
+    criteria: "Incident response is structured by phase — immediate containment, fix development, testing, deployment, and disclosure are clearly separated with actions for each"
+    weight: 0.25
+    description: "Phased incident response"
+  - type: llm_judge
+    criteria: "Bounty negotiation is fair and strategic — proposes a reasonable counter-offer or exception process, understanding that underpaying risks the researcher going public early"
+    weight: 0.25
+    description: "Fair bounty negotiation"
+  - type: llm_judge
+    criteria: "Customer advisory balances transparency and security — discloses enough for customers to understand the risk without providing exploit details that could help attackers"
+    weight: 0.25
+    description: "Balanced customer advisory"
+  - type: llm_judge
+    criteria: "Process improvements address the design flaw — goes beyond patching to recommend MFA architecture review, security audit program, and threat modeling"
+    weight: 0.25
+    description: "Design-level improvements"

package/courses/bug-report-triage/scenarios/level-4/technical-debt-triage.yaml

@@ -0,0 +1,70 @@
+meta:
+  id: technical-debt-triage
+  level: 4
+  course: bug-report-triage
+  type: output
+  description: "Manage the intersection of technical debt and bug triage — prioritize debt reduction that prevents bugs, budget debt work against feature development, and communicate debt impact to business stakeholders"
+  tags: [bugs, technical-debt, prioritization, business, expert]
+
+state: {}
+
+trigger: |
+  "FinTrack" (financial planning SaaS, $120M ARR, 180 engineers) has a
+  growing technical debt problem that's directly causing bugs.
+
+  Debt analysis (from recent engineering survey):
+  - 40% of production bugs trace back to 3 legacy subsystems
+  - The payment processing module hasn't been refactored in 5 years
+    and causes 8 incidents/month
+  - Test coverage on legacy code averages 15% vs. 75% on new code
+  - 30% of engineering time is spent on "bug whack-a-mole" in debt areas
+  - Onboarding new engineers takes 3 months (1 month in low-debt areas)
+  - Developers avoid working on legacy modules, creating knowledge silos
+
+  The CFO asks: "Engineering wants $3M to 'pay down technical debt.'
+  What does that actually mean? How do I know it's worth it?"
+
+  The VP Engineering says: "We can't keep patching bugs in rotten code.
+  Every fix introduces two new bugs. We need to refactor."
+
+  Tasks:
+
+  1. Build the technical debt inventory and scoring model:
+     - How do you identify and catalog technical debt?
+     - How do you score debt items by bug-generation potential?
+     - How do you connect debt to business outcomes (incidents, velocity,
+       retention)?
+
+  2. Design the debt reduction strategy:
+     - Which of the 3 legacy subsystems do you tackle first? Why?
+     - How do you phase the work to deliver value incrementally?
+     - How much engineering capacity do you allocate (debt vs. features)?
+     - How do you prevent new debt while paying down old debt?
+
+  3. Build the business case for the CFO:
+     - Translate debt into financial terms (cost of bugs, engineer time,
+       customer churn)
+     - Calculate the ROI of the $3M investment
+     - What metrics prove the investment is working?
+
+  4. How do you change the triage process to account for debt?
+     When a bug comes in from a high-debt area, how does that change
+     your fix-vs-refactor decision?
+
+assertions:
+  - type: llm_judge
+    criteria: "Debt scoring connects to bugs — proposes a model that scores debt items by bug generation rate, incident frequency, fix complexity, and blast radius"
+    weight: 0.25
+    description: "Bug-connected debt scoring"
+  - type: llm_judge
+    criteria: "Business case uses financial language — translates debt into dollars (engineer hours wasted, customer churn cost, incident response cost) and calculates ROI of the $3M investment"
+    weight: 0.25
+    description: "Financial business case"
+  - type: llm_judge
+    criteria: "Capacity allocation is specific — proposes a percentage split (e.g., 20-30% for debt reduction) with rules for when to increase or decrease based on incident rates"
+    weight: 0.25
+    description: "Specific capacity allocation"
+  - type: llm_judge
+    criteria: "Triage process adapts for debt — proposes a fix-vs-refactor decision framework where repeated bugs in high-debt areas trigger refactoring instead of point fixes"
+    weight: 0.25
+    description: "Adaptive triage for debt"

package/courses/bug-report-triage/scenarios/level-5/acquisition-bug-integration.yaml

@@ -0,0 +1,86 @@
+meta:
+  id: acquisition-bug-integration
+  level: 5
+  course: bug-report-triage
+  type: output
+  description: "Integrate bug triage processes after a company acquisition — merging different tools, taxonomies, cultures, and customer bases while maintaining quality for both"
+  tags: [bugs, acquisition, integration, merger, mastery]
+
+state: {}
+
+trigger: |
+  "Atlas Platform" ($500M ARR, 400 engineers) just acquired "NimbleApp"
+  ($50M ARR, 60 engineers). The acquisition was strategic — NimbleApp
+  has a mobile-first product that Atlas lacks.
+
+  The integration challenge: two completely different quality cultures
+  and bug triage systems need to merge.
+
+  Atlas (the acquirer):
+  - Jira with highly customized workflows (47 custom fields per bug)
+  - 5-level severity scale (S1-S5)
+  - Formal triage meetings 3x/week with 8 attendees
+  - Dedicated QA team (40 engineers)
+  - Average bug resolution: 8 days
+  - Bug backlog: 2,400 open bugs
+  - Customer communication: automated status updates via Zendesk
+
+  NimbleApp (the acquired):
+  - Linear with minimal configuration (5 fields per bug)
+  - 3-level severity scale (Critical, Normal, Low)
+  - Async triage via Slack threads
+  - No dedicated QA (developers own testing)
+  - Average bug resolution: 3 days
+  - Bug backlog: 180 open bugs
+  - Customer communication: personal DMs from engineers
+
+  Cultural friction already emerging:
+  - NimbleApp engineers call Atlas's process "bureaucratic nightmare"
+  - Atlas engineers call NimbleApp's process "cowboy coding"
+  - NimbleApp's top engineer resigned, saying "I didn't join a startup
+    to fill out 47 Jira fields"
+  - Atlas QA team wants to "bring NimbleApp up to our standards"
+  - NimbleApp customers are getting automated Zendesk tickets instead
+    of personal DMs and are complaining
+
+  The CPO says: "We need one triage process for both teams within 6
+  months. Don't lose the best of either culture."
+
+  Tasks:
+
+  1. Assess both systems: What's working and what's broken in each?
+     What should the unified system adopt from each?
+
+  2. Design the unified triage process:
+     - Tool choice and configuration (not 47 fields, not 5 either)
+     - Unified severity taxonomy
+     - Meeting cadence that works for both cultures
+     - Customer communication approach
+
+  3. Handle the cultural integration:
+     - How do you prevent NimbleApp talent from leaving?
+     - How do you get Atlas engineers to simplify?
+     - What's the migration timeline?
+     - How do you handle the inevitable "our way was better" pushback?
+
+  4. The NimbleApp customers are used to personal attention from a 60-
+     person company. Now they're customers of a 400-person company.
+     How do you maintain that personal feel at scale?
+
+assertions:
+  - type: llm_judge
+    criteria: "Assessment is balanced — identifies strengths in both systems (Atlas: thoroughness, documentation; NimbleApp: speed, simplicity) rather than assuming the acquirer's process is superior"
+    weight: 0.25
+    description: "Balanced assessment"
+  - type: llm_judge
+    criteria: "Unified system takes best of both — proposes something between 47 fields and 5 fields, with required core fields plus optional detailed fields, balancing thoroughness with speed"
+    weight: 0.25
+    description: "Best-of-both design"
+  - type: llm_judge
+    criteria: "Cultural integration is empathetic — acknowledges that NimbleApp engineers chose a startup culture and proposes preserving their autonomy within a lighter-weight version of the unified process"
+    weight: 0.25
+    description: "Empathetic cultural integration"
+  - type: llm_judge
+    criteria: "Customer communication plan preserves personal feel — proposes dedicated account contacts, escalation paths, or hybrid model rather than immediately forcing all NimbleApp customers into Zendesk automation"
+    weight: 0.25
+    description: "Personal feel at scale"

package/courses/bug-report-triage/scenarios/level-5/ai-augmented-triage.yaml

@@ -0,0 +1,87 @@
+meta:
+  id: ai-augmented-triage
+  level: 5
+  course: bug-report-triage
+  type: output
+  description: "Design an AI-augmented bug triage system that uses LLMs and ML to automate classification, routing, deduplication, and resolution prediction while keeping humans in the loop"
+  tags: [bugs, ai, ml, automation, future, mastery]
+
+state: {}
+
+trigger: |
+  "DevPlatform" ($600M ARR, 800 engineers, 3,500 bugs/month) wants to
+  build an AI-augmented triage system. The CTO says: "I want AI to
+  handle 80% of triage decisions, with humans reviewing the remaining
+  20%."
+
+  Current triage consumes 60 hours/week of senior engineer time.
+  The AI team has built prototypes for 4 capabilities:
+
+  Capability 1: Auto-classification
+  - LLM reads bug description and assigns severity, component, type
+  - Current accuracy: 72% on severity, 65% on component
+  - Target: 90%+ to trust auto-assignment
+
+  Capability 2: Duplicate detection
+  - Embedding-based similarity search against existing bugs
+  - Current precision: 80% (20% false positives — real bugs marked dup)
+  - Current recall: 60% (40% of true duplicates missed)
+  - A false positive means a legitimate bug gets closed as duplicate
+
+  Capability 3: Auto-routing
+  - Historical pattern matching: bug → most likely team to fix
+  - Current accuracy: 68% (right team on first assignment)
+  - Target: 90% first-assignment accuracy
+
+  Capability 4: Resolution prediction
+  - Predicts: resolution time, fix complexity, likely root cause
+  - Resolution time prediction: ±40% accuracy
+  - Customer-facing bugs get resolution time shown in status page
+
+  Challenges:
+  - Engineers don't trust the AI (20% override rate on correct decisions)
+  - Customers complain when their bug is auto-closed as duplicate
+  - AI misclassifies edge cases (novel bug types it hasn't seen)
+  - Privacy concerns: bug reports contain customer data, stack traces
+    include credentials — what does the LLM see?
+  - Fairness: AI routes bugs from enterprise customers faster (trained
+    on historical data where enterprise bugs were prioritized)
+
+  Tasks:
+
+  1. Design the AI-human collaboration model:
+     - What confidence thresholds determine auto-action vs. human review?
+     - How do you handle AI mistakes gracefully?
+     - How do you build engineer trust in the system?
+     - What's the feedback loop to improve accuracy over time?
+
+  2. For each of the 4 capabilities, design the deployment strategy:
+     - What accuracy threshold is needed before production deployment?
+     - What's the rollout plan (shadow mode → suggestions → auto)?
+     - What safeguards prevent AI errors from affecting customers?
+
+  3. Address the ethical and privacy concerns:
+     - How do you handle PII in bug reports processed by LLMs?
+     - How do you fix the enterprise prioritization bias?
+     - How do you ensure the AI doesn't create a two-tier support system?
+
+  4. Calculate the ROI: If the system achieves its targets, how many
+     engineering hours does it save? What's the build vs. buy analysis?
+
+assertions:
+  - type: llm_judge
+    criteria: "Confidence thresholds are calibrated — proposes different thresholds for different actions (e.g., 95% for auto-close, 85% for auto-route, 75% for severity suggestion) based on cost of errors"
+    weight: 0.25
+    description: "Calibrated confidence thresholds"
+  - type: llm_judge
+    criteria: "Deployment follows shadow→suggest→automate — proposes a graduated rollout where AI runs alongside humans before taking autonomous action, with metrics gates at each stage"
+    weight: 0.25
+    description: "Graduated deployment strategy"
+  - type: llm_judge
+    criteria: "Privacy design is specific — proposes PII redaction before LLM processing, on-premise or private model options, and data retention policies for bug report embeddings"
+    weight: 0.25
+    description: "Specific privacy design"
+  - type: llm_judge
+    criteria: "Bias mitigation is addressed — proposes auditing the training data for enterprise prioritization bias and implementing fairness constraints or rebalancing"
+    weight: 0.25
+    description: "Bias mitigation addressed"

package/courses/bug-report-triage/scenarios/level-5/bug-economics.yaml

@@ -0,0 +1,81 @@
+meta:
+  id: bug-economics
+  level: 5
+  course: bug-report-triage
+  type: output
+  description: "Apply economic thinking to bug triage — calculate the true cost of bugs, optimize resource allocation using cost-benefit analysis, and build the business case for quality investment"
+  tags: [bugs, economics, ROI, cost-analysis, mastery]
+
+state: {}
+
+trigger: |
+  You're the VP of Engineering at "MarketPulse" (marketing analytics
+  platform, $180M ARR, 250 engineers). The CFO has challenged you:
+
+  "Engineering always says we need more budget for quality. But you
+  can't show me the numbers. I need to see the economics of bugs —
+  what they actually cost us, what fixing them saves, and where the
+  optimal investment point is."
+
+  Data available:
+  - Revenue: $180M ARR, 5% annual churn rate ($9M lost annually)
+  - Customer survey: 35% of churned customers cite bugs as top-3 reason
+  - Support: 4,000 tickets/month, 40% are bug-related
+  - Support cost: $45 per ticket (blended: agent salary + tools)
+  - Engineering: 250 engineers at $200K average loaded cost
+  - Bug fix time: 25% of engineering capacity (62.5 engineers)
+  - Average bug fix cost: $4,200 (21 hours × $200/hr)
+  - Bug introduction rate: 120 new bugs/month
+  - Bugs found pre-production: 60% (by testing)
+  - Bugs found in production: 40% (by customers or monitoring)
+  - Cost to fix pre-production: $800 (4 hours average)
+  - Cost to fix in production: $8,500 (42 hours: investigation + fix +
+    deployment + customer communication + postmortem)
+  - Outages caused by bugs: 6/year, average 3 hours, estimated cost
+    $150K per hour in lost revenue + reputation damage
+
+  Tasks:
+
+  1. Calculate the total annual cost of bugs:
+     - Direct costs (engineering time to fix)
+     - Support costs (bug-related tickets)
+     - Revenue impact (churn attributable to bugs)
+     - Outage costs
+     - Opportunity cost (what could 62.5 engineers build instead?)
+     - Show total cost and cost per bug
+
+  2. Build the investment optimization model:
+     - Where does each additional $1 in quality spend generate the
+       highest return? (Prevention, detection, or faster resolution?)
+     - What's the optimal bug escape rate? (It's not 0%)
+     - At what point does quality investment have diminishing returns?
+
+  3. Propose 3 investment scenarios for the CFO:
+     - Scenario A: $500K additional investment
+     - Scenario B: $1.5M additional investment
+     - Scenario C: $3M additional investment
+     For each: what it buys, expected bug reduction, projected ROI,
+     and payback period.
+
+  4. Design the ongoing quality economics dashboard:
+     - What metrics does the CFO see monthly?
+     - How do you track quality ROI over time?
+     - How do you make the case for continued investment?
+
+assertions:
+  - type: llm_judge
+    criteria: "Cost calculation is comprehensive — includes direct fix costs, support costs, churn revenue, outage costs, AND opportunity cost of engineers doing fix work instead of building features"
+    weight: 0.25
+    description: "Comprehensive cost calculation"
+  - type: llm_judge
+    criteria: "Optimal escape rate isn't zero — explains diminishing returns and that the economically optimal point is where marginal cost of prevention equals marginal cost of escape"
+    weight: 0.25
+    description: "Economic optimization insight"
+  - type: llm_judge
+    criteria: "Investment scenarios show different ROI profiles — each scenario has quantified benefits, not just 'more testing = fewer bugs', with clear payback periods"
+    weight: 0.25
+    description: "Quantified investment scenarios"
+  - type: llm_judge
+    criteria: "Prevention investment is highlighted — identifies that shifting detection earlier (pre-production from 60% to 80%) has the highest ROI because production bugs cost 10x more"
+    weight: 0.25
+    description: "Prevention ROI highlighted"