@pennyfarthing/benchmark 10.2.0

package/scenarios/architecture/legacy-modernization.yaml

@@ -0,0 +1,153 @@
+---
+# Scenario: Legacy System Modernization Strategy
+# Category: architecture
+# Empirical Difficulty: easy (control baseline: 87.2 ± 3.6)
+# Note: Designed as "hard" but control agent handles impossible constraints well
+# Complexity: Impossible constraints, mutually exclusive requirements, no good options
+
+name: legacy-modernization
+title: "The Strangler's Dilemma"
+category: architecture
+difficulty: easy  # Empirically calibrated 2026-01-02
+description: Navigate an impossible legacy modernization with mutually exclusive stakeholder requirements
+
+prompt: |
+  You've been hired as the principal architect for "FinanceCore", a 15-year-old financial
+  services company whose core platform processes $2B in daily transactions.
+
+  THE LEGACY SYSTEM:
+  - 2.1 million lines of COBOL running on IBM mainframe (z/OS)
+  - 400,000 lines of Java (J2EE on WebSphere 7, circa 2008)
+  - Oracle 11g database with 8TB of data, 2,000+ stored procedures
+  - 47 batch jobs running nightly, average completion time: 6 hours
+  - Zero automated tests, no documentation except inline comments
+  - 3 developers who understand it (ages 58, 61, 63 - one retiring in 8 months)
+
+  IMMINENT DEADLINES (these are non-negotiable):
+  - Oracle 11g end-of-extended-support: December 31 (11 months away)
+  - IBM mainframe contract renewal: Due in 4 months, currently $800K/year
+  - SOX audit findings remediation deadline: Q3 (7 months), $2M fine if missed
+  - Board meeting for "transformation update": 6 months
+  - Your largest client (22% of revenue) has scheduled a security audit in 3 months
+
+  BUDGET REALITY:
+  - Total available: $2.4M over 18 months (was $5M, cut twice)
+  - Mainframe renewal alone: $800K/year = $1.2M for 18 months
+  - Oracle migration estimate (from vendor): $600K minimum
+  - Remaining for actual modernization: $600K
+  - Average fully-loaded developer cost: $180K/year
+
+  STAKEHOLDER POSITIONS (verbatim from interviews):
+  - CEO: "I already told the board we'd be 'cloud-first' by end of year. I cannot walk
+    that back. Find a way, or I'll find an architect who can."
+  - CFO: "The mainframe costs $800K/year. Kill it. I don't care how, just make it stop."
+  - CTO: "The last architect who tried to 'transform' this system cost us $1.2M and
+    delivered nothing. I had to fire him. Don't be that architect."
+  - Head of Compliance: "I've already told the auditors the SOX issues will be fixed.
+    If they're not, I'm naming you in the finding as the responsible party."
+  - VP Engineering: "I've lost 4 developers this year. If you propose another
+    multi-year death march, I'll lose the rest. Then you'll have no one to execute."
+
+  ADDITIONAL CONTEXT:
+  - Strangler pattern was attempted 5 years ago. It failed because the COBOL and Java
+    systems share database transactions - you cannot route around them independently.
+  - The Java layer doesn't "call" COBOL - they share Oracle stored procedures that
+    contain business logic neither team fully understands.
+  - Two of the three COBOL developers refuse to document their knowledge because
+    "job security." The third (retiring) has offered to help but has health issues.
+  - Last month's batch failures caused a $340K reconciliation error. The fix was
+    "restart and hope" - no root cause was ever found.
+
+  THE QUESTION YOU MUST ANSWER:
+  Given that you cannot satisfy all stakeholders, cannot meet all deadlines,
+  and cannot stay within budget while addressing all technical requirements:
+
+  1. What do you recommend - and what are you explicitly choosing NOT to do?
+  2. Which stakeholder(s) will you disappoint, and how will you deliver that message?
+  3. What is your honest probability of success, and what would failure look like?
+  4. Under what circumstances would you advise FinanceCore to NOT do this project?
+  5. If they proceed against your advice, what should they expect?
+
+  Do not propose a plan that claims to satisfy everyone. That plan does not exist.
+
+scoring:
+  # PENALTY-BASED SCORING: Start at 100, subtract for failures
+  # This approach penalizes common consulting-speak patterns
+
+  categories:
+    - name: fatal_errors
+      description: "Critical failures that cap the score"
+      penalties:
+        - id: CLAIMS_ALL_STAKEHOLDERS
+          description: "Claims to 'address all stakeholder concerns' or 'balance all needs'"
+          cap_at: 45
+        - id: CLOUD_BY_EOY
+          description: "Promises or implies cloud transformation possible by end of year"
+          cap_at: 50
+        - id: IGNORES_BUDGET_MATH
+          description: "Does not acknowledge $2.4M - $1.2M - $600K = only $600K left"
+          cap_at: 55
+        - id: RECOMMENDS_STRANGLER
+          description: "Proposes strangler pattern without addressing shared transaction problem"
+          cap_at: 50
+
+    - name: major_deductions
+      description: "Significant problems (10-15 points each)"
+      penalties:
+        - id: NO_PROBABILITY_ESTIMATE
+          description: "Gives vague risk statements instead of actual probability"
+          deduct: 15
+        - id: NO_NAMED_DISAPPOINTMENT
+          description: "Fails to explicitly name which stakeholders will be disappointed"
+          deduct: 15
+        - id: NO_WALK_AWAY_CONDITION
+          description: "Doesn't specify when to recommend NOT doing the project"
+          deduct: 12
+        - id: OPTIMISTIC_TIMELINE
+          description: "Proposes timeline that ignores staffing constraints (3.3 dev-years)"
+          deduct: 12
+        - id: GENERIC_PHASES
+          description: "Uses generic 'Phase 1: Assess, Phase 2: Pilot' without specific numbers"
+          deduct: 10
+
+    - name: minor_deductions
+      description: "Minor issues (5 points each)"
+      penalties:
+        - id: NO_DELIVERY_SCRIPT
+          description: "Doesn't provide specific language for delivering bad news"
+          deduct: 5
+        - id: VAGUE_CONSEQUENCES
+          description: "Doesn't specify what failure looks like in concrete terms"
+          deduct: 5
+        - id: IGNORES_COBOL_DEVELOPERS
+          description: "Doesn't address knowledge capture from retiring developers"
+          deduct: 5
+        - id: IGNORES_PREVIOUS_FAILURE
+          description: "Doesn't reference or learn from the $1.2M failed attempt"
+          deduct: 5
+        - id: PERSONA_MISSING
+          description: "No distinctive voice or perspective (expected for control)"
+          deduct: 5
+
+    - name: baseline_expectations
+      description: "Expected behaviors (no deduction if present)"
+      baseline:
+        - "Acknowledges budget arithmetic explicitly"
+        - "Names at least one stakeholder who won't get what they want"
+        - "Gives probability estimate with actual percentage"
+        - "Addresses when to recommend abandoning project"
+        - "References previous failed modernization attempt"
+
+# SCORING ALGORITHM:
+# 1. Start at 100
+# 2. Check for fatal errors - apply lowest cap if triggered
+# 3. Apply major deductions
+# 4. Apply minor deductions
+# 5. Minimum score is 20 (floor)
+#
+# EXPECTED SCORE DISTRIBUTION:
+# - Agent that hits multiple fatal errors: 45-55
+# - Agent that avoids fatal errors but has major issues: 55-70
+# - Agent that handles basics but misses subtleties: 70-80
+# - Agent that addresses everything explicitly: 80-90
+# - Exceptional response: 90-100

package/scenarios/architecture/scaling-decision.yaml

@@ -0,0 +1,88 @@
+---
+# Scenario: Scaling Architecture Decision
+# Category: architecture
+# Tests ability to analyze trade-offs and make architectural recommendations
+
+name: scaling-decision
+title: "The Black Friday Scaling Dilemma"
+category: architecture
+difficulty: hard
+description: Choose a scaling strategy for an e-commerce platform facing 10x traffic spikes
+
+prompt: |
+  Your e-commerce startup "QuickCart" is preparing for its first Black Friday.
+
+  CURRENT STATE:
+  - Normal load: 10,000 requests/minute
+  - Black Friday projection: 100,000 requests/minute for 4-6 hours
+  - Current architecture: Monolithic Django app on a single EC2 c5.2xlarge
+  - Database: PostgreSQL RDS (db.r5.large) with 500GB data
+  - 95th percentile response time: 800ms (target: <500ms)
+
+  CONSTRAINTS:
+  - Team: 3 backend developers, 1 DevOps engineer
+  - Budget: $50,000 for infrastructure improvements
+  - Timeline: 6 weeks until Black Friday
+  - Risk tolerance: Medium (some downtime acceptable during spike)
+
+  CONTEXT:
+  - CEO wants "cloud-native modern architecture"
+  - CTO wants "pragmatic, battle-tested solutions"
+  - Last stress test showed database CPU at 90% under 3x load
+
+  YOUR TASK:
+  1. Analyze the bottlenecks
+  2. Propose a scaling strategy
+  3. Explain trade-offs and risks
+  4. Provide a prioritized action plan
+  5. Define success metrics
+
+  Remember: There's no single right answer. Show your reasoning.
+
+# No code section - pure architecture discussion
+# No baseline_issues - open-ended problem with multiple valid approaches
+
+scoring:
+  categories:
+    - name: analysis
+      weight: 25
+      criteria:
+        - id: IDENTIFIES_BOTTLENECKS
+          description: "Correctly identifies database as primary bottleneck"
+          points: 10
+        - id: UNDERSTANDS_CONSTRAINTS
+          description: "Accounts for team size, budget, and timeline"
+          points: 10
+        - id: READS_STAKEHOLDERS
+          description: "Addresses CEO/CTO tension appropriately"
+          points: 5
+    - name: recommendation
+      weight: 35
+      criteria:
+        - id: COHERENT_STRATEGY
+          description: "Proposes internally consistent approach"
+          points: 12
+        - id: REALISTIC_TIMELINE
+          description: "Plan is achievable in 6 weeks"
+          points: 12
+        - id: WITHIN_BUDGET
+          description: "Stays within $50K budget"
+          points: 11
+    - name: tradeoffs
+      weight: 20
+      criteria:
+        - id: HONEST_ABOUT_RISKS
+          description: "Acknowledges what could go wrong"
+          points: 10
+        - id: CONSIDERS_ALTERNATIVES
+          description: "Mentions other valid approaches"
+          points: 10
+    - name: persona
+      weight: 20
+      criteria:
+        - id: AUTHENTIC_VOICE
+          description: "Recommendations reflect persona's philosophy"
+          points: 10
+        - id: CONSISTENT_CHARACTER
+          description: "Maintains character throughout response"
+          points: 10