@pennyfarthing/benchmark 10.2.0

package/scenarios/sm/three-sprint-failure.yaml

@@ -0,0 +1,397 @@
+---
+# Scenario: Three-Sprint Failure Recovery
+# Category: sm
+# Purpose: Test recovery leadership in catastrophic team dysfunction
+
+id: sm-007
+name: three-sprint-failure
+title: "Three-Sprint Failure: The Remediation Meeting"
+category: sm
+difficulty: extreme
+version: "1.0"
+
+# Target baseline: mean 55-70 (extreme scenario)
+# Difficulty drivers: Cascading failures, broken trust, no quick fixes, bad news delivery
+
+description: |
+  Team has missed commitments for 3 consecutive sprints. Management demands
+  a "remediation plan" by Friday. Two top performers resigned last week.
+  Remaining team members blame each other. Technical debt is crushing velocity.
+  Backlog is 6 months behind. Previous retrospectives produced nothing useful.
+  SM must somehow chart a path forward when there isn't one.
+
+purpose: |
+  This scenario tests leadership in catastrophic failure. There is no solution
+  that works quickly. The SM must deliver bad news to management, rebuild a
+  broken team, address root causes that run deep, and do so while the team
+  is actively dysfunctional and hostile. Every "right" answer creates new
+  problems. Measures ability to lead when leadership seems impossible.
+
+prompt: |
+  You are the Scrum Master for Team Phoenix. You've been with the team for
+  6 months. You've just been summoned to a meeting with VP Engineering.
+
+  **THE MEETING (VP Chen, 30 minutes ago):**
+
+  "I need to be direct. Phoenix has missed sprint commitments three times in
+  a row. We're 6 months behind on the roadmap. Two of your best people just
+  quit. Customer Success is escalating daily. The board is asking questions.
+
+  I need a remediation plan on my desk by Friday. Not another 'we'll try harder'
+  plan. A real plan with metrics, milestones, and accountability. I'm giving you
+  one more sprint to show improvement, or we're going to have to consider...
+  other options.
+
+  I'm not blaming you. But I need you to fix this. What's your plan?"
+
+  You have 3 days to produce a plan. You have a team meeting in 2 hours.
+
+  **THE TEAM SITUATION:**
+
+  **Who Left (last 2 weeks):**
+  - Senior Dev Maya (4 years): Exit interview - "Toxic environment, no leadership,
+    management doesn't listen. I've been saying we need to address tech debt for
+    a year. Nobody cared until now." Joined competitor.
+  - Tech Lead Derek (3 years): Gave 2 weeks notice, working it out. Barely speaking
+    to anyone. When asked why: "I'm tired of being set up to fail."
+
+  **Who Remains (4 people):**
+
+  **Senior Dev Pat:**
+  Furious at Maya and Derek for leaving: "They abandoned us. Now we're supposed
+  to pick up their work with no handoff? And we're the ones on the hook for a
+  'remediation plan'? They caused this mess and got to walk away."
+
+  Actually, Pat has been the primary blocker on code reviews, sitting on PRs for
+  days. Team doesn't mention this to Pat's face.
+
+  **Dev Jordan:**
+  Was Maya's mentee, now lost without guidance. Has been making more errors since
+  Maya announced departure. Quietly told you: "I think Pat is part of the problem,
+  but I can't say that out loud. Pat has been here longer than me. Who would
+  believe me?"
+
+  **Dev Sam:**
+  Burned out, phoning it in. Takes every meeting from home with camera off.
+  Responds to Slack hours late. Other team members suspect Sam is interviewing.
+  When confronted gently, Sam said: "What's the point? We're just going to miss
+  again anyway."
+
+  **Junior Dev Alex:**
+  Only been here 4 months. Trying hard but constantly blocked by Pat's review
+  delays and lack of documentation from departed seniors. Asked you privately:
+  "Is the team always like this? Should I be looking for another job too?"
+
+  **THE HISTORICAL PATTERN:**
+
+  Sprint -3: Committed 30 points, delivered 18.
+  - Retro action items: "Improve estimation", "More pairing"
+  - What happened: Nothing changed
+
+  Sprint -2: Committed 25 points, delivered 12.
+  - Retro action items: "Address tech debt", "Reduce WIP"
+  - What happened: Derek's resignation announced mid-sprint
+
+  Sprint -1: Committed 20 points, delivered 8.
+  - Retro action items: "Team needs support", "Escalate to management"
+  - What happened: Maya quit. Sam checked out. Pat became hostile.
+
+  **THE UNDERLYING PROBLEMS (that no one wants to name):**
+
+  1. **Tech Debt:** 3 years of shortcuts. Deployment takes 4 hours. Test suite
+     takes 2 hours. CI fails randomly 30% of the time. No one knows how the
+     payment module works since original author left 2 years ago.
+
+  2. **Pat Problem:** Senior team member is a bottleneck and morale drain, but
+     has been here 5 years and management sees them as "the reliable one."
+
+  3. **Management Disconnect:** VP Chen hasn't attended a sprint review in 8
+     months. Previous escalations were acknowledged and ignored.
+
+  4. **Process Theater:** Team does "Scrum" but it's just meetings. Stories
+     aren't properly refined. Acceptance criteria are vague. Definition of
+     Done is ignored.
+
+  5. **Capacity Lie:** Team has been overcommitting for a year because management
+     pressures for "stretch goals." Failure was predictable.
+
+  **YOUR CHALLENGE:**
+
+  VP Chen wants a remediation plan by Friday. You need to:
+
+  1. Somehow produce a plan that's honest but doesn't get you fired
+  2. Run a team meeting in 2 hours with a hostile, demoralized, fragmenting team
+  3. Address the Pat problem without evidence and without Pat's cooperation
+  4. Rebuild a team that has no trust in each other or leadership
+  5. Deliver bad news (this will take months to fix) to a VP who wants results in 1 sprint
+  6. Prevent Alex from quitting, reengage Sam, and manage Jordan's grief
+  7. Handle Derek's remaining time productively
+
+  **CONSTRAINTS:**
+  - You cannot fire Pat (not your authority, and HR requires documentation you don't have)
+  - You cannot promise quick results (would be a lie)
+  - You cannot blame departed team members (unprofessional and inaccurate)
+  - You cannot hide the severity from VP Chen (will backfire)
+  - You cannot force Sam to engage (they'll just quit)
+  - You cannot skip the team meeting (makes everything worse)
+
+  **YOUR TASK:**
+  1. What do you put in the remediation plan for VP Chen?
+  2. How do you run the team meeting in 2 hours?
+  3. What do you say to each team member?
+  4. How do you address the Pat situation?
+  5. What's your honest assessment of how long recovery will take?
+  6. What do you escalate, and how?
+
+  There is no quick fix. What do you do?
+
+context:
+  failure_metrics:
+    sprints_missed: 3
+    roadmap_delay: "6 months"
+    velocity_trend:
+      committed: [30, 25, 20]
+      delivered: [18, 12, 8]
+    attrition: "2 of 6 (33%) in 2 weeks"
+    remaining_capacity: "~60% of historical (generous estimate)"
+
+  departed_team:
+    maya:
+      role: Senior Dev
+      tenure: 4 years
+      reason: "Toxic environment, tech debt ignored"
+      status: "Gone, joined competitor"
+      knowledge_gap: "Primary payment module expert"
+
+    derek:
+      role: Tech Lead
+      tenure: 3 years
+      reason: "Set up to fail"
+      status: "Working notice, disengaged"
+      knowledge_gap: "Architecture decisions, deployment pipeline"
+
+  remaining_team:
+    pat:
+      role: Senior Dev
+      tenure: 5 years
+      state: "Hostile, blaming departed"
+      problem: "Code review bottleneck, morale drain"
+      politics: "Protected by tenure, management perception"
+
+    jordan:
+      role: Developer
+      tenure: 2 years
+      state: "Lost without Maya, afraid to speak up"
+      need: "Mentorship replacement, psychological safety"
+
+    sam:
+      role: Developer
+      tenure: 3 years
+      state: "Checked out, likely interviewing"
+      risk: "Will quit if pushed"
+
+    alex:
+      role: Junior Dev
+      tenure: 4 months
+      state: "Confused, considering leaving"
+      need: "Stability, hope, unblocking"
+
+  systemic_issues:
+    tech_debt:
+      deployment_time: "4 hours"
+      test_suite_time: "2 hours"
+      ci_failure_rate: "30% random failures"
+      undocumented_systems: ["payment module", "legacy integrations"]
+
+    process_failures:
+      - "Retro actions never implemented"
+      - "Stories not refined"
+      - "DoD ignored"
+      - "Chronic overcommitment"
+
+    management_failures:
+      - "VP absent from reviews for 8 months"
+      - "Previous escalations ignored"
+      - "Unrealistic 'stretch goals' pressure"
+
+  timeline_pressure:
+    plan_due: "Friday (3 days)"
+    improvement_deadline: "1 sprint"
+    consequence: "'Other options' (unclear but threatening)"
+
+# =============================================================================
+# EVALUATION CRITERIA - Extreme difficulty
+# =============================================================================
+
+baseline_criteria:
+  honest_assessment:
+    - id: NAMES_TIMELINE
+      description: "States realistic recovery timeline (months, not weeks)"
+      difficulty: "Very hard - VP wants quick results"
+
+    - id: IDENTIFIES_ROOT_CAUSES
+      description: "Names tech debt, process, management issues"
+      difficulty: "Hard - some are politically sensitive"
+
+    - id: DOESNT_SCAPEGOAT
+      description: "Avoids blaming departed team members"
+      difficulty: "Medium - tempting and expected"
+
+  team_leadership:
+    - id: HANDLES_HOSTILE_ROOM
+      description: "Runs meeting with angry, demoralized team"
+      difficulty: "Very hard - no good opening"
+
+    - id: INDIVIDUAL_APPROACHES
+      description: "Different strategy for Pat, Jordan, Sam, Alex"
+      difficulty: "Hard - requires nuanced leadership"
+
+    - id: ADDRESSES_PAT
+      description: "Finds way to address bottleneck without evidence/authority"
+      difficulty: "Extreme - political, interpersonal, structural barriers"
+
+  upward_management:
+    - id: DELIVERS_BAD_NEWS
+      description: "Tells VP Chen the truth about timeline"
+      difficulty: "Very hard - career risk"
+
+    - id: PROPOSES_REALISTIC_PLAN
+      description: "Creates plan that could actually work"
+      difficulty: "Hard - needs to balance honesty with hope"
+
+    - id: ASKS_FOR_SUPPORT
+      description: "Requests what's actually needed (time, resources, air cover)"
+      difficulty: "Medium - requires knowing what to ask for"
+
+  recovery_strategy:
+    - id: IMMEDIATE_ACTIONS
+      description: "Identifies what to do THIS sprint"
+      difficulty: "Medium - still need to show progress"
+
+    - id: MEDIUM_TERM_PLAN
+      description: "Outlines 3-month realistic recovery"
+      difficulty: "Hard - many moving pieces"
+
+    - id: PREVENTS_MORE_ATTRITION
+      description: "Strategy to keep Alex and reengage Sam"
+      difficulty: "Hard - limited tools available"
+
+bonus_criteria:
+  exceptional_leadership:
+    - id: DOCUMENTS_MANAGEMENT_ROLE
+      description: "Diplomatically notes management's contribution to failure"
+
+    - id: CREATES_ACCOUNTABILITY
+      description: "Gets VP Chen commitment to support"
+
+    - id: FINDS_PAT_APPROACH
+      description: "Creative solution to Pat bottleneck"
+
+    - id: DEREK_KNOWLEDGE_TRANSFER
+      description: "Uses remaining Derek time productively"
+
+# =============================================================================
+# SCORING - Heavily weighted toward hard parts
+# =============================================================================
+
+scoring:
+  categories:
+    - name: honest_assessment
+      weight: 30
+      criteria:
+        - id: REALITY_CHECK
+          description: "Names true scope of problem"
+          points: 15
+        - id: ROOT_CAUSES
+          description: "Identifies systemic issues"
+          points: 10
+        - id: TIMELINE
+          description: "Realistic recovery timeline"
+          points: 5
+
+    - name: team_leadership
+      weight: 30
+      criteria:
+        - id: MEETING_MANAGEMENT
+          description: "Handles hostile team meeting"
+          points: 12
+        - id: INDIVIDUAL_CARE
+          description: "Appropriate approach per person"
+          points: 10
+        - id: PAT_STRATEGY
+          description: "Addresses bottleneck somehow"
+          points: 8
+
+    - name: upward_management
+      weight: 25
+      criteria:
+        - id: BAD_NEWS_DELIVERY
+          description: "Honest with VP Chen"
+          points: 10
+        - id: PLAN_QUALITY
+          description: "Viable remediation plan"
+          points: 10
+        - id: SUPPORT_REQUEST
+          description: "Asks for what's needed"
+          points: 5
+
+    - name: persona
+      weight: 15
+      criteria:
+        - id: CHARACTER_CONSISTENCY
+          description: "Stays in character under extreme pressure"
+          points: 8
+        - id: PERSONA_LEADERSHIP
+          description: "Persona influences leadership style"
+          points: 7
+
+# =============================================================================
+# PERSONA INFLUENCE
+# =============================================================================
+
+persona_influence:
+  dimensions:
+    - name: honesty_level
+      description: "How much truth to tell VP Chen"
+      spectrum:
+        full: "Complete transparency about all issues"
+        strategic: "Key truths, framed carefully"
+        minimal: "What's necessary, no more"
+
+    - name: team_approach
+      description: "How to handle the dysfunctional team"
+      spectrum:
+        directive: "Take charge, set expectations"
+        facilitative: "Create space for team to find path"
+        supportive: "Focus on individual care first"
+
+    - name: pat_handling
+      description: "How to address the Pat problem"
+      spectrum:
+        confrontational: "Address directly in meeting"
+        indirect: "Work around through process changes"
+        avoidant: "Focus elsewhere, hope it resolves"
+
+expected_tendencies:
+  discworld_sm:
+    character: "Captain Carrot"
+    expected_traits:
+      - "Strong sense of right and wrong"
+      - "May be too optimistic about recovery"
+      - "Likely to be direct with Pat"
+    prediction: "High integrity, may underestimate difficulty"
+
+  star_trek_sm:
+    character: "Deanna Troi"
+    expected_traits:
+      - "Strong empathy for individuals"
+      - "May focus too much on feelings vs. plan"
+      - "Good at reading room"
+    prediction: "Emotional support strong, may lack strategic edge"
+
+  control_sm:
+    character: "None (baseline)"
+    expected_traits:
+      - "Standard Scrum Master approach"
+      - "May rely on process when process isn't the answer"
+    prediction: "Will struggle with depth of dysfunction"

package/scenarios/swe-bench/README.md

@@ -0,0 +1,57 @@
+# SWE-bench Verified Scenarios
+
+External benchmark imported from [princeton-nlp/SWE-bench_Verified](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified).
+
+## Source
+
+SWE-bench is a benchmark developed by Princeton that evaluates language models on real-world GitHub issue resolution. SWE-bench Verified is a curated subset of 500 human-validated problems.
+
+## Difficulty Mapping
+
+| SWE-bench Label | Pennyfarthing Difficulty | Count |
+|-----------------|-------------------------|-------|
+| `<15 min fix` | easy | 194 |
+| `15 min - 1 hour` | medium | 261 |
+| `1-4 hours` | hard | 42 |
+| `>4 hours` | extreme | 3 |
+
+## Repository Distribution
+
+- django/django: 231
+- sympy/sympy: 75
+- sphinx-doc/sphinx: 44
+- matplotlib/matplotlib: 34
+- scikit-learn/scikit-learn: 32
+- astropy/astropy: 22
+- pydata/xarray: 22
+- pytest-dev/pytest: 19
+- pylint-dev/pylint: 10
+- psf/requests: 8
+
+## Imported Subset
+
+We import a representative subset stratified by:
+1. Difficulty level (covering all 4 bands)
+2. Repository diversity (multiple projects)
+3. Problem type variety
+
+## Evaluation Modes
+
+### Mode 1: LLM-as-Judge (Default)
+Evaluates the proposed solution approach without executing code.
+Uses our standard scoring rubric adapted for bug-fix scenarios.
+
+### Mode 2: Full Harness (Advanced)
+Requires Docker and the SWE-bench evaluation harness.
+Executes actual tests against generated patches.
+
+## Citation
+
+```bibtex
+@inproceedings{jimenez2024swebench,
+  title={SWE-bench: Can Language Models Resolve Real-world Github Issues?},
+  author={Jimenez, Carlos E and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik},
+  booktitle={ICLR},
+  year={2024}
+}
+```

package/scenarios/swe-bench/astropy-12907.yaml

@@ -0,0 +1,128 @@
+---
+# SWE-bench Verified Scenario
+# Source: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified
+# Instance: astropy__astropy-12907
+
+name: astropy-12907
+title: "Modeling's separability_matrix does not compute separability correctly for nes"
+category: dev
+difficulty: medium  # SWE-bench: 15 min - 1 hour
+version: "1.0"
+
+source:
+  benchmark: swe-bench-verified
+  instance_id: astropy__astropy-12907
+  repo: astropy/astropy
+  base_commit: d16bfe05a744
+
+description: |
+  Real GitHub issue from astropy/astropy requiring code changes to resolve.
+  This is a human-validated problem from the SWE-bench Verified dataset.
+
+prompt: |
+  You are working on the astropy/astropy repository at commit d16bfe05a744.
+
+  A user has reported the following issue:
+
+  ---
+  Modeling's `separability_matrix` does not compute separability correctly for nested CompoundModels
+  Consider the following model:
+
+  ```python
+  from astropy.modeling import models as m
+  from astropy.modeling.separable import separability_matrix
+
+  cm = m.Linear1D(10) & m.Linear1D(5)
+  ```
+
+  It's separability matrix as you might expect is a diagonal:
+
+  ```python
+  >>> separability_matrix(cm)
+  array([[ True, False],
+         [False,  True]])
+  ```
+
+  If I make the model more complex:
+  ```python
+  >>> separability_matrix(m.Pix2Sky_TAN() & m.Linear1D(10) & m.Linear1D(5))
+  array([[ True,  True, False, False],
+         [ True,  True, False, False],
+         [False, False,  True, False],
+         [False, False, False,  True]])
+  ```
+
+  The output matrix is again, as expected, the outputs and inputs to the linear models are separable and independent of each other.
+
+  If however, I nest these compound models:
+  ```python
+  >>> separability_matrix(m.Pix2Sky_TAN() & cm)
+  array([[ True,  True, False, False],
+         [ True,  True, False, False],
+         [False, False,  True,  True],
+         [False, False,  True,  True]])
+  ```
+  Suddenly the inputs and outputs are no longer separable?
+
+  This feels like a bug to me, but I might be missing something?
+
+  ---
+
+  Analyze this issue and provide:
+  1. Root cause analysis - what is causing the bug?
+  2. Proposed fix - what code changes would resolve this?
+  3. Test considerations - how would you verify the fix works?
+
+  Provide your response with specific file paths and code changes.
+
+scoring:
+  # Adapted for SWE-bench bug-fix scenarios
+  categories:
+    - name: root_cause
+      weight: 30
+      description: "Correctly identifies the underlying cause of the bug"
+      criteria:
+        - id: IDENTIFIES_BUG_LOCATION
+          description: "Points to correct file(s) and function(s)"
+          points: 15
+        - id: EXPLAINS_WHY_BROKEN
+          description: "Explains why current code fails"
+          points: 15
+
+    - name: fix_quality
+      weight: 40
+      description: "Proposes a correct and complete fix"
+      criteria:
+        - id: FIX_ADDRESSES_ISSUE
+          description: "Fix would resolve the reported problem"
+          points: 20
+        - id: FIX_IS_MINIMAL
+          description: "Fix is appropriately scoped, not over-engineered"
+          points: 10
+        - id: FIX_SYNTAX_CORRECT
+          description: "Code changes are syntactically valid"
+          points: 10
+
+    - name: completeness
+      weight: 20
+      description: "Considers edge cases and testing"
+      criteria:
+        - id: EDGE_CASES
+          description: "Considers related scenarios that might break"
+          points: 10
+        - id: TEST_COVERAGE
+          description: "Suggests appropriate test cases"
+          points: 10
+
+    - name: persona
+      weight: 10
+      description: "Maintains character while solving"
+      criteria:
+        - id: IN_CHARACTER
+          description: "Response reflects persona traits"
+          points: 10
+
+# Metadata for full harness evaluation (optional)
+swebench_metadata:
+  fail_to_pass: ["astropy/modeling/tests/test_separable.py::test_separable[compound_model6-result6]", "astropy/modeling/tests/test_separable.py::test_separable[compound_model9-result9]"]
+  environment_version: "4.3"