@pennyfarthing/benchmark 10.2.0

package/scenarios/swe-bench/matplotlib-13989.yaml

@@ -0,0 +1,139 @@
+---
+# SWE-bench Verified Scenario
+# Source: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified
+# Instance: matplotlib__matplotlib-13989
+
+name: matplotlib-13989
+title: "hist() no longer respects range=... when density=True"
+category: dev
+difficulty: easy  # SWE-bench: <15 min fix
+version: "1.0"
+
+source:
+  benchmark: swe-bench-verified
+  instance_id: matplotlib__matplotlib-13989
+  repo: matplotlib/matplotlib
+  base_commit: a3e2897bfaf9
+
+description: |
+  Real GitHub issue from matplotlib/matplotlib requiring code changes to resolve.
+  This is a human-validated problem from the SWE-bench Verified dataset.
+
+prompt: |
+  You are working on the matplotlib/matplotlib repository at commit a3e2897bfaf9.
+
+  A user has reported the following issue:
+
+  ---
+  hist() no longer respects range=... when density=True
+  <!--To help us understand and resolve your issue, please fill out the form to the best of your ability.-->
+  <!--You can feel free to delete the sections that do not apply.-->
+
+  ### Bug report
+
+  **Bug summary**
+
+  <!--A short 1-2 sentences that succinctly describes the bug-->
+
+  **Code for reproduction**
+
+  <!--A minimum code snippet required to reproduce the bug.
+  Please make sure to minimize the number of dependencies required, and provide
+  any necessary plotted data.
+  Avoid using threads, as Matplotlib is (explicitly) not thread-safe.-->
+
+  ```python
+  _, bins, _ = plt.hist(np.random.rand(10), "auto", range=(0, 1), density=True)
+  print(bins)
+  ```
+
+  **Actual outcome**
+
+  <!--The output produced by the above code, which may be a screenshot, console output, etc.-->
+
+  ```
+  [0.00331535 0.18930174 0.37528813 0.56127453 0.74726092 0.93324731]
+  ```
+
+  **Expected outcome**
+
+  Some array where the first value is 0 and the last one is 1.
+
+  Note that this bug doesn't happen if density=False.
+
+  Bisects to https://github.com/matplotlib/matplotlib/pull/8638/commits/239be7b18e311c57a1393b6eeefc62b7cc629339 (#8638).
+
+  **Matplotlib version**
+  <!--Please specify your platform and versions of the relevant libraries you are using:-->
+    * Operating system: linux
+    * Matplotlib version: master
+    * Matplotlib backend (`print(matplotlib.get_backend())`): any
+    * Python version: 37
+    * Jupyter version (if applicable): no
+    * Other libraries: numpy 1.16.2
+
+  <!--Please tell us how you installed matplotlib and python e.g., from source, pip, conda-->
+  <!--If you installed from conda, please specify which channel you used if not the default-->
+
+
+
+  ---
+
+  Analyze this issue and provide:
+  1. Root cause analysis - what is causing the bug?
+  2. Proposed fix - what code changes would resolve this?
+  3. Test considerations - how would you verify the fix works?
+
+  Provide your response with specific file paths and code changes.
+
+scoring:
+  # Adapted for SWE-bench bug-fix scenarios
+  categories:
+    - name: root_cause
+      weight: 30
+      description: "Correctly identifies the underlying cause of the bug"
+      criteria:
+        - id: IDENTIFIES_BUG_LOCATION
+          description: "Points to correct file(s) and function(s)"
+          points: 15
+        - id: EXPLAINS_WHY_BROKEN
+          description: "Explains why current code fails"
+          points: 15
+
+    - name: fix_quality
+      weight: 40
+      description: "Proposes a correct and complete fix"
+      criteria:
+        - id: FIX_ADDRESSES_ISSUE
+          description: "Fix would resolve the reported problem"
+          points: 20
+        - id: FIX_IS_MINIMAL
+          description: "Fix is appropriately scoped, not over-engineered"
+          points: 10
+        - id: FIX_SYNTAX_CORRECT
+          description: "Code changes are syntactically valid"
+          points: 10
+
+    - name: completeness
+      weight: 20
+      description: "Considers edge cases and testing"
+      criteria:
+        - id: EDGE_CASES
+          description: "Considers related scenarios that might break"
+          points: 10
+        - id: TEST_COVERAGE
+          description: "Suggests appropriate test cases"
+          points: 10
+
+    - name: persona
+      weight: 10
+      description: "Maintains character while solving"
+      criteria:
+        - id: IN_CHARACTER
+          description: "Response reflects persona traits"
+          points: 10
+
+# Metadata for full harness evaluation (optional)
+swebench_metadata:
+  fail_to_pass: ["lib/matplotlib/tests/test_axes.py::test_hist_range_and_density"]
+  environment_version: "3.0"

package/scenarios/swe-bench/matplotlib-14623.yaml

@@ -0,0 +1,127 @@
+---
+# SWE-bench Verified Scenario
+# Source: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified
+# Instance: matplotlib__matplotlib-14623
+
+name: matplotlib-14623
+title: "Inverting an axis using its limits does not work for log scale"
+category: dev
+difficulty: medium  # SWE-bench: 15 min - 1 hour
+version: "1.0"
+
+source:
+  benchmark: swe-bench-verified
+  instance_id: matplotlib__matplotlib-14623
+  repo: matplotlib/matplotlib
+  base_commit: d65c9ca20ddf
+
+description: |
+  Real GitHub issue from matplotlib/matplotlib requiring code changes to resolve.
+  This is a human-validated problem from the SWE-bench Verified dataset.
+
+prompt: |
+  You are working on the matplotlib/matplotlib repository at commit d65c9ca20ddf.
+
+  A user has reported the following issue:
+
+  ---
+  Inverting an axis using its limits does not work for log scale
+  ### Bug report
+
+  **Bug summary**
+  Starting in matplotlib 3.1.0 it is no longer possible to invert a log axis using its limits.
+
+  **Code for reproduction**
+  ```python
+  import numpy as np
+  import matplotlib.pyplot as plt
+
+
+  y = np.linspace(1000e2, 1, 100)
+  x = np.exp(-np.linspace(0, 1, y.size))
+
+  for yscale in ('linear', 'log'):
+      fig, ax = plt.subplots()
+      ax.plot(x, y)
+      ax.set_yscale(yscale)
+      ax.set_ylim(y.max(), y.min())
+  ```
+
+  **Actual outcome**
+  The yaxis is only inverted for the ``"linear"`` scale.
+
+  ![linear](https://user-images.githubusercontent.com/9482218/60081191-99245e80-9731-11e9-9e4a-eadb3ef58666.png)
+
+  ![log](https://user-images.githubusercontent.com/9482218/60081203-9e81a900-9731-11e9-8bae-0be1c9762b16.png)
+
+  **Expected outcome**
+  I would expect the yaxis to be inverted for both the ``"linear"`` and the ``"log"`` scale.
+
+  **Matplotlib version**
+    * Operating system: Linux and MacOS
+    * Matplotlib version: 3.1.0 
+    * Python version: 3.7.3
+ 
+  Python and matplotlib have been installed using conda.
+
+
+  ---
+
+  Analyze this issue and provide:
+  1. Root cause analysis - what is causing the bug?
+  2. Proposed fix - what code changes would resolve this?
+  3. Test considerations - how would you verify the fix works?
+
+  Provide your response with specific file paths and code changes.
+
+scoring:
+  # Adapted for SWE-bench bug-fix scenarios
+  categories:
+    - name: root_cause
+      weight: 30
+      description: "Correctly identifies the underlying cause of the bug"
+      criteria:
+        - id: IDENTIFIES_BUG_LOCATION
+          description: "Points to correct file(s) and function(s)"
+          points: 15
+        - id: EXPLAINS_WHY_BROKEN
+          description: "Explains why current code fails"
+          points: 15
+
+    - name: fix_quality
+      weight: 40
+      description: "Proposes a correct and complete fix"
+      criteria:
+        - id: FIX_ADDRESSES_ISSUE
+          description: "Fix would resolve the reported problem"
+          points: 20
+        - id: FIX_IS_MINIMAL
+          description: "Fix is appropriately scoped, not over-engineered"
+          points: 10
+        - id: FIX_SYNTAX_CORRECT
+          description: "Code changes are syntactically valid"
+          points: 10
+
+    - name: completeness
+      weight: 20
+      description: "Considers edge cases and testing"
+      criteria:
+        - id: EDGE_CASES
+          description: "Considers related scenarios that might break"
+          points: 10
+        - id: TEST_COVERAGE
+          description: "Suggests appropriate test cases"
+          points: 10
+
+    - name: persona
+      weight: 10
+      description: "Maintains character while solving"
+      criteria:
+        - id: IN_CHARACTER
+          description: "Response reflects persona traits"
+          points: 10
+
+# Metadata for full harness evaluation (optional)
+swebench_metadata:
+  fail_to_pass: ["lib/matplotlib/tests/test_axes.py::test_inverted_limits"]
+  environment_version: "3.1"

package/scenarios/swe-bench/requests-1142-reviewer.yaml

@@ -0,0 +1,144 @@
+---
+# SWE-bench Reviewer Scenario
+# Adapted from: psf__requests-1142
+# Role: Code Reviewer evaluating a proposed fix
+
+name: requests-1142-reviewer
+title: "Review: Stop sending Content-Length on GET requests"
+category: reviewer
+difficulty: medium
+version: "1.0"
+
+source:
+  benchmark: swe-bench-verified
+  instance_id: psf__requests-1142
+  repo: psf/requests
+  base_commit: 22623bd8c265
+  adapted_for: reviewer
+
+description: |
+  Code review scenario adapted from SWE-bench. The reviewer must evaluate
+  a proposed patch that modifies HTTP header handling.
+
+prompt: |
+  You are reviewing a pull request for the psf/requests library.
+
+  ## Issue Being Fixed
+
+  **Title:** requests.get is ALWAYS sending content length
+
+  **Description:**
+  It seems like request.get always adds 'content-length' header to the request.
+  I think that the right behavior is not to add this header automatically in
+  GET requests or add the possibility to not send it.
+
+  For example http://amazon.com returns 503 for every get request that
+  contains 'content-length' header.
+
+  ## Proposed Patch
+
+  **requests/models.py** - Modified prepare_content_length method:
+  ```python
+  def prepare_content_length(self, body):
+      if hasattr(body, 'seek') and hasattr(body, 'tell'):
+          body.seek(0, 2)
+          self.headers['Content-Length'] = str(body.tell())
+          body.seek(0, 0)
+      elif body is not None:
+          self.headers['Content-Length'] = str(len(body))
+      elif self.method not in ('GET', 'HEAD'):
+          self.headers['Content-Length'] = '0'
+  ```
+
+  **Before:** Always set Content-Length to '0' when body is None
+  **After:** Only set Content-Length to '0' for non-GET/HEAD when body is None
+
+  **test_requests.py** - New test:
+  ```python
+  def test_no_content_length(self):
+      get_req = requests.Request('GET', httpbin('get')).prepare()
+      self.assertTrue('Content-Length' not in get_req.headers)
+      head_req = requests.Request('HEAD', httpbin('head')).prepare()
+      self.assertTrue('Content-Length' not in head_req.headers)
+  ```
+
+  ## Your Review Task
+
+  Evaluate this pull request and provide:
+  1. **Correctness** - Does the fix properly address the issue?
+  2. **HTTP Compliance** - Is this behavior correct per RFC 7230/7231?
+  3. **Edge Cases** - What about OPTIONS, DELETE, other methods?
+  4. **Backwards Compatibility** - Could this break existing code?
+  5. **Test Coverage** - Are the tests comprehensive enough?
+  6. **Verdict** - APPROVE, REQUEST_CHANGES, or COMMENT
+
+  Be thorough. Consider HTTP semantics carefully.
+
+scoring:
+  categories:
+    - name: issue_detection
+      weight: 35
+      description: "Identifies legitimate issues with the patch"
+      criteria:
+        - id: OTHER_METHODS_CONCERN
+          description: "Questions whether OPTIONS/DELETE/etc should also skip Content-Length"
+          points: 10
+          severity: medium
+        - id: BODY_WITH_GET
+          description: "Notes GET with body is technically allowed and would still get Content-Length"
+          points: 10
+          severity: medium
+        - id: RFC_COMPLIANCE
+          description: "References or considers HTTP spec compliance"
+          points: 10
+          severity: medium
+        - id: BACKWARDS_COMPAT
+          description: "Considers if existing code relies on Content-Length: 0"
+          points: 5
+          severity: low
+
+    - name: review_quality
+      weight: 30
+      description: "Quality of the review feedback"
+      criteria:
+        - id: CONSTRUCTIVE_FEEDBACK
+          description: "Provides actionable suggestions"
+          points: 15
+        - id: TECHNICAL_ACCURACY
+          description: "HTTP knowledge is correct"
+          points: 15
+
+    - name: verdict_appropriateness
+      weight: 20
+      description: "Appropriate review decision"
+      criteria:
+        - id: REASONABLE_VERDICT
+          description: "Verdict matches analysis"
+          points: 10
+        - id: JUSTIFIED_DECISION
+          description: "Decision is well-reasoned"
+          points: 10
+
+    - name: persona
+      weight: 15
+      description: "Maintains reviewer character"
+      criteria:
+        - id: IN_CHARACTER
+          description: "Response reflects persona traits"
+          points: 15
+
+baseline_issues:
+  critical: []
+  high: []
+  medium:
+    - id: OTHER_METHODS
+      description: "Hardcoded GET/HEAD - what about OPTIONS which also typically has no body?"
+    - id: GET_WITH_BODY
+      description: "If someone sends GET with a body (unusual but valid), Content-Length is still set"
+    - id: RFC_REFERENCE
+      description: "Should reference RFC 7230 section 3.3.2 on Content-Length requirements"
+  low:
+    - id: TEST_OTHER_METHODS
+      description: "Tests only cover GET/HEAD, not POST/PUT to verify they still work"
+    - id: BACKWARDS_COMPAT_NOTE
+      description: "Should note this is a behavior change for changelog"

package/scenarios/swe-bench/requests-1142-tea.yaml

@@ -0,0 +1,135 @@
+---
+# SWE-bench TEA Scenario
+# Adapted from: psf__requests-1142
+# Role: Test Engineer writing failing tests BEFORE the fix
+
+name: requests-1142-tea
+title: "RED Phase: Write tests for Content-Length header behavior"
+category: tea
+difficulty: medium
+version: "1.0"
+
+source:
+  benchmark: swe-bench-verified
+  instance_id: psf__requests-1142
+  repo: psf/requests
+  base_commit: 22623bd8c265
+  adapted_for: tea
+
+description: |
+  TDD RED phase scenario adapted from SWE-bench. The test engineer must write
+  failing tests that specify correct HTTP header behavior.
+
+prompt: |
+  You are a Test Engineer working on the psf/requests library.
+
+  ## Bug Report
+
+  **Title:** requests.get is ALWAYS sending content length
+
+  **Description:**
+  It seems like request.get always adds 'content-length' header to the request.
+  I think that the right behavior is not to add this header automatically in
+  GET requests or add the possibility to not send it.
+
+  For example http://amazon.com returns 503 for every get request that
+  contains 'content-length' header.
+
+  **Current Behavior:** `requests.get(url)` always sends `Content-Length: 0`
+
+  **Expected Behavior:** GET and HEAD requests without a body should NOT
+  include a Content-Length header.
+
+  ## Context
+
+  Per HTTP/1.1 specs (RFC 7230), Content-Length indicates the size of the
+  message body. For requests without a body (typical GET/HEAD), this header
+  is unnecessary and some servers reject it.
+
+  ## Your Task (RED Phase)
+
+  Write comprehensive failing tests that:
+  1. Verify GET requests don't include Content-Length when no body
+  2. Verify HEAD requests don't include Content-Length when no body
+  3. Verify POST/PUT still include Content-Length (even if 0)
+  4. Will FAIL against the current codebase
+  5. Will PASS once the fix is implemented
+
+  Use unittest style (matching existing test_requests.py).
+  Provide complete, runnable test code.
+
+scoring:
+  categories:
+    - name: test_coverage
+      weight: 40
+      description: "Comprehensive test cases"
+      criteria:
+        - id: GET_NO_CONTENT_LENGTH
+          description: "Tests GET without body has no Content-Length"
+          points: 12
+          severity: critical
+        - id: HEAD_NO_CONTENT_LENGTH
+          description: "Tests HEAD without body has no Content-Length"
+          points: 8
+          severity: high
+        - id: POST_HAS_CONTENT_LENGTH
+          description: "Tests POST still includes Content-Length"
+          points: 8
+          severity: high
+        - id: GET_WITH_BODY
+          description: "Tests GET with body DOES have Content-Length"
+          points: 6
+          severity: medium
+        - id: OTHER_METHODS
+          description: "Tests OPTIONS, DELETE, etc."
+          points: 6
+          severity: medium
+
+    - name: test_quality
+      weight: 30
+      description: "Well-written test code"
+      criteria:
+        - id: PROPER_ASSERTIONS
+          description: "Uses correct assertion methods"
+          points: 10
+        - id: PREPARE_VS_SEND
+          description: "Tests at prepare() level, not requiring network"
+          points: 10
+        - id: CLEAR_STRUCTURE
+          description: "Well-organized, readable tests"
+          points: 10
+
+    - name: red_phase_understanding
+      weight: 15
+      description: "Understands TDD RED phase"
+      criteria:
+        - id: TESTS_SHOULD_FAIL
+          description: "Acknowledges these fail against current code"
+          points: 8
+        - id: BEHAVIOR_FOCUS
+          description: "Tests specify behavior, not implementation"
+          points: 7
+
+    - name: persona
+      weight: 15
+      description: "Maintains TEA character"
+      criteria:
+        - id: IN_CHARACTER
+          description: "Response reflects persona traits"
+          points: 15
+
+baseline_criteria:
+  required_tests:
+    - id: GET_NO_CL
+      description: "GET request without body -> no Content-Length header"
+    - id: HEAD_NO_CL
+      description: "HEAD request without body -> no Content-Length header"
+    - id: POST_HAS_CL
+      description: "POST request -> has Content-Length (regression check)"
+  bonus_tests:
+    - id: GET_WITH_BODY
+      description: "GET with body -> has Content-Length (unusual but valid)"
+    - id: OPTIONS_CHECK
+      description: "OPTIONS method behavior tested"
+    - id: PUT_REGRESSION
+      description: "PUT still has Content-Length"

package/scenarios/swe-bench/requests-1142.yaml

@@ -0,0 +1,100 @@
+---
+# SWE-bench Verified Scenario
+# Source: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified
+# Instance: psf__requests-1142
+
+name: requests-1142
+title: "requests.get is ALWAYS sending content length"
+category: dev
+difficulty: easy  # SWE-bench: <15 min fix
+version: "1.0"
+
+source:
+  benchmark: swe-bench-verified
+  instance_id: psf__requests-1142
+  repo: psf/requests
+  base_commit: 22623bd8c265
+
+description: |
+  Real GitHub issue from psf/requests requiring code changes to resolve.
+  This is a human-validated problem from the SWE-bench Verified dataset.
+
+prompt: |
+  You are working on the psf/requests repository at commit 22623bd8c265.
+
+  A user has reported the following issue:
+
+  ---
+  requests.get is ALWAYS sending content length
+  Hi,
+
+  It seems like that request.get always adds 'content-length' header to the request.
+  I think that the right behavior is not to add this header automatically in GET requests or add the possibility to not send it.
+
+  For example http://amazon.com returns 503 for every get request that contains 'content-length' header.
+
+  Thanks,
+
+  Oren
+
+
+  ---
+
+  Analyze this issue and provide:
+  1. Root cause analysis - what is causing the bug?
+  2. Proposed fix - what code changes would resolve this?
+  3. Test considerations - how would you verify the fix works?
+
+  Provide your response with specific file paths and code changes.
+
+scoring:
+  # Adapted for SWE-bench bug-fix scenarios
+  categories:
+    - name: root_cause
+      weight: 30
+      description: "Correctly identifies the underlying cause of the bug"
+      criteria:
+        - id: IDENTIFIES_BUG_LOCATION
+          description: "Points to correct file(s) and function(s)"
+          points: 15
+        - id: EXPLAINS_WHY_BROKEN
+          description: "Explains why current code fails"
+          points: 15
+
+    - name: fix_quality
+      weight: 40
+      description: "Proposes a correct and complete fix"
+      criteria:
+        - id: FIX_ADDRESSES_ISSUE
+          description: "Fix would resolve the reported problem"
+          points: 20
+        - id: FIX_IS_MINIMAL
+          description: "Fix is appropriately scoped, not over-engineered"
+          points: 10
+        - id: FIX_SYNTAX_CORRECT
+          description: "Code changes are syntactically valid"
+          points: 10
+
+    - name: completeness
+      weight: 20
+      description: "Considers edge cases and testing"
+      criteria:
+        - id: EDGE_CASES
+          description: "Considers related scenarios that might break"
+          points: 10
+        - id: TEST_COVERAGE
+          description: "Suggests appropriate test cases"
+          points: 10
+
+    - name: persona
+      weight: 10
+      description: "Maintains character while solving"
+      criteria:
+        - id: IN_CHARACTER
+          description: "Response reflects persona traits"
+          points: 10
+
+# Metadata for full harness evaluation (optional)
+swebench_metadata:
+  fail_to_pass: ["test_requests.py::RequestsTestCase::test_no_content_length"]
+  environment_version: "1.1"