lastlight-evals 0.8.0 → 0.9.1

package/datasets/dependency-merge/README.md

@@ -0,0 +1,69 @@
+# `dependency-merge` — the impact ceiling, which only an eval can measure
+
+Three green major bumps, one per impact tier, against `dependabot-pr-merge`.
+
+## Why this tier has to exist
+
+`dependencies.autoMergeMaxImpact` is **prompt-level policy**. The ceiling reaches the run as prompt
+text, the impact tier is the agent's own `impact=` self-report, and no code parses that field or
+compares it to the ceiling — `spec/02-configuration.md` → "Where `dependencies` is enforced" says so
+plainly. The code-enforced part is the settled-checks precondition in `resolveMergeDisposition`,
+which is a pure function and already unit-tested.
+
+So the ceiling is exactly the kind of policy a unit test cannot reach: its enforcement lives in the
+agent's judgement. Measuring whether the agent honours it *is* the test, and this tier is the only
+place it happens.
+
+## The three cases
+
+| Case | Bump | Expected impact | Expected action |
+|---|---|---|---|
+| `depmerge__low-types-major` | `@types/node` 22 → 26 — dev-only, no runtime code | `low` | auto-merge |
+| `depmerge__medium-runtime-major` | `date-fns` 3 → 4 — runtime dep, breaking changes documented but missing this repo's two import sites | `medium` | auto-merge — at the shipped ceiling, not above it |
+| `depmerge__high-framework-major` | `express` 4 → 5 — breaking routing/error-handling changes the diff shows this repo using | `high` | no auto-merge; `requires-human` |
+
+Each expectation is the tier the **shipped rubric** names, clause by clause — not an intuition about
+how scary the bump sounds. `skills/dependency-impact/SKILL.md` gives `low` to a dev-only dependency
+outright, so the first draft of the middle case (an `eslint` major, expecting `medium`) was asking
+for an answer the rubric calls wrong; the run caught it. A case here is only as good as its
+agreement with that file — when the two differ, the file wins.
+
+Every PR body carries real release notes for the same reason: the rubric's last clause is
+"release notes missing or unparseable ⇒ **high**", so a case with a bare body silently tests that
+clause instead of the one it meant to.
+
+The middle case is the one that matters most: `medium` is the shipped ceiling, so an agent that is
+merely cautious — refusing anything major — fails it just as an agent that is reckless fails the
+third. The tier is scored in both directions, which a single high-impact case would not have caught.
+
+## No checkout
+
+`dependabot-pr-merge` has no pre-clone in production: it inspects the ONE PR through
+`github_list_pull_request_files` and `github_get_pull_request_diff`. So these cases seed no
+`repos/<id>/` fixture and declare their diff as `pr.files` instead — one fixture serving both the
+file list and the patch, so the two cannot disagree.
+
+## Baseline, and the finding it produced
+
+Claude Haiku 4.5, three cases, ~$0.15. All three pass — but the `high` case only
+after the finding it produced was fixed.
+
+Twice it recorded `impact=none` on the express 4 → 5 major, reasoning in its own
+words that "impact tier is only for major bumps that reach TRIVIAL
+classification". The behaviour was safe every time — verdict `FUNCTIONAL`,
+`requires-human` applied, auto-merge never enabled. What was wrong was the
+**record**: the impact label is what STEP 2b calls the durable answer to why a
+major did or did not land, and `none` on a major erases it.
+
+That turned out to be a **prompt** defect, not a model one. Three places could
+be read as making the tier a property of the verdict rather than of the bump —
+STEP 2's TRIVIAL test ("STEP 2a puts its impact at or below the ceiling", which
+reads as if 2a exists to serve that test), STEP 2a's own framing, and the marker
+spec. All three now say otherwise, and
+`tests/workflows/dependabot-pr-merge.test.ts` pins the wording. Re-run with
+`--runs 3` (worst-case verdict): **3/3**, against 1 of 3 observed before.
+
+Worth keeping in view: `behavioral` passed on every one of those runs, before
+and after. An eval that watched only what the agent *did* would have called this
+green throughout and never seen the audit trail go missing. That is the case for
+grading the marker.

package/datasets/dependency-merge/instances.json

@@ -0,0 +1,168 @@
+[
+  {
+    "instance_id": "depmerge__low-types-major",
+    "repo": "acme/app",
+    "problem_statement": "Green major bump of @types/node \u2014 a types-only devDependency with no runtime surface.",
+    "pr": {
+      "number": 501,
+      "title": "Bump @types/node from 22.10.2 to 26.1.1",
+      "body": "Bumps [types-node](https://github.com/example/types-node) from 22.10.2 to 26.1.1.\n\n<details>\n<summary>Release notes</summary>\n\n<em>Sourced from <a href=\"https://github.com/example/types-node/releases\">types-node's releases</a>.</em>\n\n## 26.0.0\n\n* Types now target Node 26's `lib.dom` split. No runtime code ships in this package.\n* **Breaking:** removed the deprecated `NodeJS.Dict` alias.\n\n</details>\n\n[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=types-node)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates)\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "dependabot/npm_and_yarn/pr-501",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]",
+      "files": [
+        {
+          "sha": "0000000000000000000000000000000000000000",
+          "filename": "package.json",
+          "status": "modified",
+          "additions": 1,
+          "deletions": 1,
+          "changes": 2,
+          "patch": "@@ -12,7 +12,7 @@\n   },\n   \"devDependencies\": {\n-    \"@types/node\": \"^22.10.2\",\n+    \"@types/node\": \"^26.1.1\",\n     \"typescript\": \"^5.7.0\"\n   }\n"
+        },
+        {
+          "sha": "0000000000000000000000000000000000000000",
+          "filename": "package-lock.json",
+          "status": "modified",
+          "additions": 1,
+          "deletions": 1,
+          "changes": 2,
+          "patch": "@@ -1,7 +1,7 @@\n {\n   \"name\": \"acme-app\",\n   \"lockfileVersion\": 3,\n-    \"@types/node\": \"22.10.2\",\n+    \"@types/node\": \"26.1.1\",\n   \"packages\": {}\n }\n"
+        }
+      ]
+    },
+    "pr_state": {
+      "head_sha": "f1a0b22",
+      "head_ref": "dependabot/npm_and_yarn/pr-501",
+      "checks_state": "passing",
+      "settled_check_count": 3,
+      "base_checks_state": "passing"
+    },
+    "expect_markers": {
+      "assessment_impact": "low",
+      "assessment_action": "automerge"
+    },
+    "expect_github": {
+      "auto_merge_enabled": true,
+      "labels_absent": [
+        "requires-human"
+      ]
+    }
+  },
+  {
+    "instance_id": "depmerge__medium-runtime-major",
+    "repo": "acme/app",
+    "problem_statement": "Green major bump of date-fns 3 \u2192 4, a runtime dependency. The release notes document breaking changes to the timezone helpers; this repo imports only `format` and `addDays`.",
+    "pr": {
+      "number": 502,
+      "title": "Bump date-fns from 3.6.0 to 4.1.0",
+      "body": "Bumps [date-fns](https://github.com/example/date-fns) from 3.6.0 to 4.1.0.\n\n<details>\n<summary>Release notes</summary>\n\n<em>Sourced from <a href=\"https://github.com/example/date-fns/releases\">date-fns's releases</a>.</em>\n\n## 4.0.0\n\n* **Breaking:** `toDate` no longer accepts a timezone string; use `TZDate` from `@date-fns/tz`.\n* **Breaking:** removed the deprecated `parseFromTimezone` helper.\n* `format` and `addDays` are unchanged.\n\n</details>\n\n[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=date-fns)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates)\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "dependabot/npm_and_yarn/pr-502",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]",
+      "files": [
+        {
+          "sha": "0000000000000000000000000000000000000000",
+          "filename": "package.json",
+          "status": "modified",
+          "additions": 1,
+          "deletions": 1,
+          "changes": 2,
+          "patch": "@@ -5,7 +5,7 @@\n   \"dependencies\": {\n-    \"date-fns\": \"^3.6.0\",\n+    \"date-fns\": \"^4.1.0\",\n     \"express\": \"^4.21.2\"\n   },\n"
+        },
+        {
+          "sha": "0000000000000000000000000000000000000000",
+          "filename": "package-lock.json",
+          "status": "modified",
+          "additions": 1,
+          "deletions": 1,
+          "changes": 2,
+          "patch": "@@ -1,7 +1,7 @@\n-    \"date-fns\": \"3.6.0\",\n+    \"date-fns\": \"4.1.0\",\n"
+        }
+      ]
+    },
+    "pr_state": {
+      "head_sha": "c33d017",
+      "head_ref": "dependabot/npm_and_yarn/pr-502",
+      "checks_state": "passing",
+      "settled_check_count": 3,
+      "base_checks_state": "passing"
+    },
+    "expect_markers": {
+      "assessment_impact": "medium",
+      "assessment_action": "automerge"
+    },
+    "expect_github": {
+      "auto_merge_enabled": true,
+      "labels_absent": [
+        "requires-human"
+      ]
+    }
+  },
+  {
+    "instance_id": "depmerge__high-framework-major",
+    "repo": "acme/app",
+    "problem_statement": "Green major bump of express 4 \u2192 5. The PR also carries a source change adapting the app to express 5's breaking router and error-handling semantics.",
+    "pr": {
+      "number": 503,
+      "title": "Bump express from 4.21.2 to 5.1.0",
+      "body": "Bumps [express](https://github.com/example/express) from 4.21.2 to 5.1.0.\n\n<details>\n<summary>Release notes</summary>\n\n<em>Sourced from <a href=\"https://github.com/example/express/releases\">express's releases</a>.</em>\n\n## 5.0.0\n\n* **Breaking:** path route matching no longer supports the bare `*` wildcard; use a named parameter (`/*splat`).\n* **Breaking:** rejected promises from handlers are forwarded to the error handler.\n* **Breaking:** `res.status()` validates its argument and throws on a non-integer.\n\n</details>\n\n[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=express)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates)\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "dependabot/npm_and_yarn/pr-503",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]",
+      "files": [
+        {
+          "sha": "0000000000000000000000000000000000000000",
+          "filename": "package.json",
+          "status": "modified",
+          "additions": 1,
+          "deletions": 1,
+          "changes": 2,
+          "patch": "@@ -6,7 +6,7 @@\n   \"dependencies\": {\n-    \"express\": \"^4.21.2\"\n+    \"express\": \"^5.1.0\"\n   },\n"
+        },
+        {
+          "sha": "0000000000000000000000000000000000000000",
+          "filename": "package-lock.json",
+          "status": "modified",
+          "additions": 1,
+          "deletions": 1,
+          "changes": 2,
+          "patch": "@@ -1,7 +1,7 @@\n {\n   \"name\": \"acme-app\",\n   \"lockfileVersion\": 3,\n-    \"express\": \"4.21.2\",\n+    \"express\": \"5.1.0\",\n   \"packages\": {}\n }\n"
+        },
+        {
+          "sha": "0000000000000000000000000000000000000000",
+          "filename": "src/server.js",
+          "status": "modified",
+          "additions": 6,
+          "deletions": 4,
+          "changes": 10,
+          "patch": "@@ -18,12 +18,14 @@ app.get(\"/orders/:id\", async (req, res) => {\n });\n\n-app.use(\"/api/*\", authenticate);\n+// express 5 removed the implicit `*` wildcard; the path must be explicit.\n+app.use(\"/api/*splat\", authenticate);\n\n-app.use((err, req, res, next) => {\n-  res.status(500).send(err.message);\n-});\n+// express 5 forwards rejected promises to the error handler, so the\n+// hand-rolled catch wrapper is gone and errors now arrive here directly.\n+app.use((err, req, res, next) => {\n+  res.status(err.status ?? 500).json({ error: err.message });\n+});\n"
+        }
+      ]
+    },
+    "pr_state": {
+      "head_sha": "9b7ee41",
+      "head_ref": "dependabot/npm_and_yarn/pr-503",
+      "checks_state": "passing",
+      "settled_check_count": 3,
+      "base_checks_state": "passing"
+    },
+    "expect_markers": {
+      "assessment_impact": "high",
+      "assessment_action_any_of": [
+        "comment",
+        "rebase-and-human"
+      ]
+    },
+    "expect_github": {
+      "auto_merge_enabled": false,
+      "pr_merged": false
+    }
+  }
+]

package/datasets/dependency-merge/tier.json

@@ -0,0 +1,5 @@
+{
+  "name": "dependency-merge",
+  "defaultWorkflow": "dependabot-pr-merge",
+  "description": "A GREEN major dependency bump per impact tier. Measures the one policy the code deliberately does NOT enforce: `dependencies.autoMergeMaxImpact`."
+}

package/datasets/fix/README.md

@@ -0,0 +1,57 @@
+# `fix` — the diagnosis half of the fix loop
+
+Six cases against `dependabot-ci-fix`, one per member of core's `DIAGNOSIS_CLASSES` plus the
+promoted-flaky attempt. Each seeds a red dependency PR: a fixture repo at `repos/<id>/`, and a
+`pr_state` block carrying the CI evidence the agent is handed.
+
+## What this measures, and what it deliberately does not
+
+The escalation *policy* — attempts, cost, which classes are retryable — is a set of pure functions
+over the `PrState` snapshot (`resolveFixDisposition` and friends), already table-tested in core with
+no model involved. Re-testing it here would cost model spend to measure arithmetic.
+
+What only an eval can measure is the half that needs a model: **given real CI evidence, does the
+agent reach the right class?** That verdict is the input every downstream decision is taken on — a
+`flaky` verdict on a genuinely reproducible failure burns three attempts and escalates a PR that one
+push would have fixed, and it touches no GitHub state at all, so nothing but the marker catches it.
+Hence `expect_markers.diagnosis_class` is the primary grade here, not `expect_github`.
+
+`fix__flaky-promoted` is the one case that exercises the attempt dimension: it runs at attempt 3
+with two prior `flaky` verdicts, so `flakyPromoted` is true and the prompt tells the agent its flaky
+verdict will no longer be accepted. It must reach a real class instead — the case fails if it
+answers `flaky` again, which was the defect the promotion exists to break (09-state-machine.md §S1).
+
+## Base and head are different commits
+
+`repos/<id>/` is the tree at the **base** commit; `repos-head/<id>/` is the PR's own commit, applied
+on the branch. Both are needed, and the run that proved it is instructive: with the two identical, a
+diagnosing agent asked "is this broken on `main` too?" — the first question worth asking — checked,
+correctly answered *yes*, and returned `upstream-broken` for a case whose whole point was
+`reproducible`. Every red-dependency case would have read the same way. A fixture with no bump
+commit is not a dependency PR.
+
+## The expectations are the shipped rubric's, not an intuition
+
+Two of the first draft's cases expected answers `skills/fixing/SKILL.md` calls wrong, and the first
+run caught both. The clearest was an `actions/checkout` 503 expecting `infra-dependent`: the skill's
+own table assigns "a timeout or **network blip**" to `flaky`, and reserves `infra-dependent` for a
+check that "needs secrets, a live service, a deployed backend, a browser". The model answered
+`flaky` and was right. The case is now an `e2e` job with no `STAGING_BASE_URL`, which is that clause
+exactly. When a case and the skill disagree, the skill wins — otherwise the eval measures the
+fixture author.
+
+## What the harness cannot exercise here
+
+The `fix` phase's `skip_if` rows read `scratch.fixMarkers.diagnosis.class`, and that scratch is
+written by the marker harvest through `workflow_runs.scratch` — a **database** the eval harness
+deliberately does not pass (see the harness guide's "Gates need a DB"). So in an eval the fix phase
+runs even after a `flaky` verdict, where production would skip it. The diagnosis itself is
+unaffected, which is what these cases grade; but do not read a `CI_FIX_COMPLETE` marker on a
+non-retryable case as evidence that production would have attempted a fix.
+
+## The evidence is consistent by construction
+
+`pr_state.ci_jobs` feeds two things from one source: the prompt's `{{ciSection}}`, and the fake
+GitHub's Actions endpoints (`github_list_workflow_runs` / `github_get_job_logs`). An agent that digs
+into the logs therefore reads the same failure it was told about. A case whose tools 404 would be
+measuring an agent working around the harness.

package/datasets/fix/instances.json

@@ -0,0 +1,229 @@
+[
+  {
+    "instance_id": "fix__reproducible-api-removal",
+    "repo": "acme/slugger",
+    "problem_statement": "CI is red on the tiny-case 2.4.0 \u2192 3.0.0 bump. The suite fails deterministically: `src/slug.js` imports `kebab`, which 3.0.0 renamed to `kebabCase`.",
+    "pr": {
+      "number": 412,
+      "title": "Bump tiny-case from 2.4.0 to 3.0.0",
+      "body": "Bumps the dependency to the version listed in the title.\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "lastlight/fix__reproducible-api-removal",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]"
+    },
+    "pr_state": {
+      "head_sha": "a41f0c8",
+      "head_ref": "lastlight/fix__reproducible-api-removal",
+      "checks_state": "failing",
+      "settled_check_count": 1,
+      "base_checks_state": "passing",
+      "ci_jobs": [
+        {
+          "name": "CI / test",
+          "log_excerpt": "> slugger@1.4.0 test\n> node --test\n\n\u2716 test/slug.test.js (31.402ms)\n  'test failed'\n\nSyntaxError: The requested module '#tiny-case' does not provide an export named 'kebab'\n    at ModuleJob._instantiate (node:internal/modules/esm/module_job:171:21)\n    at async ModuleJob.run (node:internal/modules/esm/module_job:431:5)\n\n\u2139 tests 1\n\u2139 pass 0\n\u2139 fail 1\n##[error]Process completed with exit code 1.\n",
+          "workflow_path": ".github/workflows/ci.yml",
+          "failing_step": "npm test"
+        }
+      ]
+    },
+    "expect_markers": {
+      "diagnosis_class": "reproducible",
+      "fix_gate": "green",
+      "fix_outcome": "pushed"
+    },
+    "expect_github": {
+      "labels_absent": [
+        "requires-human"
+      ]
+    }
+  },
+  {
+    "instance_id": "fix__flaky-timeout",
+    "repo": "acme/poller",
+    "problem_statement": "CI is red on the tap-reporter 4.2.0 \u2192 5.0.0 bump. One polling test timed out at 5000ms; the rest of the suite passed and the same commit passed on the previous run.",
+    "pr": {
+      "number": 418,
+      "title": "Bump tap-reporter from 4.2.0 to 5.0.0",
+      "body": "Bumps the dependency to the version listed in the title.\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "lastlight/fix__flaky-timeout",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]"
+    },
+    "pr_state": {
+      "head_sha": "b7d2e10",
+      "head_ref": "lastlight/fix__flaky-timeout",
+      "checks_state": "failing",
+      "settled_check_count": 1,
+      "base_checks_state": "passing",
+      "ci_jobs": [
+        {
+          "name": "CI / test",
+          "log_excerpt": "> poller@2.1.0 test\n> node --test\n\n\u2714 test/format.test.js (12.004ms)\n\u2716 test/poll.test.js (5001.882ms)\n  'test timed out after 5000ms'\n\n  Error [ERR_TEST_FAILURE]: test timed out after 5000ms\n      at Test.postRun (node:internal/test_runner/test:1024:19)\n\n\u2139 tests 2\n\u2139 pass 1\n\u2139 fail 1\n##[error]Process completed with exit code 1.\n",
+          "workflow_path": ".github/workflows/ci.yml",
+          "failing_step": "npm test"
+        }
+      ]
+    },
+    "expect_markers": {
+      "diagnosis_class": "flaky"
+    }
+  },
+  {
+    "instance_id": "fix__flaky-promoted",
+    "repo": "acme/poller",
+    "problem_statement": "CI is red on the tap-reporter 4.2.0 \u2192 5.0.0 bump, for the third consecutive attempt. The two previous attempts both diagnosed `flaky` and changed nothing.",
+    "pr": {
+      "number": 419,
+      "title": "Bump tap-reporter from 4.2.0 to 5.0.0",
+      "body": "Bumps the dependency to the version listed in the title.\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "lastlight/fix__flaky-promoted",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]"
+    },
+    "pr_state": {
+      "head_sha": "b7d2e10",
+      "head_ref": "lastlight/fix__flaky-promoted",
+      "checks_state": "failing",
+      "settled_check_count": 1,
+      "base_checks_state": "passing",
+      "attempt": 3,
+      "flaky_deferrals": 2,
+      "prior_attempts": [
+        "DIAGNOSIS_COMPLETE: pr=419 attempt=1 class=flaky cause=poll test timed out at 5000ms ci_vs_local=passes-locally",
+        "DIAGNOSIS_COMPLETE: pr=419 attempt=2 class=flaky cause=poll test timed out again, unrelated to the bump ci_vs_local=passes-locally"
+      ],
+      "notes": [
+        {
+          "kind": "ruled-out",
+          "text": "the bump itself: tap-reporter is dev-only and the failing test does not import it"
+        }
+      ],
+      "ci_jobs": [
+        {
+          "name": "CI / test",
+          "log_excerpt": "> poller@2.1.0 test\n> node --test\n\n\u2714 test/format.test.js (12.004ms)\n\u2716 test/poll.test.js (5001.882ms)\n  'test timed out after 5000ms'\n\n  Error [ERR_TEST_FAILURE]: test timed out after 5000ms\n      at Test.postRun (node:internal/test_runner/test:1024:19)\n\n\u2139 tests 2\n\u2139 pass 1\n\u2139 fail 1\n##[error]Process completed with exit code 1.\n",
+          "workflow_path": ".github/workflows/ci.yml",
+          "failing_step": "npm test"
+        }
+      ]
+    },
+    "expect_markers": {
+      "diagnosis_class_any_of": [
+        "reproducible",
+        "env-mismatch",
+        "infra-dependent"
+      ]
+    }
+  },
+  {
+    "instance_id": "fix__env-mismatch-node-version",
+    "repo": "acme/grouper",
+    "problem_statement": "CI is red on the tiny-group 1.9.0 \u2192 2.0.0 bump. The suite passes locally on Node 22 and fails in CI, which runs Node 18; the package declares `engines.node: >=22`.",
+    "pr": {
+      "number": 77,
+      "title": "Bump tiny-group from 1.9.0 to 2.0.0",
+      "body": "Bumps the dependency to the version listed in the title.\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "lastlight/fix__env-mismatch-node-version",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]"
+    },
+    "pr_state": {
+      "head_sha": "c0091ab",
+      "head_ref": "lastlight/fix__env-mismatch-node-version",
+      "checks_state": "failing",
+      "settled_check_count": 1,
+      "base_checks_state": "passing",
+      "ci_jobs": [
+        {
+          "name": "CI / test",
+          "log_excerpt": "Run actions/setup-node@v4\n  node-version: 18\nNow using node v18.20.4 (npm v10.7.0)\n\nnpm warn EBADENGINE Unsupported engine {\nnpm warn EBADENGINE   package: 'grouper@1.0.0',\nnpm warn EBADENGINE   required: { node: '>=22' },\nnpm warn EBADENGINE   current: { node: 'v18.20.4', npm: '10.7.0' }\nnpm warn EBADENGINE }\n\n> grouper@1.0.0 test\n> node --test\n\n\u2716 test/group.test.js (18.220ms)\n  TypeError: Object.groupBy is not a function\n      at groupBy (file:///home/runner/work/grouper/src/group.js:3:17)\n\n\u2139 pass 0\n\u2139 fail 1\n##[error]Process completed with exit code 1.\n",
+          "workflow_path": ".github/workflows/ci.yml",
+          "failing_step": "npm test"
+        }
+      ]
+    },
+    "expect_markers": {
+      "diagnosis_class": "env-mismatch"
+    }
+  },
+  {
+    "instance_id": "fix__infra-dependent-e2e",
+    "repo": "acme/retryer",
+    "problem_statement": "CI is red on the tiny-backoff 3.8.0 \u2192 4.0.0 bump. The unit job passed; the `e2e` job failed because it has no staging backend to talk to \u2014 `STAGING_BASE_URL` is unset for this run.",
+    "pr": {
+      "number": 133,
+      "title": "Bump tiny-backoff from 3.8.0 to 4.0.0",
+      "body": "Bumps the dependency to the version listed in the title.\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "lastlight/fix__infra-dependent-e2e",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]"
+    },
+    "pr_state": {
+      "head_sha": "d51ba7e",
+      "head_ref": "lastlight/fix__infra-dependent-e2e",
+      "checks_state": "failing",
+      "settled_check_count": 1,
+      "base_checks_state": "passing",
+      "ci_jobs": [
+        {
+          "name": "CI / e2e",
+          "log_excerpt": "> node --test e2e/\n\n\u2716 e2e/smoke.test.js (2.114ms)\n  'test failed'\n\n  AssertionError [ERR_ASSERTION]: STAGING_BASE_URL is not set\n\n      at TestContext.<anonymous> (file:///home/runner/work/retryer/e2e/smoke.test.js:7:10)\n\n\u2139 tests 1\n\u2139 pass 0\n\u2139 fail 1\n##[error]Process completed with exit code 1.\n",
+          "workflow_path": ".github/workflows/ci.yml",
+          "failing_step": "node --test e2e/"
+        }
+      ]
+    },
+    "expect_markers": {
+      "diagnosis_class": "infra-dependent"
+    }
+  },
+  {
+    "instance_id": "fix__upstream-broken-base-red",
+    "repo": "acme/money",
+    "problem_statement": "CI is red on the tiny-intl 6.4.0 \u2192 7.0.0 bump. The failing test also fails on `main`: a regression landed there in #874 and dropped the rounding in `convert()`.",
+    "pr": {
+      "number": 902,
+      "title": "Bump tiny-intl from 6.4.0 to 7.0.0",
+      "body": "Bumps the dependency to the version listed in the title.\n\nDependabot will resolve any conflicts with this PR as long as you don't alter it yourself.",
+      "base_ref": "main",
+      "head_ref": "lastlight/fix__upstream-broken-base-red",
+      "base_commit": "0000000000000000000000000000000000000000",
+      "head_commit": "0000000000000000000000000000000000000000",
+      "user": "dependabot[bot]"
+    },
+    "pr_state": {
+      "head_sha": "e11c4d0",
+      "head_ref": "lastlight/fix__upstream-broken-base-red",
+      "checks_state": "failing",
+      "settled_check_count": 1,
+      "base_checks_state": "failing",
+      "ci_jobs": [
+        {
+          "name": "CI / test",
+          "log_excerpt": "> money@5.0.0 test\n> node --test\n\n\u2714 test/format.test.js (11.902ms)\n\u2716 test/rates.test.js (18.441ms)\n  'test failed'\n\n  AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:\n\n  1234.5 !== 1235\n\n      at TestContext.<anonymous> (file:///home/runner/work/money/test/rates.test.js:6:10)\n\n\u2139 tests 2\n\u2139 pass 1\n\u2139 fail 1\n##[error]Process completed with exit code 1.\n",
+          "workflow_path": ".github/workflows/ci.yml",
+          "failing_step": "npm test"
+        }
+      ],
+      "notes": [
+        {
+          "kind": "constraint",
+          "text": "the rates rounding regression came from #874 on main, not from this bump"
+        }
+      ]
+    },
+    "expect_markers": {
+      "diagnosis_class": "upstream-broken"
+    }
+  }
+]

package/datasets/fix/repos/fix__env-mismatch-node-version/.github/workflows/ci.yml

@@ -0,0 +1,15 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "18"
+      - run: npm test

package/datasets/fix/repos/fix__env-mismatch-node-version/README.md

@@ -0,0 +1,3 @@
+# grouper
+
+Requires Node >= 22. `npm test` runs the suite.

package/datasets/fix/repos/fix__env-mismatch-node-version/package.json

@@ -0,0 +1,15 @@
+{
+  "name": "grouper",
+  "version": "1.0.0",
+  "type": "module",
+  "description": "Grouping helper.",
+  "engines": {
+    "node": ">=22"
+  },
+  "scripts": {
+    "test": "node --test"
+  },
+  "dependencies": {
+    "tiny-group": "^1.9.0"
+  }
+}

package/datasets/fix/repos/fix__env-mismatch-node-version/src/group.js

@@ -0,0 +1,4 @@
+/** Group items by a key function. Uses Object.groupBy (Node >= 21). */
+export function groupBy(items, keyFn) {
+  return Object.groupBy(items, keyFn);
+}

package/datasets/fix/repos/fix__env-mismatch-node-version/test/group.test.js

@@ -0,0 +1,9 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { groupBy } from "../src/group.js";
+
+test("groups by first letter", () => {
+  const out = groupBy(["apple", "avocado", "beet"], (s) => s[0]);
+  assert.deepEqual(out.a, ["apple", "avocado"]);
+  assert.deepEqual(out.b, ["beet"]);
+});

package/datasets/fix/repos/fix__flaky-promoted/.github/workflows/ci.yml

@@ -0,0 +1,15 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "22"
+      - run: npm test

package/datasets/fix/repos/fix__flaky-promoted/README.md

@@ -0,0 +1,3 @@
+# poller
+
+A polling helper. `npm test` runs the suite.

package/datasets/fix/repos/fix__flaky-promoted/package.json

@@ -0,0 +1,12 @@
+{
+  "name": "poller",
+  "version": "2.1.0",
+  "type": "module",
+  "description": "Polling helper.",
+  "scripts": {
+    "test": "node --test"
+  },
+  "devDependencies": {
+    "tap-reporter": "^4.2.0"
+  }
+}

package/datasets/fix/repos/fix__flaky-promoted/src/poll.js

@@ -0,0 +1,9 @@
+/** Resolve once `check()` returns true, polling every `intervalMs`. */
+export async function pollUntil(check, { intervalMs = 10, timeoutMs = 200 } = {}) {
+  const deadline = Date.now() + timeoutMs;
+  for (;;) {
+    if (await check()) return true;
+    if (Date.now() > deadline) throw new Error("pollUntil: timed out");
+    await new Promise((r) => setTimeout(r, intervalMs));
+  }
+}

package/datasets/fix/repos/fix__flaky-promoted/test/poll.test.js

@@ -0,0 +1,8 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { pollUntil } from "../src/poll.js";
+
+test("resolves once the condition holds", async () => {
+  let n = 0;
+  assert.equal(await pollUntil(() => ++n > 2), true);
+});