PyPI - coder-eval - Versions diffs - 0.11.5__tar.gz → 0.11.7__tar.gz - Mend

coder-eval 0.11.5tar.gz → 0.11.7tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (669) hide show

{coder_eval-0.11.5 → coder_eval-0.11.7}/.github/pages-stub/index.html RENAMED Viewed

@@ -3,7 +3,7 @@
   <head>
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
-    <title>Coder Eval — evaluate AI coding agents and Claude Code skills</title>
+    <title>Coder Eval — evaluate AI coding agents and their skills</title>
     <!--
       The description is here for link previews and for anyone reading source,
       not for ranking: this page is noindex (see below), so search engines will
@@ -12,7 +12,7 @@
     -->
     <meta
       name="description"
-      content="Coder Eval is an open-source framework for evaluating and benchmarking AI coding agents and Claude Code skills — sandboxed runs of Claude Code, Codex, and Gemini against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
+      content="Coder Eval is Playwright for coding agents: an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — sandboxed runs of Claude Code, Codex, Antigravity (Gemini), or OpenCode against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
     />
     <!--
@@ -225,9 +225,10 @@
       -->
       <h1 class="sr-only">Coder Eval</h1>
       <p class="lead">
-        An open-source framework for evaluating and benchmarking AI coding agents and their Claude
-        Code skills: it runs a real agent — Claude Code, Codex, or Gemini — in a sandbox against
-        declarative YAML tasks, then scores the files and commands the agent actually produced.
+        <strong>Playwright for coding agents.</strong> An open-source, agent-agnostic framework for
+        evaluating and benchmarking AI coding agents and their skills: it runs a real agent — Claude
+        Code, Codex, Antigravity (Gemini), or OpenCode — in a sandbox against declarative YAML
+        tasks, then scores the files and commands the agent actually produced.
       </p>
       <p class="notice">
         The documentation has moved to

{coder_eval-0.11.5 → coder_eval-0.11.7}/.github/workflows/pr-checks.yml RENAMED Viewed

@@ -997,22 +997,37 @@ jobs:
         id: dogfood
         uses: ./
         with:
+          # Exercises what tests/test_action_inputs.py cannot: it asserts the argv
+          # both step scripts build, but only a real runner proves that
+          # `working-directory:` works on a composite step and that a plugin
+          # installed via `--with` is discovered at runtime. Hence a relative
+          # run-dir (landing under tasks/), the bare task filename and the `../`
+          # plugin path, all resolved from `working-directory`.
           version: local
-          tasks: tasks/hello_date.yaml
-          model: claude-haiku-4-5-20251001
+          working-directory: tasks
           run-dir: runs/ci-action-dogfood
-          junit-path: runs/ci-action-dogfood/junit.xml
-          # Credentials go through the generic env passthrough (the only channel);
+          extra-packages: ../tests/fixtures/byoa_demo_plugin
+          # The bracketed `-D` override is the case one-argv-entry-per-line exists
+          # for: `[...]` is a bash character class, so a whitespace-split input
+          # would collapse the list whenever a file in the cwd matched. It adds
+          # Glob to hello_date.yaml's three tools, so the assertions below can tell
+          # "arrived" from "ignored".
+          args: |
+            hello_date.yaml
+            --model
+            claude-haiku-4-5-20251001
+            -D
+            agent.allowed_tools=[Read,Write,Bash,Glob]
           # ANTHROPIC_API_KEY reaching the run is proven by the API-backed task
-          # succeeding. A floor of 0.0 passes for any produced score (exercises
-          # the gate path green in CI without flakiness); the second line
-          # exercises multi-line env parsing.
-          minimum-task-score: "0.0"
+          # succeeding; the second line exercises multi-line env parsing.
           env: |
             ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
             CE_DOGFOOD_MARKER=1
+      # In `tasks/` because that is the assertion: `run-dir` is reported exactly as
+      # passed, so a relative one is relative to `working-directory`.
       - name: Verify outputs and JUnit file
+        working-directory: tasks
         env:
           JUNIT: ${{ steps.dogfood.outputs.junit-path }}
           RUNDIR: ${{ steps.dogfood.outputs.run-dir }}
@@ -1023,11 +1038,66 @@ jobs:
           # our writer emits no DTDs/entities) — stdlib ET is fine here.
           python3 -c "import sys, xml.etree.ElementTree as ET; ET.parse(sys.argv[1])" "$JUNIT"
           test -f "$RUNDIR/run.json" || { echo "run.json missing"; exit 1; }
+          # An absolute path would mean working-directory was ignored.
+          case "$RUNDIR" in /*) echo "run-dir output was rewritten to an absolute path: $RUNDIR"; exit 1 ;; esac
+      # The action does not touch $GITHUB_STEP_SUMMARY: this is both the assertion
+      # that `run-md-path` is real and the recipe the docs hand consumers.
+      - name: Append the run report to the job summary
+        if: always()
+        working-directory: tasks
+        env:
+          RUN_MD: ${{ steps.dogfood.outputs.run-md-path }}
+        run: |
+          set -euo pipefail
+          test -f "$RUN_MD" || { echo "run-md-path output does not exist: $RUN_MD"; exit 1; }
+          cat "$RUN_MD" >> "$GITHUB_STEP_SUMMARY"
+      - name: Verify the plugin was discovered and the bracketed override arrived
+        working-directory: tasks
+        env:
+          RUNDIR: ${{ steps.dogfood.outputs.run-dir }}
+        run: |
+          set -euo pipefail
+          # `coder-eval` on PATH is the action's uv tool shim, so this interrogates
+          # its environment: a task naming the fixture's agent kind validates only
+          # if the entry point was discovered there, else plan exits 1 with
+          # "No agent registered for type 'byoa-demo'".
+          cat > byoa-probe.yaml <<'YAML'
+          task_id: "action_extra_packages_probe"
+          description: "Validates only when the byoa-demo plugin is discoverable."
+          initial_prompt: "not executed - plan validates without running an agent"
+          agent:
+            type: "byoa-demo"
+          success_criteria:
+            - type: "file_exists"
+              path: "app.py"
+              description: "not executed"
+          YAML
+          coder-eval plan byoa-probe.yaml
+          rm -f byoa-probe.yaml
+          # And the `-D` value survived as a 4-element list, not word-split or
+          # glob-rewritten.
+          RUN_JSON="$RUNDIR/run.json" python3 <<'PY'
+          import json, os, sys
+          data = json.load(open(os.environ["RUN_JSON"], encoding="utf-8"))
+          rows = data.get("task_results") or []
+          if not rows:
+              sys.exit("run.json has no task_results to check the -D override against")
+          tools = (rows[0].get("agent_config") or {}).get("allowed_tools")
+          expected = ["Read", "Write", "Bash", "Glob"]
+          if tools != expected:
+              sys.exit(f"-D override did not arrive intact: allowed_tools={tools!r}, expected {expected!r}")
+          print(f"bracketed -D override resolved to {tools!r}")
+          PY
       - name: Upload dogfood run on failure
         if: failure()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a  # v7.0.1
         with:
           name: action-dogfood-runs
-          path: runs/ci-action-dogfood/
+          path: tasks/runs/ci-action-dogfood/
           retention-days: 7

{coder_eval-0.11.5 → coder_eval-0.11.7}/.github/workflows/verify-published-action.yml RENAMED Viewed

@@ -359,13 +359,11 @@ jobs:
           echo "--- task YAML:"; cat tasks/published_smoke.yaml
       # continue-on-error, because this step's exit code is NOT the gate. The action
-      # exits with coder-eval's own code (action.yml combines them), and coder-eval
-      # exits 1 on any failed task -- so a model flake failing `file_exists` would
-      # redden this workflow even with minimum-task-score at 0.0, which does not
-      # neutralize that path. This check must answer "does the published action still
-      # work", not "is the model still good": the verification step below gates on
-      # ARTIFACTS instead. A genuine model/credential outage still surfaces there, via
-      # the zero-token assertion.
+      # exits with coder-eval's own code, and coder-eval exits 1 on any failed task,
+      # so a model flake failing `file_exists` would redden this workflow. This check
+      # must answer "does the published action still work", not "is the model still
+      # good": the verification step below gates on ARTIFACTS instead. A genuine
+      # model/credential outage still surfaces there, via the zero-token assertion.
       - name: Run the published action
         id: run
         continue-on-error: true
@@ -373,11 +371,12 @@ jobs:
         with:
           # `version:` intentionally omitted -- the whole point is to exercise the
           # default pin baked into action.yml at the v0 tag.
-          tasks: tasks/published_smoke.yaml
-          model: claude-haiku-4-5-20251001
           run-dir: runs/verify-published
-          junit-path: runs/verify-published/junit.xml
-          minimum-task-score: "0.0"
+          # Task path and flags both go through `args` — the action promotes no CLI flag.
+          args: |
+            tasks/published_smoke.yaml
+            --model
+            claude-haiku-4-5-20251001
           env: |
             ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}

{coder_eval-0.11.5 → coder_eval-0.11.7}/CHANGELOG.md RENAMED Viewed

@@ -2,6 +2,181 @@
 <!-- version list -->
+## v0.11.7 (2026-09-08)
+### Bug Fixes
+- **deps**: Bump google-antigravity 0.1.7 -> 0.1.8 (Defender FP on the harness)
+  ([#158](https://github.com/UiPath/coder_eval/pull/158),
+  [`48a4d53`](https://github.com/UiPath/coder_eval/commit/48a4d5375f5cf0ce12f956ba77cbe4d48e4be543))
+- **deps**: Bump google-antigravity to 0.1.8 to clear a Defender false positive on the bundled
+  harness [PILOT-7463] ([#158](https://github.com/UiPath/coder_eval/pull/158),
+  [`48a4d53`](https://github.com/UiPath/coder_eval/commit/48a4d5375f5cf0ce12f956ba77cbe4d48e4be543))
+- **docs**: Update Antigravity version in docs
+  ([#158](https://github.com/UiPath/coder_eval/pull/158),
+  [`48a4d53`](https://github.com/UiPath/coder_eval/commit/48a4d5375f5cf0ce12f956ba77cbe4d48e4be543))
+- **evalboard**: Always show every known harness in the filter
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **evalboard**: Stop counting one model as two in the run header
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **pricing**: Refresh the rate card and add gemini 3.7/3.8 Flash
+  ([#155](https://github.com/UiPath/coder_eval/pull/155),
+  [`be98f9d`](https://github.com/UiPath/coder_eval/commit/be98f9d4bd02e12482f16cec3a57ab5c54be8149))
+### Documentation
+- Add intro video and make the README agent-agnostic
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- Widen the framing past skills-only and guard the agent roster with CE047
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **deps**: Drop the Defender rationale from the antigravity pin comment
+  ([#158](https://github.com/UiPath/coder_eval/pull/158),
+  [`48a4d53`](https://github.com/UiPath/coder_eval/commit/48a4d5375f5cf0ce12f956ba77cbe4d48e4be543))
+- **evalboard**: Trim the comments added by this branch
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **pricing**: Trim the rate-card comments to what affects an edit
+  ([#155](https://github.com/UiPath/coder_eval/pull/155),
+  [`be98f9d`](https://github.com/UiPath/coder_eval/commit/be98f9d4bd02e12482f16cec3a57ab5c54be8149))
+- **readme**: Add intro video and make the README agent-agnostic
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **readme**: Play the intro video inline, with YouTube as the fallback
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **readme**: Tell viewers to unmute the inline video
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **stub**: Make the Pages stub and package metadata agent-agnostic
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+- **tutorial**: Stop sending first-timers through the contributor toolchain
+  ([#157](https://github.com/UiPath/coder_eval/pull/157),
+  [`d715f85`](https://github.com/UiPath/coder_eval/commit/d715f856e32289be068dda7049251486660496ec))
+### Features
+- **evalboard**: Add loading skeletons to the slow routes
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+### Performance Improvements
+- **evalboard**: Cut page load time and show loading state while pages load
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **evalboard**: Move repeated per-row utility classes into the stylesheet
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+- **evalboard**: Stop re-reading the whole run store on every render
+  ([#152](https://github.com/UiPath/coder_eval/pull/152),
+  [`5e7d2b6`](https://github.com/UiPath/coder_eval/commit/5e7d2b6e69de43fbfc0c961e43c8b540da5b403f))
+## v0.11.6 (2026-09-01)
+### Bug Fixes
+- **action**: Drop the literal expression syntax from a run-step comment
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **action**: Stop the env passthrough mutating the action's own shell
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **docs**: Drop the last references to inputs that no longer exist
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **evalboard**: Even out the variant tiles and keep a task's arms adjacent
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+- **evalboard**: Resolve a variant-less task link to the task's first arm
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+### Documentation
+- **action**: Trim the comment bulk on the eight-input surface
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **evalboard**: Trim variant commentary and redundant tests
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+### Features
+- **action**: Add working-directory, extras, extra-packages, prerelease and args inputs
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **action**: Refactor github action input surface around args; harden env passthrough
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **evalboard**: Register an unlisted gha source for ad-hoc dispatch runs
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **evalboard**: Render multi-variant runs, one row per (task, arm)
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+### Refactoring
+- **action**: Drop every forwarding input; flags go through args
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **evalboard**: Drop the variant colour palette and the spread helper
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+- **evalboard**: Report pass rate per arm instead of alongside a blended one
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+### Testing
+- Drop the skip-guard unit test ([#146](https://github.com/UiPath/coder_eval/pull/146),
+  [`d679ff2`](https://github.com/UiPath/coder_eval/commit/d679ff2667f2c1e76a5d6384059107be4538c750))
+- Skip the enforcement live tests when the agent declines to try
+  ([#146](https://github.com/UiPath/coder_eval/pull/146),
+  [`d679ff2`](https://github.com/UiPath/coder_eval/commit/d679ff2667f2c1e76a5d6384059107be4538c750))
+- **action**: Record stub argv from bash so Windows argv is byte-exact
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+- **action**: Stop Git Bash rewriting the absolute path in the install-spec tests
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
 ## v0.11.5 (2026-08-28)
 ### Bug Fixes

{coder_eval-0.11.5 → coder_eval-0.11.7}/CLAUDE.md RENAMED Viewed

@@ -216,7 +216,7 @@ make plugin-reference  # the plugin's bundled criteria reference from the models
 Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction.
-Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's).
+Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed).
 When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033–CE036 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE036 enforces the `live_verdict` determinism + monotonicity contract (`criteria/base.py`) that `EarlyStopWatcher`'s latching, deferred fail-stop, and flip-attribution silently depend on: monotonicity over arbitrary Python is undecidable, so instead of a static check it REPLAYS each live criterion against every prefix of recorded trajectories (`tests/lint/live_verdict_contract.py::CASES`) — on the authored ordering AND under seeded shuffles (`permuted_violations`, which catch order-sensitive bugs the authored walk misses) — and asserts the property directly, plus registry-derived coverage — every `LiveSuccessCriterion` in the union must have cases, and every polarity its instances claim via `live_decidable_polarities()` must actually be reached by one (otherwise a single always-`undecided` fixture would "cover" a type while proving nothing). Adding a live criterion therefore means adding `ContractCase`s in the same change. CE035 resolves every `steps.<id>.outputs.<key>` / `needs.<job>.outputs.<key>` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.)

coder-eval 0.11.5__tar.gz → 0.11.7__tar.gz

coder-eval 0.11.5tar.gz → 0.11.7tar.gz